Data collection

Anonymised, subject-level information on demographics (ie, dates of birth and death, gender), diagnoses (International classification of diseases (ICD) codes), tumour histology (Surveillance, epidemiology and end results ((SEER)/International classification of diseases for oncology [ICD-O-3] tumour codes), hospital procedures (Current procedural terminology (CPT) codes based on the healthcare common procedure coding system (HCPCS)) and medicines (out-patient and in-patient), along with corresponding dates of occurrence was extracted from VUMC’s SD for the years 2012–2018.

From the EHRs of the VUMC, patients ≤25 years of age diagnosed with ALL and with their last disease measurement in year 2012 or later were identified based on at least two ICD codes for ALL from any of the ICD-9 (204·0, 204·0X) or ICD-10 (C91·0X) codes for the disease (see online tables S1 and S2 in the supplementary appendix) or on a single tumour histology code (SEER/ICD-O-3, based on tumour morphology and date of diagnosis) of 9835/3, 9836/3, 9811/3, 9812/3, 9813/3, 9814/3, 9815/3, 9816/3, 9817/3 or 9818/3 (see online table S3 in the supplementary appendix for further information on the codes used). All the medical procedures were grouped into categories to facilitate interpretation of the results (see online table S6 in the supplementary appendix).

The specific data extraction used standard hospital ICD codes and basic demographic information combined with the identification of treatments known to indicate states of disease progression. It is our belief that this data-extraction methodology is generalisable to any well-maintained hospital EHR system. The approach was platform agnostic, as ICD codes could be replaced with queries using any other standard EHR code base, such as OMOP, OHDSI, MUMPS, SNOMED or SQL (Oracle).

Patient cohorts

Patients were assigned to the CAR-T cohort or HCT cohort using multiple permutations of relevant ICD-10, CPT, HCPCS and procedure coding system codes, keyword searches for CAR-T therapy, prescriptions for infusion therapies and related physician notes captured as free text in the database. Dates of CAR-T infusion were extracted from the physician’s notes. All the data and codes were extracted along with the corresponding dates of entry.

Methodology to determine relapse-free survival

One of the key challenges of using RWD on a small population such as ALL is to ensure having enough data to derive robust endpoints on the comparative effectiveness and total cost of treatment. As ICD codes do not always accurately reflect measures such as relapse, remission or a healthy state, and as the ICD code for death was only present in 25% of the patient population records in our dataset, it was necessary to develop a robust proxy for clinical effectiveness. As CAR-T and HCT are second-line ‘salvage’ treatments for ALL, the authors opted to select relapse-free survival (RFS) as a primary endpoint of the analysis. The use of surrogate endpoints, such as RFS, for comparative effectiveness assessments is common in the field of oncology.

Relapse events for each patient were identified by the presence of a known ALL-specific drug therapy used for the treatment of an ALL remission, in combination with temporal points showing a marked increase in activity beyond the normal frequency of critical hospital procedures and the prescription codes seen during this phase of treatment. Drugs used to indicate the presence of ALL relapse included chemotherapeutics, such as asparaginase (and variants),8 clofarabine9 and/or other combinations of therapies indicated for relapsed and/or refractory ALL, such as novel, FDA-approved biological CAR-T therapies (blinatumomab (Blincyto, Amgen10) and inotuzumab ozogamicin (Besponsa, Wyeth Pharmaceuticals/Pfizer11)). The RFS period was calculated from the moment of the intervention that induced remission up to the following relapse (figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Relapses were identified by the sudden and marked increase in the number of hospital procedures (y axis), accompanied by a change in treatment associated with a relapse. The RFS period (x axis) was determined from the moment of intervention up to the following relapse. As CAR-T and HCT are second line ‘salvage’ treatments, RFS is considered a plausible surrogate endpoint for overall survival. CAR-T, chimeric antigen receptor T-cell; HCT, haematopoietic cell transplant; RFS, relapse-free survival.

Medicine and medical procedure costs

By linking the patient-level ICD codes to the HCPCS cost codes included in the US Centers for Medicare & Medicaid Services (CMS),12–15 a comprehensive patient-level pathway could be built, which allowed for the calculation of the comparative outcomes/effectiveness and total cost of treatment of CAR-T compared to HCT at the individual patient level.

Statistical analysis

The probabilities of RFS16 per patient were estimated by means of the Kaplan-Meier survival curves17 along with 95% CIs. Analyses were performed with the use of the data analytics software KNIME. For patients who received both CAR-T therapy and HCT, RFS analysis was done separately based on the RFS period achieved with each therapy. The net present value (NPV), or discounted cash flow, was annually discounted by 3%.

Patients and treatments

Using the data-extraction criteria outlined above, 637 patients treated for adolescent ALL were identified in our 5-year retrospective query. Of these, 529 patients did not have an HCT or CAR-T procedure and were removed from our dataset. Of the remaining 108 patients with relapsed adolescent ALL who had been treated with either CAR-T and/or HCT, subjects with data gaps of ≥2 months were removed from our study in order to avoid biassing the analysis with incomplete data. Our final cohorts comprised 29 HCT subjects and 14 CAR-T subjects.

Given that the prescription records from VUMC often do not include a company-specific name for the CAR-T therapy received, we assumed that patients in the CAR-T cohort infused before September 2018 had received their CAR-T therapy as part of a clinical trial.18–20 In this capacity, we see this study as an analysis of the overall performance of the current class of CAR-T treatments for adolescent ALL, and not as an analysis of a specific company’s product.

Relapse-free survival

Patients in the HCT cohort had a substantially longer RFS period and a better RFS rate when compared with those in the CAR-T cohort. The 3-year RFS probability was 46% (95% CI: 08% to 79%) in the CAR-T cohort and 68% (95% CI: 46% to 83%) in the HCT cohort (figure 2). Given these results, this study used the EHR data to determine if there was a treatment bias in the current patients selected for treatment with CAR-T compared with those who had received HCT.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Kaplan-Meier curves and 95% CIs for relapse-free survival in the CAR-T and HCT cohorts. Data censoring (slash) represents the end of available data. CAR-T, chimeric antigen receptor T-cell; HCT, haematopoietic cell transplant.

Our Kaplan-Meier curves showed that CAR-T’s 3-year outcomes were below those of HCT, which runs contrary to the results of many published CAR-T randomised controlled trial studies. We extracted further EHR data to determine if there was a selection bias or confounding variables in those patients treated with CAR-T. Even though our dataset was small, we were able to extract a subset of 8 CAR-T patients and 12 HCT patients whose records included complete RFS data on their previous first-line treatment before receiving CAR-T and HCT. What these data showed was a marked reduction in the RFS of the previous interventions of patients in the CAR-T cohort when compared with patients in the HCT cohort (figure 3). The previous RFS period in the CAR-T cohort (median, 476 days; 95% CI: 427 to 847 days) was significantly shorter than in the HCT cohort (median, 1028 days; 95% CI: 966 to 2548 days).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Kaplan–Meier curves of relapse-free survival of the CAR-T cohort (left) and the HCT cohort (right) of 8 CAR-T patients and 12 HCT patients whose preceding relapse and second-line treatment data were both available. Differences between the previous treatment responses (disease burden represented by the blue curves) were statistically tested using the log-rank test. CAR-T, chimeric antigen receptor T-cell; HCT, haematopoietic cell transplant.

As our Kaplan-Meier curves were both censored and right skewed, log-rank analysis was conducted to validate this result. The RFS period prior to CAR-T-associated or HCT-associated relapse indicated that the disease burden of the HCT and CAR-T populations was statistically different (p value <0.001). Although our dataset was limited, we were able to test the influence of patient age, average number of hospital visits per week, average amount spent on medicines and number of hospital procedures per week as factors (covariates) potentially influencing or associated with these RFS outcomes. We observed that none of these factors were correlated with pretreatment or post-treatment RFS (p values >0.8 and >0.2 in a Cox regression model). The authors assume that there are other factors, or a combination of factors, influencing RFS, and/or there is a need for a larger sample size to enable statistically relevant patient stratification and covariance analysis for the factors for which we have data. The results tables of our covariance analysis can be found in the supplementary appendix.

As HCT requires a sustained and prolonged remission before it can be administered to a patient, we hypothesised that the stem cell transplant facilities were reserving CAR-T treatment for those patients who could not sustain a remission. This hypothesis was validated and confirmed in conversations with practitioners at VUMC (see the Discussion section).

Total cost of treatment

Looking only at the internal costs within the hospital, the cost of CAR-T infusion and follow-up, excluding the actual price of the CAR-T product, is approximately 61% of the total cost of HCT, with an NPV of US$118 795 for CAR-T therapy compared with US$303 065 for HCT (table 1). However, when the price of the CAR-T therapy (tisagenlecleucel, (Kymriah, Novartis)) is also considered in the total costs, the actual CAR-T product NPV is US$579 999 (table 1). In practice, 80% of the cost of treating ALL with CAR-T therapy is due to the purchase price of the immunotherapy itself. When excluding the CAR-T product price of US$475 000, the internal hospital and medical costs of CAR-T therapy are significantly lower than those of HCT (table 1).

Limitations

Although our statistics are supported by p-values, we do understand that the cohort analysis could contain undetected biases due to our inability to run covariance analysis or propensity scores caused by both the small size of the dataset and the lack of hard data endpoints due to the censoring of a small test population. This may be seen as limiting the generalisability of the findings of this research to some readers. However, in consultation with many European HTAs and regulators, and given the accuracy of our findings which have been validated by clinicians at VUMC, we consider that this is a valuable methodology warranting further investigation.

Despite the limited population, this work presents a novel approach to quantifying the comparative value and effectiveness of new therapies to help meet the increasing challenges that are being faced by payers and HTAs. Our approach involves creative use of RWD providing actual, not estimated or extrapolated, cost assessments and outcomes measures of a new orphan therapy that are normally seen by HTAs as highly uncertain. The authors are currently in the process of establishing a follow-up study to validate the replicability of this methodological approach.