Article Text
PDF
Statistics from Altmetric.com
The challenge of accelerated clinical developments
In France, decisions for reimbursement taken by the Ministry of Health are based on appraisal by an independent Health Technology Assessment body (HTAb): the ‘Haute Autorité de santé’ (HAS). HAS grades the clinical added value of any medicinal product for which a manufacturer seeks reimbursement. This appraisal considers different types of clinical and patient-centred outcomes, including patient-reported ones. Under certain conditions, a concomitant economic assessment which accounts for patients’ preferences in the form of utility values is also performed.
As providing fast access to breakthrough therapies is a critical expectation from patients, clinicians and health policy makers, the European Medicines Agency and the Food and Drug Administration have established various accelerated approval pathways. These procedures lead to conditional approvals frequently based on uncontrolled (ie, single arm) pivotal trials.1–4 Approvals based on uncontrolled trials are also motivated by randomisation sometimes being considered unfeasible or unethical, or because the pathophysiological rationale is assumed to be important when proving effectiveness (eg, a treatment with molecular target in oncology).5 6
However, expected benefits based on uncontrolled trials as evidence are frequently not confirmed. The results of meta-epidemiological studies illustrate that appraisals based on such evidence entail a high level of uncertainty leading to ethical concerns.7 8 For patients, this may have deleterious consequences such as the use of products for which the benefits remain unknown,9 10 the overestimation of benefits with no further confirmation11 12 or the increased risks of adverse events.12 13 This high level of uncertainty may also impact health system sustainability.
While ensuring rapid access to valuable treatments is of utmost importance to patients, maintaining an adequate balance with the performance of relevant HTA in this context is highly challenging. Thus, the French Minister of Health requested HAS to provide recommendations. A consultation of patient associations, academics, manufacturers and various institutions was conducted from October 2021 to January 2022. With the support of an expert committee, a qualitative summary of the consultation has led to the prioritisation of recommendations, which are developed below (details on the consultation process are available in an online supplemental appendix 1).
Supplemental material
Need for evidence from comparative designs allowing causal interpretation of treatment effect estimation
Performing relevant HTA requires that an unbiased estimate of the treatment effect is available. Thus, the additional effect must be disentangled beyond effects due to the natural course of the medical condition, the placebo effect, various sources of bias and the effect of alternative(s). To produce such an estimate, the simplest and most consensual methodological choice is a comparison of the treatment of interest to a control by conducting a randomised controlled trial (RCT). Randomising treatment allocation balances both known and unknown confounders between groups, which enables the causal attribution of an observed effect to the investigated treatment. Other characteristics such as blinding and intention to treat analysis principles reduce the risk of bias and thereby lead to an estimation of treatment effect with the highest certainty of results.
Therefore, RCTs should still be systematically considered by manufacturers during clinical development.
Accelerating rapid access to innovative treatments by adapting the traditional RCT design
While RCTs are the gold standard for producing clinical evidence, HAS acknowledges that the strict adherence to the traditional fixed-sample RCT design can lead to long delays before patients access treatments. Several alternatives to the traditional design, such as adaptative designs, platform trials, seamless trials, pragmatic trials or cross-over designs, can be better suited choices for accelerating clinical developments. An overview of main strengths and methodological points of attention is described in table 1. All these adaptations are considered compatible with the principles of RCT design and can produce evidence with high certainty of results.
Adaptation of randomised controlled designs leading to good enough certainty of results that can accelerate treatment access to patients
They should, therefore, be encouraged first when considering the issue of balancing rapid access to beneficial medicinal products to patients while performing relevant HTA.
Increasing the certainty of results when uncontrolled trials are pivotal studies
Apart from the exceptional condition where an unequivocal response on a patient-centred outcome for almost all patients compared with an otherwise certain pejorative evolution is observed, the value of uncontrolled trials alone is strongly limited when estimating the effect of a treatment. Nonetheless, market approval relying on uncontrolled trials is prevalent,14 as is the number of dossiers submitted for HTA based on these trials. Thus, HAS proposes points of attention that should be considered providing a manufacturer wants to submit a dossier based on uncontrolled trial(s) only. They are summarised below and are detailed in box 1. Any deviation from these points of attention will further impair the certainty of results.
Methodological points of attention HAS should consider when assessing an external comparison between an uncontrolled trial and an external control
Justification of the lack of randomisation
A rationale appraised as acceptable by Haute Autorité de santé is provided.
Study design
Early planning during clinical development and before the conduct of the uncontrolled trial of the treatment of interest.
A priori definition of the clinical question, study population, intervention, comparator and outcomes in a protocol and statistical analysis plan.
An emulation of a target trial can enhance eliciting the appropriate clinical question (estimand) and designing the external comparison.
Search and selection of relevant sources of data
Well-performed systematic review identifying relevant prognostic variables, confounders and effect modifiers.
Well-performed systematic review (with eligibility criteria defined a priori) identifying relevant sources for external control.
Choice of the external control
The comparator and external source(s) of data has been chosen independently of the results of the uncontrolled trial, fit best the clinical question (does not arbitrarily favour the treatment of interest) and correspond to standard of care.
Analysis of the results
The study protocol, statistical analysis plan and report allow a transparent and appropriate assessment of the study.
A model for causal inference controlling an appropriate set of confounders and targeting the predefined estimand has been properly specified and estimated.
The model is preferably based on a method using individual patient data only such as propensity scores, g-computation or doubly robust estimation.
Underlying assumptions have been explored and seems to be met (such as positivity, sufficient overlap and sufficient balance for propensity scores).
If ‘trimming’ (ie, excluding patients in areas of the propensity score without overlap) have been performed, the resulting target population for which results can apply is described.
Residual confounding has been explored with analyses such as the use of negative and positive controls, consistency in results when using other external controls, or quantitative bias analysis and excludes a conclusion of no treatment effect.
Study characteristics of the uncontrolled trial and external control are sufficiently similar for excluding other sources of bias such as selection bias, attrition bias, measurement bias.
Safety can be properly documented for both groups.
Grading the clinical added value
The clinical added value of the treatment of interest is appraised considering the certainty of results, the relevance and magnitude of treatment effect and safety.
Justifying the suitability of the data generation programme
Lack of randomisation is a major deviation from evidence-based medicine principles and should remain an exceptional option. Therefore, a rationale for the lack of randomisation must be provided by the manufacturer. They are invited to discuss the suitability of the evidence data package during European joint scientific consultations or national early dialogues.
Gathering source(s) of data for external comparison
Performing an adequate comparison vs an external control, using for example data from cohorts or other clinical trials, can be an option for mitigating uncertainties in the absence of RCTs. Availability of external controls may be scarce (eg, in case of targeted therapies for very rare mutation(s) in oncology). Real-world data generation should therefore be anticipated by the manufacturer during the early stages of the clinical development if no pre-existing data set has been identified for the comparison.
Performing transparent and appropriate analyses
To follow an hypothetico-deductive approach, external comparison must be planned as early as possible during clinical development. The framework of the emulation of a target trial can help to define the appropriate estimand, eligibility criteria as well as exposition and outcome(s) of the targeted (ie, ideal) RCT that the external comparison tries to emulate.15 It also helps to avoid the occurrence of immortal-time bias that results from a failure to align start of follow-up.
To avoid a post hoc selection, the choice of an external control must be done prior to conducting the trial, after a well-performed systematic search. It must fit the standard of care. The retained external source of data must be chosen because it fits best the research question and not because it would arbitrarily favour the treatment of interest.
When performing an external comparison, the attribution of the observed effect to the treatment (causal statement) supposes the absence of any confounders. It requires a well performed causal inference analysis, suing appropriate methods. Approaches based on the use of individual patient data only are preferred (eg, propensity scores analysis, g-computation, doubly robust estimation). Indeed, population-based adjusted methods using individual patient data from an uncontrolled trial and aggregate statistics from external data are sometimes considered. However, they rely on the assumption of the conditional constancy of absolute effects (ie, the absolute treatment effects are assumed constant at any given level of the effect modifiers and prognostic variables),16 which is unlikely to be met. To select the appropriate set of covariables to use for conditioning the statistical analyses, a well-performed systematic search of prognosis variables, confounders and effect modifiers must be conducted.
The possibility of residual confounding must be discussed. All methods assume exhaustivity of the confounders considered, a property that will ultimately remain unverifiable. This discussion should not be based on expert opinion only and should be documented, for example, using either negative or positive controls, or a quantitative bias analysis. Sources of bias (eg, selection bias, measurement bias) due to the characteristics of the trial and external control(s) must be discussed.
Safety must be properly documented, including for the external control.
Discussion of cross-cutting implications
This paper delineates proposals for an adequate balance between fast access to valuable medicinal products and performing relevant HTA. RCT, as the simplest and most consensual design for allowing causal reasoning, must be systematically considered for clinical development. However, various adaptations of its design, as proposed in table 1, may accelerate clinical development and are encouraged whenever appropriate.
Conducting uncontrolled trials is regularly justified by arguing that randomisation is deemed unfeasible, or that there is a conflict between individual and collective ethics because there is no other alternative than best supportive care. However, results from meta-epidemiological studies showed that RCTs are regularly conducted when investigating rare diseases, even vs placebo, providing clinical equipoise (ie, the genuine uncertainty regarding the benefit of a new treatment) is appropriately explained to patients.17 Urgent response to an emergent disease, or a strong pathophysiological rationale, are also frequently argued to justify the absence of RCTs. But, randomising the first patients when an emergent disease occurs has been considered ethically appropriate for decades, as it leads to the quickest generation of evidence with the highest certainty of results.18 A strong pathophysiological rationale does not necessarily translate into clinical value and what is considered an ‘innovative’ treatment should be based on clinical evidence. Therefore, HAS cannot propose a definitive list of conditions where absence of randomisation would be de facto acceptable.
To some extent, HAS acknowledges that exceptional conditions may exist that make it unreasonable to conduct an RCT. From the HAS perspective, evidence must, however, still comply to a hypothetico-deductive approach allowing causal reasoning. Should an uncontrolled trial with a comparison vs an external control be provided, the methodological points of attention HAS should consider are described in box 1 to reduce uncertainty. Most of these points of attention (sections 2–5 of box 1) allow assessing factual elements according to good clinical and statistical principles. As such, they could be proposed as consistent points of attention applicable by different HTAbs when assessing external comparisons. However, the acceptance of the rationale justifying the lack of randomisation, as well as some elements leading to the appraisal of clinical added value (eg, relevance and magnitude of treatment effect, impact on public health), are relative to the organisation of health systems and must be assessed considering the national context.
A consequence of relying increasingly on uncontrolled trials as pivotal evidence would be a shift in demonstrating the effectiveness of medicinal products during clinical development towards post-approval. In this context, the use of real-world data, as primary or complementary source of data, is a possible way to address remaining uncertainties. Regardless of the source of data, quality of study design is a cornerstone for certainty of results. To close evidence gap quickly, appropriate studies must thus be anticipated and not limited to uncontrolled designs.19 Usage of real-world data and compliance to RCT design should not be seen as antinomic. When appropriate, the conduct of RCTs can be facilitated by collecting some of the necessary data from pre-existing databases (see table 1). In addition, HAS calls for greater transparency in the whole process of generating evidence through initiatives such as registered report publications, data and clinical study reports sharing.20
Finally, HAS will remain attentive to emerging methods to reduce evidence gaps and accelerate clinical developments. In particular, the increasing use of artificial intelligence, in silico trials and disease modelling, might become a complementary path to optimising clinical development, as illustrated by CRESIM in helping to choose the best design in rare diseases.21 22
Conclusion
Decisions for reimbursement in France are based on an assessment and appraisal process performed by an independent HAS committee which includes healthcare providers, methodologists and patients’ representatives. Members of HAS and especially this committee are eager to secure rapid patient access to valuable medicinal products and that must be weighed against the duty to propose treatment with a clear demonstration of their clinical added value. As evidence-based medicine is an impartial decision model for coverage authorisations and as causal reasoning is essential for continuous improvement of care, HAS reaffirms the importance of RCTs to establish the value of new treatments. However, to balance rapid access for patients with evidential uncertainties, the traditional fixed-sample RCT design can be adapted. Moreover, properly conducted external comparisons as detailed in this paper may represent reasonable alternatives under exceptional conditions.
Ethics statements
Patient consent for publication
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @JudithFerBen, @NaudetFlorian
Contributors AV, JF, SK and LA were responsible for the organisation and synthesis of the consultation of stakeholders as detailed in the online supplemental appendix. Preliminary scientific work on the issue of the methodological qualities of external comparisons was performed for HAS by PS and MC. All authors have participated in producing the recommendations proposed in the paper. AV, JF, SK and LA wrote the initial draft and oversaw subsequent steps of the writing process. All authors contributed to the conceptualisation and revisions of this analysis and approved the final version.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests DC reports an HIV grant from Janssen (2019-2020), personal fees from Gilead (2020) and Pfizer (2022) for lectures outside the submitted work. RT has been the coordinator of the IMI2 EBOVAC2 project in collaboration with Janssen (2014-2021). Other authors have no competing interests to disclose.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.