Article Text
PDF
Statistics from Altmetric.com
Summary box
Data transparency has become a well-established norm in biomedical research, and is especially important for broadly used public health interventions like COVID-19 vaccines.
Tax payers helped fund COVID-19 vaccine trials and should have the right to access the results.
There is inadequate availability of COVID-19 vaccine trial documents and data; individual participant data will not be available for months, perhaps years, for most vaccines.
Widespread use of interventions without full data transparency raises concerns over the rational use of COVID-19 vaccines.
Trial transparency must start early and be continuous. Trial protocols should be released once finalised, before trial results are reported, and should be accompanied with the release of trial documents and data before clinicians and the public make decisions regarding product use.
Transparency in clinical trials: an established norm across sectors
Access to data for drugs and vaccines has historically been fairly limited to journal article publications and hard-to-access and difficult to read regulatory reports.1 But the past decade has witnessed strides in clinical trial data transparency. A wide range of institutions, from pharmaceutical companies, government agencies, trade organisations, journals and not-for-profit organisations, have all acknowledged the importance of data sharing, including the release of deidentified individual participant data. Many policies, regulations and platforms now exist to facilitate data access, including landmark transparency policies from the European Medicines Agency (EMA)2 3 and Health Canada.4 Both regulators now post on their websites, sections of the licensure dossier received by the industry (https://clinicaldata.ema.europa.eu/ and https://clinical-information.canada.ca/). There are also industry and academic platforms to facilitate third-party access to trial data and documents, including ClinicalStudyDataRequest.com, Yale University Open Data Access (YODA) Project and Vivli.5 In 2013, the US and European industry trade organisations endorsed a joint statement on clinical trial data sharing, making a series of commitments that ‘recognise the importance of sharing clinical trial data in the interest of patients, healthcare and the economy”6 In 2015, the US Institute of Medicine similarly endorsed benefits of sharing clinical trial data, emphasising that ‘verification and replication of investigators’ claims’ were essential to the scientific process, and noting the numerous benefits to stakeholders ‘including payers of healthcare as well as patients, their physicians and researchers.’7
Why we need access to COVID-19 vaccine trial data and documents
Clinical trial transparency is always important, but is especially critical during the COVID-19 pandemic (or any public health emergency) where regulatory decisions are being made quickly by government health officials, novel vaccine platforms are being used, vaccines are being administered widely and taxpayer funds have contributed heavily to research and development. Critical appraisal of clinical trials is vital to inform decision making at the personal, professional and governmental level, but cannot be credibly performed on journal publications alone.8 9 Access to clinical trial data and related trial documents (see box 1) allows for independent and informed assessment of trials. By understanding details of how studies were designed and how data were collected, one can understand if endpoints were reliably operationalised and measured. Similarly, release of underlying data from clinical trials allows for independent verification of results, assessment of heterogeneity of treatment effects for specific subgroups, and facilitates the formation of new research questions.10
Types of trial documents
Case report forms (CRFs)
The original paper or electronic forms on which individual participants’ data (demographic, efficacy measurements, adverse events, etc) are recorded during the clinical trial. These documents contain structured fields which make it easier to analyse and report trial data. Access to blank case report forms (CRFs) allows for independent evaluation of how data were collected and endpoints were operationalised.
Clinical study reports(CSRs)
Unabridged, structured report of a clinical study written for regulators.30 A complete clinical study reports (CSRs), including study appendices, includes documentation of trial design, results of trial included adverse events, trial protocol, statistical analysis plans and blank CRFs. CSRs on average span thousands of pages, making them a rich source of information. European Medicines Agency and Heath Canada are the only two regulatory agencies that publish CSRs at this time.
Certificate of analysis
Provides a description of the chemical analysis and physical appearance of the study interventions actually used in the trial (both experimental and comparators).
Protocol
Document written prior to study start date which details plans on how the study will be conducted, analysed and reported. Any changes or deviations from the trial protocol should be tracked and provided, with a rationale for the change, in the form of a formal protocol amendment.
Statistical analysis plan (SAP)
A written plan of how trial data will be analysed and which statistical methods and definitions will be used.
Informed consent form (ICF)
A document required to be provided to study subjects that contains information related to the description of the study, purpose, study intervention(s), any procedures, adverse events, risk and benefits, compensation and rights of participants enrolled in the study.
Serious adverse event (SAE) narratives
Unstructured paragraphs of text providing details and context of the serious adverse events that occurred in study participants. Narratives are usually contained within a CSR.
Electronic individual participant data (IPD)
Complete electronic computerised dataset for each participant in the trial which allows for full replication of study findings using statistical software. Complete CSRs also contain participant level data, mostly in appendices, but these are in text (not dataset) form as individual line listings.
Investigational Medicinal Product Dossier (IMPD)
Document that describes the quality of the placebo and investigational product, how the product(s) were manufactured, non-clinical and clinical study results. An Investigational Medicinal Product Dossier is required for all clinical trials conducted in the European Union.
Investigator’s Brochure (IB):
A living document containing a summary of the clinical and nonclinical data of an investigational product, including its pharmacology, pharmacokinetics, toxicology and adverse event profile, among other items.
There are specific issues in COVID-19 vaccine trials that merit scrutiny. Consider blinding, an essential feature in randomised trials investigating efficacy against subjective endpoints, as in the COVID-19 vaccine trials. Assessing the reliability of blinding involves analysing endpoint definitions and data collection. The primary endpoint in many trials is laboratory-confirmed, symptomatic COVID-19. However, prior to the release of trial protocols, few details were known about this endpoint. Registry entries were vague (eg, Pfizer’s largest study11 only stated ‘confirmed COVID-19’ as one of 35 primary outcome measures), leaving unclear how the definition was operationalised. While the subsequent release of some protocols addressed some questions, it raised new questions that can only be answered with underlying data. Protocols make clear that the symptomatic component of the primary endpoint was reported by trial participants, typically via a smartphone app, and defined by one or more signs and symptoms, many of which were subjective (eg, in Pfizer’s trial, COVID-19 symptoms included at least one of the following: fever, cough, shortness of breath, chills, muscle pain, loss of taste or smell, sore throat, diarrhoea or vomiting.) The fact that the placebo was saline and the vaccines cause short term adverse events in the majority of people raises concerns about unofficial unblinding—that is, the ability of trial participants and investigators to make educated guesses as to treatment allocation. Only a thorough analysis of the underlying individual participant data will allow for an exploration of the extent to which unofficial unblinding may have occurred and biased data collection for the primary endpoint. Access to data would also allow for straightforward replication studies, perhaps particularly important when real-world results appear incompatible with reported trial results. For example, at the time of writing (27 June 2021), Seychelles, Mongolia, Bahrain, Uruguay, and Chile were experiencing COVID-19 outbreaks despite high uptake of WHO-authorised vaccines.
In addition, trial protocols from Pfizer and Moderna indicate that event adjudication committees were involved in counting COVID-19 cases. Considering that the primary endpoint was defined as a positive lab test and patient-reported symptoms, it is unclear how an adjudication committee might affect the primary endpoint evaluation process. Transparency of the committee’s charter may provide additional detail on what data committee members had access to in forming their judgements (eg, did they have access to data on patients’ symptoms in the first week after vaccination, when vaccine-related adverse events could be expected?), and what criteria they used to form their judgements. Access to such documents, therefore, is also important.
Regarding adverse events, detailed narratives of serious adverse events that occurred in a trial are a standard element found within clinical study reports and can enable a more thorough understanding of potential harms. Patterns in adverse events can be explored through access to electronic individual participant-level data.
What data have been released
For eight COVID-19 vaccines being used, or under consideration for use globally (Pfizer, Moderna, Oxford/AstraZeneca, Janssen/Johnson & Johnson, Novavax, Gamaleya Institute, Sinopharm and Sinovac), we evaluated the public availability of a variety of important pre-study documents (eg, trial protocol, statistical analysis plan, blank informed consent form, blank case report form, data monitoring board charter, and event adjudication committee charter) and post-study documents (eg, press releases with trial results, journal publication, clinical study report and individual participant data availability) at the time of writing (27 June 2021). We counted the total number of pages available as a crude proxy for the level of detail, as some documents, like clinical study reports, can be highly variable in length depending on the availability of appendices.
The overall picture is one of varied transparency. While several trials have at this point published protocols and statistical analysis plans along with the study publications, with some even released while the trials were underway, many key trial documents remain inaccessible (table 1). For example, a WHO report found that out of 86 clinical trials for 20 COVID-19 vaccines, 12% of clinical trial protocols were made publically available.12 In our analysis, trial protocols and informed consent forms were not available for trials involving special populations such as children ( NCT04816643 ) and pregnant women ( NCT04754594 ). And despite vaccine roll-out, electronic individual participant data is not available for most trials.12 Some sponsors, such as Moderna, have sent mixed messages on whether they even intend to share data, while others, such as Pfizer and Sinopharm, indicate they will not even begin accepting data requests for many months or years (table 2). In other cases, trialists have indicated very narrow time frames for sharing data, for example, ‘beginning 3 months and ending 1 year after publication’ ( ChiCTR2000032459 ).13
Currently available COVID-19 vaccine trial data for selected trials
Timing of release of individual participant data from COVID-19 vaccine trials
The greatest availability of data at present comes from the EMA, Health Canada, and Japanese Pharmaceuticals and Medical Devices Agency which have released thousands of pages from company submissions for COVID-19 vaccines, far exceeding what is available elsewhere. The Food and Drug Administration (FDA) does not routinely make any industry documents it receives publicly available. It should be noted, however, that the EMA and Health Canada’s commitment to transparency does not necessarily equate with the public availability of critical study documents. Judging from the availability of documents, regulators themselves appear to be receiving less complete and granular data than normal due to the compressed timeline from study start to regulatory decision making. For example, in July 2020, Canada granted emergency approval for the use of remdesivir, a drug used for the treatment of COVID-19, without receiving a copy of the clinical study report.14 This is also the case for the Moderna vaccine, for which a variety of trial documents have been posted by Health Canada and the EMA, but not the clinical study report.15 For Cuba, Russia, India and China state-developed vaccines, access to data prior to regulatory approval is even less clear.16
The world’s most used COVID-19 vaccines: Sinovac and Sinopharm
Sinovac and Sinopharm, developed by Chinese pharmaceutical companies, account for the majority of vaccines being in Asia, South America, the Carribean and Africa.17 They are authorised by the WHO and included in the WHO COVID-19 Vaccines Global Access initiative. However, transparency of trial data and documents is extremely limited, similar to other COVID-19 vaccines. Because neither Chinese vaccine has thus far been authorised by the EMA, Health Canada, or the Japanese Pharmaceuticals and Medical Devices Agency (the three regulators publishing data to their websites), there is also no expectation that data on these vaccines will become publicly available via regulators (table 3). Sinovac vaccines have been administered for nearly 1 year (since July 2020), and still have yet to publish clinical trial data (as at 27 June 2021). Trial results have largely been limited to government media reports and press releases.18
Availability of regulatory reviews and additional data for COVID-19 vaccines
Transparency of regulatory decision making
Apart from data and documents tied to a specific trial, credible analysis and interpretation of data may require understanding regulatory decision making. For example, the fact that many experts had originally believed the trials were designed to study a reduction in hospitalisation, intensive care utilisation and death,19 points to a need to better understand the rationale for primary endpoint selection. Regulators played a major role in shaping this. An FDA guidance document from June 2020, before phase 3 trials commenced, stated that laboratory-confirmed COVID-19 (of essentially any severity) was an acceptable primary endpoint. But there remains limited transparency on the rationale for the selection of this endpoint: why was it chosen? What other endpoints were considered? More information about the internal deliberations is essential to understand whether the decisions made were reasonable.20
Additionally, there is a need for transparency around any deliberations regarding the length of follow-up necessary to adequately assess efficacy and safety prior to licensure. Longer follow-up for investigational products such as COVID-19 vaccines and therapeutic agents are necessary to evaluate duration of protection and ensure public and professional confidence.21 The International Coalition of Medicines Regulatory Authorities (ICMRA), a global collaborative coalition of 30 medicine regulators including the FDA, Health Canada and EMA, published a statement in November 2020 stating that follow-up for treatment and placebo arms should continue ‘for as long as possible after any regulatory approval’ and recommended a follow-up period of ‘at least 1 year or more from completion of assigned doses’.22 Despite this, placebo controlled follow-up, originally planned for 2 years in many trials, was eliminated after a few months, when manufacturers began offering vaccine to placebo recipients within weeks of receiving emergency use authorisations. (Debates took place regarding the ethics of denying placebo recipients vaccine, and proposals to redesign the trials as cross-over trials were not taken up).23 In addition to ensuring the public accessibility of follow-up data from continuing trials, greater transparency is necessary into the ongoing deliberations and thinking of regulators who are currently evaluating applications from sponsors seeking to move from emergency use authorisations to actual approval or licensure.
Other regulatory documents produced by regulators that can provide greater insight into trials and the evidence development programme include scientific review memos and public assessment reports. In the USA, documents such as review memos and presentations presented to the federal advisory committee are also released. These are all invaluable, providing insight into regulatory decision making (table 3). At around 50–150 pages, they can be substantially longer than journal articles but still one or two orders of magnitude shorter than clinical study reports, and represent the regulators’ analyses of data, not the data itself. Therefore, they should be regarded as complementary to, not substitutes for, trial data.24
Real-time transparency
Transparency should begin as trials get underway, and not be left as a bureaucratic exercise to be conducted after results are announced and decisions are made. This should be the case for all trials, but especially COVID-19 vaccines given their global significance.
Before trialists begin recruiting participants, there are already a variety of pre-study documents in place, and release of these documents would allow for broader awareness and scrutiny of the trials. Are the right endpoints being studied? Are the right populations being recruited? Do informed consent forms convey sufficient information about study purpose?25 The power of real-time release of trial protocols was on display last summer when some manufacturers—Pfizer, Moderna, Janssen and AstraZeneca (for its US trial)—released study protocols for their phase 3 trials.26 That transparency not only revealed the inadequacy of what had been disclosed about the studies’ primary endpoint in trial registry entries, but helped stimulate public discussion and debate about what primary endpoint was acceptable, including at the FDA’s advisory committee the following month—all while the trials were ongoing.19 27 28
Mechanisms to stimulate greater transparency
Although trade-offs will need to be weighed in the context of a global public emergency, there are several immediate steps that governments agencies, regulators, pharmaceutical companies and professional bodies can take to achieve greater transparency of COVID-19 vaccine trials. To start, government agencies and regulators can create infrastructure to support submission and deposits of protocols, informed consent forms, committee charters and other pre-study documents. This could be done by using existing resources such as ClinicalTrials.gov. Moreover, regulatory documents, memos, and scientific reviews used to make decisions on vaccine approval can also be shared on (or linked from) the trial registry entry. Platforms for sharing individual participant data already exist and can be leveraged (eg, Vivli, ClinicalStudyDataRequest.com and YODA), reducing overall monetary cost associated with publishing of trial data.
Drug sponsors and companies can likewise create a dedicated section of their website for pre-study documentation (as some have done). This should not cause an undue burden as such documents are already written and shared with regulators, and they contain no identifying patient information. All that needs to occur is the act of publicly posting a copy. Additionally, sponsors can provide clear and updated statements regarding their timeline for access to electronic individual participant data.
Professional bodies and academic organisations can also play a large role in promoting transparency. Public statements that convey the unacceptability of promises to begin sharing months and years from now, despite vaccine roll-out, would help. More powerfully, professionals could pledge not to endorse new therapeutic agents until full access to data is provided. Doing so would send a strong message that transparency is not a ‘nice to have,’ but a fundamental component of any intervention that purports to be based on science.29 These mechanisms and levers can serve as initial starting points to create greater transparency. Longer-term solutions to advance data transparency may require further changes in policy and law.
Conclusion
Although progress has been made over the past decades in clinical trial transparency, and there are some successes for COVID-19 vaccines, there is still much room for improvement. The lack of adequate transparency about COVID-19 vaccine trials and their regulation cannot be dismissed as unfortunate, stubborn problems emblematic of the present culture in biomedicine. In a time of increasing public scrutiny, transparency of regulatory decision making leading to the approval of drug treatments and vaccines for COVID-19 is important to ensure patient and stakeholder trust. It is a scientific, moral and ethical imperative that access to complete trial data of these global public health interventions is urgently granted to patients, researchers and other key stakeholders.
Ethics statements
Patient consent for publication
Footnotes
Twitter @AnisaFarid, @RIATinitiative
Contributors ST and PD conceptualised the article and ST wrote the first draft. AR-F and KH led the data extraction and analysis. All authors (ST, AR-F, KH, TJ, PD) were involved in reviewing and editing the final manuscript.
Funding The Laura and John Arnold Foundation funds the RIAT Support Center, which supports the salaries of ST, AR-F, KH, TJ and PD.
Disclaimer The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, FDA/HHS, The University of Maryland, or the US Government.
Competing interests KH was supported by the Food and Drug Administration (FDA) of the US Department of Health and Human Services (HHS) as part of a financial assistance award U01FD005946, unrelated to this manuscript, totaling US$5000 with 100% funded by FDA/HHS. PD has received travel funds from the European Respiratory Society (2012) and Uppsala Monitoring Center (2018); grants from the FDA (through University of Maryland M-CERSI; 2020), Laura and John Arnold Foundation (2017–2022), American Association of Colleges of Pharmacy (2015), Patient-Centered Outcomes Research Institute (2014–2016), Cochrane Methods Innovations Fund (2016-2018) and UK National Institute for Health Research (2011–2014); was an unpaid IMEDS steering committee member at the Reagan-Udall Foundation for the FDA (2016–2020), and is an editor at The BMJ. The Laura and John Arnold Foundation funds the RIAT Support Center, which supports the salaries of ST, ARF, KH, TJ, and PD. TJ’s competing interests are online (https://restoringtrials.org/competing-interests-tom-jefferson/).
Provenance and peer review Commissioned; externally peer reviewed.