Article Text
PDF
Statistics from Altmetric.com
Introduction
There has been a gradual realisation that sources of evidence historically considered to be reliable (such as peer-reviewed literature) are affected by reporting bias. Reporting bias generally refers to selective reporting of research depending on the nature and direction of research results. Reporting bias includes publication bias1–3 and outcome reporting bias,1–5 among many others.6
Systematic reviews of randomised controlled trials play an important role in health decision-making. Most of these analyse data extracted from journal publications despite there being good evidence that reporting bias is widespread. As trials with unfavourable results are less likely to be published and unfavourable outcomes less likely to be reported within publications, the evidence base is often incomplete and skewed towards a positive spin. Systematic reviews that use only published data perpetuate such bias and possibly compound the issue through the credibility afforded by the systematic review, particularly if carried out by a trusted source such as Cochrane.
In a survey of 348 systematic reviews published in 2014, around three-quarters relied solely on data published in peer-reviewed journals.7 Of those that accessed other sources, data from trials registries (such as ClinicalTrials.gov), conference proceedings or contacting authors were the most used. No reviews reported using or attempting to obtain regulatory information even though the majority of the reviews evaluated drug interventions.7 A survey of 2184 Cochrane authors also found that contacting ‘trialists/investigators’ was one of the most common methods for accessing unpublished data and that data from manufacturers or from regulatory agencies were rarely obtained.8
Clinical study reports (CSRs) are documents prepared and submitted to regulators to obtain a marketing licence for a pharmaceutical. They represent the most complete account of the planning, execution and results of such trials. CSRs contain some of the same information as journal papers (ie, rationale, objectives, methods, results, discussion/conclusion), but are substantially more detailed with numerous large tables and figures, and datasets not constrained by page limits. A CSR for a single trial may be hundreds, thousands or even tens of thousands of pages in length but they are generally relatively straightforward to navigate owing to their standardised and structured format. CSRs generally contain, as appendices, important study documents including the study protocol and any amendments, statistical analysis plan and any amendments, blank case report forms, patient information sheet, blank informed consent forms and individual patient listings.9
There are indications that CSRs may be incomplete, and in some cases, may be internally inconsistent between different sections of the same CSR.10 However, when comparing different data sources for the same trial, CSRs provide the greatest breadth and depth of information compared with journal articles, trial register data and grey literature. Aggregate data on subpopulations are often found in CSRs and can provide a source of further analysis. Such a wealth of information gives a fuller and more reliable picture of a trial’s strengths and weaknesses, as well as a more reliable assessment of the benefits and harms of the studied interventions.
CSRs and other regulatory documents generally only exist for drugs and biologics. Non-pharmaceutical interventions (such as implantable devices, surgery, rehabilitation, behavioural (psychosocial) interventions and diagnostics) are responsible for a large part of healthcare expenditure and regulatory activity, but they do not generally produce CSRs. Transparency has generally been increasing in this area, although at a slower pace in the field of devices. Publicly funded trials, even of drugs and biologics, do not usually produce internationally standardised documentation, similar to a CSR.
In late 2010, the European Medicines Agency (EMA) began releasing CSRs of drugs and biologics on request under its Policy 0043.11 In October 2016, the EMA began to release CSRs prospectively under Policy 0070 (https://clinicaldata.ema.europa.eu).11 12 This policy applies only to documents received after 1 January 2015. Documents available from the EMA under Policy 0070 normally include the clinical overview, clinical summary and CSRs of individual trials.12 In 2017, Health Canada published a report announcing an initiative to publicly release clinical information concerning drugs and devices under an eventual EMA Policy 0070-like mechanism.13 In March 2018, the Food and Drug Administration (FDA) publicly released a CSR in a pilot programme that will eventually include nine new drug approvals.14 Some manufacturers are making CSRs available to reviewers (https://restoringtrials.org/insitutions-offering-data-access/). GlaxoSmithKline (GSK), for example, allows CSRs to be freely downloaded from its clinical study register, although the documents may be heavily redacted and incomplete. Other manufacturers are making CSRs available to researchers on request and after review and approval of their project proposal.
Rationale for the consideration of regulatory documents (including CSRs) as sources of data for inclusion in systematic reviews
Reporting biases can generally only be detected when two or more reports of the same trial are compared, for example, peer-reviewed publications compared with relevant regulatory documents. In addition to reporting bias, lack of transparency and lack of detail in journal publications may prevent or hinder detailed analyses of data which could be relevant to specific subpopulations potentially benefiting from or being harmed by the intervention.15 This situation is likely to be the consequence of compressing thousands of pages of text and tables into the historically restricted confines of a printed journal article.15
Table 1 contains a selected and illustrative list of studies that have compared different sources of data for the same trial, such as publication versus CSR or publication versus trial register entries. Although this is not an exhaustive list of all such studies, it covers >50 different interventions and offers insights into the ways in which reporting bias affects the biomedical literature.
Examples of studies comparing different sources of data for the same trials
The studies in table 1 strongly suggest that discrepancies in the reporting of trials across different sources of data are common. There are, however, limitations when interpreting discrepancies. First, different types of trial documents may have very different objectives. CSRs, for example, inform regulators and, by law, provide a comprehensive record of a study. Trials registers, in contrast, are primarily a visible collection of trial data, yet their reporting can be either absent or incomplete. Under some circumstances (such as for non-phase I trials of FDA-regulated drugs, devices or biologics), reporting of trials within ClinicalTrials.Gov, including the submission of results, is compulsory.16 There are also requirements for clinical trials funded by the US National Institutes of Health such as registration and reporting of results on ClinicalTrials.Gov,17 but these requirements are not always adhered to nor adequately policed.18
The generalisability of each finding of the studies in table 1 to the larger population of trials or topic areas that exist is debatable, and it is unclear whether reporting biases are lessening over time. Some journals have taken steps to limit the bias introduced by the current format of trial reporting, by requiring adherence to Consolidated Standards of Reporting Trials,19 by publishing the trial protocol, statistical analysis plan or supplementary data as an online appendix or by requiring data sharing as a condition of publication.20–22 As it is impossible to squeeze thousands of pages worth of information into a 10-page publication and the resulting information selection is based on unknown criteria, the authors of trial publications can, where these exist, provide links to the relevant CSR and other summary data.
Supplemental material
We are currently aware of four systematic reviews (a Cochrane review of neuraminidase inhibitors,23 twin reviews of recombinant human bone morphogenetic protein 2 (rhBMP-2)24 25 and a review of reboxetine26) allowing an assessment of the contribution of regulatory data compared with the same trial data from published journal articles.
In the case of rhBMP-2, both CSRs and individual participant data (supplied separately by the manufacturer via the Yale Open Data Access Project) were included in the twin reviews,24 25 while the Cochrane review of neuraminidase inhibitors and the review of reboxetine were based on CSRs.23 26 In all cases, the conclusions were that important aspects of the reviews were changed with access to the more complete data available in the CSRs. Access to the CSRs also provided a deeper understanding of the strengths and limitations of the trial evidence. In the case of the review of reboxetine, the inclusion of CSR data changed the conclusions of the review and allowed quantification of the exaggeration in favour of the effects of reboxetine compared with placebo and other SSRIs.26 The Cochrane review of neuraminidase inhibitors for influenza also found FDA drug-approval packages (medical and statistical officer reviews) to be an important source of data and detail.
As systematic reviews are considered as a gold standard of reliable research synthesis, we need to pay attention to the issue of reporting bias and to address whether, and how to decide when, accessing regulatory data, including CSRs, might offer a solution. The approach, however, is new and unfamiliar to most systematic reviewers and at the time of writing, regulatory data are not always immediately available. When available, using such documents can involve reviewing very large quantities of information, which may be time-consuming and resource intensive. Alternatively, it may be less time consuming than trying to assemble complete study data from information that is fragmented across several publications and unpublished sources such as trials registers. Thus, a framework to help identify where using data from regulatory documents is likely to matter most and prioritising those reviews which should adopt such an approach, will be helpful for groups grappling with how to respond to the increasing availability of these new sources of information.
Current practice
To raise awareness of the above issues and to assess the level of familiarity with and experiences of using data from CSRs and other regulatory documents within systematic reviews, we surveyed Cochrane and non-Cochrane authors to gauge how many had considered using regulatory data and how many had actually included such data in their reviews. There were 160 respondents with results mostly showing a lack of familiarity with regulatory sources of data, barriers to access and lack of resources to do so. The main rationale for authors seeking regulatory data, however, was minimisation of bias.27
The circumstances under which CSRs and/or other regulatory documents should be considered for inclusion in systematic reviews
We concluded from the survey findings that the systematic review community is ready to consider using data from CSRs and other regulatory documents within systematic reviews. However, owing to the additional time and resource requirements that may be required to use these data sources, use should be focused on review topics where the data are needed most. We were unable to identify any research on the topic of how to decide whether to incorporate CSRs and other regulatory documents into systematic reviews, that is, a rule for determining which reviews would most benefit from the inclusion of such data.
We therefore created an initial list of reasons (or triggers) for seeking and using such data through discussion among our author group. Our list was a product of our opinion and experience. We then carried out a follow-up targeted survey in which we asked respondents to rate the importance of each criterion in our list. This survey was sent to the 21 (of 27) systematic review authors who had used, requested or considered using regulatory data in their review and had agreed to participate in a follow-up survey. Fourteen of 21 (66%) provided a response.27
Table 2 shows our final list of criteria (after addressing review authors feedback) for assessing whether to include regulatory data of a drug or biologic in a systematic review. The variables are self-explanatory, reflecting either known or suspected bias in published results or the potential for greatest impact in terms of public health, for example, what are the human costs of acting on biased estimates of effectiveness or harm?
Criteria for assessing whether to include regulatory data of a drug or biologic in a Cochrane review (not in order of priority)
There is no proposed scoring or algorithm for combining criteria to identify priority topics or topic areas. The relative importance of criteria listed in table 2 will depend very much on context, and prioritisation is inevitably a somewhat subjective process. The list is not meant to be onerous. Systematic and formal evaluation of each of the 18 criteria is not required. We suggest reviewers instead focus on the items in the list that are most relevant to their research area of interest. The list is subject to revision and may even become obsolete over time as we learn more about the added value of using CSRs for systematic review, and if the ease of accessing CSRs becomes less problematic and burdensome in the future.
Limitations when using CSRs
There are potential limitations when using CSRs as the data source for systematic reviews. First, CSRs are written for regulators and may contain sensitive information that needs to be redacted. Redactions may delay the time it takes to obtain CSRs and the applied redactions may be extensive, masking important information for inclusion in systematic reviews. Second, although data on adverse events are required to be included in CSRs at the individual participant level, these data may not be fully provided to regulators (although regulators can request these data) or to systematic reviewers who request CSRs directly from manufacturers. For example, completed case report forms are not always held by the EMA and serious adverse event narratives are redacted from CSRs on GSK’s clinical trial register. Third, complete CSRs may not be held by the EMA, and some CSRs may not be held at all, although a complete list of all clinical trials conducted by a manufacturer does form part of the regulatory submission. Fourth, a CSR is a compiled report of a study and not the study’s underlying raw data. Despite all their strengths as a rich source of data, some investigators have identified deficiencies in the reporting in CSRs, specifically the completeness and consistency of reporting of adverse events.28–31
Conclusion
Regulatory documents are a complex and underused source of highly detailed data that could be included in systematic reviews. Although the steps to identify and extract and analyse data are broadly the same as for other sources of data, the resource implications of their use may not be. The results of our surveys and our own experience indicate that the use of regulatory documents should be considered, especially when the intervention in question is of high importance and when risk of reporting bias is great.
Acknowledgments
The authors are grateful for helpful comments on an earlier version of the draft from the following: Rene Allard, Chris Champion, Kay Dickersin, Kristina Charlotte Dietz, Jean-Mac Ferran, Mark Helfand, Toby Lasserson, Sherry Meeh, Kim Musgrave, Sarah Nevitt, Erick Turner and Beate Wieseler. The authors are very grateful to those who responded to calls for feedback and the authors endeavoured to incorporate their comments. Incorporation of feedback comments and/or listing of names above does not infer respondents’ endorsement of this work.
References
Footnotes
Contributors All authors contributed to the research reported and the writing of the article.
Funding This research was supported by funds provided by Cochrane Methods Innovation Fund.
Disclaimer The views expressed in this final report are those of the authors and not necessarily those of Cochrane or its registered entities, committees or working groups.
Competing interests All authors except AH report funding from the Cochrane Methods Innovation Fund to undertake this project. In addition, TJ was a corecipient of a UK National Institute for Health Research grant (HTA—10/80/01 Update and amalgamation of two Cochrane reviews: neuraminidase inhibitors for preventing and treating influenza in healthy adults and children; https://www.journalslibrary.nihr.ac.uk/programmes/hta/108001#/). TJ is occasionally interviewed by market research companies about phase I or II pharmaceutical products. In 2011–2014, TJ acted as an expert witness in a litigation case related to the antiviral oseltamivir, in two litigation cases on potential vaccine-related damage and in a labour case on influenza vaccines in healthcare workers in Canada. He has acted as a consultant for Roche (1997–1999), GSK (2001–2002), Sanofi-Synthelabo (2003) and IMS Health (2013). In 2014–2016, TJ was a member of three advisory boards for Boehringer Ingelheim. TJ was a member of an independent data monitoring committee for a Sanofi Pasteur clinical trial on an influenza vaccine. TJ has a potential financial conflict of interest on the drug oseltamivir. TJ was a cosignatory of a complaint to the European Ombudsman on maladministration in relation to the EMA investigation of possible harms from HPV vaccines. PD reports reimbursement from the European Respiratory Society, grants from UK NIHR, the American Association of Colleges of Pharmacy and the Laura and John Arnold Foundation, all outside the submitted work. PD is also an associate editor of The BMJ and an unpaid member of the IMEDS steering committee at the Reagan-Udall Foundation for the FDA, which focuses on drug safety research. IB is deputy director of the French EQUATOR centre; and member of the CONSORT steering committee. CH reports he has received expenses and fees for his media work. He has received expenses from WHO and holds grant funding from the NIHR, the NIHR School of Primary Care Research, THE NIHR BRC Oxford and WHO. He has received financial remuneration from an asbestos case. He has also received income from the publication of a series of toolkit books published by Blackwells. On occasion, he receives expenses for teaching EBM and is also paid for his GP work in NHS out of hours. CEBM jointly runs the Evidence Live Conference with The BMJ and the Overdiagnosis Conference with some international partners, which are based on a non-profit making model. He is Editor-in-Chief of BMJ Evidence-Based Medicine. LAS is an employee of the University of York. She previously received a research grant from YODA for her research team to independently analyse individual participant data and data from CSRs supplied by Medtronic.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.