Review Article
US Food and Drug Administration documents can provide unpublished evidence relevant to systematic reviews

https://doi.org/10.1016/j.jclinepi.2013.05.006Get rights and content

Abstract

Objectives

A key systematic review (SR) methodology is comprehensive searching. The Drug Effectiveness Review Project (DERP) SRs search US Food and Drug Administration (FDA) documents to identify unpublished evidence. This study evaluates the success of those efforts.

Study Design and Setting

We examined DERP reports published since 2003 for the use of FDA preapproval and postmarketing documents. We categorized evidence as (1) unique unpublished studies, (2) supplemental unpublished data, or (3) FDA postmarketing data analysis. Three reviewers independently assigned predetermined impact categories (e.g., qualitative or quantitative usage, fills gaps, confirms findings, and alters conclusions), resolving disagreements through consensus.

Results

Among 114 DERP reports, 19% included unpublished studies and/or supplemental data and 10% included postmarketing analyses. From 175 preapproval documents, 14% provided eligible unpublished studies and 4.0% supplemental unpublished data that helped confirm previous findings, identify important harms, and fill gaps in knowledge about understudied subpopulations, outcomes, and comparisons. Report conclusion statements changed in 9 of 33 instances of premarketing documents compared with 4 of 12 postmarketing analyses.

Conclusions

The FDA documents can provide important unpublished evidence for SRs, although in a small proportion of cases. Future research should identify attributes that predict which reviews may benefit most from review of FDA documents.

Introduction

What is new?

  • This is a comprehensive study to evaluate the impact of unpublished evidence found in Food and Drug Administration (FDA) drug review documents on conclusions of comparative drug effectiveness reviews.

  • A minority of FDA documents contain unpublished evidence that can be highly useful in resolving publication bias and selective outcome and analysis reporting, identifying important harms, and filling gaps in knowledge about understudied subpopulations, outcomes, and comparisons

  • Further work is needed to identify signals for when FDA documents are most useful in identifying unpublished evidence on drugs and other regulated products, such as time since FDA approval.

  • The FDA should work both to improve access to information available in FDA preapproval reviews of new drugs and FDA postmarketing analyses, and improve the content and organization of that material to facilitate its use in identifying and using information not available in the peer-reviewed literature.

Reducing bias is the primary goal of using a systematic approach to reviewing evidence. A key component in this process is identifying gray literature (i.e., literature not regularly indexed in databases of the published literature). In examining gray literature, reviewers are searching both for unpublished studies and unpublished data supplemental to published studies. Gray literature may provide sufficient information to allow assessment of study quality and inclusion in a systematic review (SR). At minimum, failure to account for the body of unpublished evidence can diminish the value of an SR. Comparing unpublished reports of studies from gray literature sources with published manuscripts of the same studies may uncover unreported outcomes, changes in the definition of the primary outcome, or misrepresentation of the primary outcome [11].

Evaluating unpublished study data and information help reviewers to identify reporting bias. The main types of reporting bias are publication bias, outcome reporting bias, and analysis reporting bias [2]. Reporting bias has been found to occur frequently and to have impact on the conclusions of SRs [3], [4], [5]. Potential effects include overestimation of benefits, underestimation of harms, and potentially misleading conclusions of SRs, with examples of inaccurate conclusions about the relative benefits and harms of drugs [6], [7], [8], [9], [10].

The Institute of Medicine recommends searching US Food and Drug Administration (FDA) documents as one source for identifying gray literature in pharmacotherapy reviews [11]. As a part of the approval process for a drug in the United States, manufacturers submit new drug applications (NDAs) that include complete study protocols and results, including coded individual patient data, for review, and possible reanalysis by FDA staff. The FDA reviewers write reports regarding these studies and analyses. These documents (referred to here as FDA preapproval reviews) are written for advisory panels with the approval process and product labeling in mind. In addition to these documents, analyses conducted postmarketing by the FDA (referred to here as FDA postmarketing analyses) may also be useful in identifying unpublished evidence. These are conducted by FDA analysts on an ad hoc basis, to explore the incidence of serious harms identified in postmarketing use of a drug (e.g., through MedWatch reports), using data from trials submitted during the drug approval phase. These do not include FDA-required manufacturer-conducted postmarketing studies.

There is a growing body of research documenting reporting bias in drug trials based on comparisons of FDA documents to published literature (Table 1) [1], [12], [13], [14], [15], [16], [17], [18]. In these studies, 69–83% [1], [17] of trials in FDA documents were found in the published literature. A review of patient-level data found that 60% of patient data from trials of oseltamivir had not yet been published 12 years after its approval [18]. The FDA documents have raised safety concerns not found in published studies [12], [18], [19] and demonstrated greater incidence of adverse effects [13], [16]. For benefits, the direction of reporting bias varies by outcome and drug [1], [15], [16], [17]. However, few of these studies examined the documents as part of a broad SR process.

Although the documents have the potential to help resolve at least some reporting biases, reviewers are resistant to using them on a regular basis without knowing the likelihood that the time and effort involved will result in useful information. Searching FDA documents is standard for the Drug Effectiveness Review Project (DERP) and the Agency for Healthcare Research and Quality Effective Health care Program, but so far very little has been published about these experiences. The DERP has more than 8 years of experience in using these documents in producing comparative effectiveness reviews of drug therapies. The DERP is a collaboration of primarily state Medicaid agencies that pool funds to commission comparative SRs of drug therapy topics from two evidence-based practice centers. The participants use the reports in various ways to support their Medicaid-preferred drug lists, primarily serving as the evidence base for decisions about which drugs are preferred.

The purpose of this work is to help answer the question, “When conducting a review that includes drug therapies, should FDA documents be searched and reviewed for unpublished evidence?” Specifically, we wanted to identify the frequency of inclusion of information found in FDA documents as a proxy measure of effort and to determine the impact of the information found on conclusions.

Section snippets

Methods

A full description of methods used to identify and use FDA documents for DERP is presented in Appendix at www.jclinepi.com. We identified all DERP reports containing information from FDA documents by searching the results sections in reports using the keywords: “FDA,” “Food,” “Center for Drug,” “CDER,” “statistical review,” and “medical review.” We included data from all SRs from 2003 to July 2011. Within a given SR, multiple unique occurrences of information could be identified and counted

Results

Since 2003, 114 DERP reports have been completed, covering 39 topics and including 75 updates. For example, attention deficit hyperactivity disorder (ADHD) is one topic for which we produced one original report that looked at 27 brand name drug products and four updates that added six new drug products [20], [21], [22], [23]. From the perspective of the reviewer, the yield for the workload can be viewed on a per-report basis or relative to the number of drugs as some reviews will include few

Discussion

Although the problem of publication bias is widely acknowledged [5], [15], [39], few studies have looked specifically at the impact of including unpublished data from FDA documents on the conclusions of comparative effectiveness reviews. To date, our study is the most comprehensive to assess the impact of including such information for a wide variety of drug classes on the findings and conclusions of reviews. Of the 175 FDA preapproval reviews, 30 (17%) led to the identification of unpublished

Conclusion

Review of evidence in FDA documents can provide important unpublished evidence on benefits and harms that is highly relevant to SRs, although in a small proportion of cases. Further work is needed to identify attributes that may predict which reviews will most likely benefit from the review of FDA documents.

Acknowledgments

The authors would like that thank Allison Low, BS, for work done in collecting and preparing data for this report, and Leah Williams, BS, for work in editing and formatting the manuscript.

References (54)

  • J. Lenzer

    FDA is incapable of protecting US "against another Vioxx"

    BMJ

    (2004)
  • J.P. McCormack et al.

    Digging for data from the COX-2 trials

    CMAJ

    (2002)
  • T.J. Moore

    Deadly medicine: why tens of thousands of heart patients died in America's worst drug disaster

    (1995)
  • Finding what works in health care: standards for systematic reviews

    (2011)
  • J.S. Floyd et al.

    Prasugrel as a potential cancer promoter: review of the unpublished data

    Arch Intern Med

    (2010)
  • M. Man-Son-Hing et al.

    Quinine for nocturnal leg cramps: a meta-analysis including unpublished data

    J Gen Intern Med

    (1998)
  • S.E. Nissen et al.

    Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes

    N Engl J Med

    (2007)
  • K. Rising et al.

    Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation

    PLoS Med

    (2008)
  • B. Hart et al.

    Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses

    BMJ

    (2012)
  • E.H. Turner et al.

    Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database

    PLoS Med

    (2012)
  • T. Jefferson et al.

    Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children

    Cochrane Database Syst Rev

    (2012)
  • S.E. Nissen et al.

    Effect of muraglitazar on death and major adverse cardiovascular events in patients with type 2 diabetes mellitus

    JAMA

    (2005)
  • M. McDonagh et al.

    Drug class review: pharmacologic treatments for ADHD. Update #2

  • M. McDonagh et al.

    Drug class review: pharmacologic treatments for attention deficit hyperactivity disorder. Update #3

  • M. McDonagh et al.

    Drug class review: pharmacologic treatments for ADHD. Update #1

  • M. McDonagh et al.

    Drug class review: pharmacologic treatments for attention deficit hyperactivity disorder. Update #4

  • Medical review of Nexium (Esomeprazole Magnesium) delayed-release Capsules [CR]

    (2001)
  • Cited by (0)

    View full text