What is new?
- •
This is a comprehensive study to evaluate the impact of unpublished evidence found in Food and Drug Administration (FDA) drug review documents on conclusions of comparative drug effectiveness reviews.
- •
A minority of FDA documents contain unpublished evidence that can be highly useful in resolving publication bias and selective outcome and analysis reporting, identifying important harms, and filling gaps in knowledge about understudied subpopulations, outcomes, and comparisons
- •
Further work is needed to identify signals for when FDA documents are most useful in identifying unpublished evidence on drugs and other regulated products, such as time since FDA approval.
- •
The FDA should work both to improve access to information available in FDA preapproval reviews of new drugs and FDA postmarketing analyses, and improve the content and organization of that material to facilitate its use in identifying and using information not available in the peer-reviewed literature.
Reducing bias is the primary goal of using a systematic approach to reviewing evidence. A key component in this process is identifying gray literature (i.e., literature not regularly indexed in databases of the published literature). In examining gray literature, reviewers are searching both for unpublished studies and unpublished data supplemental to published studies. Gray literature may provide sufficient information to allow assessment of study quality and inclusion in a systematic review (SR). At minimum, failure to account for the body of unpublished evidence can diminish the value of an SR. Comparing unpublished reports of studies from gray literature sources with published manuscripts of the same studies may uncover unreported outcomes, changes in the definition of the primary outcome, or misrepresentation of the primary outcome [11].
Evaluating unpublished study data and information help reviewers to identify reporting bias. The main types of reporting bias are publication bias, outcome reporting bias, and analysis reporting bias [2]. Reporting bias has been found to occur frequently and to have impact on the conclusions of SRs [3], [4], [5]. Potential effects include overestimation of benefits, underestimation of harms, and potentially misleading conclusions of SRs, with examples of inaccurate conclusions about the relative benefits and harms of drugs [6], [7], [8], [9], [10].
The Institute of Medicine recommends searching US Food and Drug Administration (FDA) documents as one source for identifying gray literature in pharmacotherapy reviews [11]. As a part of the approval process for a drug in the United States, manufacturers submit new drug applications (NDAs) that include complete study protocols and results, including coded individual patient data, for review, and possible reanalysis by FDA staff. The FDA reviewers write reports regarding these studies and analyses. These documents (referred to here as FDA preapproval reviews) are written for advisory panels with the approval process and product labeling in mind. In addition to these documents, analyses conducted postmarketing by the FDA (referred to here as FDA postmarketing analyses) may also be useful in identifying unpublished evidence. These are conducted by FDA analysts on an ad hoc basis, to explore the incidence of serious harms identified in postmarketing use of a drug (e.g., through MedWatch reports), using data from trials submitted during the drug approval phase. These do not include FDA-required manufacturer-conducted postmarketing studies.
There is a growing body of research documenting reporting bias in drug trials based on comparisons of FDA documents to published literature (Table 1) [1], [12], [13], [14], [15], [16], [17], [18]. In these studies, 69–83% [1], [17] of trials in FDA documents were found in the published literature. A review of patient-level data found that 60% of patient data from trials of oseltamivir had not yet been published 12 years after its approval [18]. The FDA documents have raised safety concerns not found in published studies [12], [18], [19] and demonstrated greater incidence of adverse effects [13], [16]. For benefits, the direction of reporting bias varies by outcome and drug [1], [15], [16], [17]. However, few of these studies examined the documents as part of a broad SR process.
Although the documents have the potential to help resolve at least some reporting biases, reviewers are resistant to using them on a regular basis without knowing the likelihood that the time and effort involved will result in useful information. Searching FDA documents is standard for the Drug Effectiveness Review Project (DERP) and the Agency for Healthcare Research and Quality Effective Health care Program, but so far very little has been published about these experiences. The DERP has more than 8 years of experience in using these documents in producing comparative effectiveness reviews of drug therapies. The DERP is a collaboration of primarily state Medicaid agencies that pool funds to commission comparative SRs of drug therapy topics from two evidence-based practice centers. The participants use the reports in various ways to support their Medicaid-preferred drug lists, primarily serving as the evidence base for decisions about which drugs are preferred.
The purpose of this work is to help answer the question, “When conducting a review that includes drug therapies, should FDA documents be searched and reviewed for unpublished evidence?” Specifically, we wanted to identify the frequency of inclusion of information found in FDA documents as a proxy measure of effort and to determine the impact of the information found on conclusions.