Case report 1: reporting in peer-reviewed literature versus clinical trials registries and clinical study reports—melanoma

The primary peer-reviewed publication for METRIC is Flaherty et al.16 A secondary publication reports a quality-of-life analysis only and is not discussed further here.17 We identified the associated ClinicalTrials.gov webpage using the NCT01245062 trial identifier and the key outcomes investigated (this trial was not registered on EudraCT). We obtained a redacted version of the clinical study report from the GSK clinical study register (http://www.gsk-clinicalstudyregister.com).18 Aggregate data were not redacted and all end points from the trial were available.

Table 1 summarises the efficacy outcomes reported in each data source. An important difference was in the reporting of end point by treatment line, whereas Flaherty et al16 reported on the intention-to-treat (ITT) population (including patients who were chemotherapy naïve (first-line) and those who were chemotherapy experienced (second-line)), ClinicalTrials.gov reported progression-free survival (PFS) separately for first-line and second-line patients, data particularly useful in systematic reviews, and NMAs investigating treatments in melanoma for a specific treatment line. Furthermore, Flaherty et al16 reported results only for the ITT population, whereas ClinicalTrials.gov reported outcomes for several subpopulations including presence of brain metastases (table 1).

We found further discrepancies: Flaherty et al16 reported hazard ratios (HRs) for PFS for the subpopulations, but did not report median PFS. In contrast, ClinicalTrials.gov reported median PFS for these subpopulations but did not report HRs. Different presentations of data may consequently inhibit or complicate evidence syntheses.19

While all study outcomes considered in this example were available from the redacted clinical study report, as this consisted of 2236 pages, considerable time was needed to source these data. Additional end points were also available from the clinical study report with several results reported for multiple subgroups. However, due to the limited benefit to this case study and the extensive nature of the document, these additional end points were not extracted.

Case report 2: reporting in peer-reviewed literature versus clinical trials registries and clinical study reports—JIA

Key outcomes for the CHERISH trial were identified from the peer-reviewed publication20 and from ClinicalTrials.gov again using the unique study identifier NCT00988221 (table 2). No results were available from the EudraCT entry, and the clinical study report was not publically available from the Roche website.

All results reported in Brunner et al20 were available through ClinicalTrials.gov, but several outcomes available on ClinicalTrials.gov were not reported in the peer-reviewed publication.

A major difference was that ClinicalTrials.gov reported 104-week outcome data, whereas the latest time point reported by Brunner et al20 was week 40. Longer follow-up data are important as they permit assessment of long-term efficacy and safety; this is particularly true for trials in which survival is a measure of efficacy.20 Reporting multiple time points can also be informative when conducting longitudinal NMA.21

Efficacy was assessed using the JIA American College of Rheumatology (ACR) response criteria, within which there are JIA core response variables and JIA-ACR 30/50/70/90 responses (achieved if patients have a 30/50/70/90% or greater improvement in three or more of the JIA-core response variables (CRVs)). Brunner et al20 reported only the proportion of patients achieving the core response variables at week 16, whereas ClinicalTrials.gov reported, in addition, change in each JIA core response variables from baseline to week 16. Furthermore, ClinicalTrials.gov reported all efficacy outcomes at week 104, none of which were reported in the peer-reviewed publication.

Case report 3: augmenting the evidence base with data from regulatory authorities—HCV

To address the lack of head-to-head studies comparing simeprevir with boceprevir or telaprevir in treatment-naïve HCV patients, our group indirectly compared the efficacy and safety of these regimens in an NMA.14 Results were presented at the American Association for the Study of Liver Disease 2013 meeting,14 including a subgroup analysis to determine the effect of baseline METAVIR score on sustained virological response to treatment. Although METAVIR subgroup data for telaprevir were available in the primary publication, those for boceprevir were not;22 these latter were reported in the EMA EPAR (table 3). Without the addition of these EPAR data, an indirect comparison of simeprevir versus telaprevir and boceprevir, in accordance with the licensing in patients with specific stages of fibrosis would not have been possible.

Case report 4: augmenting the evidence base with data from regulatory authorities—COPD

Although mortality is an increasingly important consideration in COPD, publications of trial data do not always report on-trial mortality. During a review by our research group,23 we identified Dong et al15 which conducted a systematic review and NMA of RCTs in COPD reporting overall death or cardiovascular death. Overall and cardiovascular mortality data were unavailable from four peer-reviewed publications, but these data were sourced from the FDA website allowing a more comprehensive review than would have been possible using peer-reviewed publications only (table 4).

Limited availability of end point data is a constraint of many systematic reviews and meta-analyses. In the context of healthcare decision-making, it is vital to have all available data to make the best informed decisions. Where evidence is lacking from conventional peer-reviewed sources, efforts should be made to retrieve data from alternative sources in order of usability.

Health technology assessment (HTA) agencies expect evidence submissions to incorporate all the available, relevant evidence and responsibility lies with the systematic reviewer and meta-analyst to ensure this evidence is captured.

Through our examples, we show differences exist in the level of reporting between peer-reviewed publications and the corresponding clinical trials registries webpage, the website of regulatory agencies and also the trial clinical study report where available. Arguably the most important difference identified in reporting was end point availability by previous therapies for the METRIC trial. The peer-reviewed publication reported PFS results for the ITT population regardless of their previous exposure to chemotherapy. ClinicalTrials.gov reported results by chemotherapy status: naïve or experienced. In the current landscape of oncology drug approval, where products are often licensed for a specific set of patients based on their previous treatment exposure, these results could impact on any systematic reviews or NMAs in melanoma.28 As the EMA and FDA now require all trials within their jurisdiction to be registered on either EudraCT or ClinicalTrials.gov, results will increasingly be made available through these registries.

Schroll et al29 recently investigated the accessibility and usefulness of data from EMA and FDA reports and concluded while FDA reports provide more substantial outcome data, they are more difficult to navigate than EMA reports, are often considerably larger and lack standardisation in reporting.29 Locating these documents on the EMA and FDA websites can be challenging; there is a need for further improvement in ease of website navigation and in clarity of reporting.30 ,31

Although several authors have investigated the influence of unpublished data from FDA reports on systematic reviews and NMAs, disparity exists in the conclusions reached. MacLean et al32 searched for FDA reviews of new drug applications for selected non-steroidal anti-inflammatory drugs (NSAIDs) and compared these with published trial data, focussing on dyspepsia as a toxicity outcome. They found no significant difference between the published and unpublished data with respect to pooled estimates of the risk of dyspepsia. Hart et al33 recalculated selected meta-analyses that were conducted on trials that had unpublished FDA data available. Comparison of the summary data from reanalyses incorporating unpublished FDA data revealed increased and reduced drug efficacy in approximately equal numbers. McDonagh et al34 investigated Drug Effectiveness Review Project (DERP) reports, including data from FDA documents. They reported that 30 of 175 FDA reviews contained unpublished data that were used in 34 DERP reports. Conclusions were affected by the inclusion of these unpublished data in 11 of the DERP reviews, 9.6% of the total DERP reports investigated. Including unpublished data may not always influence the results of evidence syntheses, but it is apparent that the potential for it to do so exists.

Our case studies were selected based on our previous experience in systematic reviews in these disease areas. Although not systematic in our selection, these isolated examples illustrate the need to consider the additional sources of unpublished data discussed in this paper.