Refining the E in EBM ===================== * Tom Jefferson * epidemiology * information management This is the second in a series of notes addressing the need to change our evidence sources for assessing the effects of pharmaceuticals and biologics because of the danger of unrecognised reporting bias. In this note I shall discuss the possibility of using a shorter way of redefining the E than using clinical study reports. In the first note—‘Redefining the ‘E’ in EBM’— I made the point that trial publications (if they exist) are the tip of the data iceberg.1 The ‘iceberg’ is made up of a complex and massive reality of thousands of pages of clinical study reports, manuals, forms and slides of meetings invisible below the waterline. The submerged part is likely to represent anything between 1 and 8000 pages of published material.2 Even more importantly, the submerged material has the potential to change the conclusions of systematic reviews. To prove this point we have presented a list of references in our first editorial1 and at the Restoring Invisible and Abandoned Trials (RIAT) ([https://restoringtrials.org/whatisriat/](https://restoringtrials.org/whatisriat/)) initiative. I have another reason to be wary of the published written word, especially in prestigious journals. The reason is not new, but is not widely known. It is the refusal of some authors and editors to correct misreporting in their trials, even when presented with the evidence of misreporting. BMJ-EBM recently published the evidence of misreporting of an active ingredient as an ‘inert placebo’ in pivotal trials of human papillomavirus vaccines.3 Long before we embarked in the restoration, following RIAT rules, we contacted sponsors and authors of the misreported articles publicly inviting them to take action.4 We got no response. Once the restoration was complete we sent our manuscript to the same journal who had presided over the publication of three of the misreported trials. No action was taken. We encountered the same problem with the journals which published ghost-written pivotal trials of the antiviral Tamiflu.5 In our initial note,1 we suggested addressing reporting bias by including regulatory data in our reviews and briefly mentioned “different types of regulatory data now on offer”,1 such as regulators’ reports. We first came across regulators’ reports in the preparation of our Cochrane review on neuraminidase inhibitors, but we were not the first to use them.6 Because clinical study reports had not yet been released to us,6 we accessed 2000 pages of letters, medical and statistical officer reports and minutes of meetings preparatory to the granting of the licence by the US Food and Drugs Administration (FDA). The documents (which are cited by file number and content in our review) provided a unique insight into the workings of regulators and their reviews of what we could not (yet) see: the thousands of pages of clinical study reports on both neuraminidase inhibitors, Relenza and Tamiflu.6 The FDA documents were extremely detailed and impressive king-sized peer-review reports. Unlike journal peer-review reports, they addressed several trials in the same development programme, and provided tables listing all the trials in the programme of interest to the regulator. The listing sometimes eliminated the problem of identification and (at times) publication bias altogether. These reports (all with different names) are now available from several sources and easily downloadable. Some are very difficult to follow but others are eminently readable and authoritative.7 Over time, FDA reports have become more readable, although there is a worry that recent proposal to conflate various reports might affect detailing.8 Table 1 summarises the main regulators’ reports on offer and their sources. View this table: [Table 1](http://ebm.bmj.com/content/25/6/189/T1) Table 1 Easily accessible regulators’ assessment reports of pharmaceuticals by National Competent Authority The average length and complexity of these reports is way below that of an evidence development programme of clinical study reports, typically being in the low hundreds of pages. However, unlike most clinical study reports, regulator’s comments reports are easily accessible without registration and can be very useful as a halfway house between publications and clinical study reports. Their biggest drawback is that they represent a third party’s view of what you cannot see: the clinical study reports. Some National Competent Authorities’ websites provide further information such as assessment history, labels patient information leaflets, postmarketing activities which could be invaluable in assessing the long-term effects of an intervention (table 1). Regulators’ reports should be routinely introduced as a source of data and information in evidence synthesis of pharmaceuticals when time is short and clinical study reports are not available. The halfway house represents a practical refining of the Evidence in EBM in the current scenario of universal reliance on inscrutable dubious publications and unaccountable authors and editors. ## Footnotes * Contributors TJ is the sole author. * Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors. * Competing interests TJ has provided a link to a webpage with extensive disclosure in the manuscript. * Patient consent for publication Not required. * Provenance and peer review Commissioned; internally peer reviewed. ## References 1. Jefferson T , Jørgensen L . Redefining the ‘E’ in EBM. BMJ Evid Based Med 2018;23:46–7.[doi:10.1136/bmjebm-2018-110918](http://dx.doi.org/10.1136/bmjebm-2018-110918) [FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NToiZWJtZWQiO3M6NToicmVzaWQiO3M6NzoiMjMvMi80NiI7czo0OiJhdG9tIjtzOjIwOiIvZWJtZWQvMjUvNi8xODkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 2. Doshi P , Jefferson T . Clinical study reports of randomised controlled trials: an exploratory review of previously Confidential industry reports. BMJ Open 2013;3:e002496. [doi:10.1136/bmjopen-2012-002496](http://dx.doi.org/10.1136/bmjopen-2012-002496) pmid:http://www.ncbi.nlm.nih.gov/pubmed/23447465 [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=23447465&link_type=MED&atom=%2Febmed%2F25%2F6%2F189.atom) 3. Doshi P , Bourgeois F , Hong K , et al . Adjuvant-containing control arms in pivotal quadrivalent human papillomavirus vaccine trials: restoration of previously unpublished methodology. BMJ Evid Based Med 2020;25:213–9.[doi:10.1136/bmjebm-2019-111331](http://dx.doi.org/10.1136/bmjebm-2019-111331) pmid:32184277 [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiZWJtZWQiO3M6NToicmVzaWQiO3M6ODoiMjUvNi8yMTMiO3M6NDoiYXRvbSI7czoyMDoiL2VibWVkLzI1LzYvMTg5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 4. Doshi P . Rapid response: call to action: RIAT restoration of a previously unpublished methodology in Gardasil vaccine trials. Available: [https://www.bmj.com/content/346/bmj.f2865/rr-7](https://www.bmj.com/content/346/bmj.f2865/rr-7) 5. Jefferson T , Jones MA , Doshi P , et al . Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database of Systematic Reviews 2012. 6. Turner EH , Matthews AM , Linardatos E , et al . Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252–60.[doi:10.1056/NEJMsa065779](http://dx.doi.org/10.1056/NEJMsa065779) pmid:http://www.ncbi.nlm.nih.gov/pubmed/18199864 [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1056/NEJMsa065779&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=18199864&link_type=MED&atom=%2Febmed%2F25%2F6%2F189.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000252375600006&link_type=ISI) 7. Turner EH . How to access and process FDA drug approval packages for use in research. BMJ 2013;347:f5992. [doi:10.1136/bmj.f5992](http://dx.doi.org/10.1136/bmj.f5992) pmid:http://www.ncbi.nlm.nih.gov/pubmed/24126858 [FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDcvb2N0MTRfMi9mNTk5MiI7czo0OiJhdG9tIjtzOjIwOiIvZWJtZWQvMjUvNi8xODkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 8. US Food and Drugs Administration. New drugs regulatory program modernization: improving approval package documentation and communication. A notice by the food and drug administration. Available: [https://www.federalregister.gov/documents/2019/06/27/2019-13751/new-drugs-regulatory-program-modernization-improving-approval-package-documentation-and](https://www.federalregister.gov/documents/2019/06/27/2019-13751/new-drugs-regulatory-program-modernization-improving-approval-package-documentation-and) [Accessed 27 Jun 2019].