Catalogue of bias: publication bias =================================== * Nicholas J DeVito * Ben Goldacre * medical ethics ## Background Dickersin and Min define publication bias as the failure to publish the results of a study ‘on the basis of the direction or strength of the study findings’.1 This non-publication introduces a bias which impacts the ability to accurately synthesise and describe the evidence in a given area.2 Publication bias is a type of reporting bias and closely related to dissemination bias, although dissemination bias generally applies to all forms of results dissemination, not simply journal publications. A variety of distinct biases are often grouped into the overall definition of publication bias.3 4 There are a number of risk factors and causes for publication bias identified in the literature.5 Research has shown causes of publication bias ranging from trialist motivation, past experience, and competing commitments; perceived or real lack of interest in results from editors, reviewers, or other colleagues; or conflicts of interest that would lead to the suppression of results not aligned with a specific agenda.3 6–9 The role of journal editors is particularly complex as the gatekeepers to publication. Significant results are more widely cited in medicine aligning the incentives of both investigators and editors towards these studies.10 A review by Song and colleagues reports studies showing that strength and direction of study results do not impact the acceptance rates of submitted manuscripts; however, this research may not account for researchers selectively withholding poorly conducted or presented research with non-significant findings.3 Whether there is an editorial bias or not, the persistence of this belief among investigators may also impact which studies are submitted for publication. ## Examples In his 1986 piece on publication bias in clinical research, R.J. Simes compared data reported to a cancer trial registry with data from the published literature on the survival impact of two cancer therapies. Simes found that in both instances, the survival impact of the therapies either disappeared or was substantially less when the subset of data published in the academic literature was compared against the more complete data from a registry.11 Publication bias is commonly assessed in cohort studies, such as the Simes example, where publication status is ascertained for a group of known completed trials. Research into treatments for depression provides a more recent example. Turner and colleagues reported that 31% of a cohort of studies for antidepressant drugs registered and reported to the Food and Drug Administration (FDA) were never published. The literature included 91% positive studies while the larger FDA cohort only contained 51% positive studies.12 Driessen and colleagues reviewed all NIH grants for psychological treatments for depression from 1972 to 2008. When publications were not found, the data were requested from the grant recipients. Thirteen out of 55 trials (23.6%) arising from this cohort were never published. The effect size of psychological treatments was reduced by 25% when unpublished data were included in the pooled analysis with the published data.7 Results of cohort studies such as these have been collected in systematic reviews. A 2013 systematic review by Dwan and colleagues reviewed 20 cohort studies on publication bias in randomised controlled trials and showed ‘statistically significant outcomes had a higher odds of being fully reported compared with non-significant outcomes (range of OR: 2.2 to 4.7)’.13 A 2014 systematic review by Schmucker and colleagues examined studies of publication and dissemination bias conducted using research approved by ethics committees or registered on a trial registry. They found that across 23 cohort studies, ‘statistically significant results were more likely to be published than those without (pooled OR 2.8; 95% CI 2.2 to 3.5)’.14 ## Impact The above examples help illustrate the impact of publication bias. This can vary from the non-publication of a single notable study to compromising the complete assessment of a therapeutic area. However, as with many biases, large-scale quantitative research has tended to focus on documenting the prevalence of publication bias, rather than its impact, and assessing the direction and magnitude of bias can be difficult.15 Schmucker and colleagues conducted a systematic review examining studies on publication bias that additionally estimated the impact of unpublished studies on pooled effects. They found seven studiescomparing pooled treatment effect estimates according to different publication status; two of these showed a statistically significant effect of unpublished or grey literature data on the pooled estimates.16 ## Preventive steps Prevention of publication bias can take many forms. Certain journals, such as *Trials*, have made the solicitation and publication of null results a part of their core mission.17 However, previously documented barriers to publication, such as time and investigator interest,3 cannot be addressed by the presence of journals receptive to null results. The preceding decade has seen various initiatives in the US and EU requiring certain trials to report results directly onto clinical trial registries in structured data format within 12 months of completion, providing an additional avenue for dissemination outside of academic journals. Sadly, there is growing evidence that these laws and guidelines are undermined by loopholes and poor compliance.18–21 Authors of systematic reviews and meta-analyses can also take steps to reduce the impact of non-publication on their work. The search for evidence should not be limited to only journal articles indexed in repositories such as PubMed or Ovid. Authors can and should search for results through other routes including trial registries, regulatory documents, and contacting trialists of known or suspected unpublished work.22 23 They can also use statistical methods to estimate if their sample of studies is likely impacted by publication bias. Funnel plots are a common way to visualise a skew in the publication of findings. While useful, their interpretation must be carefully considered based on the methods used to construct the plot.24–26 More rigorous statistical methods for assessing publication bias exist and should be considered for use in meta-research.23 27 28 While there is evidence that asymmetry tests for publication bias are underutilised, there is also evidence suggesting that they are not applicable to many meta-analyses.29 30 ## Footnotes * Contributors All authors contributed equally to manuscript drafting and revision. * Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors * Competing interests None declared. * Provenance and peer review Not commissioned; externally peer reviewed. ## References 1. 1. Dickersin K , Min YI . Publication bias: the problem that won’t go away. Ann N Y Acad Sci 1993;703:135–48.[doi:10.1111/j.1749-6632.1993.tb26343.x](http://dx.doi.org/10.1111/j.1749-6632.1993.tb26343.x) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1111/j.1749-6632.1993.tb26343.x&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=8192291&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=A1993BA22S00016&link_type=ISI) 2. 2. Guyatt GH , Oxman AD , Montori V , et al . GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol 2011;64:1277–82.[doi:10.1016/j.jclinepi.2011.01.011](http://dx.doi.org/10.1016/j.jclinepi.2011.01.011) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2011.01.011&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=21802904&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) 3. 3. Song F , Parekh S , Hooper L , et al . Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010;14:iii, ix-xi, 1-193.[doi:10.3310/hta14080](http://dx.doi.org/10.3310/hta14080) 4. 4. Müller KF , Briel M , D’Amario A , et al . Defining publication bias: protocol for a systematic review of highly cited articles and proposal for a new framework. Syst Rev 2013;2:34.[doi:10.1186/2046-4053-2-34](http://dx.doi.org/10.1186/2046-4053-2-34) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=23692820&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) 5. 5. Dickersin K . The existence of publication bias and risk factors for its occurrence. JAMA 1990;263:1385–9.[doi:10.1001/jama.1990.03440100097014](http://dx.doi.org/10.1001/jama.1990.03440100097014) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1001/jama.1990.03440100097014&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=2406472&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=A1990CQ72400014&link_type=ISI) 6. 6. Cowley AJ , Skene A , Stainer K , et al . The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction: an example of publication bias. Int J Cardiol 1993;40:161–6.[doi:10.1016/0167-5273(93)90279-P](http://dx.doi.org/10.1016/0167-5273(93)90279-P) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1016/0167-5273(93)90279-P&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=8349379&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=A1993LP28200012&link_type=ISI) 7. 7. Driessen E , Hollon SD , Bockting CL , et al . Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of us national institutes of health-funded trials. PLoS One 2015;10:e0137864.[doi:10.1371/journal.pone.0137864](http://dx.doi.org/10.1371/journal.pone.0137864) 8. 8. Lexchin J , Bero LA , Djulbegovic B , et al . Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003;326:1167–70.[doi:10.1136/bmj.326.7400.1167](http://dx.doi.org/10.1136/bmj.326.7400.1167) [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjYvNzQwMC8xMTY3IjtzOjQ6ImF0b20iO3M6MTk6Ii9lYm1lZC8yNC8yLzUzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 9. 9. Bekelman JE , Li Y , Gross CP . Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA 2003;289:454–65.[doi:10.1001/jama.289.4.454](http://dx.doi.org/10.1001/jama.289.4.454) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1001/jama.289.4.454&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=12533125&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000180472300027&link_type=ISI) 10. 10. Fanelli D . Positive results receive more citations, but only in some disciplines. Scientometrics 2013;94:701–9.[doi:10.1007/s11192-012-0757-y](http://dx.doi.org/10.1007/s11192-012-0757-y) 11. 11. Simes RJ . Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;4:1529–41.[doi:10.1200/JCO.1986.4.10.1529](http://dx.doi.org/10.1200/JCO.1986.4.10.1529) [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiamNvIjtzOjU6InJlc2lkIjtzOjk6IjQvMTAvMTUyOSI7czo0OiJhdG9tIjtzOjE5OiIvZWJtZWQvMjQvMi81My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 12. 12. Turner EH , Matthews AM , Linardatos E , et al . Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252–60.[doi:10.1056/NEJMsa065779](http://dx.doi.org/10.1056/NEJMsa065779) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1056/NEJMsa065779&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=18199864&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000252375600006&link_type=ISI) 13. 13. Dwan K , Gamble C , Williamson PR , et al . Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One 2013;8:e66844.[doi:10.1371/journal.pone.0066844](http://dx.doi.org/10.1371/journal.pone.0066844) 14. 14. Schmucker C , Schell LK , Portalupi S , et al . Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries. PLoS One 2014;9:e114023.[doi:10.1371/journal.pone.0114023](http://dx.doi.org/10.1371/journal.pone.0114023) 15. 15. Murad MH , Chu H , Lin L , et al . The effect of publication bias magnitude and direction on the certainty in evidence. BMJ Evid Based Med 2018;23:84–6.[doi:10.1136/bmjebm-2018-110891](http://dx.doi.org/10.1136/bmjebm-2018-110891) 16. 16. Schmucker CM , Blümle A , Schell LK , et al . Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One 2017;12:e0176210.[doi:10.1371/journal.pone.0176210](http://dx.doi.org/10.1371/journal.pone.0176210) 17. 17.Aims and scope. Trials. [https://trialsjournal.biomedcentral.com/submission-guidelines/aims-and-scope](https://trialsjournal.biomedcentral.com/submission-guidelines/aims-and-scope) (Accessed 30 Aug 2018). 18. 18. Prayle AP , Hurley MN , Smyth AR . Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ 2012;344:d7373.[doi:10.1136/bmj.d7373](http://dx.doi.org/10.1136/bmj.d7373) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1136/bmj.d7373&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=22214756&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) 19. 19. Miller JE , Korn D , Ross JS . Clinical trial registration, reporting, publication and FDAAA compliance: a cross-sectional analysis and ranking of new drugs approved by the FDA in 2012. BMJ Open 2015;5:e009758.[doi:10.1136/bmjopen-2015-009758](http://dx.doi.org/10.1136/bmjopen-2015-009758) 20. 20. DeVito NJ , Bacon S , Goldacre B . FDAAA TrialsTracker: a live informatics tool to monitor compliance with fda requirements to report clinical trial results.  View ORCID Profile  2018. 21. 21. Goldacre B , DeVito NJ , Heneghan C , et al . Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource. BMJ 2018;362:k3218.[doi:10.1136/bmj.k3218](http://dx.doi.org/10.1136/bmj.k3218) [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNjIvc2VwMTJfOC9rMzIxOCI7czo0OiJhdG9tIjtzOjE5OiIvZWJtZWQvMjQvMi81My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 22. 22. Chan AW . Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ 2012;344:d8013.[doi:10.1136/bmj.d8013](http://dx.doi.org/10.1136/bmj.d8013) [FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDQvamFuMDNfMi9kODAxMyI7czo0OiJhdG9tIjtzOjE5OiIvZWJtZWQvMjQvMi81My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 23. 23.1. 2. Higgins JPT , 3. Green S Sterne JAC , Egger M , Moher D . Chapter 10: Addressing reporting biases. In: Higgins JPT , Green S , eds. Cochrane handbook for systematic reviews of interventions: The Cochrane Collaboration, 2011. 24. 24. Tang JL , Liu JL . Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol 2000;53:477–84.[doi:10.1016/S0895-4356(99)00204-8](http://dx.doi.org/10.1016/S0895-4356(99)00204-8) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1016/S0895-4356(99)00204-8&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=10812319&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000087075500005&link_type=ISI) 25. 25. Sterne JA , Egger M . Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol 2001;54:1046–55. [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1016/S0895-4356(01)00377-8&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=11576817&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000171047800012&link_type=ISI) 26. 26. Terrin N , Schmid CH , Lau J . In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol 2005;58:894–901.[doi:10.1016/j.jclinepi.2005.01.006](http://dx.doi.org/10.1016/j.jclinepi.2005.01.006) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2005.01.006&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=16085192&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) [Web of Science](http://ebm.bmj.com/lookup/external-ref?access_num=000231543200005&link_type=ISI) 27. 27. Egger M , Davey Smith G , Schneider M , et al . Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315:629–34.[doi:10.1136/bmj.315.7109.629](http://dx.doi.org/10.1136/bmj.315.7109.629) [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTUvNzEwOS82MjkiO3M6NDoiYXRvbSI7czoxOToiL2VibWVkLzI0LzIvNTMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 28. 28. Schneck A . Examining publication bias-a simulation-based evaluation of statistical tests on publication bias. PeerJ 2017;5:e4115.[doi:10.7717/peerj.4115](http://dx.doi.org/10.7717/peerj.4115) 29. 29. Onishi A , Furukawa TA . Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study. J Clin Epidemiol 2014;67:1320–6.[doi:10.1016/j.jclinepi.2014.07.002](http://dx.doi.org/10.1016/j.jclinepi.2014.07.002) [CrossRef](http://ebm.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2014.07.002&link_type=DOI) [PubMed](http://ebm.bmj.com/lookup/external-ref?access_num=25194857&link_type=MED&atom=%2Febmed%2F24%2F2%2F53.atom) 30. 30. Ioannidis JP , Trikalinos TA . The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ 2007;176:1091–6.[doi:10.1503/cmaj.060410](http://dx.doi.org/10.1503/cmaj.060410) [Abstract/FREE Full Text](http://ebm.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMDoiMTc2LzgvMTA5MSI7czo0OiJhdG9tIjtzOjE5OiIvZWJtZWQvMjQvMi81My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=)