Original Article
Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study

https://doi.org/10.1016/j.jclinepi.2018.05.022Get rights and content

Abstract

Objective

To assess the effects of abbreviated literature searches on evidence syntheses conclusions.

Study design and setting

We randomly selected 60 Cochrane reviews of clinical interventions and repeated literature searches using 14 abbreviated approaches (combinations of MEDLINE, Embase, CENTRAL with and without searches of reference lists). If abbreviated searches missed included studies, we recalculated meta-analyses. Cochrane authors determined whether the new evidence base would change conclusions. We assessed the noninferiority of abbreviated searches allowing for a maximum of 10% changed conclusions.

Results

We conducted 840 abbreviated literature searches. Noninferiority varied based on the definition of “changed conclusion”. When the reduction of the certainty of a conclusion was of concern, all abbreviated searches were inferior. Searching Embase only rendered the greatest proportion of changed conclusions (27%, 95% confidence interval [CI]: 16%–40%); combining MEDLINE, Embase, CENTRAL with searches of references lists the lowest (8%, 95% CI 3%–18%). When falsely reaching an opposite conclusion was of concern, combining one database with another or with searches of reference lists was noninferior to comprehensive searches (2%, 95% CI: 0%–9%).

Conclusion

If decision-makers are willing to accept less certainty and a small risk for opposite conclusions, some abbreviated searches are viable options for rapid evidence syntheses. Decisions demanding high certainty require comprehensive searches.

Introduction

High-quality systematic reviews are the best option to inform and guide decision-making in health care because they employ high methodological standards in searching, selecting, appraising, and synthesizing primary research. Ensuring methodological rigor, however, is time- and labor-intense and can result in systematic reviews taking up to 24 months to be completed [1], [2]. Such long production times often do not meet the time-sensitive needs of decision-makers. As a consequence, rapid reviews have become a pragmatic alternative to systematic reviews. They are accelerated knowledge syntheses that make results available to decision-makers in a shorter time frame (within a few weeks to a few months) by streamlining certain methodological aspects of systematic reviews. One potential trade-off of rapid reviews, however, is that they might have less reliable results than systematic reviews. Case studies comparing individual rapid reviews with systematic reviews report mixed results concerning the consistency of conclusions [3], [4], [5], [6].

Methodological shortcuts in rapid reviews usually address time-consuming processes such as screening and selection of studies [7], [8], data extraction, critical appraisal of the risk of bias, evidence synthesis [9], and literature searches [10]. Abbreviated search approaches as used in rapid reviews are diverse and limit the sources (e.g., the number of electronic databases) or specifications of the search (e.g., publication year, languages, or study types) [7]. For example, searching only the most relevant electronic databases without searches of gray literature, trial registries, and reference lists, substantially reduces the time and effort spent on the search itself and reduces the number of records retrieved. This in turn reduces the time spent on record screening. Usually, abbreviated searches do not detect all relevant studies that a comprehensive literature search would identify [11], [12], [13], [14], [15]. The impact of missed studies on the results and conclusions of a systematic review, however, remains unclear. Studies on abbreviated search approaches for reviews on therapeutic interventions concluded that not finding all studies did not markedly change the meta-analysis results [15], [16]. A study focusing mainly on observational studies concluded that searching only one electronic database was not sufficient to identify all relevant studies in a research area [17].

These three studies focused mainly on recall and precision of searches and the impact of missed studies on results of meta-analyses but not on the impact on overall conclusions of the evidence syntheses. To date, no sufficiently powered study has investigated, across a range of different clinical topics, the impact of abbreviated literature searches (i.e., limiting the number of sources searched) on the conclusions of evidence syntheses. Therefore, the aim of our study was to assess the effect of various abbreviated search approaches on the overall conclusions of evidence syntheses of clinical interventions. Specifically, our research question was “Do bodies of evidence that are based on abbreviated literature searches lead to different conclusions about benefits and harms of interventions compared with bodies of evidence that are based on comprehensive, systematic literature searches?”

Section snippets

Materials and methods

A detailed description of the methods of our study has been published elsewhere [18]. We summarize the most important methodological steps in the following sections.

Results

The 60 Cochrane reviews included a total of 1,335 primary studies [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83], [84], [85]. The number of studies included in a review ranged from 1 to 124. Half of

Discussion

To the best of our knowledge, this is the first study to assess the impact of various abbreviated literature searches on conclusions across multiple bodies of evidence of clinical interventions. Our results indicate that inferiority or noninferiority of abbreviated searches depend on the willingness of decision-makers to accept or not accept a lower degree of certainty when making conclusions and a small remaining risk of falsely making an opposite conclusion.

If decision-makers need rapid

Conclusion

Abbreviating a systematic literature search can offer viable alternatives to comprehensive searches if decision-makers are willing to accept a lower degree of certainty for conclusions and a small remaining risk of falsely making an opposite conclusion. Decisions that demand the highest possible certainty should be based on comprehensive, systematic searches of the literature including specialized literature databases. Searching only a single electronic database is never a reliable method for

Acknowledgments

The authors would like to thank the authors of the included Cochrane Reviews for taking the time to participate in the survey. They also gratefully acknowledge the Cochrane Editorial Unit for providing data files of the Cochrane reviews and the Cochrane Information Specialists for providing search documentation. Furthermore, they would like to thank the following individuals for their contributions to this project: Sandra Hummel, Birgit Teufer, MA, Tuba Karakaya, Sarina Schwarz, Christoph von

References (87)

  • P. Corabian et al.

    Rapid assessments provide acceptable quality advice

    Annu Meet Int Soc Technol Assess Health Care Int Soc Technol Assess Health Care Meet

    (2002)
  • S. Van de Velde et al.

    Medicinal use of potato-derived products: conclusions of a rapid versus full systematic review

    Phytother Res

    (2011)
  • A.M. Abou-Setta et al.

    Methods for developing evidence reviews in short periods of time: a scoping review

    PLoS One

    (2016)
  • J. Polisena et al.

    Rapid review programs to support health care and policy decision making: a descriptive analysis of processes and methods

    Syst Rev

    (2015)
  • J. Polisena et al.

    Rapid review summit: an overview and initiation of a research agenda

    Syst Rev

    (2015)
  • A.C. Tricco et al.

    A scoping review of rapid review methods

    BMC Med

    (2015)
  • A.V. Kulkarni et al.

    Comparisons of citations in web of science, scopus, and google scholar for articles published in general medical journals

    JAMA

    (2009)
  • M. Egger et al.

    How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empiracal study

    Health Technology Assess

    (2003)
  • S.E. Bayliss et al.

    Where and how to search for information on the effectiveness of public health interventions–a case study for prevention of cardiovascular disease

    Health Info Libr J

    (2014)
  • A. Westphal et al.

    Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression

    J Public Health Res

    (2014)
  • B. Nussbaumer-Streit et al.

    Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study

    Syst Rev

    (2016)
  • Francisco, U.o.C.S

    Sample size calculator

    (2017)
  • G. Wagner et al.

    Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey

    BMC Med Res Methodol

    (2017)
  • J.P.T. Higgins et al.

    Standards for the conduct of new Cochrane intervention reviews, Methodological expectations of Cochrane intervention reviews

    (2016)
  • V. King et al.
  • Clarivate Analytics, EndNote, Version X8 (computer software)

    (2018)
  • SurveyMonkey

    Software to conduct web-based surveys

  • P. Adams Stephen et al.

    Lipid-lowering efficacy of rosuvastatin

    Cochrane Database Syst Rev

    (2014)
  • A. Akl Elie et al.

    Oral anticoagulation in patients with cancer who have no therapeutic or prophylactic indication for anticoagulation

    Cochrane Database Syst Rev

    (2014)
  • A. Akl Elie et al.

    Parenteral anticoagulation in ambulatory patients with cancer

    Cochrane Database Syst Rev

    (2014)
  • A. Akl Elie et al.

    Anticoagulation for the long-term treatment of venous thromboembolism in patients with cancer

    Cochrane Database Syst Rev

    (2014)
  • S. Alkhawaja et al.

    Post-pyloric versus gastric tube feeding for preventing pneumonia and improving nutritional outcomes in critically ill adults

    Cochrane Database Syst Rev

    (2015)
  • L. Anderson et al.

    Exercise-based cardiac rehabilitation for coronary heart disease

    Cochrane Database Syst Rev

    (2016)
  • B.N.G. Andriolo et al.

    Early versus late tracheostomy for critically ill patients

    Cochrane Database Syst Rev

    (2015)
  • J. Barlow et al.

    Parent-infant psychotherapy for improving parental and infant mental health

    Cochrane Database Syst Rev

    (2015)
  • S. Birks Jacqueline et al.

    Evans Rivastigmine for Alzheimer's disease

    Cochrane Database Syst Rev

    (2015)
  • H. Blessberger et al.

    Perioperative beta-blockers for preventing surgery-related mortality and morbidity

    Cochrane Database Syst Rev

    (2014)
  • H.M.P. Boardman et al.

    Hormone therapy for preventing cardiovascular disease in post-menopausal women

    Cochrane Database Syst Rev

    (2015)
  • J. Bradt et al.

    Music for stress and anxiety reduction in coronary heart disease patients

    Cochrane Database Syst Rev

    (2013)
  • R. Buchbinder et al.

    Percutaneous vertebroplasty for osteoporotic vertebral compression fracture

    Cochrane Database Syst Rev

    (2015)
  • J. Chen et al.

    Sulfasalazine for ankylosing spondylitis

    Cochrane Database Syst Rev

    (2014)
  • J.C. Claro et al.

    Amiodarone versus other pharmacological interventions for prevention of sudden cardiac death

    Cochrane Database Syst Rev

    (2015)
  • G. Cormick et al.

    Calcium supplementation for prevention of primary hypertension

    Cochrane Database Syst Rev

    (2015)
  • Cited by (0)

    Conflict of interest: None of the authors report any conflicts of interest with respect to the topic of this manuscript.

    Funding source: This work was supported by internal funds of Cochrane Austria.

    View full text