Original ArticleAbbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study
Introduction
High-quality systematic reviews are the best option to inform and guide decision-making in health care because they employ high methodological standards in searching, selecting, appraising, and synthesizing primary research. Ensuring methodological rigor, however, is time- and labor-intense and can result in systematic reviews taking up to 24 months to be completed [1], [2]. Such long production times often do not meet the time-sensitive needs of decision-makers. As a consequence, rapid reviews have become a pragmatic alternative to systematic reviews. They are accelerated knowledge syntheses that make results available to decision-makers in a shorter time frame (within a few weeks to a few months) by streamlining certain methodological aspects of systematic reviews. One potential trade-off of rapid reviews, however, is that they might have less reliable results than systematic reviews. Case studies comparing individual rapid reviews with systematic reviews report mixed results concerning the consistency of conclusions [3], [4], [5], [6].
Methodological shortcuts in rapid reviews usually address time-consuming processes such as screening and selection of studies [7], [8], data extraction, critical appraisal of the risk of bias, evidence synthesis [9], and literature searches [10]. Abbreviated search approaches as used in rapid reviews are diverse and limit the sources (e.g., the number of electronic databases) or specifications of the search (e.g., publication year, languages, or study types) [7]. For example, searching only the most relevant electronic databases without searches of gray literature, trial registries, and reference lists, substantially reduces the time and effort spent on the search itself and reduces the number of records retrieved. This in turn reduces the time spent on record screening. Usually, abbreviated searches do not detect all relevant studies that a comprehensive literature search would identify [11], [12], [13], [14], [15]. The impact of missed studies on the results and conclusions of a systematic review, however, remains unclear. Studies on abbreviated search approaches for reviews on therapeutic interventions concluded that not finding all studies did not markedly change the meta-analysis results [15], [16]. A study focusing mainly on observational studies concluded that searching only one electronic database was not sufficient to identify all relevant studies in a research area [17].
These three studies focused mainly on recall and precision of searches and the impact of missed studies on results of meta-analyses but not on the impact on overall conclusions of the evidence syntheses. To date, no sufficiently powered study has investigated, across a range of different clinical topics, the impact of abbreviated literature searches (i.e., limiting the number of sources searched) on the conclusions of evidence syntheses. Therefore, the aim of our study was to assess the effect of various abbreviated search approaches on the overall conclusions of evidence syntheses of clinical interventions. Specifically, our research question was “Do bodies of evidence that are based on abbreviated literature searches lead to different conclusions about benefits and harms of interventions compared with bodies of evidence that are based on comprehensive, systematic literature searches?”
Section snippets
Materials and methods
A detailed description of the methods of our study has been published elsewhere [18]. We summarize the most important methodological steps in the following sections.
Results
The 60 Cochrane reviews included a total of 1,335 primary studies [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] [61], [62], [63], [64], [65], [66], [67], [68], [69], [70], [71], [72], [73], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83], [84], [85]. The number of studies included in a review ranged from 1 to 124. Half of
Discussion
To the best of our knowledge, this is the first study to assess the impact of various abbreviated literature searches on conclusions across multiple bodies of evidence of clinical interventions. Our results indicate that inferiority or noninferiority of abbreviated searches depend on the willingness of decision-makers to accept or not accept a lower degree of certainty when making conclusions and a small remaining risk of falsely making an opposite conclusion.
If decision-makers need rapid
Conclusion
Abbreviating a systematic literature search can offer viable alternatives to comprehensive searches if decision-makers are willing to accept a lower degree of certainty for conclusions and a small remaining risk of falsely making an opposite conclusion. Decisions that demand the highest possible certainty should be based on comprehensive, systematic searches of the literature including specialized literature databases. Searching only a single electronic database is never a reliable method for
Acknowledgments
The authors would like to thank the authors of the included Cochrane Reviews for taking the time to participate in the survey. They also gratefully acknowledge the Cochrane Editorial Unit for providing data files of the Cochrane reviews and the Cochrane Information Specialists for providing search documentation. Furthermore, they would like to thank the following individuals for their contributions to this project: Sandra Hummel, Birgit Teufer, MA, Tuba Karakaya, Sarina Schwarz, Christoph von
References (87)
- et al.
A retrospective comparison of systematic reviews with same-topic rapid reviews
J Clin Epidemiol
(2018) - et al.
Meta-epidemiologic analysis indicates that MEDLINE searches are sufficient for diagnostic test accuracy systematic reviews
J Clin Epidemiol
(2014) - et al.
Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions
J Clin Epidemiol
(2015) - et al.
Searching one or two databases was insufficient for meta-analysis of observational studies
J Clin Epidemiol
(2005) Optimal two-stage designs for phase II clinical trials
Control Clin Trials
(1989)- et al.
ROBIS: a new tool to assess risk of bias in systematic reviews was developed
J Clin Epidemiol
(2016) - et al.
Testing the effectiveness of simplified search strategies for updating systematic reviews
J Clin Epidemiol
(2017) - et al.
Expediting systematic reviews: methods and implications of rapid reviews
Implement Sci
(2010) - et al.
EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews
(2015) Rapid versus full systematic reviews: an inventory of current methods and practice in Health Technology Assessment. ASERNIP-S Report No. 60
(2007)
Rapid assessments provide acceptable quality advice
Annu Meet Int Soc Technol Assess Health Care Int Soc Technol Assess Health Care Meet
Medicinal use of potato-derived products: conclusions of a rapid versus full systematic review
Phytother Res
Methods for developing evidence reviews in short periods of time: a scoping review
PLoS One
Rapid review programs to support health care and policy decision making: a descriptive analysis of processes and methods
Syst Rev
Rapid review summit: an overview and initiation of a research agenda
Syst Rev
A scoping review of rapid review methods
BMC Med
Comparisons of citations in web of science, scopus, and google scholar for articles published in general medical journals
JAMA
How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empiracal study
Health Technology Assess
Where and how to search for information on the effectiveness of public health interventions–a case study for prevention of cardiovascular disease
Health Info Libr J
Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression
J Public Health Res
Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study
Syst Rev
Sample size calculator
Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey
BMC Med Res Methodol
Standards for the conduct of new Cochrane intervention reviews, Methodological expectations of Cochrane intervention reviews
Clarivate Analytics, EndNote, Version X8 (computer software)
Software to conduct web-based surveys
Lipid-lowering efficacy of rosuvastatin
Cochrane Database Syst Rev
Oral anticoagulation in patients with cancer who have no therapeutic or prophylactic indication for anticoagulation
Cochrane Database Syst Rev
Parenteral anticoagulation in ambulatory patients with cancer
Cochrane Database Syst Rev
Anticoagulation for the long-term treatment of venous thromboembolism in patients with cancer
Cochrane Database Syst Rev
Post-pyloric versus gastric tube feeding for preventing pneumonia and improving nutritional outcomes in critically ill adults
Cochrane Database Syst Rev
Exercise-based cardiac rehabilitation for coronary heart disease
Cochrane Database Syst Rev
Early versus late tracheostomy for critically ill patients
Cochrane Database Syst Rev
Parent-infant psychotherapy for improving parental and infant mental health
Cochrane Database Syst Rev
Evans Rivastigmine for Alzheimer's disease
Cochrane Database Syst Rev
Perioperative beta-blockers for preventing surgery-related mortality and morbidity
Cochrane Database Syst Rev
Hormone therapy for preventing cardiovascular disease in post-menopausal women
Cochrane Database Syst Rev
Music for stress and anxiety reduction in coronary heart disease patients
Cochrane Database Syst Rev
Percutaneous vertebroplasty for osteoporotic vertebral compression fracture
Cochrane Database Syst Rev
Sulfasalazine for ankylosing spondylitis
Cochrane Database Syst Rev
Amiodarone versus other pharmacological interventions for prevention of sudden cardiac death
Cochrane Database Syst Rev
Calcium supplementation for prevention of primary hypertension
Cochrane Database Syst Rev
Cited by (0)
Conflict of interest: None of the authors report any conflicts of interest with respect to the topic of this manuscript.
Funding source: This work was supported by internal funds of Cochrane Austria.