Article Text
PDF
Abstract
This paper is part of a series of methodological guidance from the Cochrane Rapid Reviews Methods Group. Rapid reviews (RR) use modified systematic review methods to accelerate the review process while maintaining systematic, transparent and reproducible methods. In this paper, we address considerations for RR searches. We cover the main areas relevant to the search process: preparation and planning, information sources and search methods, search strategy development, quality assurance, reporting, and record management. Two options exist for abbreviating the search process: (1) reducing time spent on conducting searches and (2) reducing the size of the search result. Because screening search results is usually more resource-intensive than conducting the search, we suggest investing time upfront in planning and optimising the search to save time by reducing the literature screening workload. To achieve this goal, RR teams should work with an information specialist. They should select a small number of relevant information sources (eg, databases) and use search methods that are highly likely to identify relevant literature for their topic. Database search strategies should aim to optimise both precision and sensitivity, and quality assurance measures (peer review and validation of search strategies) should be applied to minimise errors.
- Methods
- Evidence-Based Practice
- Systematic Reviews as Topic
- Information Science
Data availability statement
No data are available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Compared with systematic reviews, rapid reviews (RR) often abbreviate or limit the literature search in some way to accelerate review production. However, RR guidance rarely specifies how to select topic-appropriate search approaches.
WHAT THIS STUDY ADDS
This paper presents an overview of considerations and recommendations for RR searching, covering the complete search process from the planning stage to record management. We also provide extensive appendices with practical examples, useful sources and a glossary of terms.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
There is no one-size-fits-all solution for RR literature searching: review teams should consider what search approaches best fit their RR project.
Introduction
This paper is part of a series from the Cochrane Rapid Reviews Methods Group (RRMG) providing methodological guidance for rapid reviews (RRs).1–3 While the RRMG’s guidance4 5 on Cochrane RR production includes brief advice on literature searching, we aim to provide in-depth recommendations for the entire search process.
Literature searching is the foundation for all reviews; therefore, it is important to understand the goals of a specific RR. The scope of RRs varies considerably (from focused questions to overviews of broad topics).6 As with conventional systematic reviews (SRs), there is not a one-size-fits-all approach for RR literature searches. We aim to support RR teams in choosing methods that best fit their project while understanding the limitations of modified search methods. Our recommendations derive from current systematic search guidance, evidence on modified search methods and practical experience conducting RRs.
This paper presents considerations and recommendations, described briefly in table 1. The table also includes a comparison to the SR search process based on common recommendations.7–10 We provide supplemental materials, including a list of additional resources, further details of our recommendations, practical examples, and a glossary (explaining the terms written in italics) in online supplemental appendices A–C.
Supplemental material
Recommendations for rapid review literature searching
Preparation and planning
Given that the results of systematic literature searches underpin a review, planning the searches is integral to the overall RR preparation. The RR search process follows the same steps as an SR search; therefore, RR teams must be familiar with the general standards of systematic searching. Templates (see online supplemental appendix B) and reporting guidance11 for SR searches can also be adapted to structure the RR search process.
Developing a plan for the literature search forms part of protocol development and should involve an information specialist (eg, librarian). Information specialists can assist in refining the research question, selecting appropriate search methods and resources, designing and executing search strategies, and reporting the search methods. At minimum, specialist input should include assessing information sources and methods and providing feedback on the primary database search strategy.
Two options exist for abbreviating the search process: (1) reducing time spent on conducting searches (eg, using automation tools, reusing existing search strategies, omitting planning or quality assurance steps) and (2) reducing the size of the search result (eg, limiting the number of information sources, increasing the precision of search strategies, using study design filters). Study selection (ie, screening search results) is usually more resource-intensive than searching,12 particularly for topics with complex or broad concepts or diffuse terminology; thus, the second option may be more efficient for the entire RR. Investing time upfront in optimising search sensitivity (ie, completeness) and precision (ie, positive predictive value) can save time in the long run by reducing the screening and selection workload.
Preliminary or scoping searches are critical to this process. They inform the choice of search methods and identify potentially relevant literature. Texts identified through preliminary searching serve as known relevant records that can be used throughout the search development process (see sections on database selection, development and validation of search strategies).
In addition to planning the search itself, the review team should factor in time for quality assurance steps (eg, search strategy peer review) and the management of search results (eg, deduplication, full-text retrieval).
Information sources and methods
To optimise the balance of search sensitivity and precision, RR teams should prioritise the most relevant information sources for the topic and the type of evidence required. These can include bibliographic databases (eg, MEDLINE/PubMed), grey literature sources and targeted supplementary search methods. Note that this approach differs from the Methodological Expectations of Cochrane Intervention Reviews Standards9 where the same core set of information sources is required for every review and further supplemented by additional topic-specific and evidence-specific sources.
Choosing bibliographic databases
For many review topics, most evidence is found in peer-reviewed journal articles, making bibliographic databases the main resource of systematic searching. Limiting the number of databases searched can be a viable option in RRs, but it is important to prioritise topic-appropriate databases.
MEDLINE has been found to have high coverage for studies included in SRs13 14 and is an appealing database choice because access is free via PubMed. However, coverage varies depending on topics and relevant study designs.15 16 Additionally, even if all eligible studies for a topic were available in MEDLINE, search strategies will usually miss some eligible studies because search sensitivity is lower than database coverage.13 17 This means searching MEDLINE alone is not a viable option, and additional information sources or search methods are required. Known relevant records can be used to help assess the coverage of selected databases (see also online supplemental appendix C).
Further information sources and search techniques
Supplementary systematic search methods have three main goals, to identify (1) grey literature, (2) published literature not covered by the selected bibliographic databases and (3) database-covered literature that was not retrieved by the database searches.
When RRs search only a small number of databases, supplementary searches can be particularly important to pick up eligible studies not identified via database searching. While supplementary methods might increase the time spent on searching, they sometimes better optimise search sensitivity and precision, saving time in the long run.18 Depending on the topic and relevant evidence, such methods can offer an alternative to adding additional specialised database searches. To decide if and what supplementary searches are helpful, it is important to evaluate what literature might be missed by the database searches and how this might affect the specific RR.
Study registries and other grey literature
Some studies indicate that the omission of grey literature searches rarely affects review conclusions.17 19 However, the relevance of study registries and other grey literature sources is topic-dependent.16 19–21 For example, randomised controlled trials (RCTs) on newly approved drugs are typically identified in ClinicalTrials.gov.20 For rapidly evolving topics such as COVID-19, preprints are an important source.21 For public health interventions, various types of grey literature may be important (eg, evaluations conducted by local public health agencies).22
Further supplementary search methods
Other supplementary techniques (eg, checking reference lists, reviewing specific websites or electronic table of contents, contacting experts) may identify additional studies not retrieved by database searches.23 One of the most common approaches involves checking reference lists of included studies and relevant reviews. This method may identify studies missed by limited database searches.12 Another promising citation-based approach is using the ‘similar articles’ option in PubMed, although research has focused on updating existing SRs.24 25
Considerations for RRs of RCTs
Databases and search methods to identify RCTs have been particularly well researched.17 20 24 26 27 For this reason, it is possible to give more precise recommendations for RRs based on RCTs than for other types of review. Table 2 provides an overview of the most important considerations; additional information can be found in online supplemental appendix C.
Information sources for identification of randomised controlled trials (RCTs)
Search strategies
We define ‘search strategy’ as a Boolean search query in a specific database (eg, MEDLINE) using a specific interface (eg, Ovid). When several databases are searched, this query is usually developed in a primary database and interface (eg, Ovid MEDLINE) and translated to other databases.
Developing search strategies
Optimising search strategy precision while aiming for high sensitivity is critical in reducing the number of records retrieved. Preliminary searches provide crucial information to aid efficient search strategy development. Reviewing the abstracts and subject headings used in known relevant records will assist in identifying appropriate search terms. Text analysis tools can also be used to support this process,28 29 for example, to develop ‘objectively derived’ search strategies.30
Reusing or adapting complete search strategies (eg, from SRs identified by the preliminary searches) or selecting elements of search strategies for reuse can accelerate search strategy development. Additionally, validated search filters (eg, for study design) can be used to reduce the size of the search result without compromising the sensitivity of a search strategy.31 However, quality assurance measures are necessary whether the search strategy is purpose-built, reused or adapted (see the ‘Quality assurance’ section.)
Database-specific and interface-specific functionalities can also be used to improve searches’ precision and reduce the search result’s size. Some options are: restricting to records where subject terms have been assigned as the major focus of an article (eg, major descriptors in MeSH), using proximity operators (ie, terms adjacent or within a set number of words), frequency operators (ie, terms have to appear a minimum number of times in an abstract) or restricting search terms to the article title.32–34
Automated syntax translation can save time and reduce errors when translating a primary search strategy to different databases.35 36 However, manual adjustments will usually be necessary.
The time taken to learn how to use supporting technologies (eg, text analysis, syntax translation) proficiently should not be underestimated. The time investment is most likely to pay off for frequent searchers. A later paper in this series will address supporting software for the entire review process.
Limits and restrictions
Limits and restrictions (eg, publication dates, language) are another way to reduce the number of records retrieved but should be tailored to the topic and applied with caution. For example, if most studies about an intervention were published 10 years ago, then an arbitrary cut-off of ‘the last 5 years’ will miss many relevant studies.37 Similarly, limiting to ‘English only’ is acceptable for most cases, but early in the COVID-19 pandemic, a quarter of available research articles were written in Chinese.38 Depending on the RR topic, certain document types (eg, conference abstracts, dissertations) might be excluded if not considered relevant to the research question.
Note also that preset limiting functions in search interfaces (eg, limit to humans) often rely on subject headings (eg, MeSH) alone. They will miss eligible studies that lack or have incomplete subject indexing. Using (validated) search filters 31 is preferable.
Updating existing reviews
One approach to RR production involves updating an existing SR. In this case, preliminary searches should be used to check if new evidence is available. If the review team decide to update the review, they should assess the original search methods and adapt these as necessary.
One option is to identify the minimum set of databases required to retrieve all the original included studies.39 Any reused search strategies should be validated and peer-reviewed (see below) and optimised for precision and/or sensitivity.
Additionally, it is important to assess whether the topic terminology or the relevant databases have changed since the original SR search.
In some cases, designing a new search process may be more efficient than reproducing the original search.
Quality assurance and search strategy peer review
Errors in search strategies are common and can impact the sensitivity and comprehensiveness of the search result.40 If an RR search uses a small number of information sources, such errors could affect the identification of relevant studies.
Validation of search strategies
The primary database search strategy should be validated using known relevant records (if available). This means testing if the primary search strategy retrieves eligible studies found through preliminary searching. If some known studies are not identified, the searcher assesses the reasons and decides if revisions are necessary. Even a precision-focused systematic search should identify the majority—we suggest at least 80%–90%—of known studies. This is based on benchmarks for sensitivity-precision-maximising search filters41 and assumes that the set of known studies is representative of the whole of relevant studies.
Peer review of search strategies
Ideally, an information specialist should review the planned information sources and search methods and use the PRESS (Peer Review of Electronic Search Strategies) checklist42 to assess the primary search strategy. Turnaround time has to be factored into the process from the outset (eg, waiting for feedback, revising the search strategy). PRESS recommends a maximum turnaround time of five working days for feedback, but in-house peer review often takes only a few hours.
If the overall RR time plan does not allow for a full peer review of the search strategy, a review team member with search experience should check the search strategy for spelling errors and correct use of Boolean operators (AND, OR, NOT) at a minimum.
Reporting and record management
Record management requirements of RRs are largely identical to SRs and have to be factored into the time plan. Teams should develop a data management plan and review the relevant reporting standards at the project’s outset. PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses literature search extension)11 is a reporting standard for SR searches that can be adapted for RRs.
Reference management software (eg, EndNote,43 Zotero44) should be used to track search results, including deduplication. Note that record management for database searches is less time-consuming than for many supplementary or grey literature searches, which often require manual entry into reference management software.12
Additionally, software platforms for SR production (eg, Covidence,45 EPPI-Reviewer,46 Systematic Review Data Repository Plus47) can provide a unified way to keep track of records throughout the whole review process, which can improve management and save time. These platforms and other dedicated tools (eg, SRA Deduplicator)48 also offer automated deduplication. However, the time and cost investment necessary to appropriately use these tools have to be considered.
Conclusion
Decisions about search methods for an RR need to consider where time can be most usefully invested and processes accelerated. The literature search should be considered in the context of the entire review process, for example, protocol development and literature screening: Findings of preliminary searches often affect the development and refinement of the research question and the review’s eligibility criteria. In turn, they affect the number of records retrieved by the searches and therefore the time needed for literature selection.
For this reason, focusing only on reducing time spent on designing and conducting searches can be a false economy when seeking to speed up review production. While some approaches (eg, text analysis, automated syntax translation) may save time without negatively affecting search validity, others (eg, skipping quality assurance steps, using convenient information sources without considering their topic appropriateness) may harm the entire review. Information specialists can provide crucial aid concerning the appropriate design of search strategies, choice of methods and information sources.
For this reason, we consider that investing time at the outset of the review to carefully choose a small number of highly appropriate search methods and optimise search sensitivity and precision likely leads to better and more manageable results.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @micaelaescb
Collaborators On behalf of the Cochrane Rapid Reviews Methods Group: Declan Devane, Gerald Gartlehner, Isolde Sommer.
Contributors IK, SR, AB, CME-L and SW contributed to the conceptualisation of this paper. IK, AB and CME-L wrote the first draft of the manuscript. All authors critically reviewed and revised the manuscript. IK is responsible for the overall content.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests AB is co-convenor of the Cochrane Qualitative and Implementation Methods Group. In the last 36 months, he received royalties from Systematic Approaches To a Successful Literature Review (Sage 3rd edn), payment or honoraria form the Agency for Healthcare Research and Quality, and travel support from the WHO. DD works part time for Cochrane Ireland and Evidence Synthesis Ireland, which are funded within the University of Ireland Galway (Ireland) by the Health Research Board (HRB) and the Health and Social Care, Research and Development (HSC R&D) Division of the Public Health Agency in Northern Ireland.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Linked Articles
- Research methods and reporting
- Research methods and reporting
- Research methods and reporting