Article Text
PDF
Abstract
Living systematic reviews (LSRs) are an increasingly common approach to keeping reviews up to date, in which new relevant studies are incorporated as they become available, so as to inform healthcare policy and practice in a timely manner. While journal publishers have been exploring the publication of LSRs using different updating and publishing approaches, readers cannot currently assess if the evidence underpinning a published LSR is up to date, as neither the search details, the selection process, nor the list of identified studies is made available between the publication of updates. We describe a new method to transparently report the living evidence surveillance process that occurs between published LSR versions. We use the example of the living Cochrane Review on nirmatrelvir combined with ritonavir (Paxlovid) for preventing and treating COVID-19 to illustrate how this can work in practice. We created a publicly accessible spreadsheet on the Open Science Framework platform, linking to the living Cochrane Review, that details the search and study selection process, enabling readers to track the progress of eligible ongoing or completed studies. Further automation of the evidence surveillance process should be explored.
- Systematic Reviews as Topic
- Methods
- Information Science
- COVID-19
Data availability statement
Data are available in a public, open access repository.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Readers of living systematic reviews (LSRs) cannot currently assess if the evidence underpinning a published LSR is up to date, as information on the evidence surveillance process is not available between updates.
WHAT THIS STUDY ADDS
The living Cochrane Review on ‘Nirmatrelvir combined with ritonavir for preventing and treating COVID-19’ is the first LSR published as a standard journal publication that makes information on the evidence surveillance process available on a monthly basis in-between published updates. By using a publicly accessible, continually updated data file on Open Science Framework that documents the study selection process following each update search, it addresses a major limitation in the current reporting of LSRs.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
LSR teams that decide to adopt our open science-based approach of reporting the new underpinning evidence of an LSR in the form of a publicly accessible data file can increase transparency, improve reliability and timeliness, and thus the value for readers, especially in an emerging evidence scenario. Possibilities for automation of the evidence surveillance process should be explored.
Introduction
Living systematic reviews (LSRs), in which new studies are continually incorporated as they become available, represent a new approach to keeping reviews up to date and have become increasingly common in recent years.1–3 Because the underpinning evidence is continually being updated, LSRs can inform healthcare policy and practice in a timely manner—a situation clearly suited to the COVID-19 pandemic when the evidence base is changing rapidly.4
Since 2015 Cochrane and other journal publishers have been exploring the publication of LSRs using different updating and publishing approaches.5 Having been involved in the production of several LSRs on COVID-19 for Cochrane, we used an approach adopted by many other author teams, which consists of running searches and screening results on a regular basis and publishing an updated version of the review as new evidence is identified.6 7 However, details of the ongoing surveillance between published LSR updates are not reported, leaving readers in the dark about whether the review is up to date and which new underpinning evidence, for example, relevant new studies, has become available since the last publication. Other LSR author teams in Cochrane have used the ‘version history’ in the Cochrane Library to inform readers about the update status of the review, thereby adding transparency to its living process.8 9 They report on recent searches and summarise the impact of any newly identified or incorporated studies in a brief one paragraph ‘revision event’. But again, neither the details of the search and selection process nor the list of identified studies is made available to the readers.
Other journals, for example, Annals of Internal Medicine, use two types of publications to indicate the nature of the update. The first are ‘update alerts’ (published as letters) which provide a succinct narrative summary of the impact new evidence has on the review’s findings, with links to supplementary files, as appropriate. The second are ‘major updates’ in which the review is republished, fully incorporating new studies. The living review of remdesivir for adults with COVID-19 illustrates these two types.10 11 Both update alerts and major updates flag any proposed changes to search frequency and study inclusion criteria. Although the information about search frequency allows readers to estimate when the next update might be published, there is no information on what evidence may have been identified between updates.
Another example of a frequently updated review is the LSR of drug treatments for COVID-19 published in the BMJ.12 This LSR uses a visual summary to illustrate the results of the search, screening and update process. In contrast to other LSRs, the study flow diagram indicates the included studies as well as those that will be included in the next update (presented in a table in the review). However, between updates, there is the same reporting issue that leaves readers without information about what new evidence has been identified. A selection of the examples mentioned is depicted in figure 1.
Examples of available information about study identification and selection in Living Systematic Reviews
The aim of this paper is to describe a new open science-based approach to increase the transparency of the living search and study selection process that occurs between published LSR versions. We use the example of the Cochrane Review of ‘Nirmatrelvir combined with ritonavir for preventing and treating COVID-19’ to illustrate how reporting new underpinning evidence of an LSR can work in practice.
Making new evidence underpinning an LSR publicly available: the Cochrane Review on nirmatrelvir/ritonavir
Pfizer’s new drug combination of nirmatrelvir/ritonavir (Paxlovid) poses an excellent case for establishing an LSR with a standardised monthly search and study selection process. The antiviral drug combination aims to avoid severe COVID‐19 in asymptomatic people or those with mild symptoms, thus decreasing hospitalisation and death. In light of the ongoing potential for evolving virus variants, limited effective therapies for the treatment of COVID‐19 and prevention of SARS‐CoV‐2 infection in the outpatient setting, as well as global vaccination coverage issues, the role of effective oral therapies for patients at high risk of severe disease is of global interest.
Besides several ongoing trials of nirmatrelvir/ritonavir for both treatment and post‐exposure prophylaxis being conducted by the pharmaceutical company itself,13 we expect many new investigator-initiated trials of the drug combination worldwide after its Emergency Use Authorization by the Food and Drug Administration (FDA) in December 202114 and its marketing authorisation by the European Medicines Agency (EMA) in January 2022.15 Therefore, the Cochrane Review evaluating Pfizer’s new antiviral was designed as an LSR with continuous monitoring of new and ongoing studies.16
Kahale and colleagues recently proposed to document the search and selection process for each LSR update through tailored Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagrams.17 However, they do not address the lack of transparency between updates. To improve transparency between published updates of the review, we created an Excel spreadsheet—publicly accessible on the Open Science Framework (OSF) platform and linked from the results section of the Cochrane Review—that details the living evidence surveillance process, enabling readers to easily track the progress of eligible studies, whether completed or ongoing.18
Monthly searches for the LSR began on 11 April 2022, the day the protocol was published in the Cochrane Library.16 The information specialist (MIM) provides the author team with the search results the same day, having first removed duplicates as well as all previously screened records using EndNote. Two members of the author team then independently screen the search results in Covidence and document the outcomes of the selection process in the Excel spreadsheet. The data available in the publicly accessible OSF file are summarised in table 1.
Data elements of the LSR’s evidence surveillance process available in the public spreadsheet
After the authors have finished screening and assessing any newly identified studies, the information specialist uploads the Excel file to OSF. Readers of the review can easily access the review’s new underpinning evidence via the OSF link to find out whether any new studies have been identified that might affect the results, as well as track the progression of studies through the review. Figure 2 depicts three of the four tabs available in the publicly accessible spreadsheet.
Living Systematic Review surveillance process available in the public spreadsheet as of 25 September 2022
Further details of the living methods used in the Cochrane Review, which have recently been termed ‘living mode parameters’ in the context of living guidelines,19 are available in the section on Methods for future updates—Living systematic review considerations.16 There, we specify our criteria for republishing the review—these include adding studies that contribute data to one or more prioritised outcomes; changes to the credibility (e.g., GRADE (Grading of Recommendations, Assessment, Development and Evaluations) rating) of these outcomes and the addition of new settings, populations, interventions, comparisons or outcomes. When these criteria apply, we highlight the respective study triggering the republication of the review in the publicly available data file. Likewise, a study will be highlighted if it changes the living mode of the review. In case of retraction of an included study, we will follow standard retraction recommendations by Cochrane and document the retraction and changed number of included studies. Finally, the spreadsheet incorporates a link to the latest published version of the review, as well as previous versions.
By implementing this transparent evidence surveillance approach in a Cochrane Review that can currently only draw on the first regulatory study of the drug combination in an unvaccinated population (while several investigator-initiated trials are on the horizon with likely consequential changes to the body of evidence due to the changing immunisation status of the population), we showcase how the reporting process can be enhanced in-between published updates of an LSR.
Discussion
This paper presents a novel open science-based mechanism for LSRs, which we have developed to provide readers with up-to-date information on the status of the evidence that is actively being considered for inclusion in-between published updates, as well as an audit trail of studies that have already been assessed. We propose the use of a publicly accessible, easy to set up and continually updated data file that documents the results of the study selection process following each update search to address a major limitation in the current reporting of LSRs.
Clinicians, guideline developers and health policymakers who use LSRs as a basis for guidance need to know what is going on between published updates and, crucially, whether new evidence has an impact on the results or conclusions of the review. In typical circumstances, delays of several weeks or months between identifying new studies and the publication of an update are unlikely to dramatically affect policy or practice. But as we have seen with COVID-19, when the evidence base is virtually non-existent or very uncertain, new studies can have an immediate and substantial impact on practice and health outcomes.20
Of the many living reviews that appeared during the early period of the pandemic, some have continued to be updated regularly, others only intermittently, and some never progressed beyond the initial base version.21 If LSRs are to deliver on their promise of ‘staying alive’, readers ought to be able to rely on them for the latest evidence. To what extent this is possible depends on several factors, including: how frequently updates are planned (and whether this is evident to readers); how rapidly new evidence is identified and incorporated in the review; and how easy is it to see what evidence is awaiting inclusion, and thus what impact it might have. This last factor—which we address with our new approach—is critical in determining the usefulness of a living review.
Unless readers have confidence that an LSR is up to date and can readily see (and make judgements about) what evidence has not yet been included, the effort invested by author teams may be wasted and does little to avoid the duplication of reviews that has been a feature of the pandemic.22 The example of remdesivir illustrates how susceptible LSRs are to losing relevance if this issue is not addressed—when results of the influential WHO Solidarity trial were posted as a preprint in mid-October 2020, it took 2–5 months for LSRs to be updated.11 12 23
Alongside the growing numbers of LSRs, the pandemic has also seen the prominence of living guidelines, living updates and online living evidence systems, such as the Australian National COVID-19 Clinical Evidence Taskforce (covid19evidence.net.au), PAHO’s Ongoing Living Update of Potential COVID-19 Therapeutics Options (iris.paho.org/handle/10665.2/52719), COVID-19 LNMA (www.covid19lnma.com) and the COVID-NMA repository (covid-nma.com). These dynamic and web-based approaches allow for a more rapid updating and presentation of summaries of evidence, free of the constraints that LSR authors face in preparing reviews for publication in traditional journals.24 A recently published framework for the development of living practice guidelines describes aspects of the planning, production, reporting and dissemination process in more detail.19
The feasibility of sustaining LSRs is largely determined by the capacity of the author team (e.g., personnel, resources and commitment) and the flow of new evidence that is generated.4 Our suggested reporting approach is predominantly manual. Automation is often proposed to help reduce the workload and could be particularly beneficial for the searching, deduplication and screening components of LSRs. From our experience as information professionals (MIM, SM), overall automation is currently not possible. However, we elucidate the possibility for automation of these three tasks in the following.
First, with regard to searching, we are not aware of any tool that can fully automate running ready-designed searches from several licensed or free access databases and export results into one file. Automating this part of the search process should be explored through collaborations between systematic review tool developers and database vendors. Implementation should be feasible, at least with the freely available sources, such as PubMed, ClinicalTrials.gov and WHO International Clinical Trials Registry Platform, assuming appropriate application programming interfac s (APIs) are in place. In addition, specialised databases such as the Cochrane COVID-19 Study Register and Epistemonikos’ L·OVE platform25 26 have helped to reduce sources to be searched when identifying evidence during the pandemic.
Second, concerning deduplication, important progress has been made in recent months. New and refined tools such as Deduklick27 and the SRA Deduplicator (https://sr-accelerator.com/%23/deduplicator) can now assist with the maintenance of LSRs by semi-automating the removal of duplicates between databases within each search iteration as well as removing already screened records from an update search.
Third, screening automation is already partially addressed by the RCT classifier, a validated tool which automatically classifies publications as RCTs (randomised controlled trials). It is implemented in Cochrane’s Screen4Me service, Covidence and Robot Search28 29 and could be used when a high number of new records are expected. In light of this ongoing progress, we encourage LSR teams to explore automation according to their own needs and to collaborate with experts, such as information specialists and tool developers.
Conclusions
Currently there is no mechanism for knowing if the evidence underpinning a published LSR is up to date. The living Cochrane Review on ‘Nirmatrelvir combined with ritonavir for preventing and treating COVID-19’ provides a complete evidence profile that is easily accessible. To our knowledge, this is the first LSR published as a standard journal publication that makes information on the search and selection process of relevant studies publicly available on OSF in-between publication of the updated versions of the review. For readers, this open science-based solution increases transparency and improves the reliability and usefulness of the review. It could therefore be adopted more widely by systematic reviewers who wish to enhance the value of LSRs in an emerging evidence scenario. Possibilities for automation of the process should be explored by LSR teams, information specialists and developers of systematic review tools.
Data availability statement
Data are available in a public, open access repository.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Acknowledgments
The authors would like to thank the Cochrane Infectious Diseases Group, especially Paul Garner, for the opportunity to implement our idea into a Cochrane Review. We would also like to thank the co-authors of the Cochrane Review Rebecca Kuehn, Maria Popp, Ildikó Gágyor, Peter Kranke, Patrick Meybohm and Nicole Skoetz for agreeing to the new LSR reporting approach. MIM would like to thank Bernd Richter, who mentioned the idea of providing a publicly available list of studies relevant to a systematic review in-between its update cycles in a conversation from several years ago.
References
Footnotes
Twitter @mintimetz
Contributors MIM wrote the first draft of the manuscript. SM, SW and SR revised the manuscript. All authors read and approved the final version. MIM is the guarantor and attests that all authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests MIM is an Associate Editor of BMJ Evidence Based Medicine and has conducted the search for multiple living systematic reviews published by Cochrane. SM is the Senior Information Specialist with the Australian National COVID‑19 Clinical Evidence Taskforce guidelines. SR is the first author and SW the corresponding author of the living Cochrane Review on ‘Nirmatrelvir combined with ritonavir for preventing and treating COVID‐19’.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.