Article Text
PDF
Statistics from Altmetric.com
The problem
Searches conducted a year or more before submission of a systematic review (SR) paper can result in journal editors or peer-reviewers rejecting it. Their concerns are that findings from SRs with ‘out-of-date’ searches might provide decision-makers with misleading evidence.1 Although recent technological advances have helped to speed up some review processes,2 other methodological advances have increased the work required such that reviews often require longer than 12 months to produce useful and rigorous findings. This puts many SRs at risk of rejection by journal editors.
We argue that a blanket 12-month cut-off point for searches is not appropriate, that it may hinder the dissemination of important research, and may have a knock-on impact on reviewers’ willingness to undertake the most ambitious reviews. We also argue that not all SRs are equally at risk of being ‘out of date’ at 12 months; while intervention effectiveness reviews in fast-moving areas may become outdated well before 12 months,3 others, such as qualitative evidence syntheses, are unlikely to have their findings substantially changed by the inclusion of new evidence. We focus on recent developments in SR designs, methods and technologies, to reflect on whether existing journal publishing guidelines are at odds with current SR approaches designed to improve review quality and usefulness.
Background
SRs are a recognised research approach for identifying, synthesising and analysing published evidence on topics of interest.4 Their aim is to provide a comprehensive, unbiased and trustworthy assessment of available evidence to support decision-makers in adopting policies that ‘do more good than harm’.5 Indeed, policy makers, researchers, practitioners and public stakeholders from within and outside of healthcare, place increased emphasis on the importance of conducting SRs.4 Although the rationale for conducting SRs remains the same, the process of systematic reviewing has changed significantly in recent years.6 Technological advances and automation have helped to speed up processes such as study identification.2 However, additional steps in the review process, reflecting new understandings about how to ensure SR findings are robust and relevant to policy decision-makers’ questions, have increased the work required. Continued increases in available published evidence have also increased the volume of research included and analysed in SRs.7 We suggest that there are two key factors that have significant time implications for reviewers today.
Breadth and depth of review questions impacts on the extent of work required
While historically SR questions were designed to be narrow, the Cochrane Handbook recognises that ‘Increasingly, reviews are becoming broader, aiming, for example, to identify which intervention—out of a range of treatment options—is most effective, or to investigate how an intervention varies depending on implementation and participant characteristics’.8 Review methods have evolved, resulting in a rich diversity of SRs in relation to the types of questions, review approaches and research methods.9 In figure 1, we provide examples to characterise different types of reviews that explore intervention effectiveness and the questions they seek to answer.6 Although we focus on reviews addressing intervention effectiveness, the breadth and complexity of reviews addressing other types of question is also likely to vary.6 10
Variation in the breadth and complexity of reviews addressing intervention effectiveness.24–27
There are numerous factors that may impact on the time required to complete broad, complex or multicomponent reviews and thus increase their risk of remaining unpublished. Below we discuss six key factors.
First, the increased breadth and depth of these reviews inevitably increases the work required. As figure 1 illustrates, broad effectiveness reviews cover more ground than simple effectiveness reviews. They may also require comparative work to assess the relative effectiveness of different intervention types. Thus, while the Cochrane Handbook recommends that searches are ‘as up to date as possible’, specifying within 12 months but ideally within 6 months of publication, it also acknowledges that reviews with a broad scope will ‘require more resources’.11
Complex effectiveness reviews often require significant conceptual development and interpretation, which can be time-consuming. The purpose of these reviews is to understand the critical components and mechanisms of multicomponent or multilevel interventions.12 Since methods to examine intervention complexity are relatively recent, this work also often requires methodological innovation, which further increases the time required.
Multicomponent approaches are often designed to understand intervention mechanisms and contextual complexity in addition to intervention effectiveness. As such, these approaches, such as mixed-method research synthesis or realist synthesis, often involve multiple synthesis stages.
Second, these types of review may also increase the extent of work required for effective patient and public involvement (PPI), which is vital to ensure that SRs produce useful findings.13 PPI can advance fields of study and help SRs address real-world questions from patient-based perspectives.14 However, the extent of PPI undertaken in any review will necessarily vary. As illustrated in figure 2, PPI is typically undertaken at the beginning of a review to identify relevant questions to answer. Further input during the synthesis stage and beyond can help to interpret emerging findings and disseminate findings to interested audiences. Input at the synthesis stage can assist with interpretation of the findings and is therefore particularly useful when broad questions are asked, or when conceptual development is a feature of the review. For example, in broad reviews examining multiple interventions, PPI can help interpret findings on effectiveness by providing insights about the likely acceptability of the various options reviewed. As such, the breadth and complexity of reviews and the extent of PPI required may be positively correlated.
Review procedures and impact on review completion time. GRADE, Grading of Recommendations, Assessment, Development and Evaluation; PPI, patient and public involvement; SRs, systematic review.
Third, to ensure SRs also address practice and policy interests, engagement activities may extend beyond the bounds of PPI and can include consultation with practitioners, research commissioners, policy makers and academics. At the EPPI Centre, we have found it helpful to engage with policy commissioners throughout the review, to understand how emerging findings may be applied. Involvement of policy makers and other stakeholders alongside patient and public stakeholders, will likely increase the review’s relevance and utility, but involving multiple stakeholders will inevitably increase the time required to complete a review.
Fourth, as the SR community continually strives to enhance and improve review methods, the steps required for any review have increased. Figure 2 illustrates several recently recommended review procedures for improving the quality and usability of SRs.15–18 While such advances are welcomed, these additional steps inevitably increase the time required for completion, even for single synthesis reviews. However, for broad or multicomponent reviews, this extra work may be multiplied. For example, in a multicomponent review containing both an effectiveness synthesis and a qualitative evidence synthesis, both Grading of Recommendations, Assessment, Development and Evaluation (GRADE) (a tool for summarising confidence in effects of interventions by outcome across studies)16 and GRADE-CerQUAL (for assessing the confidence of evidence from reviews of qualitative research)17 may be applied. In a broad review using GRADE, it will need to be applied to the findings about each type of intervention examined.
Fifth, as the extent of available research literature continues to grow,7 this can result in larger numbers of studies to screen for inclusion in each review. The impact is exponential for broad or multicomponent reviews. Though there are tools, such as machine learning and priority screening, that can reduce screening time, these technologies are unlikely to significantly impact on reviews that require iterative searching or that include ‘views studies’, which are more challenging to identify.19
Sixth, the report and publication process can result in additional time challenges. For example, sourcing peer-reviewers with expertise in complex reviews and novel synthesis methods, may take considerable time. At the EPPI Centre there have been occasions when it has taken 6 months to receive completed peer reviews, due to a lack of peer reviewers qualified to comment. Also, policy commissioners often require reviews to be written up as project reports and peer-reviewed prior to being written up for journal publication.
As broad, complex and multicomponent reviews will inevitably take longer, the risks of being out of date, and therefore, misleading, may appear to be increased. However, we argue that the risks are less applicable to these types of reviews.
Broad, complex and multicomponent reviews may not age as quickly as simple effectiveness reviews
The findings of reviews solely involving meta-analysis may be substantially affected by being out of date. Indeed, a single new study may completely alter the review's conclusion if the direction or precision of the pooled estimate of effect is changed. However, not all meta-analyses will be affected. Areas where reviews go out of date quickly usually involve rapidly evolving treatments or new diseases. Indeed, proponents of Living Systematic Reviews—that is, a SR that is continually updated, incorporating relevant new evidence as it becomes available—acknowledge that such reviews are only warranted in certain circumstances. These include when certainty in existing evidence is low and when the research field covered by the SR is moving relatively quickly, and new evidence is emerging.20 A key challenge for Living Systematic Reviews is the coordinated and continuous effort required to maintain them. Broader effectiveness reviews that examine a range of interventions are less at risk, though portions of them may be (where new interventions appear). The nuanced findings arising from complex or multi-component reviews are arguably even less likely to be compromised by ‘out-of-date’ searches. Reviews which examine perspectives and understandings (eg, patient or provider views) are less likely to date as quickly, as cultural norms and expectations tend to evolve more slowly over time. As proposed in a consensus statement published in the BMJ on whether and when to update SRs, reviewers should consider ‘whether new research or new methods will affect the findings’.21
Recommendations
Systematic reviewing has changed, adapting to the needs of policy makers and practitioners across many fields. Unless and until the time savings from automation of review processes can fully mitigate the increased workload due to added complexity and procedures to ensure rigour, there are justifiable reasons for extended review timelines. Review teams often face the dilemma of balancing the need for useful and rigorous, complex reviews with the need for up-to-date evidence. Given this balancing act, we argue that reviewers and publishers should take a more nuanced view to SRs with searches over a year old.
Journal editors and peer reviewers should consider the potential utility of reviews with so called ‘out-of-date’ searches, and how likely any new or additional studies are to change the review findings. Further, they should consider the extent of work required to update a review, particularly if the review is broad or addresses multiple questions. It is becoming common practice to update searches towards the end of a review to see whether any newer papers have been published. However, this is easier to conduct for narrow effectiveness reviews, as studies for this type of review are more easily identified and ‘slotted in’ to the analysis. Reviews which develop theory will likely involve more complex searching and interpretation.
The impact of unpublished reviews, in terms of research waste, also needs consideration. Non-publication of reviews results in lost effort and resources. A related concern is that publishing pressure is shaping the review landscape in favour of narrow, straightforward effectiveness reviews. For example, if review teams are focused on completing a review within 12 months, they might avoid engaging in PPI, or refrain from addressing research questions that require a broader scope or deeper analysis, ultimately leading to the generation of evidence that is less useful for policy decisions. A 12-month rule means that review authors who opt to undertake more complex reviews, or engage in extensive PPI, run the risk of non-publication of their findings. Conversely, less ambitious reviews have a greater chance of meeting journals’ time constraints. We recognise that in certain circumstances rapid reviews may be warranted. For example, in the current COVID-19 pandemic, SRs have been conducted in as little as 2 weeks.22 However, evidence indicates that these rapidly produced reviews actually increase the risk of misleading evidence being used to inform decision-making.23
Because of these concerns, we suggest some possible approaches that journal editors could adopt to address this problem:
To provide space in the submission form for authors to justify why their last search date is more than 12 months old and/or why their findings are unlikely to be altered by new research evidence.
To explicitly encourage peer reviewers to consider whether a request for an update is justified, considering the extent, nature and complexity of the work.
Conclusion
Some reviews will be compromised by searches that are more than 12 months old. However, others, such as complex multicomponent reviews, which often take longer than 12 months to complete, are less likely to be compromised by being out of date. As such, journal editors and peer reviewers should be encouraged to make a more nuanced assessment of when reviews may be considered out of date.
SRs have developed methods to produce more rigorous and useful evidence. We encourage journal editors to develop policies and practices to support this endeavour.
Ethics statements
Patient consent for publication
Acknowledgments
This work is not directly funded but builds on methodological work conducted for The London/York NIHR Policy Research Programme Reviews Facility funded through the NIHR PRP contract with the EPPI Centre at UCL (PR-R6-0113-11003).
References
Footnotes
Twitter @gillianstokesm
Correction notice Since this article was first published online, figure 1 has been updated with the correct references.
Contributors GS and KS wrote the initial draft of the manuscript and JT commented on it. GS, KS and JT revised the manuscript. All authors read and approved the final manuscript. The corresponding author attests that all listed authors meet the authorship criteria and that there are no additional authors who have been omitted.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Disclaimer The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the DHSC. We would like to thank our colleagues at the EPPI Centre involved in the novel methods development that inspired this paper.
Competing interests None declared.
Patient and public involvement statement As the paper makes clear, authentic and rigorous patient and public involvement (PPI) is an essential element of conducting high quality systematic reviews, however, as this piece is essentially theoretical, PPI was not sought.
Provenance and peer review Not commissioned; externally peer reviewed.