Original Article
Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial

https://doi.org/10.1016/j.jclinepi.2020.01.005Get rights and content
Under a Creative Commons license
open access

Abstract

Objectives

To determine the accuracy of single-reviewer screening in correctly classifying abstracts as relevant or irrelevant for literature reviews.

Study Design and Setting

We conducted a crowd-based, parallel-group randomized controlled trial. Using the Cochrane Crowd platform, we randomly assigned eligible participants to 100 abstracts each of a pharmacological or a public health topic. After completing a training exercise, participants screened abstracts online based on predefined inclusion and exclusion criteria. We calculated sensitivities and specificities of single- and dual-reviewer screening using two published systematic reviews as reference standards.

Results

Two hundred and eighty participants made 24,942 screening decisions on 2,000 randomly selected abstracts from the reference standard reviews. On average, each abstract was screened 12 times. Overall, single-reviewer abstract screening missed 13% of relevant studies (sensitivity: 86.6%; 95% confidence interval [CI], 80.6%–91.2%). By comparison, dual-reviewer abstract screening missed 3% of relevant studies (sensitivity: 97.5%; 95% CI, 95.1%–98.8%). The corresponding specificities were 79.2% (95% CI, 77.4%–80.9%) and 68.7% (95% CI, 66.4%–71.0%), respectively.

Conclusions

Single-reviewer abstract screening does not appear to fulfill the high methodological standards that decisionmakers expect from systematic reviews. It may be a viable option for rapid reviews, which deliberately lower methodological standards to provide decision makers with accelerated evidence synthesis products.

Keywords

Randomized controlled trial
Literature screening
Systematic reviews
Rapid reviews
Accuracy

Cited by (0)

Funding: This work was supported by funding from Cochrane to the Cochrane Rapid Reviews Methods Group and by internal funds of Cochrane Austria.

Ethics approval and consent to participate: The local institutional review board at Danube University Krems concluded that no ethical approval was necessary for our study. All participants provided electronic consent to be part of the study.

Potential conflicts of interest: Gerald Gartlehner is a co-convener of the Cochrane Rapid Reviews Methods Group. All authors declare that they have no financial conflicts of interest regarding the topic of this study.

Availability of data and materials: The datasets used and analyzed during the current study are available from the corresponding author upon reasonable request.

Trial registration: Open Science Framework: https://osf.io/3jyqt.