• evidence-based medicine

Clinicians are faced with the challenge of keeping abreast of the rapidly growing clinical literature. One method that can help clinicians meet this challenge is the provision of synthesised evidence by such groups as the Cochrane Collaboration. The Cochrane Library contains the full text of Cochrane reviews, which bring together information on controlled studies (mostly randomised controlled trials) in a standard format. The first 1000 reviews were prepared by early 2001, and there are now more than 1600 available.¹ The task of synthesising the clinical literature is a huge undertaking, and in this editorial we will use the number of studies in a typical Cochrane review to provide an estimate of the number of Cochrane reviews that are needed to cover all trials currently in the Cochrane Central Register of Controlled Trials (CENTRAL).^2–3

CENTRAL is the main source used by Cochrane reviewers to locate studies. It contains citations to more than 300 000 reports of studies, which may be eligible for inclusion in Cochrane reviews. CENTRAL is reference-based rather than study-based, and therefore multiple references may be included for a single trial.

We examined 1000 reviews from the Cochrane Database of Systematic Reviews in Issue 1, 2001, of the Cochrane Library to determine the average number of studies, and references to the studies, included in a typical Cochrane review. Comparing this with an estimate for the number of relevant references in CENTRAL, it is possible to predict how many more Cochrane reviews are needed to cover the healthcare research already in CENTRAL.

Of the 1000 reviews, 11 had been replaced by notices of withdrawal, leaving 989 complete reviews. The included studies in these reviews were hand counted by 1 author (SM), and this process was confirmed by a computerised count of the database files. We used median values to express averages because the small number of reviews with unusually high numbers of studies would lead to a positive skewing of mean values. We used a random sample of 104 reviews to obtain the average number of references per study in a Cochrane review.

The 989 Cochrane reviews included information on a total of 9778 studies. The number of studies per review ranged from 0–136, with a median of 6 (interquartile range 3 to 12). 48 reviews (5%) did not include data from any studies, and 19 reviews (2%) included ≥50 studies. In the 104 reviews where the number of references for each included study was counted, 1051 studies were reported in 1514 references. The median number of references per study per review was 1.1 (interquartile range 1.0–1.5).

An average of 1.1 references per study suggests that 300 000 references in CENTRAL might refer to about 270 000 studies. Thus, if the average Cochrane review includes 6 studies, 45 000 Cochrane reviews would be needed to cover 300 000 references. However, the 1.1 references per study might be an underestimate and some of the records in CENTRAL might not relate to studies that are suitable for Cochrane reviews. For example, the Cochrane Stroke Group reported an average of 2.1 references per study in their Specialized Register of Stroke Trials (4525 reports corresponding to 2132 stroke trials).⁴ On the basis of the Stroke Group figure, the estimated number of additional Cochrane reviews needed to cover all references in CENTRAL would be 24 000. However, this still does not take into account the probability that some references in CENTRAL would not be eligible for Cochrane reviews.

Comparing the typical size of one of the first 1000 Cochrane reviews with that found by descriptive studies of other types of systematic reviews is problematic. Such studies have found averages of 10 or 14 studies per review.⁵ Instead of trying to estimate the appropriate size of an average systematic review, our finding of the typical size of a Cochrane review based on the first 1000 reviews combined with information on the content of CENTRAL can provide a reasonable estimate of how much work remains to be done.

Even with the assumption that future Cochrane reviews will each include about 6 studies, the above estimates of 24 000 or 45 000 additional reviews are likely to be excessive, because not all the references in CENTRAL (or the studies they report) might be suitable for inclusion in new Cochrane reviews. For example, there will be records for references to studies that are already included in Cochrane reviews but for which the reference is not yet linked appropriately to the study in the review, and some references appear more than once in CENTRAL. Some of the studies will relate to interventions that are no longer used or will only have information that is not relevant to healthcare decisions, and some references do not relate to studies of healthcare interventions. Finally, there will be some references to studies that are eligible for existing Cochrane reviews but have not yet been included.

Therefore, if 45 000 or 24 000 are accepted as overestimates, some correction is needed. At the moment, no data exist to help with this estimation, but we think it is reasonable to predict that at least 10 000 Cochrane reviews are needed to cover a substantial proportion of the studies relevant to health care that have already been identified, taking into account the typical size of a Cochrane review, number of included studies and their references, and the probable number of eligible studies in CENTRAL. This estimation is a crude approximation but is probably a reasonable minimum. Reviews will also be needed on the many thousands of new studies that are published each year, and the 10 000 reviews will need to be kept up to date.

When will 10 000 reviews be ready? The Cochrane Database of Systematic Reviews is updated every quarter to include new Cochrane reviews, as well as those that have been updated. Currently, there are more than 1600 Cochrane reviews and 1200 protocols for reviews in progress in the Cochrane Database of Systematic Reviews.¹ The figure shows the recent growth in the number of reviews. Fitting a simple model to the data allows us to predict when this growth will reach 10 000 Cochrane reviews. We estimate that this will be between 2010 and 2015, and we look forward to either being proved correct or pessimistic in our estimate. If, however, 24 000 or 45 000 reviews are needed to cover all existing randomised trials of health care, it will take until about the 2030s and 2040s, respectively, if the current rate of growth continues.

• Download figure
• Open in new tab
• Download powerpoint

The need to produce and use reviews of healthcare interventions should also be faced by trialists. If reports of new research were routinely discussed within the context of updated systematic reviews, users of this new research would, quite correctly, be presented immediately with a key part of the evidence needed to make decisions about health care.⁶

The implications of our study represent considerable challenges not only to the Cochrane Collaboration and others involved in the production of systematic reviews, but also to organisations involved in policy decisions about health care that aim to base treatment guidelines and guidance on up to date reviews of healthcare evidence. The challenges to be addressed in producing 10 000 systematic reviews and keeping them up to date centre on the engagement of enthusiasm, skills, time, and resources to undertake and support such work. The alternatives, however, might be to rely on non-systematic, out of date summaries of evidence supplemented by an occasional systematic review when it is available, or to make decisions without concern for reliable evidence on the likely effects of the interventions used.⁷ The authors would be very pleased to be contacted by anyone wishing to contribute to growth in the number of Cochrane reviews.