Survey of claims of no effect in abstracts of Cochrane reviews
BMJ 2003; 326 doi: https://doi.org/10.1136/bmj.326.7387.475 (Published 01 March 2003) Cite this as: BMJ 2003;326:475
- Phil Alderson, associate director (training) (palderson{at}cochrane.co.uk),
- Iain Chalmers, former director
- Correspondence to: P Alderson
- Accepted 23 December 2002
It is never correct to claim that treatments have no effect or that there is no difference in the effects of treatments. It is impossible to prove a negative or that two treatments have the same effect. There will always be some uncertainty surrounding estimates of treatment effects, and a small difference can never be excluded.1
Claims of no effect or no difference may mean that patients continue to be denied or exposed to interventions with important effects, either beneficial or harmful. They may also suggest that further research is unnecessary, so delaying satisfactory estimates of treatment effects.
The impossibility of proving no effect or no difference should be distinguished from the concept used for equivalence trials, where bounds are set on the differences that are deemed practically important. An analysis of 45 reports of trials purporting to test equivalence found that only a quarter set boundaries on their concept of equivalence.2 Given the rationale for avoiding claims of no effect or difference, such claims should be infrequent. We measured their prevalence in abstracts of systematic reviews published in the Cochrane Database of Systematic Reviews.
Methods and results
We downloaded the abstract for each of 989 complete reviews in the Cochrane Database of Systematic Reviews in issue 1, 2001, and 80 reviews published for the first time in issue 2, 2001. We extracted the sections headed Main Results and Reviewers' Conclusions. One of us (IC) read these sections, looking for claims of no effect. Only those reviews that stated “there was no difference” or “there was no effect” without any qualification about clinical or statistical significance were classified as claiming no effect or difference. The process was repeated for 143 Cochrane abstracts published for the first time in issues 1 and 2 of 2002, except that PA assessed them. Both authors then reviewed all the abstracts of reviews identified as claiming no effect or difference to agree on categorisation.
Claims of no effect or difference were made in 240 (22.5%) abstracts published in the 2001 issues of the Cochrane Database of Systematic Reviews and in 19 (13.3%) abstracts first published in 2002. The difference in proportions was −9.2%, 95% confidence interval −2.2% to −14.5%).
Comment
Inappropriate claims of no effect or no difference occurred in about a fifth of abstracts of Cochrane reviews. These claims may have been due to careless wording rather than a mistaken belief that no effect or difference had been shown. It is encouraging that these errors seem to be decreasing.
The decrease may reflect our unconscious use of more lenient evaluations in more recent publications or be due to a higher proportion of significant differences being detected. Alternatively, those reporting and editing Cochrane reviews may have become more aware of the inappropriateness of using no effect or difference because of recent initiatives aimed at improving the quality of these reviews. If so, we hope that there will soon be no Cochrane reviews making claims of no effect or difference and that this will not result in an increase in vague wording. Acceptable phrases include “no significant differences were detected” and “there is insufficient evidence either to support or to refute.”
We decided to concentrate on identifying errors in abstracts of Cochrane reviews because they are widely available. We have reported the errors to the relevant editorial teams. Inappropriate wording may have been used in other parts of the review, however, and we urge readers to promote improvements by using the electronic feedback system in the Cochrane Database of Systematic Reviews.
Acknowledgments
We thank Mike Clarke, Paul Garner, Peter Gøtzsche, Philippa Middleton, Jim Neilson, Andy Smith, and Liz Wager for comments on earlier drafts of this manuscript. The views expressed in this article represent those of the authors and are not necessarily the views or the official policy of the Cochrane Collaboration.
Contributors: The study was conceived and conducted by both authors, and both drfated the manuscript. PA is the guarantor.
Footnotes
-
Funding The work was carried out as part of the programme of the UK Cochrane Centre, which is funded by the NHS Research and Development Programme. The guarantor accepts full responsibility for the conduct of the study, had access to the data, and controlled the decision to publish.
-
Competing interests None declared.