Article Text

Download PDFPDF

Claims of ‘no difference’ or ‘no effect’ in Cochrane and other systematic reviews
  1. Phoebe Rose Marson Smith1,
  2. Lynda Ware2,
  3. Clive Adams3,
  4. Iain Chalmers4
  1. 1 Science Policy Research Unit, University of Sussex, Brighton, Brighton and Hove, UK
  2. 2 UK Cochrane Centre, Oxford, Oxfordshire, UK
  3. 3 School of Medicine, University of Nottingham, Nottingham, Nottinghamshire, UK
  4. 4 Centre for Evidenced-Based Medicine, University of Oxford, Oxford, Oxfordshire, UK
  1. Correspondence to Phoebe Rose Marson Smith, Science Policy Research Unit, University of Sussex, Brighton BN1 9RH, UK; phoebemarsonsmith{at}gmail.com

Abstract

Estimates of treatment effects/differences derived from controlled comparisons are subject to uncertainty, both because of the quality of the data and the play of chance. Despite this, authors sometimes use statistical significance testing to make definitive statements that ‘no difference exists between’ treatments. A survey to assess abstracts of Cochrane reviews published in 2001/2002 identified unqualified claims of ‘no difference’ or ‘no effect’ in 259 (21.3%) out of 1212 review abstracts surveyed. We have repeated the survey to assess the frequency of such claims among the abstracts of Cochrane and other systematic reviews published in 2017. We surveyed the 643 Cochrane review abstracts published in 2017 and a random sample of 643 abstracts of other systematic reviews published in the same year. We excluded review abstracts that referred only to a protocol, lacked a conclusion or did not contain any relevant information. We took steps to reduce biases during our survey. "No difference/no effect" was claimed in the abstracts of 36 (7.8%) of 460 Cochrane reviews and in the abstracts of 13 (6.0%) of 218 other systematic reviews. Incorrect claims of no difference/no effect of treatments were substantially less common in Cochrane reviews published in in 2017 than they were in abstracts of reviews published in 2001/2002. We hope that this reflects greater efforts to reduce biases and inconsistent judgements in the later survey as well as more careful wording of review abstracts. There are numerous other ways of wording treatment claims incorrectly. These must be addressed because they can have adverse effects on healthcare and health research.

  • public health
  • health policy
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Evidence generated from controlled comparisons of treatments is used in making treatment choices and recommendations. Although the data yielded by treatment comparisons are subject to uncertainty both because of the quality of the data and the play of chance, authors sometimes use methods such as statistical significance testing to make definitive statements that there is ‘no effect of’ or ’no difference between’ treatments.

‘Absence of evidence is not evidence of absence’. In 2003, a survey of abstracts of 1212 Cochrane reviews published in 2001/2002 found that 259 (21.3%) had made claims of ‘no difference’ or ‘no effect’ of treatments, without any qualification about clinical or statistical significance.1

Claims that ‘there is no difference between the effects of two or more treatments (or between treatment and no treatment)’ continue to be made, both in systematic reviews and in reports of individual treatment comparisons. A recent survey showed that such claims were made in 9 (7.5%) of 120 reports of randomised trials published during 2016/207 in the BMJ, JAMA, Lancet and the New England Journal of Medicine. 2

We have surveyed the prevalence of claims of no difference or no effect of treatments in the abstracts of Cochrane reviews and of other systematic reviews published in 2017.

Materials and methods

Abstracts of the 643 Cochrane reviews published in 2017 were extracted from the Cochrane Library. We drew a random sample of other systematic reviews published in 2017 from PubMed using the following search term:

(((((((“systematic review”(Publication Type)) AND (“2017”[Date - Publication])) AND “English”(Language))) NOT Cochrane) AND Review (ptyp) AND (“2017_01_01”(PDat) / “2017_12_31”(PDat))), and assessed the first 643 reports retrieved.

We distinguished between the terms ‘meta-analysis’ and ‘systematic review’ and evaluated only abstracts of reviews that claimed to have been ‘systematic’. An abstract was excluded if it referred only to a protocol, lacked a conclusion or did not contain any relevant information.

Before data extraction began, we endeavoured to reduce biases and increase consistency by

  1. Establishing agreement about which forms of words would be considered incorrect and producing a document with examples of claims of no difference and no effects.

  2. Giving each of the two assessors (PRMS and LW) a document arranging abstracts in an order assigned using a random number generator. One data extractor evaluated from the first abstract listed to the last abstract listed; the other evaluated the abstracts in the reverse order. This aimed to reduce the probability of any trend over time in assessing the articles more or less leniently due to learning effects.

  3. Preparing abstracts without author information as plain text and assigning a random number in an attempt to reduce biases by blinding the provenance of the review.

Results

One hundred eighty-three (28 %) of the 643 abstracts of Cochrane reviews and 425 (66 %) of 643 of the abstracts of other systematic reviews were excluded because they did not mention results or a conclusion (table 1).

Table 1

Results of survey of claims in abstracts of Cochrane and other systematic reviews published in 2017

Discussion

Table 2 presents four estimates of inappropriate claims of no difference or no effect: two from our survey and two from previously published surveys.1 2 Inappropriate claims were substantially less common in Cochrane reviews published in 2017 than among those published in 2001/2002 (7.8% vs 21.3%, respectively). This dramatic difference is consistent with greater care being taken in the wording of reports of Cochrane and other systematic reviews. But it may also reflect our attempts to reduce sources of biases compared with the earlier survey of Cochrane reviews,1 even though our attempt to blind judges to the provenance of the abstracts did not appear to have been successful.

Table 2

Frequency of claims of ‘no treatment differences/effects’ in 2001/202 and 2016/207

Although it may be reasonable to take some satisfaction in our findings, they do not justify complacency. During our survey, we became aware of many ways in addition to the two (no difference and no effect) on which we had focused our attention. Our survey threw up 71 examples of wording deemed unacceptable by a majority of six experienced colleagues whom we invited to assess them. These phrases included ‘evidence for no effect’, ‘does not affect’, ‘no more common’ and ‘found no beneficial or harmful effects’. Gates and Ealing were also critical of claims of additional examples of unacceptable wording—such as ‘no treatment benefit’—found in their survey.2

These findings suggest that pedagogical and editorial efforts to deal with this problem remain. We are pleased to note that invalid claims of ‘no difference/effect’ are now addressed in the Cochrane handbook.3 Another example of a relevant initiative is the RevManHAL software.4 This uses the analyses in the data tables of a completed review; the outcome labels, numerical findings and CIs; and the number of studies and participants contributing to the outcome; and then formats these and pastes them into the Results section of the review. The software also holds two banks of phrases relevant to findings that are either clearly different or are not. Depending on the result, the software randomly selects a single phrase from the appropriate bank of phrases and adds this to the Results section under the appropriate heading and before the relevant formatted numerical data, thus producing text that is less repetitive and more readable.5 6 We are grateful to an anonymous reviewer of this article for suggesting the phrases ‘we did not find any eligible evidence of a difference’, and ‘we did not find any eligible evidence of a clear difference’, and these have now been included in the RevManHAL software.

It remains important to recognise the uncertainties inherent in statistical estimates of treatment differences, and the need to distinguish between ‘no evidence of a difference/effect’ and ‘evidence of no difference/effect’. In practical terms, this implies using CIs to assess how confidently important treatment differences can be ruled out and using wording that reflects the probabilistic approach entailed.

Systematic reviews of healthcare interventions need to be as clear as the evidence will support. Many people will read only the abstracts of systematic reviews. Leaving readers with the impression that there is no difference between alternative treatments may result in dangerously misinformed clinical decisions and failure to address important uncertainties in additional research.

Acknowledgments

We thank Phil Alderson, Martin Burton, Claire Glenton and Toby Lasserson for their contributions to the survey. We also thank Paul Glasziou, Martin Burton, Nancy Santesso, Robert Matthews and two anonymous referees for helpful comments on earlier drafts of this paper.

References

Footnotes

  • Contributors IC conceived and oversaw the project. PRMS and LW contributed to the data collection, analysis and writing of the paper. We thank PA, MB, CG, CA and TL for their contributions to the survey. We also thank PG, MB and NS for comments on earlier drafts of this paper.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.