Article Text

Download PDFPDF

Claims of ‘no difference’ or ‘no effect’ in Cochrane and other systematic reviews
  1. Phoebe Rose Marson Smith1,
  2. Lynda Ware2,
  3. Clive Adams3,
  4. Iain Chalmers4
  1. 1Science Policy Research Unit, University of Sussex, Brighton, Brighton and Hove, UK
  2. 2UK Cochrane Centre, Oxford, Oxfordshire, UK
  3. 3School of Medicine, University of Nottingham, Nottingham, Nottinghamshire, UK
  4. 4Centre for Evidenced-Based Medicine, University of Oxford, Oxford, Oxfordshire, UK
  1. Correspondence to Phoebe Rose Marson Smith, Science Policy Research Unit, University of Sussex, Brighton BN1 9RH, UK; phoebemarsonsmith{at}gmail.com

Abstract

Estimates of treatment effects/differences derived from controlled comparisons are subject to uncertainty, both because of the quality of the data and the play of chance. Despite this, authors sometimes use statistical significance testing to make definitive statements that ‘no difference exists between’ treatments. A survey to assess abstracts of Cochrane reviews published in 2001/2002 identified unqualified claims of ‘no difference’ or ‘no effect’ in 259 (21.3%) out of 1212 review abstracts surveyed. We have repeated the survey to assess the frequency of such claims among the abstracts of Cochrane and other systematic reviews published in 2017. We surveyed the 643 Cochrane review abstracts published in 2017 and a random sample of 643 abstracts of other systematic reviews published in the same year. We excluded review abstracts that referred only to a protocol, lacked a conclusion or did not contain any relevant information. We took steps to reduce biases during our survey. 'No difference/no effect' was claimed in the abstracts of 36 (7.8%) of 460 Cochrane reviews and in the abstracts of 13 (6.0%) of 218 other systematic reviews. Incorrect claims of no difference/no effect of treatments were substantially less common in Cochrane reviews published in in 2017 than they were in abstracts of reviews published in 2001/2002. We hope that this reflects greater efforts to reduce biases and inconsistent judgements in the later survey as well as more careful wording of review abstracts. There are numerous other ways of wording treatment claims incorrectly. These must be addressed because they can have adverse effects on healthcare and health research.

  • public health
  • health policy
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

View Full Text

Statistics from Altmetric.com

Footnotes

  • Contributors IC conceived and oversaw the project. PRMS and LW contributed to the data collection, analysis and writing of the paper. We thank PA, MB, CG, CA and TL for their contributions to the survey. We also thank PG, MB and NS for comments on earlier drafts of this paper.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.