Discussion

Table 2 presents four estimates of inappropriate claims of no difference or no effect: two from our survey and two from previously published surveys.1 2 Inappropriate claims were substantially less common in Cochrane reviews published in 2017 than among those published in 2001/2002 (7.8% vs 21.3%, respectively). This dramatic difference is consistent with greater care being taken in the wording of reports of Cochrane and other systematic reviews. But it may also reflect our attempts to reduce sources of biases compared with the earlier survey of Cochrane reviews,1 even though our attempt to blind judges to the provenance of the abstracts did not appear to have been successful.

Although it may be reasonable to take some satisfaction in our findings, they do not justify complacency. During our survey, we became aware of many ways in addition to the two (no difference and no effect) on which we had focused our attention. Our survey threw up 71 examples of wording deemed unacceptable by a majority of six experienced colleagues whom we invited to assess them. These phrases included ‘evidence for no effect’, ‘does not affect’, ‘no more common’ and ‘found no beneficial or harmful effects’. Gates and Ealing were also critical of claims of additional examples of unacceptable wording—such as ‘no treatment benefit’—found in their survey.2

These findings suggest that pedagogical and editorial efforts to deal with this problem remain. We are pleased to note that invalid claims of ‘no difference/effect’ are now addressed in the Cochrane handbook.3 Another example of a relevant initiative is the RevManHAL software.4 This uses the analyses in the data tables of a completed review; the outcome labels, numerical findings and CIs; and the number of studies and participants contributing to the outcome; and then formats these and pastes them into the Results section of the review. The software also holds two banks of phrases relevant to findings that are either clearly different or are not. Depending on the result, the software randomly selects a single phrase from the appropriate bank of phrases and adds this to the Results section under the appropriate heading and before the relevant formatted numerical data, thus producing text that is less repetitive and more readable.5 6 We are grateful to an anonymous reviewer of this article for suggesting the phrases ‘we did not find any eligible evidence of a difference’, and ‘we did not find any eligible evidence of a clear difference’, and these have now been included in the RevManHAL software.

It remains important to recognise the uncertainties inherent in statistical estimates of treatment differences, and the need to distinguish between ‘no evidence of a difference/effect’ and ‘evidence of no difference/effect’. In practical terms, this implies using CIs to assess how confidently important treatment differences can be ruled out and using wording that reflects the probabilistic approach entailed.

Systematic reviews of healthcare interventions need to be as clear as the evidence will support. Many people will read only the abstracts of systematic reviews. Leaving readers with the impression that there is no difference between alternative treatments may result in dangerously misinformed clinical decisions and failure to address important uncertainties in additional research.