Article Text

Download PDFPDF

Mental health
Evaluation of spin in abstracts of papers in psychiatry and psychology journals
Free
  1. Samuel Jellison1,
  2. Will Roberts1,
  3. Aaron Bowers1,
  4. Tyler Combs1,
  5. Jason Beaman2,3,
  6. Cole Wayant1,
  7. Matt Vassar1
  1. 1 College of Osteopathic Medicine, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
  2. 2 Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
  3. 3 Department of Psychiatry, Oklahoma State University Medical Center, Tulsa, Oklahoma, USA
  1. Correspondence to Samuel Jellison, Oklahoma State University Center for Health Sciences, Tulsa, OK 74107, USA; samuesj{at}okstate.edu

Abstract

We have identified ‘spin’ in abstracts of randomised controlled trials (RCTs) with nonsignificant primary endpoints in psychiatry and psychology journals. This is a cross-sectional review of clinical trials with nonsignificant primary endpoints published in psychiatry and psychology journals from January 2012 to December 2017. The main outcome was the frequency and manifestation of spin in the abstracts. We define spin as the ‘use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results’. We have also assessed the relationship between industry funding and spin. Of the 486 RCTs examined, 116 were included in our analysis of spin. Spin was identified in 56% (n=65) of those included. Spin was found in 2 (2%) titles, 24 (21%) abstract results sections and 57 (49.1%) abstract conclusion sections. Evidence of spin was simultaneously identified in both results and conclusions sections in 15% of RCTs (n=17). Twelve articles reported industry funding (10%). Industry funding was not associated with increased odds of spin in the abstract (unadjusted OR: 1.0; 95% CI: 0.3 to 3.2). We found no relationship between industry funding and spin in abstracts. These findings raise concerns about the effects spin may have on clinicians. Further steps could be taken to address spin, including inviting reviewers to comment on the presence of spin and updating Consolidated Standards of Reporting Trials guidelines to contain language discouraging spin.

  • data reporting
  • psychiatry
  • bias
  • research methodology
  • psychiatry in literature

Statistics from Altmetric.com

Introduction

Randomised controlled trials (RCTs) serve as the gold standard in psychiatry. Given the importance of such trials to clinical practice, it is imperative that results be reported objectively.

Researchers are encouraged to conduct studies and report findings according to the highest ethical standards.1 2 This standard means reporting results completely, in accordance with a protocol that outlines primary and secondary endpoints and prespecified subgroups and statistical analyses. However, authors are free to choose how to report or interpret study results. In an abstract, authors may include only the results they want to highlight or the conclusions they wish to draw. These results and conclusions, however, may not accurately summarise the findings of the study. When such a misrepresentation of study results occurs, there is said to be spin. Spin has been defined as, ‘the use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results’.3 Many practices contribute to spin, including the selective reporting of outcomes,4 5 p-hacking,6 7 inappropriate application of statistical measures like relative risk8 and manipulation of figures or graphs.9 10

Spin in abstracts has recently been discussed in a systematic review.11 Evidence suggests that abstract information alone is capable of changing a majority of clinicians’ care decisions.12 For example, when unadjusted analyses or secondary outcomes are given undue attention in abstracts, readers’ overall appraisal of the contents of a manuscript is altered.13 Additionally, a previous systematic review showed there to be a higher rate of favourable conclusions in industry-funded studies compared with other sponsorships.14

We have evaluated the prevalence of spin in abstracts of RCTs with nonsignificant primary endpoints in the psychology and psychiatry literature and have explored the association between spin and industry funding.

Methods

We followed the methods of Boutron et al in developing training material, defining spin, and devising extraction forms.3 The protocol for the study was published in advance.15 We searched PubMed on 21 May 2018 for RCTs of psychiatry and behavioural treatments in humans published in 2012 to 2017 in key journals: JAMA Psychiatry, American Journal of Psychiatry, Journal of Child Psychology and Psychiatry, Psychological Medicine, British Journal of Psychiatry and Journal of the American Academy of Child and Adolescent Psychiatry. These journals were selected as the top journals that publish clinical trials from the Psychiatry section of Google Scholar. Search results were added to a PubMed collection and exported to Rayyan16 for screening by title and abstract. After training, SJ and WR screened records for inclusion and extracted data for spin. Training videos were created by CW to define spin and explain the screening and data extraction procedures. If, after discussion, SJ and WR did not agree, differences were resolved by a third adjudicator.

To be included, a trial had to randomise humans to an intervention, statistically compare two or more groups and have a nonsignificant primary endpoint. Records were excluded if they did not meet these criteria. Data extraction was done using a pilot-tested Google Form. Items extracted from each included trial were the title, journal, funding source, comparator arm, primary endpoint, the statistical analysis of the primary endpoint, secondary endpoints, the statistical analysis of secondary endpoints and trial registration number (if reported). The primary endpoints evaluated had to be clearly stated by the author as the primary endpoint. If a primary endpoint was not clearly stated, the trial registry was referenced. If a primary endpoint was still not determined, the trial was excluded. Extractors were then asked whether spin was present in the abstract of the randomised trial. Spin in the title, abstract results, abstract conclusions and selection of reported endpoints was considered.

We considered there to be evidence of spin if trial authors focused on statistically significant results, interpreted statistically nonsignificant results as equivalent or noninferior, used favourable rhetoric in the interpretation of nonsignificant results (eg, ‘trend toward significance’) or claimed benefit of an intervention despite statistically nonsignificant results. Our assessments of spin were conducted irrespective of whether the reported endpoints were related to treatment efficacy or safety. Rather, we judged any strategy of spin in the context of a nonsignificant primary endpoint to be a strategy aimed at making the trial results appear more favourable. We used the study’s prespecified alpha value or confidence intervals to determine what constituted a statistically significant result. If authors focused on statistically significant results, we catalogued whether it was a within-group comparison, subgroup analysis, statistically significant secondary endpoints or modified treatment population. All other strategies of spin that were apparent, but did not fall under one of the above categories, were recorded. Spin was considered to be associated with industry funding if its funding source was documented as either ‘Industry’ or ‘Multiple with industry’. Summary statistics (frequencies and proportions) were calculated using Google Sheets. ORs were calculated using STATA V.13.1 (StataCorp, LLC, College Station, Texas, USA). No changes were made to the protocol while the study was conducted.

Results

We included 116 papers with statistically nonsignificant results for the primary endpoints (figure 1).

Spin was identified in 56% (65/116) of included RCTs. Evidence of spin was found in 2 (2%) titles, 24 (21%) abstract results sections and 57 (49%) abstract conclusion sections. Evidence of spin was simultaneously identified in both results and conclusions sections in 17 RCTs (15%). Spin was identified more commonly in trials that used placebo or care-as-usual as the comparator arm (table 1).

Table 1

Treatment groups of articles containing spin

The types of spin used in titles and abstracts are listed in table 2.

Table 2

Manifestation of spin in title and abstract results

The types of spin used in abstract conclusions are listed in table 3.

Table 3

Spin in abstract conclusions

The main funding sources are listed in table 4. The frequency of spin found for each funding source is listed in table 5. Industry funding was not associated with increased odds of spin in the abstract (unadjusted OR: 1.0; 95% CI: 0.3 to 3.2).

Table 4

Report characteristics

Table 5

Presence of spin by funding source

Discussion

In our survey of RCT abstracts published in six top psychiatry/psychology journals over 5 years, we found that the majority of articles (56%) contained spin in one form or another. Most often, spin was found in the results and conclusion sections of the abstract. We did not find any relationship between the presence of spin and industry funding. Spin was most commonly associated with public funding. In the context of previous investigations of spin, our results are comparable. Lazarus et al identified evidence of spin in 84% of abstracts of non-randomised trials of therapeutic interventions.17 In a similar survey of RCTs in robotic colorectal surgery, Patel et al found that 82% of trials contained spin in the abstract or conclusion section.18 Lastly, in a study evaluating only misleading primary outcome reporting, Mathieu et al found that 23% of rheumatology trials had conclusions that disagreed with the results of the abstract.19

Because the concept of spin is relatively new to the scientific community, there are few studies that have analysed the impact that spin has on clinical decision-making or the funding of subsequent trials. We have found only one.11 In that study, Boutron et al conducted an RCT in which they analysed the interpretation of abstract results by 300 oncologists.20 Each participant was given two abstracts to read, one with spin in the conclusions and one without, both with statistically nonsignificant results. For abstracts with spin, participants rated the experimental treatment as being more beneficial and were more interested to read the full text of the article.20 While definitive conclusions about the effect of spin on real-life clinical practice are difficult to ascertain based off one trial of spin, the results of the Boutron trial suggest that there is significant potential for misinterpretation of results when spin is introduced to article abstracts. Conversely, another 2017 RCT of 622 clinicians found that when the primary outcome of a trial is clear in the abstract, the presence of spin or overstatement of nonsignificant results does not change the perception of the efficacy of the treatment.21 In fact, clinicians found trials without spin in the abstract to be more valid than those with overstated results.

Researchers have an ethical obligation to honestly and clearly report the results of their research.22 Adding spin to the abstract of an article may mislead physicians who are attempting to draw conclusions about a treatment for patients. Most physicians read only the article abstract the majority of the time, while up to 25% of editorial decisions are based on the abstract alone.23–25 Those who write clinical trial manuscripts know that they have a limited amount of time and space in which to capture the attention of the reader. Positive results are more likely to be published,26 and many manuscript authors have turned to questionable reporting practices in order to beautify their results.13 Worst of all, Lazarus et al found that 15% of peer-reviewers asked authors to add spin to their manuscripts.27

To solve potential misinterpretation of research findings caused by spin, journal editors should consider inviting reviewers to comment on the presence of spin. This may help ensure that abstracts are free from spin by the time they reach researchers, clinicians and lay people. Additionally, reporting guidelines are already being used by several journals to ensure accurate and transparent reporting of clinical trial results,28 29 and the use of such guidelines improves trial reporting.30 31 While the recently published Consolidated Standards of Reporting Trials for abstracts does not contain language discouraging spin,32 research reporting could be improved by discouraging spin in abstracts.

Strengths and limitations

Our study has several key strengths and limitations. First, we used two independent data extractors to mitigate errors in data analysis. Second, we evaluated 5 years of psychology and psychiatry literature to promote the generalisability of our findings. However, despite our focus on highly ranked and potentially highly read journals, our selection of articles may not be generalisable to all psychiatry RCTs or other types of studies. Further, despite our use of objective criteria for what defines spin and dual, independent data extraction, all assessments of spin are prone to subjective interpretation.

Conclusion

Over 50% of trials published in top psychiatry/psychology journals contain spin in the abstract. No association was found between spin and industry funded studies. Further research is needed to establish the effects of spin on clinical decision-making and the funding of future studies. We suggest future studies to assess the frequency of spin within other specialty journals, different study designs and its effect on lay people. Authors, journal editors and peer-reviewers should continue to be vigilant for spin to reduce the risk of biased reporting of trial results.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.

Footnotes

  • Twitter @samjellison

  • Contributors All authors substantially contributed to the conception or design of the work, or the acquisition, analysis or interpretation of data. All authors contributed to the drafting of the work or revising it critically for important intellectual content. All authors gave final approval of the version published. All authors agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Patient consent for publication Not required.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.