Article Text
PDF
Abstract
Objective To compare the prevalence of ‘spin’, and specific reporting strategies for spin, between infographics, abstracts and full texts of randomised controlled trials (RCTs) reporting non-significant findings in the field of health and medicine and to assess factors associated with the presence of spin.
Design Cross-sectional observational study.
Data source Publications in top quintile health and medical journals from August 2018 to October 2020 (Journal Citation Reports database).
Eligibility criteria Infographics, abstracts and full texts of RCTs with non-significant results for a primary outcome.
Main outcome(s) and measure(s) Presence of spin (any spin and spin in the results and conclusions of infographics, abstracts and full texts).
Exposure(s) Conflicts of interest, industry sponsorship, trial registration, journal impact factor, spin in the abstract, spin in the full text.
Results 119 studies from 40 journals were included. One-third (33%) of infographics contained spin. Infographics were not more likely to contain any spin than abstracts (33% vs 26%, OR 1.4; 95% CI 0.8 to 2.4) or full texts (33% vs 26%, OR 1.4; 95% CI 0.8 to 2.4). Higher journal impact factor was associated with slightly lower odds of spin in infographics and full texts, but not abstracts. Infographics, but not abstracts or full texts, were less likely to contain spin if the trial was prospectively registered. No other significant associations were found.
Conclusions Nearly one-third of infographics spin the findings of RCTs with non-significant results for a primary outcome, but the prevalence of spin is not higher than in abstracts and full texts. Given the increasing popularity of infographics to disseminate research findings, there is an urgent need to improve the reporting of research in infographics.
- Methods
- Health
- Publishing
Data availability statement
Data are available on reasonable request. Original data may be made available by the authors on reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Many infographics summarising health and medical research do not report sufficient information to allow for accurate interpretation of study results, despite their increasing use to summarise research findings.
WHAT THIS STUDY ADDS
This investigation of spin in 119 infographics, abstracts and full texts of randomised controlled trials reporting non-significant findings for a primary outcome found that nearly one-third of infographics (33%) spin research findings. This was not statistically different to the prevalence of spin in abstracts (26%) and full texts (26%). Spin was substantially less likely in infographics if the study was prospectively registered, and slightly less likely in infographics and full texts if it was published in a higher impact factor journal.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The findings of this study underscore a need to improve the reporting of infographics summarising health and medical research, as many infographics spin research findings and health professionals, researchers and patients often use infographics as a substitute for reading full-text articles.
Introduction
Infographics (or ‘information graphics’) are becoming increasingly popular tools to summarise health and medical research and increase the attention research receives.1–5 However, many health professionals, researchers and patients use infographics as a substitute for reading full-text articles and view infographics as tools to help them save time by not having to read the full text.6 This could present a considerable issue if infographics do not accurately portray information from the full-text article and misrepresent study results.
In research, ‘spin’ is defined as a misrepresentation of study results that overemphasises the beneficial effects of an intervention or overstates safety compared with that shown by the results.7 8 Spin may be a result of inadvertent bias, ignorance of this scientific issue, expectation of specific outcomes or of wilful intent.8 9 Infographics may be uniquely prone to spin, as compared with abstracts and full-text articles, for several reasons. For example, infographic developers may want to limit text to improve visual appeal, which, in turn, might lead to selective reporting. Additionally, until recently, there has been little guidance for infographic developers on what to report in infographics of health and medical research.10 As such, important details may be left out in the reporting of infographics.
A recent study investigated the proportion of infographics in health and medical research that report key characteristics of the full-text article (eg, participant and intervention characteristics, benefits and harms of an intervention, effect estimates and measures of precision).1 Key characteristics included some aspects of spin which were found to varying degrees, such as whether the infographic’s conclusion acknowledged the risk of bias/certainty, had no issues of indirectness and was based on the primary outcome. However, there are many other examples of spin that were not investigated and may appear in infographics summarising health and medical research such as the omission of the primary outcome, selective reporting of positive results and omission of negative results and overenthusiastic interpretation of statistically non-significant findings as being effective.
The aim of this study was to compare the prevalence of spin (overall and by type of spin/specific reporting strategy) in infographics, abstracts and full texts of randomised controlled trials (RCTs) reporting non-significant findings for a primary outcome in the field of health and medicine. This study also aimed to assess factors associated with the presence of spin in infographics, abstracts and full texts.
Methods
Reporting
This cross-sectional study was reported following the Strengthening the Reporting of Observational Studies in Epidemiology guidelines (online supplemental file).11
Supplemental material
Data sources and search strategy
This study employed a rigorous approach to select studies with infographics from a range of high-quality health and medical journals in accordance with a previously published study.1 The Journal Citation Reports database was used to identify 597 journals ranked in the top quintile (based on 2019 impact factor) of 35 unique fields related to health and medical research. Subsequently, two researchers independently screened each journal’s website to determine whether the journal published infographics. Searches were conducted using the search bar (searching “infographic”, “graphic abstract”, “visual abstract”), and by manually searching all issues from August 2018 to October 2020 as well as other potentially relevant parts of the website (eg, designated section for infographics). Through this process, 69 journals were identified as having published infographics within the time frame. Two researchers then manually searched the 69 journals identified as having published infographics (starting from the October 2020 issue and working backwards in August 2018) for studies meeting inclusion criteria.
Eligibility criteria and selection of infographics
Published RCTs with negative findings for at least one primary outcome (ie, the intervention being tested had no significant effect compared with the control group) and an infographic summarising findings were eligible for inclusion. A maximum of four studies were included per journal to avoid biasing results towards journals publishing more infographics. RCTs with positive findings (findings that were statistically significant for a primary outcome), secondary analyses of RCTs, pilot or feasibility RCTs, non-inferiority RCTs, adaptive RCTs that were stopped before reaching the primary endpoint, RCTs with coprimary efficacy and safety outcomes where the efficacy outcome was positive but the safety outcome was negative (ie, no difference in adverse events), animal studies, studies where a full text could not be obtained and studies in which the primary outcome was not clear from the full text were excluded. Studies were also excluded if the infographic was a duplicate of a table or figure from the full text or if the infographic did not portray any results of the study. These exclusions meant only ‘visual abstracts’ or ‘graphical abstracts’—hereafter referred to as ‘infographics’ for simplicity—were included.
Infographic characteristics
Studies were characterised, based on publishing journal, into their corresponding unique fields (eg, allergy, anaesthesiology, clinical neurology, orthopaedics, sports sciences and surgery) related to medicine and health research listed in the Journal Citation Reports database (table 1). Additional characteristics extracted to describe the sample included whether the study had multiple primary outcomes, multiple time points, multiple intervention groups and if a primary time point was specified.
Number of included journals and studies per field
Outcomes
For the purposes of this study, spin was defined as the use of specific reporting strategies, regardless of motive, to portray the experimental intervention as effective (despite a statistically non-significant effect on a primary outcome(s)) or to distract readers from statistically non-significant effects.7 Both text and graphics (ie, graphs, figures and tables) were assessed for spin in infographics. Only text was assessed for spin in abstracts and full texts. The specific reporting strategies of interested were identified from previous literature7 and included:
Reporting focusing on statistically significant effects for within-group comparisons, secondary outcomes, subgroup analyses or a modified population of analyses (eg, per-protocol analyses).
Interpreting statistically non-significant effects a primary outcome as showing treatment equivalence or comparable effectiveness (ie, ‘both are effective’).
Reporting highlighting the benefit of an intervention despite statistically non-significant effects on a primary outcome.
Reporting focusing on statistically significant effects for one primary outcome while ignoring non-statistically significant effects for other primary outcomes.
Reporting focusing on statistically significant effects for a primary outcome at a non-primary time point or the primary outcome at one time point while ignoring non-statistically significant effects for the primary outcome at other time points (when no primary time point is specified).
Two researchers independently read the results and conclusions section of each included infographic, abstract and full text and coded whether the specific types of spin were present, absent or not applicable when either the infographic, abstract or full text did not include a particular section (eg, some infographics did not have a conclusion). Disagreements were resolved by discussion between the two researchers (two of either RM, GB, JK or JS) with a third researcher (JZ) being consulted if necessary.
Exposure variables
Additional variables extracted were the 2019 Journal Impact Factor of the journal included studies were published in and whether studies were prospectively registered, industry sponsored or reported conflicts of interest. Each of these variables was rated as yes, no or unclear and then collapsed to yes versus no/unclear for analysis. Studies were recorded as prospectively registered if the infographic, abstract or full text mentioned registration of the trial to an online database (eg, ClinicalTrials.gov) and the registry report showed it was ‘prospectively’ registered. If registry information was not available in these places, studies were recorded as not registered. Industry sponsorship was considered present if a non-academic or research organisation/institution funded any part of the study. In cases where funding was not mentioned, studies were reported as not having an industry sponsor. Studies were recorded as not having conflicts of interest if none were reported. All exposure variables were coded independently by two researchers (RM and GB), with disagreements resolved by discussion or consultation with a third researcher (JZ).
Data analysis
Study characteristics and the prevalence of spin (overall and in the results and conclusion sections specifically) were described using counts and percentages. Univariable logistic regression was performed to investigate differences in the prevalence of all types of spin between infographics, abstracts and full texts and investigate associations between exposure variables (see the ‘Exposure variables’ section) and any evidence of spin in the infographic, abstract and full text. All analyses were performed by using Stata V.16.1 (StataCorp).
Patient and public involvement
Patients or the public were not involved in the design, conduct, reporting or dissemination of this research.
Results
Selection of studies
A total of 263 studies were identified and screened from the search for negative RCTs with infographics. Of these, 144 were excluded. Common reasons for exclusion were non-inferiority trials (n=40) or secondary analyses (n=34), and infographics being identical to a table or figure in the full-text article (n=26). Following exclusions, 119 studies from 40 journals were included (figure 1).
Study flow diagram. RCT, randomised controlled trial; JCR, Journal Citation Reports.
Characteristics of included studies
Fields with the highest number of journals and studies included were Cardiac & Cardiovascular Systems (7 journals; 16 included studies), Medicine, General & Internal (6 journals; 24 included studies), Surgery (6 journals; 13 included studies) and Urology & Nephrology (5 journals; 20 included studies). The other 10 fields contributed fewer journals and studies, ranging from 1 to 4 and 1 to 13, respectively (table 1). Four studies from over half (n=22) of the journals found to have eligible infographics were included. Additionally, four journals included three studies, five journals included two studies and nine journals included one study. Three infographics did not contain a results section, 37 infographics did not contain a conclusions section and 2 full texts did not contain a conclusions section.
Of the 119 studies meeting the eligibility criteria, 34 (28.6%) had multiple primary outcomes and 26 (22%) had multiple interventions. 51 (43%) studies assessed outcomes at multiple time points, with 37 (73%) of these specifying a primary time point. Most included trials were prospectively registered (n=108, 91%), 42 (35%) were industry funded and 53 (44.5%) reported having conflicts of interest. The median journal impact factor was 7.3 (IQR: 4.9–20.6).
Spin in infographics, abstracts and full-texts articles
Spin was present in 39 (33%) infographics, 31 (26%) abstracts and 31 (26%) full texts (table 2). While any evidence of spin occurred more frequently in infographics, infographics were not significantly more likely to contain spin than abstracts (OR 1.4; 95% CI 0.8 to 2.4) or full texts (OR 1.8; 95% CI 0.8 to 2.4). Additionally, abstracts were not more likely to contain spin than full texts (OR 1.0; 95% CI 0.6 to 1.8). The distribution of spin in infographics, abstracts and full texts only, combinations of these, and no spin in any section, is reported in online supplemental table 1.
Supplemental material
Spin in 119 infographics, abstracts and full-text articles*
Spin in results section
Infographics were significantly more likely to contain spin in the results than both abstracts (OR 2.1; 95% CI 1.1 to 4.1) and full texts (OR 4.3; 95% CI 1.9 to 9.5). Abstracts were not more likely to contain spin in the results than full texts (OR 2.0; 95% CI 0.9 to 4.8) (table 2).
The most common strategies used to spin results sections of infographics and abstracts were highlighting the benefit of an intervention despite statistically non-significant effects (infographics: n=12 (10%); abstracts: n=6 (5%)) and focusing on statistically significant effects for secondary outcomes (infographics: n=10 (9%); abstracts: n=6 (5%)). Similarly, the strategy most used to spin results of full texts was highlighting the benefit of an intervention despite statistically non-significant effects (n=5, 4%). Infographics were significantly more likely than full texts to spin results by focusing on statistically significant effects for secondary outcomes (OR 5.5; 95% CI 1.2 to 25.8). No other comparisons were statistically significant (online supplemental table 3).
Supplemental material
Spin in conclusions section
Infographics were not significantly more likely to contain spin in the conclusion than abstracts (OR 0.8; 95% CI 0.4 to 1.6) or full texts (OR 0.7; 95% CI 0.3 to 1.4) and abstracts were not more likely to contain spin in the conclusion than full texts (OR 0.9; 95% CI 0.5 to 1.6) (table 2). The most common strategy used to spin the conclusions sections of infographics, abstracts and full texts was highlighting the benefit of an intervention despite statistically non-significant effects (infographics: n=6 (7%); abstracts: n=13 (11%); full texts: n=13 (11%)). Specific strategies to spin conclusions sections were not more likely to occur in infographics compared with abstracts or full texts, or abstracts compared with full texts (online supplemental table 2).
Supplemental material
Factors associated with spin
Higher journal impact factor was associated with slightly lower odds of spin in infographics (OR 0.96 for a 1 unit increase in impact factor; 95% CI 0.92 to 0.99) and full texts (OR 0.94; 95% CI 0.89 to 0.99), but not abstracts (OR 0.96; 95% CI 0.92 to 1.01). Only infographics were less likely to contain spin (OR 0.2; 95% CI 0.1 to 0.9) if the trial was prospectively registered. No other significant associations were found (table 3).
Factors associated with any spin in an infographic, abstract or full text
Discussion
Summary of main findings
One-third of infographics summarising negative RCTs in journals in the top quintile of health and medical journals contain spin. Although the prevalence of spin was higher in infographics as compared with the corresponding abstract and full text, this difference was not statistically significant. Overall, the prevalence of spin in infographics, abstracts and full texts is problematically high, with greater than a quarter of each containing some form of spin.
The most common strategies used to spin results and conclusions were highlighting the benefit of interventions despite statistically non-significant effects and focusing on statistically significant effects for secondary outcomes. Infographics were 2 and 4 times more likely to contain spin in the results section than both abstracts and full texts, respectively. Preregistered trials and trials published in high-impact factor journals were less likely to contain spin in their infographics. To help readers avoid spin when designing an infographic, table 4 describes the different types of spin we investigated, with an explanation of each type and brief advice on how to avoid each type of spin.
Explanation of different types of spin and tips on how to avoid them
Interpretation
Results from this study support previous work demonstrating that most infographics do not report sufficient information for readers to appropriately interpret findings.1 While infographics can increase the attention research receives,2–5 12 many people—especially those not involved in research/academia—use infographics as a substitute for reading full-text articles.6 As such, the use of infographics in health and medical research may attract more readers, but at the cost of presenting insufficient information to appropriately interpret study findings. This issue is particularly worrisome in healthcare as an inappropriate interpretation of study findings could affect healthcare provider and patient decision-making.
Spin was not unique to infographics in the present study and was highly prevalent in abstracts and full texts as well. These findings support the notion that although initiatives like public research protocols, prespecified endpoints, and peer review exist to improve integrity and reporting of studies, researchers may possess considerable flexibility in the manner in which they present outcomes.8 Consequently, the data do not always ‘speak for themselves’ and may be distorted in scientific publications.7 8 13 While RCTs reduce sources of bias and are considered the ‘gold standard’ in effectiveness research,14 considerable spin still exists in the reporting of these studies, particularly when outcomes are not statistically significant.7 15
A previous study by Boutron et al assessed the prevalence of spin in abstracts and full texts of RCTs with non-significant findings for primary outcomes.7 It found substantially higher prevalences of spin compared with the present study in abstracts (results: 37.5% vs 14% in this analysis; conclusion: 58.0% vs 24% in this analysis) and full texts (results: 29.2% vs 8% in this analysis; conclusion: 50.0% vs 26% in this analysis).7 There are several explanations for these differences. The present study investigated more RCTs (119 RCTs from 40 journals vs 72) compared with the study by Boutron et al and likely has more representative results. In addition, our assessment only included studies published in journals in the top quintile of medicine and health, whereas journal rank was not part of Boutron et al’s inclusion criteria. Therefore, the present study may have included higher-quality articles that were less likely to spin findings. A search containing less-cited journals may have resulted in higher prevalences of spin, as demonstrated by the association between journal impact factor and the likelihood of spin in the infographic, abstract and full text that was found.
The high prevalence of spin in infographics published alongside articles in top health and medical journals underscores the need to improve the reporting of research in infographics, similar to ongoing efforts to improve the reporting of abstracts and full texts.11 16 17 The Reporting of Infographics and Visual Abstracts of Comparative studies (RIVA-C) checklist and guide was recently developed following a review of how infographics report research and a two-stage, modified Delphi survey of 92 infographic developers/designers, researchers, health professionals and other key stakeholders.10 The checklist and guide include 10 items to facilitate the creation of clear, transparent and sufficiently detailed infographics which summarise comparative studies of health and medical interventions. Appropriate dissemination and uptake of this checklist and guide have the potential to improve the completeness with which research findings are presented in infographics and subsequently reduce spin.
Limitations
While this assessment of spin involved the identification of specific strategies used to spin findings, the assessment of spin required the interpretation of data, which is inevitably subjective.18 Measures were taken to reduce subjectivity by having two researchers independently extract data using a standardised data extraction form, compare responses and resolve disagreements by discussion or consultation with one-third researcher. Trial findings were dichotomised as positive versus negative for inclusion based on p values. However, there are known criticisms of the interpretation of trial findings based solely on these thresholds.19 Inclusion of eligible studies was limited to four per journal to avoid biasing results towards journals publishing more infographics. As such, the total number of trials meeting our study’s inclusion criteria is unknown.
Conclusion
One-third of infographics published in top health and medical journals summarising negative RCTs contain spin. Although the prevalence of spin was higher in infographics as compared with the corresponding abstract and full text, this difference was not statistically significant. A higher journal impact factor was associated with slightly decreased odds of spin in infographics and full texts, but not in abstracts. Infographics, but not abstracts or full texts, were less likely to contain spin if the trial was prospectively registered. Conflicts of interest and industry sponsorship were not associated with spin in infographics, abstracts or full texts. Given the increasing popularity of infographics to disseminate research findings, there is an urgent need to improve the reporting of research in infographics.
Data availability statement
Data are available on reasonable request. Original data may be made available by the authors on reasonable request.
Ethics statements
Patient consent for publication
Footnotes
X @giovanni_ef, @zadro_josh
Contributors JZ and GF conceptualised the study. JZ, RM, GB, ARG, JS and JK extracted data. JZ analysed the data. JZ and RM had substantial contributions to interpretation of the data. RM and JZ drafted the manuscript. GB, GF, ARG, JS and JK critically revised the manuscript for important intellectual content. All authors approved the final version of the manuscript for submission. All authors had access to the data in the study and took responsibility for the integrity of the data and accuracy of reporting and data analysis. RM is the guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. RM is supported by a Health Services Research and Development postdoctoral fellowship (Connecticut Veterans Health Administration, United States). GF is supported by an NHMRC fellowship (APP2009808). JZ is supported by an NHMRC fellowship (APP1194105).
Disclaimer Findings will be disseminated at scientific conferences and through media.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.