Ratio measures in leading medical journals: structured review of accessibility of underlying absolute risks
BMJ 2006; 333 doi: https://doi.org/10.1136/bmj.38985.564317.7C (Published 14 December 2006) Cite this as: BMJ 2006;333:1248
- Lisa M Schwartz, associate professor of medicine1,
- Steven Woloshin, associate professor of medicine1,
- Evan L Dvorin, instructor of medicine2,
- H Gilbert Welch, professor of medicine1
- 1Veteran Affairs Outcomes Group, White River Junction, VT 05009, USA
- 2Dartmouth Medical School, Hanover, NH
- Correspondence to: S Woloshin steven.woloshin{at}dartmouth.edu
- Accepted 29 August 2006
Abstract
Objective To examine the accessibility of absolute risk in articles reporting ratio measures in leading medical journals.
Design Structured review of abstracts presenting ratio measures.
Setting Articles published between 1 June 2003 and 1 May 2004 in Annals of Internal Medicine, BMJ, Journal of the American Medical Association, Journal of the National Cancer Institute, Lancet, and New England Journal of Medicine.
Participants 222 articles based on study designs in which absolute risks were directly calculable (61 randomised trials, 161 cohort studies).
Main outcome measure Accessibility of the absolute risks underlying the first ratio measure in the abstract.
Results 68% of articles (150/222) failed to report the underlying absolute risks for the first ratio measure in the abstract (range 55−81% across the journals). Among these articles, about half did report the underlying absolute risks elsewhere in the article (text, table, or figure) but half did not report them anywhere. Absolute risks were more likely to be reported in the abstract for randomised trials compared with cohort studies (62% v 21%; relative risk 3.0, 95% confidence interval 2.1 to 4.2) and for studies reporting crude compared with adjusted ratio measures (62% v 21%; relative risk 3.0, 2.1 to 4.3).
Conclusion Absolute risks are often not easily accessible in articles reporting ratio measures and sometimes are missing altogether—this lack of accessibility can easily exaggerate readers' perceptions of benefit or harm.
Introduction
For good reasons, ratio measures, such as relative risks and odds ratios, have become a common way to compare outcomes in two groups. For observational studies they serve as the central metric of the strength of association between exposure and outcome, a key criterion for establishing causality in classic epidemiology. For both observational and experimental studies they serve as a convenient mechanism to express the magnitude of an effect on baseline risk—a relative change. Furthermore, the ratio is often believed to be transportable—that is, the relative change may be applied to different populations with different baseline risks. Finally, they have the appealing feature of summarising two numbers (the risk in one group and the risk in the other) into one, which in turn facilitates comparisons of the effect of various exposures. But this feature of ratio measures also represents their major weakness, that the underlying absolute risks are concealed.
Unless ratio measures are reported with the underlying absolute risks, readers cannot judge the clinical significance of the effect. Consider the following example. Readers may be told that the relative risk of death with drug A compared with placebo is 0.5; in other words, people who take drug A are half as likely to die as people who take placebo. But without the underlying absolute risks—the chance of death in each group—the information is incomplete.1 A relative risk of 0.5, for example, is compatible with a wide range of changes in the risk of death: from 20% to 10%, from 1% to 0.5%, and from 0.0004% to 0.0002%. Effects presented in relative terms alone have been repeatedly shown to seem more impressive than the same effects presented in absolute terms in studies of physicians,2 3 policy makers,4 and patients.5 6 Moreover, providing absolute risks has been shown to improve patients' comprehension of statements of both absolute risk reduction and relative risk reduction.7
The importance of providing absolute risks has been recognised in the requirements of groups trying to improve reporting in the medical literature. The consolidated standards of reporting trials (CONSORT), an international effort initiated in the mid-1990s to improve the reporting of randomised trials, explicitly calls on researchers to state the results using absolute numbers when feasible.8 A more recent initiative organised in 2003, strengthening the reporting of observational studies in epidemiology (STROBE), does as well.9 We systematically determined how frequently the absolute risks comprising ratio measures are reported in the medical literature.
Methods
Search strategy
We searched Medline using the search terms rate ratio*, relative ratio*, relative risk*, risk ratio*, or odds ratio* (the * ensures that any suffix such as “s” is included), and identified 320 articles with ratio measures in the abstract published between 1 June 2003 and 1 May 2004 in six leading medical journals: Annals of Internal Medicine, BMJ, Journal of the American Medical Association, Journal of the National Cancer Institute, Lancet, and New England Journal of Medicine. We excluded 98 articles with study designs in which absolute risks might not be directly calculable (52 case-control studies, 46 meta-analyses). Thus the final sample consisted of 222 articles with study designs where absolute risks were directly calculable: 61 randomised trials, 161 cohort studies.
Article review process
We reviewed each article using a standardised data extraction form (see fig A on bmj.com). The coder began by recording the study design, the number of ratio measures reported in the results of the abstract, and the value and name of the first ratio measure in the abstract.
To facilitate understanding of the ratio measure, the coder specified the exposure groups being compared, categorised the exposure groups as discrete (for example, drug versus placebo) or continuous (for example, years of age), and specified the outcome variable (for example, death rate).
The coder then searched for the absolute risks underlying the first ratio measure. Although some of these numbers were true rates—that is, events per person year of observation in the exposed and unexposed—most were risks—that is, proportion of the exposed and unexposed who had an event in a specified time period, whereas a few others were means (for example, mean time to falling asleep in the exposed and unexposed). For simplicity we use the phrase “absolute risks” to refer to the basic numbers comprising the numerator and denominator of the ratio.
We first determined if the absolute risks were in the abstract. For discrete exposures we simply looked for the two absolute risks underlying the ratio (for example, the absolute risk for those taking drug A and the absolute risk for those taking placebo). For continuous exposures (for example, relative risk of 1.05 for each year of age) we looked for an absolute risk grounding the outcome at any single exposure level (for example, risk of death for patients aged 65 years).
If the absolute risks were in the abstract, no further review was done (n=72). If the absolute risks were not in the abstract, we looked for them in the article, searching the entire text and all tables and figures (n=150). If the absolute risks were not reported anywhere in the article, the coder attempted to calculate them based on the data that were reported. These calculations entailed extracting data from one or more places in the article and ranged from simple division to a series of calculations (see fig B on bmj.com).
Each author coded a subset of articles. Thirty articles were double coded to establish inter-rater reliability. Agreement for all items included in this report was “substantial” or greater—that is, the kappa statistic ranged from 0.7 to1.0.10 Disagreements were resolved by consensus between the coders.
Analyses
We used the χ2 test to compare differences in proportions and t tests to compare means. All analyses were done using Stata version 9.0 and α was set at 0.05.
Results
Use of ratio measures
Of the original research articles with abstracts during the study period, 30% (320/1051) included at least one ratio measure in the abstract. After excluding articles with study designs in which absolute risks may not be directly calculable (46 meta-analyses, 52 case-control studies), 222 articles were eligible for the study. The box highlights the multiplicity of language used to name the ratio measures. About 30 terms were found, many synonymous.
Synonyms used to describe various ratio measures
Ratio of risks
-
Relative risk (or RR)*; post-trial relative risk, risk ratio; “times more likely”; and unadjusted risk ratio
Ratio of risks: adjusted
-
Adjusted relative risk*; multivariable adjusted relative risk; multivariable analysis relative risk; multivariable relative risk; multivariate relative risk; after adjustment risk ratio; and adjusted risk ratio
One minus the ratio of risks
-
Relative risk reduction*; reduction in relative risk*; risk reduction; proportional reduction*; and per cent reduction
Ratio of rates
-
Rate ratio*; incidence rate ratio; and incidence density ratio
Ratio of rates: adjusted
-
Adjusted rate ratio* and standardised incidence ratio
Ratio of odds
-
Odds ratio*; unadjusted odds ratio; and exposure odds ratio
Ratio of odds: adjusted
-
Adjusted odds ratio*; multivariate odds ratio; and multivariate adjusted odds ratio
*Most commonly used language
The table⇓ shows the characteristics of articles overall and by journal. On average, article abstracts included 3.3 distinct ratio measures, with a range of 1-14. Most of the articles identified were cohort studies (73% cohort studies v 27% randomised trials), and the abstracts of cohort studies included more ratio measures than the abstracts of randomised trials (mean number of ratio measures 3.6 v 2.5; P=0.002). Overall, 72% (159/222) of the ratio measures were reported as being adjusted (that is, based on a model controlling for a set of covariates) and 28% (63/222) were crude.
Description of articles with ratio measures in abstract. Values are numbers (percentages) unless stated otherwise
Accessibility of absolute risks for first ratio measure in abstract
Overall, 68% of articles (n=150) failed to report the underlying absolute risks for the first ratio measure in the abstract, ranging from 55-81% across the journals (table⇑ and fig 1⇓). Among articles failing to report the absolute risks in the abstract, about half reported them elsewhere in the article (text, table, or figure); but half did not report them anywhere. When absolute risks were not reported, the coders tried to calculate them from the data presented but were often unable to do so. As shown in figure 1⇓, the final breakdown of the 222 articles was: 32% (n=72) had underlying absolute risks that were easily accessible (in the abstract), 32% (n=72) had somewhat accessible absolute risks (reported elsewhere in the article), 22% (n=49) had absolute risks that were not easily accessible (not reported but calculable), and 13% (n=29) had inaccessible absolute risks (not reported and not calculable).
Fig 1 Proportion of articles where absolute risks for first ratio measure were not reported in the abstract
Figure 1⇑ also shows the effect of study design and adjustment. Randomised trials were more likely than cohort studies to report the absolute risks in the abstract (62% v 21%; relative risk 3.0, 95% confidence interval 2.1 to 4.2). Absolute risks were also more likely to be reported for crude as opposed to adjusted ratio measures (62% v 21%; relative risk 3.0, 2.1 to 4.3).
Finally, we compared the accessibility of absolute risks for discrete exposures (for example, taking or not taking a drug) and continuous exposures (for example, years of age). Nine per cent (21/222) of the ratio measures involved continuous exposures; all occurred in the subset of 136 articles that were cohort studies and presented adjusted ratio measures. When the analysis was restricted to this subset, absolute risks were more likely to be reported in the abstract for discrete rather than for continuous exposures (21% v 5%, relative risk 4.1, 0.6 to 28.9).
Discussion
Although ratio measures are commonly reported in the medical literature, the underlying absolute risks are not. In our review of ratio measures in six major medical journals, we found that the underlying absolute risks were often difficult to access or were missing altogether. The lack of accessibility of these fundamental data may well lead journal readers (doctors, policy makers, journalists, and patients) to have exaggerated perceptions of the reported effect sizes.
Some may worry that our suggestion—reporting absolute risks—solves one problem by creating its opposite—that is, that absolute risks may lead people to minimise important effects. For example, whereas most doctors would consider the benefit of a drug that reduced all-cause mortality among relatively healthy outpatients over five years from 4% to 3% to be important, many patients might dismiss the 1% point change as trivial. But the challenge here is calibration—to provide the context that this effect is big compared with most drugs in this setting. The solution is not to revert to the language of a “25% reduction,” because that fails to distinguish between a change from 4% to 3% and a change from 0.004% to 0.003%, but to calibrate people by exposing them to absolute risks (with and without the intervention) for a variety of medical therapies.
We also found that accessibility of absolute risks is more of a problem for cohort studies than for randomised trials (although it is still substantial in randomised trials, 38% failed to report absolute risks with the first ratio measure in the abstract, and in 12% of cases these numbers were not reported anywhere in the article). The more complete reporting for randomised trials may reflect the widespread acceptance of the CONSORT statement first issued over 10 years ago.8 With the advent of newer organisations such as STROBE,9 focusing on observational studies, the reporting of cohort studies may improve.
Our study has several limitations. Firstly, our analysis was limited to six major medical journals. We deliberately chose this limited sample of journals because of their reputation for quality. Thus we believe our data represent a best case scenario: we think it is unlikely that less prestigious journals with less statistical support and fewer full time editors would be doing a better job. Secondly, our search strategy did not include some study designs (for example, meta-analyses) and some ratio measures (for example, hazard ratios). This was intentional as we wanted to focus on designs where the absolute rates could be directly calculated, and on the most familiar ratio measures. Finally, we evaluated only the first ratio measure in the abstract (rather than all ratios in the abstract or article). This was also intentional to ensure consistency in coding. But there is no reason to believe that subsequent ratio measures are better reported than the first. In fact we argue that because it is first, and may receive extra prominence, the initial ratio measure should be communicated as clearly as possible.
Although authors can do better, ultimately we think it is the responsibility of journal editors to ensure that the absolute risks are easily accessible to readers. In fact journal editors have come to a consensus on this point. More than 190 medical journals in the United States and elsewhere (including our study journals)11 endorse the CONSORT statement for randomised trials, which calls for reporting absolute risks. Many have also endorsed the STROBE statement for cohort studies. Moreover, journal editors are best positioned to make sure that absolute risks are easily accessible. Just as they have (successfully) insisted that authors write a structured abstract, journal editors can insist that authors provide absolute risks adjacent to the corresponding ratio measure. If authors cannot do so, the paper should not be published.
Figure 2⇓ highlights our suggestions for ensuring that absolute risks underlying ratio measures are made easily accessible in journal abstracts. In many instances the process is straightforward. For example, when there are two groups and a crude ratio measure, editors can simply require that the risks in each group be included. Other settings are more challenging, in particular when the ratio measure is adjusted. Two possible methods are considered in figure 2⇓. Authors could provide the crude absolute risk in each group and the crude and adjusted relative risk to highlight the effect of adjustment. An alternative would be to explicitly provide both the crude and adjusted absolute risks, in addition to the crude and adjusted relative risks. Here editors would have to specify how authors should calculate adjusted absolute risks for each group (and state these rules in the instructions for authors). Three possible standards to use for adjustment are the study population as a whole (which is typically done to provide the risk for the average patient), the unexposed group, or the exposed group (having the advantage of holding one absolute risk constant to facilitate comparison).
Fig 2 Suggested approaches to improve communication of ratio measures
Editors will also need to provide guidance for how authors should report the underlying absolute risks for ratio measures on the basis of continuous exposures. We suggest that authors report the absolute risk for at least one level of exposure.
Figure 2⇑ also provides suggestions for several other issues identified in our review: the reporting of multiple ratio measures in a single abstract (four of the six journals studied had abstracts with over 10 distinct ratio measures); the multiplicity of language used to refer to ratio measures, which frequently does not provide unique information and may simply confuse readers (for example, proportional reduction versus percentage reduction versus risk reduction versus RR); and the confusion about whether a ratio measure without a modifier is crude or adjusted.
To improve the reporting of ratio measures in the medical literature we believe that journal editors need to ensure that absolute risks are routinely included in the abstract adjacent to the corresponding ratio measure. Without the absolute risks, ratio measures alone may leave readers with an exaggerated sense of what studies find. Exaggerated perceptions matter because they may lead to either unwarranted enthusiasm for new medical interventions or unwarranted concern about potentially harmful exposures.
What is already known on this topic
-
Ratio measures without the underlying absolute risks often exaggerate readers' perceptions of benefit or harm
What this study adds
-
In major medical journals, more often than not ratio measures are reported in the abstract of a medical research article without the underlying absolute risks
-
The inaccessibility of absolute risks is a bigger problem for cohort studies than for randomised trials, and for studies reporting adjusted compared with crude ratio measures
Footnotes
-
The data extraction form and calculations carried out by coders are on bmj.com
-
Contributors: LMS and SW analysed the data and wrote the first draft. They contributed equally to this project and the order of their names is arbitrary. HGW participated in the design of the analyses and made important contributions to the presentation of the work. ELD participated in design, data collection, and analyses. SW is the guarantor.
-
Funding: SW and LMS were supported by Veterans Affairs career development awards in health services research and development and Robert Wood Johnson generalist faculty scholar awards. This study was supported by a grant from the National Cancer Institute (R01CA104721) and from a research enhancement award from the Department of Veterans Affairs. The views expressed here do not necessarily represent the views of the Department of Veterans Affairs or the United States government.
-
Competing interests: None declared.
-
Ethical approval: Not required.