Original articleFunnel plots for detecting bias in meta-analysis: Guidelines on choice of axis
Introduction
Funnel plots—scatter plots in which the treatment effects estimated from individual studies on the horizontal axis are plotted against a measure of study precision on the vertical axis—have been proposed as a means of detecting publication bias in meta-analysis [1]. In the absence of bias the graph resembles a symmetrical inverted funnel because the treatment effect estimates from smaller studies scatter more widely at the bottom of the graph, with the spread narrowing with increasing precision among larger studies. If there is publication bias because smaller studies which show no statistically significant effects remain unpublished 2, 3, then the funnel plot will appear asymmetrical 4, 5. Funnel plot asymmetry cannot, however, be interpreted as proof of publication bias in meta-analysis [6]. Asymmetry could also result from the overestimation of treatment effects in smaller studies of inadequate methodological quality [7]. Furthermore, heterogeneity of treatment effects will lead to funnel plot asymmetry if the true treatment effect is larger in the smaller trials 6, 8. For example, if a combined outcome is considered then substantial benefit may be seen only in patients at high risk for the component of the combined outcome which is affected by the intervention [9]. Trials conducted in high-risk patients will also tend to be smaller, because of the difficulty in recruiting such patients.
Funnel plots were first used in educational research and psychology [1], mainly for meta-analyses of continuous outcome variables in which standardized mean difference was plotted against sample size. In medical research vertical axes based on the standard error or variance of the treatment effect estimate have been increasingly used. A majority of trials in medicine have binary outcomes, and treatment effects are usually expressed as risk or odds ratios, although risk differences may also be used to measure treatment effects.
Meta-analysts thus face a wide array of choices for both vertical and horizontal axes in funnel plots. This leads to the danger that the funnel plot chosen for a particular meta-analysis may be that which best conveys the message desired by the investigator, or may not be appropriate for detecting bias [10]. The purpose of this article is to provide guidelines for the choice of axes in funnel plots of meta-analyses with binary outcomes.
Section snippets
Choice of vertical axis in funnel plots: case study
The randomized controlled trials of magnesium treatment in the prevention of death following myocardial infarction (Table 1) are a well known example where publication bias, demonstrated by an asymmetrical funnel plot 5, 6, has been suggested as an explanation for the discrepancy between meta-analyses which showed a clear beneficial effect of magnesium therapy on mortality 11, 12 and a subsequent large trial which showed no effect [13]. Fig. 1 shows funnel plots for these 16 trials, using six
Choice of horizontal axis in funnel plots: empirical study
The choice of treatment effect measure may affect the interpretation of randomized trials and meta-analyses [16]. In practice, most meta-analyses use ratio measures of treatment effect (odds ratio or relative risk), although risk differences are sometimes also used. We conducted an empirical study to investigate whether the prevalence of funnel plot asymmetry in published meta-analyses depends on the choice of treatment effect.
As described in detail elsewhere [8], we hand searched volumes
Discussion
The potential for bias in the location, selection or conduct of the component studies included in meta-analyses is increasingly recognized. Funnel plots are a useful graphic means of checking whether “small study effects”—a tendency for treatment effect estimates in small studies to differ from those in larger studies—may have distorted the results of a meta-analysis [19]. This could be due to publication bias, other reporting biases, low methodological quality of smaller studies or true
References (39)
- et al.
Publication bias in clinical research
Lancet
(1991) - et al.
Publication and related bias in meta-analysispower of statistical tests and prevalence in the literature
J Clin Epidemiol
(2000) - et al.
Misleading funnel plot for detection of bias in meta-analysis
J Clin Epidemiol
(2000) - et al.
Long term outcome after magnesium sulphate in suspected acute myocardial infarctionThe Second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2)
Lancet
(1994) - et al.
Meta-analysis of the literaure or of individual patient datais there a difference?
Lancet
(1993) - et al.
The fate of abstracts submitted to a cancer meetingfactors which influence presentation and subsequent publication
Ann Oncol
(1992) - et al.
Summing up. The science of reviewing research
(1984) - et al.
Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards
JAMA
(1992) - et al.
Publication biasa problem in interpreting medical data
J R Stat Soc A
(1988) - et al.
Misleading meta-analysis. Lessons from “an effective, safe, simple” intervention that wasn't
Br Med J
(1995)