Original article
A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis

https://doi.org/10.1016/S0895-4356(01)00414-0Get rights and content

Abstract

Objectives: To compare meta-analysis of summary study level data with the equivalent individual patient data (IPD) analysis when interest lies in identification of binary patient characteristics related to treatment efficacy. Design: A simulation study comparing meta-regression with IPD analyses of randomized controlled trials. Methods: Twenty-seven different meta-analysis situations were simulated with 1000 repetitions in each case. The following parameters were varied: (1) the treatment effect magnitude for different patient risk groups; (2) sample sizes of individual studies; and (3) number of studies. The meta-regression and IPD results were then compared for each situation. Results: The statistical power of meta-regression was dramatically and consistently lower than that of IPD analysis, with little agreement between the parameter estimates obtained from the two methods. Only in meta-analyses of large numbers of large trials, did meta-regression detect differential treatment effects between risk groups with any consistency. Conclusions: Meta-analysis of summary data may be adequate when estimating a single pooled treatment effect or investigating study level characteristics. However, when interest lies in investigating whether patient characteristics are related to treatment, IPD analysis will generally be necessary to discover any such relationships. In these situations practitioners should try to obtain individual-level data.

Introduction

The use of meta-analysis in assessing the potential benefits of health care interventions has greatly increased over recent years. The main benefit of meta-analysis is that a pooled estimate of the intervention effect is more precise than that obtained from any individual study. Other benefits include the study of heterogeneity, risk quantification and an explicit framework for the synthesis process [1]. When performing a meta-analysis it may become apparent that there is more variation (heterogeneity) between the results of the studies than one might expect by chance. When such heterogeneity is observed it is important to explore why it exists [2].

Heterogeneity between study results may occur if there are differences in the characteristics of the patients included in each of the studies and these characteristics are related to treatment efficacy. For example, there may be differences in the inclusion/exclusion criteria between studies (e.g., patient age, severity of disease, levels of risk). The identification of patient characteristics as causes of heterogeneity may help to identify types of patients who would benefit more or less from an intervention, which ultimately may allow better tailoring of interventions to patients 3, 4.

Alternatively, study design characteristics, such as differences in the method of administration of the intervention, length of follow-up, or design quality factors, between studies, could lead to heterogeneity being observed in a meta-analysis. Additionally, variations in effects across geographical areas may lead to heterogeneity being observed [5].

In an attempt to explain such heterogeneity, meta-regression is often used. This involves including study-level covariates in a regression-type analysis to try to explain variation in the outcome, such as a (loge) odds ratio, in terms of study or aggregated patient characteristics. If study design characteristics (e.g., geographical area of study or year of publication) are being investigated this procedure is straightforward and covariates are included in a standard way. The exploration of patient-level characteristics through meta-regression is more problematic because only aggregated summary values of the factor of interest may be obtained from each study, for example the percentage of patients with a certain characteristic. The relationship between treatment effect and the characteristic can be explored using meta-regression [6], including a variable indicating the proportion of the patients in each study with the characteristics of interest. The effect of age (or other variables that may be considered continuous) may be investigated in a meta-regression by including a variable for the mean age of the patients in each trial. Several recent analyses have used meta-regression techniques on aggregate patient-level data 7, 8, 9.

Use of such patient aggregated covariates may lead to aggregation bias, or the ecological fallacy 10, 11: associations between aggregated values may not be representative of the true relationships in the data at the individual level.

An alternative approach to meta-analysis, which avoids this problem, is to obtain and combine individual patient data (IPD) for each of the trials 12, 13. IPD has greater statistical power than meta-regression, but the extent of this is not fully known. Although, IPD is acknowledged to be the gold standard methodology for carrying out a meta-analysis, it is usually relatively costly and time consuming. Currently there is no consensus regarding if and when such an analysis is worth the extra cost and effort, because a variety of differences in results between summary and IPD analyses have been noted 14, 15, 16.

Here we compare the results of meta-regression analyses, including aggregated covariate data with IPD regression analysis, through simulation methods. This allows us to investigate and quantify the problems of using aggregated patient level covariates in a meta-analysis when interest lies in identifying patient characteristics that are related to treatment efficacy (effect modification).

Section snippets

Methods

Results from a fixed number of randomized controlled trials comparing a standard treatment with a new treatment were simulated. The outcome was defined as a dichotomous variable, indicating the occurrence or not of the event of interest. In all trials, the treatment effect varied, depending on whether an individual patient was in a high- or low-risk group (i.e., defined in terms of an effect modifier). In addition, the proportion of high-risk patients varied from study to study. For simplicity,

Results

For illustration, scatter plots of the loge odds ratio versus the proportion of high-risk patients for the first eight simulations of the meta-analyses incorporating 10 studies of size 500 with a moderate effect size in the high-risk group are presented in Fig. 2. In addition to plotting the individual study estimates, the fitted meta-regression line and the estimates of the loge odds ratios for low- and high-risk patients obtained from the IPD regression analysis have been added to the plot.

Discussion

It is generally acknowledged that IPD meta-analyses are the gold standard [13]; however they are in general lengthy and costly to perform. It is currently unclear when the added value of an IPD analysis over a summary measure analysis generally justifies the extra time and cost. In this article we have investigated the situation where there are differential treatment effects related to a measured patient characteristic and compared the results of using only aggregated study data and complete

References (24)

  • J. Lau et al.

    Summing up evidenceone answer is not always enough

    Lancet

    (1998)
  • L.A. Stewart et al.

    Meta-analysis of the literature or of individual patient datais there a difference?

    Lancet

    (1993)
  • A.J. Sutton et al.

    Methods for meta-analysis in medical research

    (2000)
  • S.G. Thompson

    Why sources of heterogeneity in meta-analysis should be investigated

    BMJ

    (1994)
  • J.P.A. Ioannids et al.

    Uncontrolled pearls, controlled evidence, meta-analysis and the individual patient

    J Clin Epidemiol

    (1998)
  • P.P. Glasziou et al.

    An evidence based approach to individualizing treatment

    BMJ

    (1995)
  • G.A. Colditz et al.

    The efficacy of bacillus Calmette-Guerin vaccination of newborns and infants in the prevention of tuberculosismeta-analyses of the published literature

    Pediatrics

    (1995)
  • S.G. Thompson et al.

    Explaining heterogeneity in meta-analysisa comparison of methods

    Stats Med

    (1999)
  • G. Ioannis et al.

    Effect of angiotensin-converting enzyme inhibitors on the progression of nondiabetic renal diseasea meta-analysis of randomised trials

    Ann Intern Med

    (1997)
  • M. Barza et al.

    Single or multiple daily doses of aminoglycosidesa meta-analysis

    BMJ

    (1996)
  • C. Cammà et al.

    Preoperative radiotherapy for resectable rectal cancera meta-analysis

    JAMA

    (2000)
  • S. Greenland

    Quantitative methods in the review of epidemiological literature

    Epidemiol Rev

    (1987)
  • Cited by (324)

    • Long-term outcomes of cognitive behavioural therapy for social anxiety disorder: A meta-analysis of randomised controlled trials

      2022, Journal of Anxiety Disorders
      Citation Excerpt :

      There was little indication that treatment-specific moderating variables influenced treatment efficacy. The usage of aggregate data, which does not adequately represent relationships on the individual level (Lambert et al., 2002), may contribute to this lack of effect. For instance, there may be factors associated with differential treatment response (e.g., participant characteristics) that potentiate intervention-covariate interactions.

    • Collaborative hand surgery clinical research without sharing individual patient data; proof of principle study

      2022, Journal of Plastic, Reconstructive and Aesthetic Surgery
      Citation Excerpt :

      For example, identifying prognostic factors is, to our knowledge, not possible using meta-analysis of summary statistics. While meta-regression has been proposed for this purpose, a simulation study by Lambert et al. demonstrated that meta-regression outcomes do not correspond to results from analyzing patient-level data.28 For answering this type of research question, combining patient-level data is still required.

    View all citing articles on Scopus
    View full text