You are here

  • Home
  • Archive
  • Volume 28, Issue 2
  • Different meta-analysis methods can change judgements about imprecision of effect estimates: a meta-epidemiological study

Article Text

Article menu
Download PDFPDF
Original research
Different meta-analysis methods can change judgements about imprecision of effect estimates: a meta-epidemiological study
  1. Zhen Wang1,2,
  2. Muayad A Alzuabi1,2,
  3. Rebecca L Morgan3,4,
  4. Reem A Mustafa3,5,
  5. Yngve Falck-Ytter6,
  6. Philipp Dahm7,8,
  7. Shahnaz Sultan9,10,
  8. Mohammad Hassan Murad1,2
  1. 1 Mayo Clinic Evidence-Based Practice Center, Rochester, Minnesota, USA
  2. 2 Mayo Clinic Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Rochester, Minnesota, USA
  3. 3 Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
  4. 4 Department of Population and Quantitative Health Sciences, Case Western Reserve University, Cleveland, Ohio, USA
  5. 5 Departments of Internal Medicine and Population Health, University of Kansas Medical Center, Kansas City, Kansas, USA
  6. 6 Case Western Reserve University, Case Western Reserve University, Cleveland, Ohio, USA VA Northeast Ohio Healthcare System
  7. 7 Urology Section, Minneapolis VA Health Care System, Minneapolis, Minnesota, USA
  8. 8 Department of Urology, University of Minnesota Twin Cities, Minneapolis, Minnesota, USA
  9. 9 Minneapolis VA Health Care System, Minneapolis, Minnesota, USA
  10. 10 University of Minnesota Twin Cities, Minneapolis, Minnesota, USA
  1. Correspondence to Zhen Wang, Mayo Clinic Evidence-Based Practice Center, Rochester, Minnesota, USA; Wang.Zhen{at}mayo.edu

Abstract

Objectives To empirically evaluate five commonly used meta-analysis methods and their impact on imprecision judgements about effect estimates. The two fixed-effect model methods were the inverse variance method based on normal distribution and the Mantel-Haenszel method. The three random-effects model methods were the DerSimonian and Laird, the Hartung-Knapp-Sidik-Jonkman and the profile likelihood approaches.

Design Meta-epidemiological study.

Setting Meta-analyses published between 2007 and 2019 in the 10 general medical journals with the highest impact factors that evaluated a medication or device for chronic medical conditions and included at least 5 randomised trials.

Main outcome measures Discordance in the judgements of imprecision of effect estimates based on two definitions: when either boundary of 95% CI of the OR changed by more than 15% or changed in relation to the null.

Results We analysed 88 meta-analyses including 1114 trials with an average of 12.60 trials per meta-analysis and average I2 of 26% (range: 0%–96%). The profile likelihood failed to converge in three meta-analyses (3%). Discordance in imprecision judgements based on the two definitions, respectively, occurred between the fixed normal distribution and fixed Mantel-Haenszel method (8% and 2%), between the DerSimonian and Laird and Hartung-Knapp-Sidik-Jonkman methods (19% and 10%), between the DerSimonian and Laird and profile likelihood methods (9% and 5%), and between the Hartung-Knapp-Sidik-Jonkman and profile likelihood methods (5% and 13%). Discordance was greater when fewer studies and greater heterogeneity was present.

Conclusion Empirical evaluation of studies of chronic medical conditions showed that conclusions about the precision of the estimates of the efficacy of a drug or device frequently changed when different pooling methods were used, particularly when the number of studies within a meta-analysis was small and statistical heterogeneity was substantial. Sensitivity analyses using more than one method may need to be considered in these two scenarios.

  • methods

Data availability statement

Data are available on reasonable request. Available on request.

https://doi.org/10.1136/bmjebm-2022-112053

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Data availability statement

    Data are available on reasonable request. Available on request.

    View Full Text

    Footnotes

    • Twitter @EBMUrology

    • Contributors ZW and MHM initiated the study, analysed data and wrote the first manuscript draft. ZW, MAA, RLM, RAM, YF-Y, PD, SS and MHM interpreted the findings and critically revised the manuscript draft. ZW and MHM act as guarantors for the manuscript. All authors read, commented on and approved the final manuscript.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.