Meta-Analysis and Causal Inference: A Case Study of Benzene and Non-Hodgkin Lymphoma

https://doi.org/10.1016/j.annepidem.2010.02.001Get rights and content

Meta-analysis is an important method in the practice of occupational epidemiology, with a legitimate, but limited role to play in causal inference. Meta-analysis provides an assessment of consistency—one of several classic causal criteria—through tests of heterogeneity and an assessment of differences across studies. It can also provide an increase in the precision of effect estimates, including the precision of dose response relationships. Causal inference, however, involves much more: a complete assessment of the classic causal criteria, for example. Causal claims, therefore, should not emerge from meta-analyses as such. A recent meta-analysis of epidemiological studies of benzene exposure and non-Hodgkin lymphoma (NHL), however, does exactly that. Using studies from a previous narrative review in which the authors made no causal claim, the same authors performed a meta-analysis and concluded that it represented new evidence that benzene causes NHL. Despite a lack of consistency (i.e., significant heterogeneity), weak associations, no evidence of dose-response, no effort to provide an assessment of biological plausibility, and no new epidemiological evidence, the authors, nevertheless, changed their conclusion from association to causation. By using case study as an illustrative platform, this commentary provides cautionary and critical comments about the use of meta-analysis and causal inference in occupational epidemiology.

Introduction

In early 2007, a comprehensive review of the epidemiologic evidence on benzene and non-Hodgkin lymphoma (NHL) appeared (1). The investigators concluded that “the evidence supports an association between occupational benzene exposure and NHL” (1, p. 385). One year later, the same investigators, plus one, published a meta-analysis of benzene and NHL using the same literature search strategy and concluded that their study provided new evidence that benzene causes NHL (2).

In light of the rapid rise of NHL incidence over two decades and the difficulties the scientific community has faced in attempting to explain this epidemic, it seems reasonable to carefully examine how these investigators went from association to causation 3, 4. More important is the question of whether such an inferential shift is scientifically justified. As Austin Bradford Hill emphasized in his now classic article on causal inference, a key scientific concern in occupational epidemiology is distinguishing between association and causation (5). Associations are numerical (statistical) features of data (6). Causation, on the other hand, is a feature of the material world; causation connects exposures and diseases in ways amenable to preventive interventions. Distinguishing between association and causation, therefore, is a fundamental scientific concern with important implications for public health.

At least three possibilities could explain the shift from association to causation in the 2008 meta-analysis. First, new epidemiologic studies may have been published after the 2007 review. But, according to the authors of the 2008 meta-analysis, only one case-control study appeared in 2007 and it reported null results (7). Thus, new studies of benzene and NHL cannot explain or justify the causal claim found in the meta-analysis. A single null study simply cannot tip the scales in that way.

A second possibility is that the addition of a new member of the investigative team—Dr. Steinmaus—could have been responsible for the shift from association to causation. But this possibility is unlikely, unless we accept the fact that in this particular situation personal subjective opinion and individual judgment mattered more to inference-making than methodology (8).

The third, and most likely, possibility to explain the shift to causation is methodological, specifically, the use of meta-analysis. The major difference between the 2007 publication—by Smith et al. (1)—and the 2008 publication—by Steinmaus et al. (2)—is the use of meta-analysis. Simply put, the investigators went from association to causation on the basis of a meta-analysis applied to essentially the same epidemiological evidence.

But can meta-analysis by itself generate a causal claim? That is the central methodological question here. And specifically, does the meta-analysis of benzene and NHL by Steinmaus et al. (2) provide sufficient rationale to move the scientific consensus on this controversial topic from association to causation?

In order to answer these questions, some historical and methodological background will be presented. Conclusions from other recent reviews and meta-analyses on the same topic will be reviewed along with a systematic review of the extent to which causal claims emerge from meta-analysis in the broader practice of occupational epidemiology, as well as what the methodological literature says about the role of meta-analysis in causal inference.

My purpose in this study is not to perform a full causal analysis of the relationship between NHL and benzene, reanalyze the studies, or examine all the many dimensions of the practice of meta-analysis. Rather, the purpose of this article is to point out several cautionary and critical concerns about meta-analysis and causal inference in occupational epidemiology, using the report of Steinmaus et al. (2) as a case study.

Section snippets

Prior Reviews and Meta-Analyses on Benzene and NHL

The conclusions found in the review by Smith et al. (1) and in the meta-analysis by Steinmaus et al. (2) do not exist in a scientific vacuum. At least 16 other reviews and two additional meta-analyses were published in the past 10 years, from 1999 through 2008 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, reflecting the intense interest in the benzene and NHL relationship as well as in the rapid rise in NHL incidence that remains largely unexplained.

The conclusions of

Can the meta-analysis on benzene and NHL by steinmaus et al. Justify a causal claim?

Although meta-analysis has a role to play in causal inference, one should not use meta-analysis as the primary method for a causal evaluation; that is not its purpose. Even when a meta-analysis is of high quality, it provides at best an assessment of the consistency of results across studies and increased precision regarding the summary effect measures, as discussed in the previous paragraphs. If causal inference were the aim, then Steinmaus et al. (2) should have at least discussed in detail

Discussion

Meta-analysis now holds a legitimate and increasingly popular place in the epidemiologist's toolbox of methods. Its methodological robustness does not extend very far, however, into the practice of causal inference. There, meta-analysis has a limited role to play. At its core, meta-analysis provides a way to average a group of numbers, numbers that happen to be the results of individual scientific studies, but numbers nevertheless. Meta-analysis also provides an assessment of the consistency of

Conclusion

Examining the role of meta-analysis in making causal claims is important because meta-analysis has emerged as a legitimate and widely used methodology in occupational epidemiology. Using meta-analysis alone, however, to generate causal claims is not appropriate. As illustrated herein, this critical analysis revealed that meta-analysis was used to make a claim about benzene and NHL from the same scientific evidence that many others determined falls short of causality. That consensus remains

References (108)

  • M. Meyer-Baron et al.

    Occupational aluminum exposure: evidence in support of its neurobehavioral impact

    Neurotoxicology

    (2007)
  • F. Tuttelmann et al.

    Gene polymorphisms and male infertility—a meta-analysis and literature review

    Reprod Biomed Online

    (2007)
  • F. Amsler et al.

    Compression therapy for occupational leg syndromes and chronic venous disorders—a meta-analysis of randomised controlled trials

    Eur J Vasc Endovasc Surg

    (2008)
  • M.M. Lehtola et al.

    The effectiveness of interventions for preventing injuries in the construction industry: a systematic review

    Am J Prev Med

    (2008)
  • M. Meyer-Baron et al.

    The impact of solvent mixtures on neurobehavioral performance: conclusions from epidemiological data

    Neurotoxicology

    (2008)
  • G. Van Maele-Fabry et al.

    Risk of leukaemia among pesticide manufacturing workers: a review and meta-analysis of cohort studies

    Environ Res

    (2008)
  • M.T. Smith et al.

    Benzene Exposure and Risk of Non-Hodgkin Lymphoma

    Cancer Epidemiol Biomarkers Prev

    (2007)
  • C. Steinmaus et al.

    Meta-analysis of benzene exposure and non-Hodgkin lymphoma: biases could mask an important association

    Occup Environ Med

    (2008)
  • C.A. Clarke et al.

    Changing incidence of Non-Hodgkin Lymphoma in the United States

    Cancer

    (2002)
  • K. Garber

    Lymphoma Rate Rise Continues to Baffle Researchers

    J Natl Cancer Inst

    (2001)
  • A.B. Hill

    The environment and disease: association or causation?

    Proc R Soc Med

    (1965)
  • J.M. Last

    Dictionary of epidemiology

    (2001)
  • A. Seidler et al.

    Solvent exposure and malignant lymphoma: a population-based case-control study in Germany

    J Occup Med Toxicol

    (2007)
  • D.L. Weed

    The nature and necessity of scientific judgment

    J Law Policy

    (2007)
  • S.R. O'Connor et al.

    Benzene and non-Hodgkin's lymphoma

    J Pathol

    (1999)
  • O. Wong et al.

    Non-Hodgkin's lymphoma and exposure to benzene in a multinational cohort of more than 308,000 petroleum workers, 1937-1996

    J Occup Environ Med

    (2000)
  • D. Baris et al.

    Epidemiology of lymphomas

    Curr Opin Oncol

    (2000)
  • R.B. Hayes et al.

    Benzene and lymphohematopoietic malignancies in humans

    Am J Ind Med

    (2001)
  • A.J. Swerdlow

    Epidemiology of Hodgkin's disease and non-Hodgkin's lymphoma

    Eur J Nucl Med Mol Imaging

    (2003)
  • M.T. Smith et al.

    Causal models of leukemia and lymphoma

    IARC Sci Publ

    (2004)
  • D. Pyatt

    Benzene and hematopoietic malignancies

    Clin Occup Environ Med

    (2004)
  • S.G. Fisher et al.

    The epidemiology of non-Hodgkin's lymphoma

    Oncogene

    (2004)
  • A.M. Muller et al.

    Epidemiology of non-Hodgkin's lymphoma (NHL): trends, geographic distribution, and etiology

    Ann Hematol

    (2005)
  • M.A. Mehlman

    Causal relationship between non-Hodgkin's lymphoma and exposure to benzene and benzene-containing solvents

    Ann NY Acad Sci

    (2006)
  • P. Hartge et al.

    Non-Hodgkin lymphoma

  • K. Ekstrom-Smedby

    Epidemiology and etiology of non-Hodgkin lymphoma–a review

    Acta Oncol

    (2006)
  • D.D. Alexander et al.

    The non-Hodgkin lymphomas: a review of the epidemiologic literature

    Int J Epidemiol

    (2006)
  • S. Greenland

    Meta-analysis

  • D.L. Weed

    Interpreting epidemiologic evidence: how meta-analysis and causal inference methods are related

    Int J Epidemiol

    (2000)
  • B. Armstrong et al.

    Lung cancer risk after exposure to polycyclic aromatic hydrocarbons: a review and meta-analysis

    Environ Health Perspect

    (2004)
  • M. Goodman et al.

    Mesothelioma and lung cancer among motor vehicle mechanics: a meta-analysis

    Ann Occup Hyg

    (2004)
  • M.W. Heymans et al.

    Back schools for non-specific low-back pain

    Cochrane Database Syst Rev

    (2004)
  • G. Kwakkel et al.

    Effects of augmented exercise therapy time after stroke: a meta-analysis

    Stroke

    (2004)
  • G. Smedslund et al.

    The effectiveness of workplace smoking cessation programmes: a meta-analysis of recent studies

    Tob Control

    (2004)
  • G. Van Maele-Fabry et al.

    Prostate cancer among pesticide applicators: a meta-analysis

    Int Arch Occup Environ Health

    (2004)
  • M.F. Walker et al.

    Individual patient data meta-analysis of randomized controlled trials of community occupational therapy for stroke patients

    Stroke

    (2004)
  • G.C. Wendel-Vos et al.

    Int J Epidemiol

    (2004)
  • A. Buja et al.

    Cancer incidence among male military and civil pilots and flight attendants: an analysis on published data

    Toxicol Ind Health

    (2005)
  • E.B. Faragher et al.

    The relationship between job satisfaction and health: a meta-analysis

    Occup Environ Med

    (2005)
  • J.A. Hayden et al.

    Meta-analysis: exercise therapy for nonspecific low back pain

    Ann Intern Med

    (2005)
  • Cited by (0)

    Financial support for this research was provided by CONservation of Clean Air and Water in Europe (CONCAWE) with the explicit (contractual) agreement that the author had full and final control over the design, analysis, and interpretation of the findings and the contents of the manuscript.

    View full text