Meta-Analysis and Causal Inference: A Case Study of Benzene and Non-Hodgkin Lymphoma
Introduction
In early 2007, a comprehensive review of the epidemiologic evidence on benzene and non-Hodgkin lymphoma (NHL) appeared (1). The investigators concluded that “the evidence supports an association between occupational benzene exposure and NHL” (1, p. 385). One year later, the same investigators, plus one, published a meta-analysis of benzene and NHL using the same literature search strategy and concluded that their study provided new evidence that benzene causes NHL (2).
In light of the rapid rise of NHL incidence over two decades and the difficulties the scientific community has faced in attempting to explain this epidemic, it seems reasonable to carefully examine how these investigators went from association to causation 3, 4. More important is the question of whether such an inferential shift is scientifically justified. As Austin Bradford Hill emphasized in his now classic article on causal inference, a key scientific concern in occupational epidemiology is distinguishing between association and causation (5). Associations are numerical (statistical) features of data (6). Causation, on the other hand, is a feature of the material world; causation connects exposures and diseases in ways amenable to preventive interventions. Distinguishing between association and causation, therefore, is a fundamental scientific concern with important implications for public health.
At least three possibilities could explain the shift from association to causation in the 2008 meta-analysis. First, new epidemiologic studies may have been published after the 2007 review. But, according to the authors of the 2008 meta-analysis, only one case-control study appeared in 2007 and it reported null results (7). Thus, new studies of benzene and NHL cannot explain or justify the causal claim found in the meta-analysis. A single null study simply cannot tip the scales in that way.
A second possibility is that the addition of a new member of the investigative team—Dr. Steinmaus—could have been responsible for the shift from association to causation. But this possibility is unlikely, unless we accept the fact that in this particular situation personal subjective opinion and individual judgment mattered more to inference-making than methodology (8).
The third, and most likely, possibility to explain the shift to causation is methodological, specifically, the use of meta-analysis. The major difference between the 2007 publication—by Smith et al. (1)—and the 2008 publication—by Steinmaus et al. (2)—is the use of meta-analysis. Simply put, the investigators went from association to causation on the basis of a meta-analysis applied to essentially the same epidemiological evidence.
But can meta-analysis by itself generate a causal claim? That is the central methodological question here. And specifically, does the meta-analysis of benzene and NHL by Steinmaus et al. (2) provide sufficient rationale to move the scientific consensus on this controversial topic from association to causation?
In order to answer these questions, some historical and methodological background will be presented. Conclusions from other recent reviews and meta-analyses on the same topic will be reviewed along with a systematic review of the extent to which causal claims emerge from meta-analysis in the broader practice of occupational epidemiology, as well as what the methodological literature says about the role of meta-analysis in causal inference.
My purpose in this study is not to perform a full causal analysis of the relationship between NHL and benzene, reanalyze the studies, or examine all the many dimensions of the practice of meta-analysis. Rather, the purpose of this article is to point out several cautionary and critical concerns about meta-analysis and causal inference in occupational epidemiology, using the report of Steinmaus et al. (2) as a case study.
Section snippets
Prior Reviews and Meta-Analyses on Benzene and NHL
The conclusions found in the review by Smith et al. (1) and in the meta-analysis by Steinmaus et al. (2) do not exist in a scientific vacuum. At least 16 other reviews and two additional meta-analyses were published in the past 10 years, from 1999 through 2008 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, reflecting the intense interest in the benzene and NHL relationship as well as in the rapid rise in NHL incidence that remains largely unexplained.
The conclusions of
Can the meta-analysis on benzene and NHL by steinmaus et al. Justify a causal claim?
Although meta-analysis has a role to play in causal inference, one should not use meta-analysis as the primary method for a causal evaluation; that is not its purpose. Even when a meta-analysis is of high quality, it provides at best an assessment of the consistency of results across studies and increased precision regarding the summary effect measures, as discussed in the previous paragraphs. If causal inference were the aim, then Steinmaus et al. (2) should have at least discussed in detail
Discussion
Meta-analysis now holds a legitimate and increasingly popular place in the epidemiologist's toolbox of methods. Its methodological robustness does not extend very far, however, into the practice of causal inference. There, meta-analysis has a limited role to play. At its core, meta-analysis provides a way to average a group of numbers, numbers that happen to be the results of individual scientific studies, but numbers nevertheless. Meta-analysis also provides an assessment of the consistency of
Conclusion
Examining the role of meta-analysis in making causal claims is important because meta-analysis has emerged as a legitimate and widely used methodology in occupational epidemiology. Using meta-analysis alone, however, to generate causal claims is not appropriate. As illustrated herein, this critical analysis revealed that meta-analysis was used to make a claim about benzene and NHL from the same scientific evidence that many others determined falls short of causality. That consensus remains
References (108)
- et al.
An update of the epidemiology of non-Hodgkin's lymphoma
Clin Lymphoma
(2003) - et al.
Exposure to benzene and non-Hodgkin lymphoma: an epidemiologic overview and an ongoing case-control study in Shanghai
Chem Biol Interact
(2005) - et al.
Non-Hodgkin lymphoma and benzene exposure: A systematic literature review
Chem Biol Interact
(2005) - et al.
The epidemiology of non-Hodgkin lymphoma
Pathology
(2005) - et al.
Epidemiology of hematological malignancies
Ann Oncol
(2007) Epidemiologic evidence and causal inference
Hematol Oncol Clin North Am
(2000)- et al.
Impairments of colour vision induced by organic solvents: a meta-analysis study
Neurotoxicology
(2004) - et al.
Meta-analysis of risk factors for cutaneous melanoma: II. Sun exposure
Eur J Cancer
(2005) - et al.
Effect of smoking on hearing loss: quality assessment and meta-analysis
Prev Med
(2005) - et al.
Intermittent and chronic ultraviolet light exposure and uveal melanoma: a meta-analysis
Ophthalmology
(2005)
Occupational aluminum exposure: evidence in support of its neurobehavioral impact
Neurotoxicology
Gene polymorphisms and male infertility—a meta-analysis and literature review
Reprod Biomed Online
Compression therapy for occupational leg syndromes and chronic venous disorders—a meta-analysis of randomised controlled trials
Eur J Vasc Endovasc Surg
The effectiveness of interventions for preventing injuries in the construction industry: a systematic review
Am J Prev Med
The impact of solvent mixtures on neurobehavioral performance: conclusions from epidemiological data
Neurotoxicology
Risk of leukaemia among pesticide manufacturing workers: a review and meta-analysis of cohort studies
Environ Res
Benzene Exposure and Risk of Non-Hodgkin Lymphoma
Cancer Epidemiol Biomarkers Prev
Meta-analysis of benzene exposure and non-Hodgkin lymphoma: biases could mask an important association
Occup Environ Med
Changing incidence of Non-Hodgkin Lymphoma in the United States
Cancer
Lymphoma Rate Rise Continues to Baffle Researchers
J Natl Cancer Inst
The environment and disease: association or causation?
Proc R Soc Med
Dictionary of epidemiology
Solvent exposure and malignant lymphoma: a population-based case-control study in Germany
J Occup Med Toxicol
The nature and necessity of scientific judgment
J Law Policy
Benzene and non-Hodgkin's lymphoma
J Pathol
Non-Hodgkin's lymphoma and exposure to benzene in a multinational cohort of more than 308,000 petroleum workers, 1937-1996
J Occup Environ Med
Epidemiology of lymphomas
Curr Opin Oncol
Benzene and lymphohematopoietic malignancies in humans
Am J Ind Med
Epidemiology of Hodgkin's disease and non-Hodgkin's lymphoma
Eur J Nucl Med Mol Imaging
Causal models of leukemia and lymphoma
IARC Sci Publ
Benzene and hematopoietic malignancies
Clin Occup Environ Med
The epidemiology of non-Hodgkin's lymphoma
Oncogene
Epidemiology of non-Hodgkin's lymphoma (NHL): trends, geographic distribution, and etiology
Ann Hematol
Causal relationship between non-Hodgkin's lymphoma and exposure to benzene and benzene-containing solvents
Ann NY Acad Sci
Non-Hodgkin lymphoma
Epidemiology and etiology of non-Hodgkin lymphoma–a review
Acta Oncol
The non-Hodgkin lymphomas: a review of the epidemiologic literature
Int J Epidemiol
Meta-analysis
Interpreting epidemiologic evidence: how meta-analysis and causal inference methods are related
Int J Epidemiol
Lung cancer risk after exposure to polycyclic aromatic hydrocarbons: a review and meta-analysis
Environ Health Perspect
Mesothelioma and lung cancer among motor vehicle mechanics: a meta-analysis
Ann Occup Hyg
Back schools for non-specific low-back pain
Cochrane Database Syst Rev
Effects of augmented exercise therapy time after stroke: a meta-analysis
Stroke
The effectiveness of workplace smoking cessation programmes: a meta-analysis of recent studies
Tob Control
Prostate cancer among pesticide applicators: a meta-analysis
Int Arch Occup Environ Health
Individual patient data meta-analysis of randomized controlled trials of community occupational therapy for stroke patients
Stroke
Int J Epidemiol
Cancer incidence among male military and civil pilots and flight attendants: an analysis on published data
Toxicol Ind Health
The relationship between job satisfaction and health: a meta-analysis
Occup Environ Med
Meta-analysis: exercise therapy for nonspecific low back pain
Ann Intern Med
Cited by (0)
Financial support for this research was provided by CONservation of Clean Air and Water in Europe (CONCAWE) with the explicit (contractual) agreement that the author had full and final control over the design, analysis, and interpretation of the findings and the contents of the manuscript.