Elsevier

Environment International

Volumes 92–93, July–August 2016, Pages 597-604
Environment International

The relationship between study sponsorship, risks of bias, and research outcomes in atrazine exposure studies conducted in non-human animals: Systematic review and meta-analysis

https://doi.org/10.1016/j.envint.2015.10.011Get rights and content

Highlights

  • A critical component of systematic reviews is the assessment of risks of bias of included studies.

  • There is controversy about whether funding source should be included in risk of bias assessments.

  • This study examined the association of industry sponsorship with the methods, conclusions, or results of animal studies of atrazine.

  • Industry sponsored studies were more likely than studies with other funders to report results and conclusions that favored the sponsor although there were no differences in the risks of bias.

  • Research sponsorship should be included as a criterion in tools for assessing risks of bias.

Abstract

Background

A critical component of systematic review methodology is the assessment of the risks of bias of studies that are included in the review. There is controversy about whether funding source should be included in a risk of bias assessment of animal toxicology studies.

Objective

To determine whether industry research sponsorship is associated with methodological biases, the results, or conclusions of animal studies examining the effect of exposure to atrazine on reproductive or developmental outcomes.

Methods

We searched multiple electronic databases and the reference lists of relevant articles to identify original research studies examining the effect of any dose of atrazine exposure at any life stage on reproduction or development in non-human animals. We compared methodological risks of bias, the conclusions of the studies, the statistical significance of the findings, and the magnitude of effect estimates between industry sponsored and non-industry sponsored studies.

Results

Fifty-one studies met the inclusion criteria. There were no differences in methodological risks of bias in industry versus non-industry sponsored studies. 39 studies tested environmentally relevant concentrations of atrazine (11 industry sponsored, 24 non-industry sponsored, 4 with no funding disclosures). Non-industry sponsored studies (12/24, 50.0%) were more likely to conclude that atrazine was harmful compared to industry sponsored studies (2/11, 18.1%) (p value = 0.07). A higher proportion of non-industry sponsored studies reported statistically significant harmful effects (8/24, 33.3%) compared to industry-sponsored studies (1/11; 9.1%) (p value = 0.13). The association of industry sponsorship with decreased effect sizes for harm outcomes was inconclusive.

Conclusion

Our findings support the inclusion of research sponsorship as a risk of bias criterion in tools used to assess risks of bias in animal studies for systematic reviews. The reporting of other empirically based risk of bias criteria for animal studies, such as blinded outcome assessment, randomization, and all animals included in analyses, needs to improve to facilitate the assessment of studies for systematic reviews.

Introduction

Results from animal studies are a critical, and often the only, input to assessing potential harm from exposure to chemicals. However, the lack of reproducibility of findings from animal research has reduced public confidence in the utility of animal experiments (van der Worp et al., 2010) and led to claims that animal research is a waste of financial resources (Macleod et al., 2014). These problems with animal research have resulted in significant debate about how to assess biases in animal studies used in systematic reviews, risk assessments and other regulatory decisions (Woodruff et al., 2011, Rooney et al., 2014, National Academies of Science, 2014). A critical component of systematic review methodology is the assessment of the risks of bias of studies that are included in the review.

Risk of bias occurs when the methodological characteristics of a study produce a systematic error in the magnitude or direction of the results (Higgins and Green, 2011). Bias can shift effect estimates to be larger or smaller. For example, in controlled human clinical drug trials, studies with a high risk of bias (such as those lacking randomization, allocation concealment, or blinding of participants and outcome assessors) produce larger treatment effect sizes, thus falsely inflating the efficacy of the test interventions, compared to studies that have these design features (Schulz and Grimes, 2002a, Schulz and Grimes, 2002b). However, biased human studies assessing the harms of drugs are more likely to report smaller estimates of adverse effects (Nieto et al., 2007).

Less is known about methodological risks of bias in animal studies, although a systematic review of instruments for assessing risks of bias in animal studies identified criteria that have been shown empirically to bias effect estimates in animal models (Krauth et al., 2013). For example, analyses of animal studies examining interventions for stroke, multiple sclerosis and trauma have shown that lack of randomization, blinding, specification of inclusion and exclusion criteria, statistical power, and failure to use comorbid animals are associated with inflated effect estimates of pharmaceutical interventions (Bebarta et al., 2003, Crossley et al., 2008, Sena et al., 2010a).

Industry funding for research and industry relationships with academic researchers pose an additional risk of bias. Considerable evidence shows a strong association between industry funding, investigator financial conflicts of interest, and biased outcomes in clinical research, even when controlling for methodological characteristics of the studies (Lundh et al., 2012). There is little evidence regarding the influence of these conflicts of interest on the outcomes of animal research (Krauth et al., 2014, Bennett et al., 2010, Abdel-Sattar et al., 2014). There are conflicting results concerning the association of industry funding and research outcomes among the small cohorts of animal studies that have been examined and further research on the influence of conflicts of interest on animal studies is needed (Bennett et al., 2010, Abdel-Sattar et al., 2014). There is controversy about whether funding source should be included in risk of bias assessments for studies included in systematic reviews (Bero, 2013).

Atrazine (6-chloro-N-ethyl-N′-(1-methylethyl)-1,3,5-triazine-2,4-diamine) is used as an herbicide. Atrazine is commonly found in drinking water in the United States. The EPA has concluded that “atrazine is an endocrine disruptor” (Agency, 2007) but not that atrazine affects amphibian sexual development (Agency, 2010). As of 2013, the EPA has not altered these conclusions (Agency, 2013). Atrazine studies are a good topic for an analysis of funding bias because concerns have been raised about the influence of industry sponsorship on the design and results of studies examining the effects of atrazine on reproductive and developmental outcomes (Hayes, 2004).

The objective of this study is to determine whether industry research sponsorship is associated with the methods, conclusions, or results of animal studies examining the effect of exposure to atrazine on reproductive or developmental outcomes. We test three specific hypotheses. First, we hypothesize that industry sponsored studies will be less likely to have conclusions indicating harm from atrazine than non-industry sponsored studies. Second, we test the hypothesis that industry sponsored studies will be less likely to report statistically significant results indicating harm from atrazine than non-industry sponsored studies. Third, we test the hypothesis that industry sponsored studies will have smaller effect estimates of harm than non-industry sponsored studies. In addition, we compare the methodological risks of bias of industry sponsored vs. non-industry sponsored studies to determine if there are differences in the methods of the studies.

Section snippets

Methods

We searched for studies that addressed the following question: “Does exposure to atrazine have adverse reproductive or developmental effects in non-human animals”? We searched for studies that had non-human animal subjects that were exposed to any dose of atrazine during any life stage. Exposure levels of atrazine were classified and adverse outcomes were grouped as described below.

Results

Our searches identified 51 studies that met the inclusion criteria (Fig. 1). There were 12 industry funded studies, 29 non-industry and 10 with no disclosures. The studies were published between 1984 and 2013 with the majority (41 studies) published between 2005 and 2010. Amphibians, fish and reptiles were most commonly studied (n = 39), followed by rats (n = 8), birds/fowl (n = 3) and mice (n = 1).

Conclusions

We tested three specific hypotheses to determine whether industry sponsorship of research is associated with the results or conclusions of non-human animal studies examining the effect of atrazine exposure on reproductive or developmental outcomes. First, we tested the hypothesis that industry sponsored studies would be less likely to have conclusions indicating harm from atrazine than non-industry sponsored studies. This hypothesis was supported; 81% of industry supported studies did not

Funding source

National Institute of Environmental Health Sciences (Grant # R21ES021028) (http://www.niehs.nih.gov). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conflict of interest

The authors declare no competing conflicts of interest.

References (28)

  • R. DerSimonian et al.

    Meta-analysis in clinical trials

    Control. Clin. Trials

    (1986)
  • M.R. Macleod

    Biomedical research: increasing value, reducing waste

    Lancet

    (2014)
  • K.F. Schulz et al.

    Blinding in randomised trials: hiding who got what

    Lancet

    (2002)
  • K.F. Schulz et al.

    Allocation concealment in randomised trials: defending against deciphering

    Lancet

    (2002)
  • M. Abdel-Sattar

    The relationship between risk of bias criteria, research outcomes, and study sponsorship in a cohort of preclinical thiazolidinedione animal studies: a meta-analysis

    Evid Based Preclin Med

    (2014)
  • Agency, E.P., Atrazine chemical summary. 2007. p....
  • Agency, E.P. Atrazine updates: amphibians. 2010 July 23, 2014]; Available from:...
  • Agency, E.P. Atrazine updates: atrazine evaluation process. 2013 July 23, 2014]; Available from:...
  • V. Bebarta et al.

    Emergency medicine animal research: does use of randomization and blinding affect the results?

    Acad. Emerg. Med.

    (2003)
  • C.L. Bennett

    Association between pharmaceutical support and basic science research on erythropoiesis-stimulating agents

    Arch. Intern. Med.

    (2010)
  • L.A. Bero

    Why the Cochrane risk of bias tool should include funding source as a standard item

    Cochrane Database Syst. Rev.

    (2013)
  • N.A. Crossley

    Empirical evidence of bias in the design of experimental stroke studies: a metaepidemiologic approach

    Stroke

    (2008)
  • J.M. Drazen

    Toward more uniform conflict disclosures: the updated ICMJE conflict of interest reporting form

    Natl. Med. J. India

    (2010)
  • T. Hayes

    There is no denying this: Defusing the confusion about Atrazine

    Bioscience

    (2004)
  • Cited by (0)

    View full text