Original ArticleNon-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study
Introduction
Publication bias occurs when “investigators, reviewers, and editors submit or accept manuscripts for publication based on the direction or strength of the study findings” [1]. Studies with particular characteristics (e.g., statistically positive results, large effect sizes) are more likely to be published than those without these characteristics [2]. The consequences of publication bias are severe; a recent meta-analysis has shown that publication bias exaggerated the estimate of antidepressant's effectiveness by 32%, on average [3].
Cross-sectional samples of the published literature have found that the proportion of statistically positive results in individual studies (e.g., randomized controlled trials [RCTs]; observational studies) ranges from 35% to 97%, depending on the sample used [4], [5], [6], [7], [8], [9], [10], [11], [12]. Only a few studies have examined the proportion of statistically positive results and conclusions in systematic reviews (SRs). In one study, 60% of the 193 included reviews had positive conclusion statements, whereas 13% had negative conclusions [13]. Only 17% assessed for publication bias and 27% included unpublished material [13]. Another study found that SRs published in traditional journals reported more beneficial results compared to those published in the Cochrane Database of Systematic Reviews (P = 0.007) [14].
These studies provide indirect evidence of publication bias because factors other than publication bias may have led to a higher proportion of studies with statistically positive results and positive conclusions [13]. For example, in individual studies it may indicate that the sample included studies with high power (e.g., included a large number of participants), which were able to show a true difference of the null hypothesis [15]. For SRs, it may indicate that the individual studies included had high power. Previous research examining indirect publication bias for SRs used a specific sample of SRs (e.g., SRs reported in four general medicine journals and four specific medicine journals [14]) and only examined a few “predictors” (i.e., SR characteristics that predict positive results and conclusions). Furthermore, only one independent investigator categorized the results and conclusions in these studies.
Indirect evidence of SR publication bias is particularly important, as decision makers increasingly rely on SRs [16], [17]. If evidence of SR publication bias exists, this may influence the utility of SRs. We aimed to examine a broad spectrum of “predictors” to determine which SR characteristics were associated with favorable results and positive conclusions and to determine the level of concordance between the results and conclusions in this sample of SRs using a large sample of all SRs published in MEDLINE in November 2004.
Section snippets
Sample of SRs
We identified a sample of SRs indexed in MEDLINE (November 2004) and published in English. The methods of ascertaining this sample have been described elsewhere [18]. Briefly, the SRs were identified through a modified version of Montori's strategy (see Appendix A on the journal's web site at www.elsevier.com) [19]. Two independent reviewers (A.C.T., J.T.) first screened the citations (i.e., titles and abstracts) from the literature search. Subsequently, one reviewer screened full-text articles
Sample acquisition
A total of 1,046 records were identified and screened. Of these, 758 full-text articles were obtained and 300 SRs met the eligibility criteria for the previous study [18]. Four SRs were excluded from the present study, as they were not health related, leaving 296 SRs in our sample (Fig. 2).
SR characteristics
A little less than half of the SRs (141/296, 47.6%) included a meta-analysis of the primary outcome, the majority of which assessed between-study inconsistency (134/141, 95.0%; Table 1). Cochrane reviews were
Discussion
Indirect publication bias occurs when one observes a high proportion of statistically positive results and/or conclusions in the published literature [13]. In our sample of SRs, approximately 1/3 had favorable results and positive conclusions, which increased to approximately 2/3 for SRs including a meta-analysis of the primary outcome. To evaluate whether indirect publication bias is present in our sample of SRs, comparison with other cross-sectional studies is likely beneficial.
In our study,
Acknowledgments
This research was funded by the Canadian Agency for Drugs and Technologies in Health. The funding agreement ensured the authors' independence in designing the study, interpreting the data, writing, and publishing the report. The research was conducted as part of a PhD dissertation in Population Health at the University of Ottawa.
We thank Jacqueline Tetroe and Dr. Mario Cappelli for their input on the original protocol of this study; Margaret Sampson for assistance with the literature searches;
References (29)
How important is publication bias? A synthesis of available data
AIDS Educ Prev
(1997)Publication bias: recognizing the problem, understanding its origins and scope, and preventing harm
- et al.
Selective publication of antidepressant trials and its influence on apparent efficacy
N Engl J Med
(2008) The importance of negative results in psychological research
Can Psychol
(1964)- et al.
Signifying significant significance
Am Psychol
(1972) - et al.
Publication bias against null results
Psychol Rep
(1997) Consequences of prejudice against the null hypothesis
Psychol Bull
(1975)Source of funding and outcome of clinical trials
J Gen Intern Med
(1986)- et al.
Positive-outcome bias: comparison of emergency medicine and general medicine literatures
Acad Emerg Med
(1994) - et al.
Statistical power, sample size, and their reporting in randomized controlled trials
JAMA
(1994)
Sample-size of randomized double-blind trials 1976-1991
Dan Med Bull
The file drawer problem of non-significant results—does it apply to biological research?
Oikos
Publication and related biases
Health Technol Assess
Cited by (0)
This research was funded by the Canadian Agency for Drugs and Technologies in Health. Ms. Tricco is funded by a Canadian Institutes of Health Research Canada Graduate Scholarship and a University of Ottawa National Excellence Scholarship. Dr. Brehaut is funded by an Ontario Ministry of Health Career Scientist Award. Dr Moher is funded by a University of Ottawa Research Chair.