Article Text

Download PDFPDF
General medicine
The magnitude of small-study effects in the Cochrane Database of Systematic Reviews: an empirical study of nearly 30 000 meta-analyses
  1. Lifeng Lin1,
  2. Linyu Shi1,
  3. Haitao Chu2,
  4. Mohammad Hassan Murad3
  1. 1 Department of Statistics, Florida State University, Tallahassee, Florida, USA
  2. 2 Division of Biostatistics, University of Minnesota, Minneapolis, Minnesota, USA
  3. 3 Evidence-Based Practice Center, Mayo Clinic, Rochester, Minnesota, USA
  1. Correspondence to Dr Lifeng Lin, Department of Statistics, Florida State University, Tallahassee, FL 32306, USA; linl{at}


Publication bias, more generally termed as small-study effect, is a major threat to the validity of meta-analyses. Most meta-analysts rely on the p values from statistical tests to make a binary decision about the presence or absence of small-study effects. Measures are available to quantify small-study effects’ magnitude, but the current literature lacks clear rules to help evidence users in judging whether such effects are minimal or substantial. This article aims to provide rules of thumb for interpreting the measures. We use six measures to evaluate small-study effects in 29 932 meta-analyses from the Cochrane Database of Systematic Reviews. They include Egger’s regression intercept and the skewness under both the fixed-effect and random-effects settings, the proportion of suppressed studies, and the relative change of the estimated overall result due to small-study effects. The cut-offs for different extents of small-study effects are determined based on the quantiles in these distributions. We present the empirical distributions of the six measures and propose a rough guide to interpret the measures’ magnitude. The proposed rules of thumb may help evidence users grade the certainty in evidence as impacted by small-study effects.

  • epidemiology

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


  • LL and LS contributed equally.

  • Contributors LL, HC and MHM conceived and designed the analysis; LL and LS collected the data, performed the data analysis and drafted the manuscript; all authors critically revised the manuscript and approved the final version.

  • Funding This research was supported in part by NIH NLM R21 012197 (HC, LL), NLM R21 012744 (HC), and AHRQ R03 HS024743 (HC, LL).

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Patient consent for publication Not required.