Objective: Meta-analysis yields a biased result if published studies represent a biased selection of the evidence. Copas proposed a selection model to assess the sensitivity of meta-analysis conclusions to possible selection bias. An alternative proposal is the trim-and-fill method. This article reports an empirical comparison of the two methods.
Study design and setting: We took 157 meta-analyses with binary outcomes, analyzed each one using both methods, then performed an automated comparison of the results. We compared the treatment estimates, standard errors, associated P-values, and number of missing studies estimated by both methods.
Results: Both methods give similar point estimates, but standard errors and P-values are systematically larger for the trim-and-fill method. Furthermore, P-values from the trim-and-fill method are typically larger than those from the usual random effects model when no selection bias is detected. By contrast, P-values from the Copas selection model and the usual random effects model are similar in this setting. The trim-and-fill method reports more missing studies than the Copas selection model, unless selection bias is detected when the position is reversed.
Conclusions: The assumption that the most extreme studies are missing leads to excessively conservative inference in practice for the trim-and-fill method. The Copas selection model appears to be the preferable approach.