PT  - JOURNAL ARTICLE
AU  - Ying, Xiangji
AU  - Robinson, Karen A
AU  - Ehrhardt, Stephan
TI  - Re-evaluating the role of pilot trials in informing effect and sample size estimates for full-scale trials: a meta-epidemiological study
AID  - 10.1136/bmjebm-2023-112358
DP  - 2023 Dec 01
TA  - BMJ Evidence-Based Medicine
PG  - 383--391
VI  - 28
IP  - 6
4099  - http://ebm.bmj.com/content/28/6/383.short
4100  - http://ebm.bmj.com/content/28/6/383.full
SO  - BMJ EBM2023 Dec 01; 28
AB  - Background Some have argued that pilot trials have little value for informing the expected effect size of a subsequent large trial. This study aims to empirically evaluate the roles of pilot trials in informing the effect and sample size estimates of a full-scale trial.Methods We conducted a search in PubMed on 19 February 2022, for all pilot trials published between 2005 and 2018 and their subsequent full-scale trials. We analysed the agreement in results by comparing the direction and magnitude of the effect size in the pilot trial and full-scale trial. Logistic regression was used to explore whether a significant pilot trial and other characteristics were associated with a significant full-scale trial.Results A total of 248 pairs of pilot and full-scale trials were analysed. Full-scale trials with a significant pilot trial were 2.72 times more likely to find a significant result for the primary efficacy outcome than those with a non-significant pilot trial (95% CI 1.52 to 4.86, p=0.001). The association remained significant irrespective of changes made to the trial design. In 73% of the pairs, the pilot trial produced a larger point estimate than the subsequent full-scale trial, but 87% of pairs had a 95% CI estimated by the pilot trial that covered the full-scale trial point estimate. Full-scale trials with a sample size estimated using the SD from the pilot trial were less likely to yield a significant result (OR=0.26, 95% CI 0.10 to 0.65, p=0.004).Conclusion Pilot trials can provide strong signals on intervention efficacy. When determining the sample size for full-scale trials, using the CI bounds from the pilot trials instead of the point estimate may improve power estimation.Data are available upon reasonable request. The datasets generated during the current study are available from the corresponding author on reasonable request.