You are here

  • Home
  • Archive
  • Volume 28, Issue 6
  • Re-evaluating the role of pilot trials in informing effect and sample size estimates for full-scale trials: a meta-epidemiological study

Article Text

Article menu
Download PDFPDF
Original research
Re-evaluating the role of pilot trials in informing effect and sample size estimates for full-scale trials: a meta-epidemiological study
  1. Xiangji Ying1,
  2. Karen A Robinson1,2,3,
  3. Stephan Ehrhardt1
  1. 1 Department of Epidemiology, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland, USA
  2. 2 Department of Medicine, School of Medicine, Johns Hopkins University, Baltimore, Maryland, USA
  3. 3 Section Evidence-Based Practice, Western Norway University of Applied Sciences, Bergen, Norway
  1. Correspondence to Dr Stephan Ehrhardt, Department of Epidemiology, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD 21205, Maryland, USA; sehrhar6{at}jhu.edu

Abstract

Background Some have argued that pilot trials have little value for informing the expected effect size of a subsequent large trial. This study aims to empirically evaluate the roles of pilot trials in informing the effect and sample size estimates of a full-scale trial.

Methods We conducted a search in PubMed on 19 February 2022, for all pilot trials published between 2005 and 2018 and their subsequent full-scale trials. We analysed the agreement in results by comparing the direction and magnitude of the effect size in the pilot trial and full-scale trial. Logistic regression was used to explore whether a significant pilot trial and other characteristics were associated with a significant full-scale trial.

Results A total of 248 pairs of pilot and full-scale trials were analysed. Full-scale trials with a significant pilot trial were 2.72 times more likely to find a significant result for the primary efficacy outcome than those with a non-significant pilot trial (95% CI 1.52 to 4.86, p=0.001). The association remained significant irrespective of changes made to the trial design. In 73% of the pairs, the pilot trial produced a larger point estimate than the subsequent full-scale trial, but 87% of pairs had a 95% CI estimated by the pilot trial that covered the full-scale trial point estimate. Full-scale trials with a sample size estimated using the SD from the pilot trial were less likely to yield a significant result (OR=0.26, 95% CI 0.10 to 0.65, p=0.004).

Conclusion Pilot trials can provide strong signals on intervention efficacy. When determining the sample size for full-scale trials, using the CI bounds from the pilot trials instead of the point estimate may improve power estimation.

  • randomized controlled trial
  • pilot trial
  • sample size calculation
  • power calculation
  • efficacy
  • meta-epidemiological study

Data availability statement

Data are available upon reasonable request. The datasets generated during the current study are available from the corresponding author on reasonable request.

https://doi.org/10.1136/bmjebm-2023-112358

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Data availability statement

    Data are available upon reasonable request. The datasets generated during the current study are available from the corresponding author on reasonable request.

    View Full Text

    Footnotes

    • Contributors XY and SE conceptualised and designed the study and lead the acquisition, analysis and interpretation of the data. KR significantly contributed to the data analysis and interpretation. XY composed the initial draft of the manuscript, with all authors commenting on subsequent drafts and contributing to its revision. All authors have given approval for the final version to be published and have agreed to be accountable for all aspects of the work. XY is the guarantor of this work and, as such, had full access to all the data in the study and take responsibility for the integrity and accuracy of the data and data analyses.

    • Funding This study is funded by the Mary B. Meyer Memorial Fund Scholarship (XY). The funding organisations had no role in the design and conduct of the study; collection, management, analysis and interpretation of the data; preparation, review or approval of the manuscript; and decision to submit the manuscript for publication.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.