You are here

Article Text

Article menu
Original research
Assessing the magnitude of reporting bias in trials of homeopathy: a cross-sectional study and meta-analysis
  1. Gerald Gartlehner1,2,
  2. Robert Emprechtinger1,
  3. Marlene Hackl3,
  4. Franziska L Jutz3,
  5. Jacob E Gartlehner4,
  6. Julian N Nonninger4,
  7. Irma Klerings1,
  8. Andreea Iulia Dobrescu1
  1. 1 Department for Evidence-based Medicine and Evaluation, Danube University Krems, Krems, Austria
  2. 2 RTI International, Research Triangle Park, North Carolina, USA
  3. 3 Karl Landsteiner University of Health Sciences, Krems, Austria
  4. 4 Medical University of Vienna, Wien, Austria
  1. Correspondence to Dr Gerald Gartlehner, Department for Evidence-based Medicine and Evaluation, Danube University Krems, Krems, Austria; Gerald.Gartlehner{at}donau-uni.ac.at

Abstract

Objectives To assess the magnitude of reporting bias in trials assessing homeopathic treatments and its impact on evidence syntheses.

Design A cross-sectional study and meta-analysis. Two persons independently searched Clinicaltrials.gov, the EU Clinical Trials Register and the International Clinical Trials Registry Platform up to April 2019 to identify registered homeopathy trials. To determine whether registered trials were published and to detect published but unregistered trials, two persons independently searched PubMed, Allied and Complementary Medicine Database, Embase and Google Scholar up to April 2021. For meta-analyses, we used random effects models to determine the impact of unregistered studies on meta-analytic results.

Main outcomes and measures We report the proportion of registered but unpublished trials and the proportion of published but unregistered trials. We also assessed whether primary outcomes were consistent between registration and publication. For meta-analyses, we used standardised mean differences (SMDs).

Results Since 2002, almost 38% of registered homeopathy trials have remained unpublished, and 50% of published randomised controlled trials (RCTs) have not been registered. Retrospective registration was more common than prospective registration. Furthermore, 25% of primary outcomes were altered or changed compared with the registry. Although we could detect a statistically significant trend toward an increase of registrations of homeopathy trials (p=0.001), almost 30% of RCTs published during the past 5 years had not been registered.

A meta-analysis stratified by registration status of RCTs revealed substantially larger treatment effects of unregistered RCTs (SMD: −0.53, 95% CI −0.87 to −0.20) than registered RCTs (SMD: −0.14, 95% CI −0.35 to 0.07).

Conclusions Registration of published trials was infrequent, many registered trials were not published and primary outcomes were often altered or changed. This likely affects the validity of the body of evidence of homeopathic literature and may overestimate the true treatment effect of homeopathic remedies.

Data availability statement

Data are available upon reasonable request. Data available upon reasonable request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjebm-2021-111846

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Summary box

    What is already know about this subject?

    • Because homeopathy is exempt from most regulatory requirements, any assessment of the effectiveness must rely on published evidence.

    • Between 2000 and 2013, only 46% of registered randomised controlled trials (RCTs) on homeopathy were published.

    • The proportion of published but unregistered RCTs on homeopathy is unknown.

    What are the new findings?

    • Large proportions of homeopathy trials remain unpublished (38%) or unregistered (50%).

    • 25% of registered trials altered or changed the primary outcome in the publication.

    • Unregistered homeopathy trials tended to have larger treatment effects than registered trials.

    How might it impact clinical practice in the foreseeable future?

    • Clinicians need to be aware that published homeopathy trials represent only a selected proportion of research with mostly positive results.

    • Effect estimates of meta-analyses of homeopathy trials might substantially overestimate the true treatment effect of homeopathic remedies and need to be interpreted cautiously.

    Introduction

    Homeopathy, developed by Samuel Hahnemann in Germany almost 200 years ago, is largely inconsistent with current scientific concepts.1 For example, the principle of similarity (like cures like) and the claim that an increasing dilution of a substance leads to a stronger treatment effect (potentiation) lack evidence and contradict medical and physical principles.2Nevertheless, in many western countries, homeopathy’s popularity as a safe, holistic and comparable alternative to modern medicine persists. A systematic review reported that up to 9.2% of adults in mostly western countries have relied on homeopathic remedies during the past 12 months.3 According to a market research report, the global homeopathic product market reached US$5.5 billion in 2018 with most sales in North America and Europe.4

    For years, sceptics and homeopathic practitioners have engaged in a fierce debate on whether homeopathic treatments are more effective than placebo.5–11 Advocates of homeopathy often refer to two systematic reviews and meta-analyses by Mathie et al that reported statistically significant differences of homeopathic treatments compared with placebo.12 13 According to Google Scholar, these reviews have been cited up to 200 times in other research publications. Sceptics counter that homeopathy’s effectiveness is no different from a placebo’s effectiveness when effectiveness is based on methodologically sound studies.14 The Australian National Health and Medical Research Council concluded that according to an assessment of 57 systematic reviews on 68 conditions, ‘there are no health conditions for which there is reliable evidence that homeopathy is effective’.15 Institutions in the European Union16 and the UK17 have issued similar statements that support this stance.

    An important factor often missing from the debate, however, is that published RCTs of homeopathic treatments might not represent the totality of conducted scientific studies but rather only a selected proportion with positive results. This phenomenon, known as reporting bias, occurs when the publication or non-publication of studies or outcomes depend on the nature and direction of results, with statistically significant findings having a higher likelihood of publication than non-significant findings.18 Because statistical methods to detect or correct for reporting bias have limitations, meta-analyses of published studies like the ones from Mathie et al 12 13 can lead to inflated and misleading results because positive trials are overrepresented.

    To reduce reporting bias, public trial registries such as ClinicalTrials.gov in the USA and ClinicalTrialsRegister.eu in the European Union have been founded to promote the prospective registration of all trials. Since an amendment of the Declaration of Helsinki in 2008, prospective trial registration and publication of results are regarded as an ethical obligation of investigators.19 In 2005, the International Committee of Medical Journal Editors (ICMJE) adopted a policy that the journals they oversee would only publish results of clinical trials which have been prospectively recorded in a public registry.20 These measures have led to an increase in the number of clinical trials that are prospectively registered21; however, researchers are not obligated to publish results of such trials22 and the proportion of non-publication remains high.

    Because regulatory agencies do not require proof of effectiveness for homeopathic products, little attention has been paid to the non-publication of homeopathy trials and its consequences. An assessment by Thomas et al reported that up to the year 2013, only 46% of registered homeopathic trials (16/35) were published.23 When no information is publicly available about the majority of homeopathic trials, sound conclusions about the efficacy and the risks of using homeopathic medicinal products for treating health conditions are impossible.24 25

    To conduct a comprehensive assessment of the state of registration and publication of trials on homeopathy, our study had four objectives: (1) to determine the proportion of registered trials assessing homeopathy that remains unpublished, (2) to examine whether registered primary outcomes are consistent with published primary outcomes, (3) to assess the proportion of published randomised controlled trials (RCTs) on homeopathy that have been registered in a public clinical trial registry and (4) to gauge the impact of reporting bias on evidence syntheses of homeopathy trials.

    Methods

    For the purpose of this project, we adopted the following definition of a clinical trial by the ICMJE: ‘Any research project that prospectively assigns human subjects to intervention and comparison groups to study the cause-and-effect relationship between a medical intervention and a health outcome’.20

    Identification of registered trials and corresponding publications

    To identify registered trials assessing homeopathy, we searched the following trial registries up to 19 April 2021: Clinicaltrials.gov (https://clinicaltrials.gov), EU Clinical Trials Register (www.clinicaltrialsregister.eu) and the International Clinical Trials Registry Platform (ICTRP) (https://ictrptest.azurewebsites.net) of the WHO. The ICTRP is a meta-register, which draws data from 17 national trial registries.26

    We included registrations of all trials (randomised or non-randomised; controlled or uncontrolled) that assessed the effectiveness of individualised or non-individualised homeopathic treatments for any medical condition in humans. A recruitment status of ‘completed’ or ‘unknown’ was eligible for inclusion, trials classified as ‘terminated’ or ‘withdrawn’ were excluded from analyses. To ensure that investigators had at least 2 years to publish their research, we adopted a cut-off date of 19 April 2019. WHO recommends that clinical trials are published within 2 years of completion.27 To assess whether trials were registered prospectively, during enrolment, or retrospectively, we compared the date of first submission of the registration with the date of first enrolment based on the ICTRP.

    To determine whether registered trials were published, we used publication information in registries (eg, links to publications) or searched PubMed, Ovid Allied and Complementary Medicine Database (AMED), Embase.com and Google Scholar using the unique trial registration number, the title of the registered trial or the name of the principal investigator to detect publications. Two persons independently conducted searches. We classified trials that we detected as published manuscripts in a searchable electronic database as ‘published’. If we could not detect a publication that corresponded to the registration, we contacted the investigator named as the person responsible for the registration per email. Registration records for which we could not detect publications were classified as ‘unpublished’. In case a published study was later retracted, we classified it as unpublished.

    Assessment of the consistency of primary outcomes

    For each study, we identified the registered primary outcome and its point of time of assessment as reported in the registry. We compared this information with the primary outcome and its point of time of assessment in the publication. If a publication did not explicitly state the primary outcome, we viewed the outcome most prominently reported in the abstract and the results of the manuscript as the primary outcome. Two persons assessed the consistency of outcomes.

    Identification of published trials and corresponding registrations

    We limited our searches for published trials to RCTs that were published in 2002 or later. We applied 2002 as a cut-off because the first clinical trials registry (ClinicalTrials.gov) was established in 2000.28 With this cut-off, we allowed for 2 years to register trials before publication.

    To identify published RCTs, we reviewed reference lists of four landmark systematic reviews by Mathie et al that assessed homeopathic treatments for any medical condition.12 13 29 30 and nine Cochrane reviews on homeopathy for specific indications.31–38 In addition, we conducted systematic literature searches from January 2013 to July 2021 to update the systematic reviews by Mathie et al.12 13 An experienced information specialist searched PubMed, Cochrane Central Register of Controlled Trials (Cochrane Library/Wiley), Embase.com, Ovid AMED, CORE-Hom (https://www.carstens-stiftung.de/datenbanken-zur-integrativen-medizin.html#/) and Scopus.com. Online supplemental table 1 presents the search strategy. We limited searches to humans and RCTs. Two investigators independently screened abstracts and full-text articles of the literature searches for eligibility. We resolved conflicts by discussion and consensus.

    Supplemental material

    To determine whether published RCTs were registered, we checked full-text publications for information about registration or searched Clinicaltrials.gov, EU Clinical Trials Register and ICTRP to detect registration records of each published RCT. The first two registries are the most commonly used registries in North America and Europe and the third registry, ICTRP, is a meta-register that includes national registries from a more economically diverse group of countries throughout the world. We classified trials as registered if we were able to detect a registraiton record that was dated earlier than the publication of the manuscript. Ethical approval numbers did not qualify as registration numbers.

    Data extraction and data management

    We organised data in Microsoft Excel (Office 365). Extracted data from registries included the unique registration number, the status of the registered trial (ongoing, completed and terminated), whether the registration was prospective or retrospective, and characteristics of the registered trials (eg, individualised or non-individualised homeopathy, intended sample size, primary outcome and point of time of assessment, type of control intervention and others).

    Extracted data from published trials included author and publication information (unique trial registration number, digital object identifier and PubMed identification number), medical condition of the population, intervention and control characteristics, sample size and study design.

    Quantitative analyses

    Based on the total number of registered trials, we calculated the proportion of trials that were published or remained unpublished. Likewise, based on the total number of published RCTs assessing homeopathy, we determined the proportion of registered and unregistered RCTs. We assessed time trends of registrations and publications using a logistic regression predicting whether studies were published or registered depending on the year of publication or registration.

    For the meta-analytic example, we used data by Mathie et al.13 We re-analysed the meta-analysis using a random effects model (DerSimonian and Laird) and standardised mean differences (SMDs) as the outcome measure. For our analyses, we used the same outcomes as Mathie et al who selected outcomes in the following hierarchical order based on availability in the publications: mortality, morbidity, health impairment, limitation of activity, restriction of participation and surrogate outcomes. We stratified the meta-analysis by registration status (registered vs unregistered RCTs).39 40

    We conducted all statistical analyses with R V.4.1.0 (R Core Team, 2021).

    Results

    In the following sections, we first summarise the publication status of registered homeopathy trials and present the degree of consistency of primary outcomes in registries and publications. We then present the registration status of published homeopathy trials. Finally, we present an example illustrating the potential impact of missing registration of RCTs on results of a meta-analysis of homeopathy trials.

    Publication status of registered homeopathy trials

    Of 116 registered homeopathy trials, 90 met our inclusion criteria. Figure 1 presents the deduplicated search results and the number of included and excluded records. Online supplemental table 2 summarises registration characteristics and, if available, publication identifiers of included trials. Most registered trials were RCTs (72%) or uncontrolled trials (27%). A single study was a non-randomised controlled trial. The first clinical trial on homeopathy was registered in July 2003. We contacted 25 investigators for further information about the publication status of their registered studies, of whom 5 responded to our query.

    Figure 1
    Figure 1

    Disposition of search results in clinical trial registries (created by authors). ICTRP, International Clinical Trials Registry Platform; RCT, randomised clinical trial.

    Fifty-five registered trials reported that non-individualised homeopathy was the intervention of choice, 34 used individualised homeopathy and one trial did not report the type of homeopathy. Intended sample sizes ranged from 1 (n-of-1 RCT) to 906 participants, with a median sample size of 60. Forty-three trials (47.8%) were registered before the start of the study or during recruitment, 47 trials (52.2%) were registered retrospectively. Most trials were registered in ClinicalTrials.gov (66%).

    Overall, 57.8% (n=52) of registered trials were published in a journal that was listed in PubMed, Embase or Google Scholar, 4.4% (n=4) were published as grey literature and 37.8% (n=34) of registered homeopathy trials remained unpublished. The grey literature publications were master theses that were available through university servers and could be detected when searching Google Scholar for the registration number. Of trials registered in 2008 (when the Declaration of Helsinki started to require trial registration and publication) or later (n=78), only 24 trials (30.7%) were prospectively registered and had published results, and thus adhered to the Declaration of Helsinki19 on ethical principles of medical research.

    Figure 2 depicts the proportions of published and unpublished trials per year and the trend for publication of registered trials over time. In this graph, we classified grey literature publications as ‘published’. Between 2003 and 2018 (we did not detect eligible registrations for 2019), the probability of completed and subsequently published trials did not show a statistically significant increase over time (p for trend=0.817).

    Figure 2
    Figure 2

    Proportions of published and unpublished registered trials with trend by registration year (created by authors).

    Consistency of Primary Outcomes

    Of the 56 registered and published trials (including the four trials published in the grey literature), 25.0% (n=14) modified or switched the primary outcome measure or the point of time of assessment. Most commonly, outcomes that were not registered as the primary outcomes were reported as such in the study publications. In four instances, the point of time of assessment was not the same as pre-specified in the register.

    Registration status of published homeopathy trials

    Our literature searches rendered 193 published RCTs assessing the efficacy or effectiveness of homeopathic treatments between 2002 and 2021 (figure 1). Online supplemental table 3 presents publication characteristics and registration status of included RCTs. Of 193 RCTs, 49.7% (n=96) had been registered (prospectively or retrospectively), for the remaining 50.3% (n=97), no registration could be detected. Between 2002 and 2021, the proportion of registered RCTs statistically significantly increased over time (p<0.001 for trend). Figure 3 depicts the proportions of registered and unregistered RCTs by year of publication and the trend of registration over time.

    Figure 3
    Figure 3

    Proportions of registered and unregistered trials with trend by publication year (created by authors).

    The impact of missing trial registration and selective publication on results of a meta-analysis

    To illustrate the potential bias introduced by missing registration, we used data from a meta-analysis by Mathie et al 13 as an example. This meta-analysis pooled data from 54 double-blinded, low or unclear risk of bias RCTs that compared non-individualised homeopathy with placebo for any medical indication in humans. Included studies were published between 1976 and 2014. The original meta-analysis by Mathie et al yielded a statistically significantly greater efficacy of homeopathy than placebo (SMD: −0.33, 95% CI −0.44 to – 0.21).13

    For our example, we included the 19 RCTs,41–59 which were published between 2002 and 2014. We chose 2002 as a threshold because ClinicalTrials.gov was introduced in 2000. By 2002, investigators would have had 2 years to register a trial. 2014 was the end of literature search of the Mathie et al’s review. In our example, we conducted a stratified meta-analysis of these 19 RCTs by registration status: 6 RCTs54–59 were registered and 13 were not.41–53

    A meta-analysis of unregistered RCTs yielded a statistically significant treatment effect favouring homeopathy (SMD: −0.53, 95% CI −0.87 to −0.20). By contrast, a meta-analysis of registered RCTs did not show a statistically significant difference between homeopathy and placebo (SMD: −0.14, 95% CI −0.35 to 0.07). Meta-regression revealed that the difference in effect sizes between registered and unregistered studies did not reach statistical significance (difference in SMDs: 0.39, 95% CI −0.09 to 0.87). Figure 4 presents the meta-analyses of registered and unregistered RCTs.

    Figure 4
    Figure 4

    Random effects meta-analyses stratified by registered and unregistered RCTs (created by authors). RCTs, randomised controlled trials; SMD, standardised mean difference.

    Discussion

    Our study revealed a poor adherence of investigators assessing homeopathic treatments to the Declaration of Helsinki, which requires prospective registration of trials and dissemination of results. Almost 38% of registered homeopathy trials remained unpublished, and 50% of published RCTs had not been registered. Retrospective registration was more common than prospective registration. Furthermore, 25% of primary outcomes were altered or changed compared with the registry. Although we could detect a statistically significant trend toward an increase of registrations of homeopathy trials, almost 30% of RCTs published during the past 5 years had not been registered.

    Overall, these findings suggest a concerning lack of scientific and ethical standards in the field of homeopathy and a high risk for reporting bias. The high proportions of unregistered or retrospectively registered trials indicate that publication frequently depends on the nature of results. Presumably, the number of unknown cases of unregistered and unpublished homeopathy trials is high. Findings also indicate that journals publishing homeopathy trials do not adhere to policies by the ICMJE, which demand that only registered RCTs should be published.20

    The non-publication of trial results and selective outcome reporting, however, is not a phenomenon that is limited to homeopathy.60 Over the past years, numerous studies reported that, despite registration, large proportions of completed trials remained unpublished.22 61–65 For example, an assessment of 2132 registered clinical trials in Germany between 2009 and 2013 showed that 33% remained unpublished after 5 years.62 Likewise, of 4347 clinical trials conducted in academic centres in the USA, 34% remained unpublished.61

    Homeopathy is often viewed as a safe alternative to pharmaceutical treatments. For the approval of pharmaceutical interventions, however, the industry is required to submit all trial data to regulatory agencies, regardless of the publication in the medical literature. Homeopathy, however, is exempt from most regulatory requirements with the consequence that no independent regulatory agency reviews individual patient data of trials, assesses statistical analyses methods or requests long-term follow-up data. Any assessment of the effectiveness of homeopathic treatments, therefore, must rely only on published evidence.

    To the best of our knowledge, Thomas et al were the first who assessed the extent of non-publication of trials in the homeopathic literature.23 Their assessment covered the trial literature from 2002 to 2014 and was less comprehensive than ours. Overall, however, they reported similar findings as ours. In their sample, 46% of registered trials remained unpublished and in 25% of published homeopathy trials, the primary outcome measures had been switched or modified.23 The consistency of these results indicates that over almost a decade, the situation has not substantially improved

    Consequently, reporting biases are likely to have a substantial impact on the estimated treatment effect of homeopathy. Using data from a highly cited meta-analysis of homeopathy RCTs, our example showed that unregistered trials yielded substantially larger treatment effects than registered trials. This finding is consistent with a meta-epidemiological study of Cochrane reviews by Dechartres et al.66 Based on 32 meta-analyses of various medical topics, unregistered or retrospectively registered trials showed larger treatment effects than prospectively registered trials (ratio of ORs=0.81, 95 % CI 0.65 to 1.02).

    Our study has several limitations. First, although our searches covered 17 trial registries, chances are high that we missed records because of limited recall (ie, the ability of a search strategy to identify relevant records) in these registries. Another reason we may have missed registrations of published trials concerns the registries themselves. Time lags often occur between national registry entries and exportation od the information to ICTRP. And registry information is occasionally incomplete or unclear. Likewise, despite the considerable effort of two investigators to detect publications of registered trials, we might have overlooked some, particularly if they were available only as grey literature. Second, our example is based on a single meta-analysis on non-individualised homeopathic treatments that might not be applicable to individualised homeopathy. We attempted to use data from a meta-analysis on individualised homeopathy, but in this body of evidence only a single RCT was registered, therefore, making it impossible to conduct any meaningful analysis. Third, we excluded registered trials from our analyses that were categorised as ‘terminated’ or ‘suspended’. One could argue that this excludes trials that were terminated early for either benefit or harm. We deemed it unlikely, however, that homeopathy trials were terminated for these reasons. A more common reason for termination is the lack of recruitment of participants. For these reasons, we chose to exclude terminated studies as not to bias results toward a lower publication rate. Finally, it is conceivable that a substantial number of unregistered (and thus untraceable) homeopathy trials remain unpublished. Unfortunately, due to a lack of registration requirement for homeopathic trials, such trials remain impossible for researchers like us to detect. Without a required registration, the actual magnitude of non-publication of homeopathic trials is impossible to fully assess. Our findings are, therefore, likely an underestimation of the true extent of non-publication of homeopathy trials.

    Overall, it is paramount that the homeopathic research community commits to the adherence of scientific standards and requires the registration of all trials assessing homeopathic treatments. Furthermore, journals publishing homeopathic research must adhere to basic standards of quality by adopting or adhering to ICMJE policies and only publishing the results of clinical trials, which have been prospectively recorded in a public registry.

    Conclusions

    Registration of published trials was infrequent, many registered trials were not published and primary outcomes were often altered or changed. This likely affects the validity of the body of evidence of homeopathic literature and may substantially overestimate the true treatment effect of homeopathic remedies.

    Data availability statement

    Data are available upon reasonable request. Data available upon reasonable request.

    Ethics statements

    Patient consent for publication

    Ethics approval

    This study does not involve human participants.

    Acknowledgments

    We would like to thank Liliya Ziganshina from Cochrane Russia for providing input on a Russian publication. We are also grateful to Petra Wellemsen for administrative support and to Dawn Gartlehner for editing and improving the readability of our manuscript.

    References

    1. ↵
      2. Grimes DR
      . Proposed mechanisms for homeopathy are physically impossible. Focus on Alternative and Complementary Therapies 2012;17:149–55.doi:10.1111/j.2042-7166.2012.01162.x
      OpenUrl
    2. ↵
      2. Grams N
      . Homeopathy-where is the science? A current inventory on a pre-scientific artifact. EMBO Rep 2019;20. doi:doi:10.15252/embr.201947761. [Epub ahead of print: 14 02 2019].pmid:http://www.ncbi.nlm.nih.gov/pubmed/30765373
    3. ↵
      2. Relton C ,
      3. Cooper K ,
      4. Viksveen P , et al
      . Prevalence of homeopathy use by the general population worldwide: a systematic review. Homeopathy 2017;106:69–78.doi:10.1016/j.homp.2017.03.002 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28552176
      OpenUrlPubMed
    4. ↵
    5. ↵
      2. Ernst E ,
      3. Homeopathy EE
      . Homeopathy, a "helpful placebo" or an unethical intervention? Trends Pharmacol Sci 2010;31:1.doi:10.1016/j.tips.2009.10.005 pmid:http://www.ncbi.nlm.nih.gov/pubmed/19963285
      OpenUrlCrossRefPubMed
    6. ↵
      2. Ernst E ,
      3. Homeopathy EE
      . Homeopathy, non-specific effects and good medicine. Rheumatology 2011;50:1007–8.doi:10.1093/rheumatology/keq265 pmid:http://www.ncbi.nlm.nih.gov/pubmed/21076130
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      2. Iacobucci G
      . Homeopathy should have professional accreditation revoked, NHS leaders urge. BMJ 2019;367:l6248.doi:10.1136/bmj.l6248 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31662308
      OpenUrlFREE Full Text
    8. ↵
      2. Jacobs J
      . Homeopathic prevention and management of epidemic diseases. Homeopathy 2018;107:157–60.doi:10.1055/s-0038-1649487 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29753299
      OpenUrlPubMed
    9. ↵
      2. Matthiessen P
      . Homöopathie und intellektuelle Redlichkeit – Eine Stellungnahme. Deutsche Zeitschrift fur Onkologie 2019;50:172–7.
      OpenUrl
    10. ↵
      2. Pannek J ,
      3. Pannek-Rademacher S
      . Time to say good-bye? Homeopathy, skeptics and thoughts on how to proceed. J Complement Integr Med 2021. doi:doi:10.1515/jcim-2020-0491. [Epub ahead of print: 06 May 2021].pmid:http://www.ncbi.nlm.nih.gov/pubmed/33957710
    11. ↵
      2. Spence D
      . Good medicine: homeopathy. BMJ 2012;345:e6184.doi:10.1136/bmj.e6184 pmid:http://www.ncbi.nlm.nih.gov/pubmed/22983533
      OpenUrlFREE Full Text
    12. ↵
      2. Mathie RT ,
      3. Lloyd SM ,
      4. Legg LA , et al
      . Randomised placebo-controlled trials of individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev 2014;3:142.doi:10.1186/2046-4053-3-142 pmid:http://www.ncbi.nlm.nih.gov/pubmed/25480654
      OpenUrlCrossRefPubMed
    13. ↵
      2. Mathie RT ,
      3. Ramparsad N ,
      4. Legg LA , et al
      . Randomised, double-blind, placebo-controlled trials of non-individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev 2017;6:63.doi:10.1186/s13643-017-0445-3 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28340607
      OpenUrlCrossRefPubMed
    14. ↵
      2. Shang A ,
      3. Huwiler-Müntener K ,
      4. Nartey L , et al
      . Are the clinical effects of homoeopathy placebo effects? comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet 2005;366:726–32.doi:10.1016/S0140-6736(05)67177-2 pmid:http://www.ncbi.nlm.nih.gov/pubmed/16125589
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      Homeopathy: NHMRC. Available: https://www.nhmrc.gov.au/about-us/resources/homeopathy [Accessed 24 Jun 2021].
    16. ↵
      Homeopathic products and practices: assessing the evidence and ensuring consistency in regulating medical claims in the EU: European academies' science Advisory Council. Available: https://easac.eu/fileadmin/PDF_s/reports_statements/EASAC_Homepathy_statement_web_final.pdf [Accessed 27 Jun 2021].
    17. ↵
      Evidence check 2: Homeopathy house of commons science and technology Committee, 2010. Available: https://publications.parliament.uk/pa/cm200910/cmselect/cmsctech/45/45.pdf [Accessed 27 Jun 2021].
    18. ↵
      2. Hopewell S ,
      3. Loudon K ,
      4. Clarke MJ
      . Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009;1:MR000006.
      OpenUrlCrossRefPubMed
    19. ↵
      World Medical Association IW. WMA Declaration of Helsinki - Ethical Principles for Medical Research Involving Human Subjects Le Keynes: World Medical Association, Inc. (WMA), 2018. Available: https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/
    20. ↵
      2. De Angelis CD ,
      3. Drazen JM ,
      4. Frizelle FA , et al
      . Is this clinical trial fully registered? A statement from the International Committee of medical Journal editors. Lancet 2005;365:1827–9.doi:10.1016/S0140-6736(05)66588-9 pmid:http://www.ncbi.nlm.nih.gov/pubmed/15924965
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      2. Trinquart L ,
      3. Dunn AG ,
      4. Bourgeois FT
      . Registration of published randomized trials: a systematic review and meta-analysis. BMC Med 2018;16:173.doi:10.1186/s12916-018-1168-6 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30322399
      OpenUrlPubMed
    22. ↵
      2. Ross JS ,
      3. Mulvey GK ,
      4. Hines EM , et al
      . Trial publication after registration in ClinicalTrials.gov: a cross-sectional analysis. PLoS Med 2009;6:e1000144.doi:10.1371/journal.pmed.1000144 pmid:http://www.ncbi.nlm.nih.gov/pubmed/19901971
      OpenUrlCrossRefPubMed
    23. ↵
      2. Thomas ET ,
      3. Clark J ,
      4. Glasziou P
      . Publication and outcome reporting of homeopathy trials registered in ClinicalTrials.gov. Focus on Alternative and Complementary Therapies 2016;21:127–33.doi:10.1111/fct.12278
      OpenUrl
    24. ↵
      2. Kicinski M ,
      3. Springate DA ,
      4. Kontopantelis E
      . Publication bias in meta-analyses from the Cochrane database of systematic reviews. Stat Med 2015;34:2781–93.doi:10.1002/sim.6525 pmid:http://www.ncbi.nlm.nih.gov/pubmed/25988604
      OpenUrlCrossRefPubMed
    25. ↵
      2. Thornton A ,
      3. Lee P
      . Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol 2000;53:207–16.doi:10.1016/S0895-4356(99)00161-4 pmid:http://www.ncbi.nlm.nih.gov/pubmed/10729693
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. WHO
      . Primary registries in the WHO registry network: World Health organization. Available: https://www.who.int/clinical-trials-registry-platform/network/primary-registries [Accessed 9 Jun 2021].
    27. ↵
      1. World Health organization
      . Joint statement on public disclosure of results from clinical trials: World Health Organization (WHO), 2017. Available: https://www.who.int/news/item/18-05-2017-joint-statement-on-registration [Accessed 9 Mar 2021].
    28. ↵
      Clinicaltrials.Gov background: U.S. National library of medicine. Available: https://clinicaltrials.gov/ct2/about-site/background [Accessed 9 Jun 2021].
    29. ↵
      2. Mathie RT ,
      3. Fok YYY ,
      4. Viksveen P , et al
      . Systematic review and meta-analysis of randomised, Other-than-Placebo controlled, trials of Non-Individualised homeopathic treatment. Homeopathy 2019;108:088–101.doi:10.1055/s-0038-1677481 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30699444
      OpenUrlPubMed
    30. ↵
      2. Mathie RT ,
      3. Ulbrich-Zürni S ,
      4. Viksveen P , et al
      . Systematic review and meta-analysis of randomised, Other-than-Placebo controlled, trials of individualised homeopathic treatment. Homeopathy 2018;107:229–43.doi:10.1055/s-0038-1667129 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30121049
      OpenUrlPubMed
    31. ↵
      2. Smith CA
      . Homoeopathy for induction of labour. Cochrane Database Syst Rev 2003;4:CD003399.doi:10.1002/14651858.CD003399 pmid:http://www.ncbi.nlm.nih.gov/pubmed/14583972
      OpenUrlPubMed
    32. ↵
      2. McCarney R ,
      3. Warner J ,
      4. Fisher P , et al
      . Homeopathy for dementia. Cochrane Database Syst Rev 2003;1:CD003803.doi:10.1002/14651858.CD003803 pmid:http://www.ncbi.nlm.nih.gov/pubmed/12535487
      OpenUrlPubMed
    33. ↵
      2. McCarney RW ,
      3. Linde K ,
      4. Lasserson TJ
      . Homeopathy for chronic asthma. Cochrane Database Syst Rev 2004;1:CD000353.
      OpenUrl
    34. ↵
      2. Peckham EJ ,
      3. Cooper K ,
      4. Roberts ER , et al
      . Homeopathy for treatment of irritable bowel syndrome. Cochrane Database Syst Rev 2019;9:CD009710.doi:10.1002/14651858.CD009710.pub3 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31483486
      OpenUrlPubMed
    35. ↵
      2. Heirs M ,
      3. Dean ME , Cochrane Developmental, Psychosocial and Learning Problems Group
      . Homeopathy for attention deficit/hyperactivity disorder or hyperkinetic disorder. Cochrane Database Syst Rev 2007;164.doi:10.1002/14651858.CD005648.pub2
    36. ↵
      2. Mathie RT ,
      3. Frye J ,
      4. Fisher P
      . R) for preventing and treating influenza and influenza-like illness. Cochrane Database Syst Rev 2015;1:CD001957.
      OpenUrl
    37. ↵
      2. Kassab S ,
      3. Cummings M ,
      4. Berkovitz S
      . Homeopathic medicines for adverse effects of cancer treatments. Cochrane Database Syst Rev 2009;2:CD004845.
      OpenUrl
    38. ↵
      2. Rada G ,
      3. Capurro D ,
      4. Pantoja T , et al
      . Non-Hormonal interventions for hot flushes in women with a history of breast cancer. Cochrane Database Syst Rev 2010;9:CD004923.doi:10.1002/14651858.CD004923.pub2 pmid:http://www.ncbi.nlm.nih.gov/pubmed/20824841
      OpenUrlPubMed
    39. ↵
      2. Kim NY ,
      3. Bangdiwala SI ,
      4. Thaler K , et al
      . SAMURAI: sensitivity analysis of a meta-analysis with unpublished but registered analytical investigations (software). Syst Rev 2014;3:27.doi:10.1186/2046-4053-3-27 pmid:http://www.ncbi.nlm.nih.gov/pubmed/24641974
      OpenUrlPubMed
    40. ↵
      2. Polanin JR ,
      3. Tanner-Smith EE ,
      4. Hennessy EA
      . Estimating the difference between published and unpublished effect sizes. Rev Educ Res 2016;86:207–36.doi:10.3102/0034654315582067
      OpenUrlCrossRef
    41. ↵
      2. Baker DG ,
      3. Myers SP ,
      4. Howden I , et al
      . The effects of homeopathic Argentum nitricum on test anxiety. Complement Ther Med 2003;11:65–71.doi:10.1016/S0965-2299(03)00059-1 pmid:http://www.ncbi.nlm.nih.gov/pubmed/12801490
      OpenUrlPubMed
    42. ↵
      2. Belon P ,
      3. Banerjee P ,
      4. Choudhury SC , et al
      . Can administration of potentized homeopathic remedy, Arsenicum album, alter antinuclear antibody (ANA) titer in people living in high-risk arsenic contaminated areas? I. A correlation with certain hematological parameters. Evid Based Complement Alternat Med 2006;3:99–107.doi:10.1093/ecam/nek013 pmid:http://www.ncbi.nlm.nih.gov/pubmed/16550230
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      2. Frass M ,
      3. Dielacher C ,
      4. Linkesch M , et al
      . Influence of potassium dichromate on tracheal secretions in critically ill patients. Chest 2005;127:936–41.doi:10.1378/chest.127.3.936 pmid:http://www.ncbi.nlm.nih.gov/pubmed/15764779
      OpenUrlCrossRefPubMed
    44. ↵
      2. Jacobs J ,
      3. Fernandez EA ,
      4. Merizalde B , et al
      . The use of homeopathic combination remedy for dengue fever symptoms: a pilot RCT in Honduras. Homeopathy 2007;96:22–6.doi:10.1016/j.homp.2006.10.004 pmid:http://www.ncbi.nlm.nih.gov/pubmed/17227744
      OpenUrlCrossRefPubMed
    45. ↵
      2. Khuda-Bukhsh AR ,
      3. Pathak S ,
      4. Guha B , et al
      . Can homeopathic arsenic remedy combat arsenic poisoning in humans exposed to groundwater arsenic contamination?: a preliminary report on first human trial. Evid Based Complement Alternat Med 2005;2:537–48.doi:10.1093/ecam/neh124 pmid:http://www.ncbi.nlm.nih.gov/pubmed/16322812
      OpenUrlCrossRefPubMedWeb of Science
    46. ↵
      2. Khuda-Bukhsh AR ,
      3. Banerjee A ,
      4. Biswas SJ , et al
      . An initial report on the efficacy of a millesimal potency Arsenicum album LM 0/3 in ameliorating arsenic toxicity in humans living in a high-risk arsenic village. Zhong Xi Yi Jie He Xue Bao 2011;9:596–604.doi:10.3736/jcim20110604 pmid:http://www.ncbi.nlm.nih.gov/pubmed/21669162
      OpenUrlPubMed
    47. ↵
      2. Kim LS ,
      3. Riedlinger JE ,
      4. Baldwin CM , et al
      . Treatment of seasonal allergic rhinitis using homeopathic preparation of common allergens in the southwest region of the US: a randomized, controlled clinical trial. Ann Pharmacother 2005;39:617–24.doi:10.1345/aph.1E387 pmid:http://www.ncbi.nlm.nih.gov/pubmed/15741420
      OpenUrlCrossRefPubMed
    48. ↵
      2. Kotlus BS ,
      3. Heringer DM ,
      4. Dryden RM
      . Evaluation of homeopathic Arnica montana for ecchymosis after upper blepharoplasty: a placebo-controlled, randomized, double-blind study. Ophthalmic Plast Reconstr Surg 2010;26:395–7.doi:10.1097/IOP.0b013e3181cd93be pmid:http://www.ncbi.nlm.nih.gov/pubmed/20683279
      OpenUrlCrossRefPubMed
    49. ↵
      2. Malapane E ,
      3. Solomon EM ,
      4. Pellow J
      . Efficacy of a homeopathic complex on acute viral tonsillitis. J Altern Complement Med 2014;20:868–73.doi:10.1089/acm.2014.0189 pmid:http://www.ncbi.nlm.nih.gov/pubmed/25238506
      OpenUrlPubMed
    50. ↵
      2. Naidoo P ,
      3. Pellow J
      . A randomized placebo-controlled pilot study of cat saliva 9cH and histaminum 9cH in cat allergic adults. Homeopathy 2013;102:123–9.doi:10.1016/j.homp.2013.02.007 pmid:http://www.ncbi.nlm.nih.gov/pubmed/23622262
      OpenUrlPubMed
    51. ↵
      2. Robertson A ,
      3. Suryanarayanan R ,
      4. Banerjee A
      . Homeopathic Arnica montana for post-tonsillectomy analgesia: a randomised placebo control trial. Homeopathy 2007;96:17–21.doi:10.1016/j.homp.2006.10.005 pmid:http://www.ncbi.nlm.nih.gov/pubmed/17227743
      OpenUrlCrossRefPubMed
    52. ↵
      2. Wolf M ,
      3. Tamaschke C ,
      4. Mayer W , et al
      . [Efficacy of Arnica in varicose vein surgery: results of a randomized, double-blind, placebo-controlled pilot study]. Forsch Komplementarmed Klass Naturheilkd 2003;10:242–7.doi:10.1159/000074778 pmid:http://www.ncbi.nlm.nih.gov/pubmed/14605480
      OpenUrlPubMed
    53. ↵
      2. Zabolotnyi DI ,
      3. Kneis KC ,
      4. Richardson A , et al
      . Efficacy of a complex homeopathic medication (Sinfrontal) in patients with acute maxillary sinusitis: a prospective, randomized, double-blind, placebo-controlled, multicenter clinical trial. Explore 2007;3:98–109.doi:10.1016/j.explore.2006.12.007 pmid:http://www.ncbi.nlm.nih.gov/pubmed/17362845
      OpenUrlPubMed
    54. ↵
      2. Colau J-C ,
      3. Vincent S ,
      4. Marijnen P , et al
      . Efficacy of a non-hormonal treatment, BRN-01, on menopausal hot flashes: a multicenter, randomized, double-blind, placebo-controlled trial. Drugs R D 2012;12:107–19.doi:10.2165/11640240-000000000-00000 pmid:http://www.ncbi.nlm.nih.gov/pubmed/22852580
      OpenUrlPubMed
    55. ↵
      2. Cornu C ,
      3. Joseph P ,
      4. Gaillard S , et al
      . No effect of a homoeopathic combination of Arnica montana and Bryonia alba on bleeding, inflammation, and ischaemia after aortic valve surgery. Br J Clin Pharmacol 2010;69:136–42.doi:10.1111/j.1365-2125.2009.03574.x pmid:http://www.ncbi.nlm.nih.gov/pubmed/20233176
      OpenUrlPubMed
    56. ↵
      2. Oberbaum M ,
      3. Galoyan N ,
      4. Lerner-Geva L , et al
      . The effect of the homeopathic remedies Arnica montana and Bellis perennis on mild postpartum bleeding--a randomized, double-blind, placebo-controlled study--preliminary results. Complement Ther Med 2005;13:87–90.doi:10.1016/j.ctim.2005.03.006 pmid:http://www.ncbi.nlm.nih.gov/pubmed/16036165
      OpenUrlPubMed
    57. ↵
      2. Padilha RQ ,
      3. Riera R ,
      4. Átallah AN
      . Homeopathic Plumbum metallicum for lead poisoning: a randomized clinical trial. Homeopathy 2011;100:116–21.doi:10.1016/j.homp.2010.11.007 pmid:http://www.ncbi.nlm.nih.gov/pubmed/21784327
      OpenUrlPubMed
    58. ↵
      2. Paris A ,
      3. Gonnet N ,
      4. Chaussard C , et al
      . Effect of homeopathy on analgesic intake following knee ligament reconstruction: a phase III monocentre randomized placebo controlled study. Br J Clin Pharmacol 2008;65:180–7.doi:10.1111/j.1365-2125.2007.03008.x pmid:http://www.ncbi.nlm.nih.gov/pubmed/18251757
      OpenUrlCrossRefPubMedWeb of Science
    59. ↵
      2. Singer SR ,
      3. Amit-Kohn M ,
      4. Weiss S , et al
      . Traumeel S for pain relief following hallux valgus surgery: a randomized controlled trial. BMC Clin Pharmacol 2010;10:9.doi:10.1186/1472-6904-10-9 pmid:http://www.ncbi.nlm.nih.gov/pubmed/20380750
      OpenUrlPubMed
    60. ↵
      2. Song F ,
      3. Parekh S ,
      4. Hooper L , et al
      . Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010;14:1–193.doi:10.3310/hta14080 pmid:http://www.ncbi.nlm.nih.gov/pubmed/20181324
      OpenUrlCrossRefPubMed
    61. ↵
      2. Chen R ,
      3. Desai NR ,
      4. Ross JS , et al
      . Publication and reporting of clinical trial results: cross sectional analysis across academic medical centers. BMJ 2016;352:i637.doi:10.1136/bmj.i637 pmid:http://www.ncbi.nlm.nih.gov/pubmed/26888209
      OpenUrlAbstract/FREE Full Text
    62. ↵
      2. Wieschowski S ,
      3. Riedel N ,
      4. Wollmann K , et al
      . Result dissemination from clinical trials conducted at German University medical centers was delayed and incomplete. J Clin Epidemiol 2019;115:37–45.doi:10.1016/j.jclinepi.2019.06.002 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31195110
      OpenUrlCrossRefPubMed
    63. ↵
      2. Ross JS ,
      3. Tse T ,
      4. Zarin DA , et al
      . Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis. BMJ 2012;344:d7292.doi:10.1136/bmj.d7292 pmid:http://www.ncbi.nlm.nih.gov/pubmed/22214755
      OpenUrlCrossRefPubMed
    64. ↵
      2. Gordon D ,
      3. Taddei-Peters W ,
      4. Mascette A , et al
      . Publication of trials funded by the National heart, lung, and blood Institute. N Engl J Med 2013;369:1926–34.doi:10.1056/NEJMsa1300237 pmid:http://www.ncbi.nlm.nih.gov/pubmed/24224625
      OpenUrlCrossRefPubMedWeb of Science
    65. ↵
      2. Saito H ,
      3. Gill CJ
      . How frequently do the results from completed US clinical trials enter the public domain?--A statistical analysis of the ClinicalTrials.gov database. PLoS One 2014;9:e101826.doi:10.1371/journal.pone.0101826 pmid:http://www.ncbi.nlm.nih.gov/pubmed/25025477
      OpenUrlCrossRefPubMed
    66. ↵
      2. Dechartres A ,
      3. Ravaud P ,
      4. Atal I , et al
      . Association between trial registration and treatment effect estimates: a meta-epidemiological study. BMC Med 2016;14:100.doi:10.1186/s12916-016-0639-x pmid:http://www.ncbi.nlm.nih.gov/pubmed/27377062
      OpenUrlCrossRefPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Correction notice This article has been corrected since it first published. We were contacted by a reader who identified a problem in Supplementary Table 3. We contacted the authors who reported that, in error, they had uploaded the wrong version of the table. In checking the intended version of the table carefully, the authors found that they had misclassified six published trials as unregistered (when they were registered). Upon correction, this changes the total number of unregistered trials from 103 to 97 of the 193 trials. This changes the result of the proportion of unregistered studies from 53.4% to 50.3%; this does not materially alter the conclusions of the study. In the course of investigating the reader’s query BMJ also asked the authors to make clearer to readers how they defined registration of the study and this has been added to the text.

    • Contributors GG and AID planned the study and were responsible for quality assurance. MH, FLJ, JEG and JNN searched registries and extracted data. IK conducted literature searches. RE ran statistical analyses. GG drafted the manuscript and acts as the guarantor. All authors commented on and revised the draft manuscript.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.