Article Text

Assessing the magnitude of reporting bias in trials of homeopathy: a cross-sectional study and meta-analysis
  1. Gerald Gartlehner1,2,
  2. Robert Emprechtinger1,
  3. Marlene Hackl3,
  4. Franziska L Jutz3,
  5. Jacob E Gartlehner4,
  6. Julian N Nonninger4,
  7. Irma Klerings1,
  8. Andreea Iulia Dobrescu1
  1. 1Department for Evidence-based Medicine and Evaluation, Danube University Krems, Krems, Austria
  2. 2RTI International, Research Triangle Park, North Carolina, USA
  3. 3Karl Landsteiner University of Health Sciences, Krems, Austria
  4. 4Medical University of Vienna, Wien, Austria
  1. Correspondence to Dr Gerald Gartlehner, Department for Evidence-based Medicine and Evaluation, Danube University Krems, Krems, Austria; Gerald.Gartlehner{at}donau-uni.ac.at

Abstract

Objectives To assess the magnitude of reporting bias in trials assessing homeopathic treatments and its impact on evidence syntheses.

Design A cross-sectional study and meta-analysis. Two persons independently searched Clinicaltrials.gov, the EU Clinical Trials Register and the International Clinical Trials Registry Platform up to April 2019 to identify registered homeopathy trials. To determine whether registered trials were published and to detect published but unregistered trials, two persons independently searched PubMed, Allied and Complementary Medicine Database, Embase and Google Scholar up to April 2021. For meta-analyses, we used random effects models to determine the impact of unregistered studies on meta-analytic results.

Main outcomes and measures We report the proportion of registered but unpublished trials and the proportion of published but unregistered trials. We also assessed whether primary outcomes were consistent between registration and publication. For meta-analyses, we used standardised mean differences (SMDs).

Results Since 2002, almost 38% of registered homeopathy trials have remained unpublished, and 50% of published randomised controlled trials (RCTs) have not been registered. Retrospective registration was more common than prospective registration. Furthermore, 25% of primary outcomes were altered or changed compared with the registry. Although we could detect a statistically significant trend toward an increase of registrations of homeopathy trials (p=0.001), almost 30% of RCTs published during the past 5 years had not been registered.

A meta-analysis stratified by registration status of RCTs revealed substantially larger treatment effects of unregistered RCTs (SMD: −0.53, 95% CI −0.87 to −0.20) than registered RCTs (SMD: −0.14, 95% CI −0.35 to 0.07).

Conclusions Registration of published trials was infrequent, many registered trials were not published and primary outcomes were often altered or changed. This likely affects the validity of the body of evidence of homeopathic literature and may overestimate the true treatment effect of homeopathic remedies.

Data availability statement

Data are available upon reasonable request. Data available upon reasonable request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Summary box

What is already know about this subject?

  • Because homeopathy is exempt from most regulatory requirements, any assessment of the effectiveness must rely on published evidence.

  • Between 2000 and 2013, only 46% of registered randomised controlled trials (RCTs) on homeopathy were published.

  • The proportion of published but unregistered RCTs on homeopathy is unknown.

What are the new findings?

  • Large proportions of homeopathy trials remain unpublished (38%) or unregistered (50%).

  • 25% of registered trials altered or changed the primary outcome in the publication.

  • Unregistered homeopathy trials tended to have larger treatment effects than registered trials.

How might it impact clinical practice in the foreseeable future?

  • Clinicians need to be aware that published homeopathy trials represent only a selected proportion of research with mostly positive results.

  • Effect estimates of meta-analyses of homeopathy trials might substantially overestimate the true treatment effect of homeopathic remedies and need to be interpreted cautiously.

Introduction

Homeopathy, developed by Samuel Hahnemann in Germany almost 200 years ago, is largely inconsistent with current scientific concepts.1 For example, the principle of similarity (like cures like) and the claim that an increasing dilution of a substance leads to a stronger treatment effect (potentiation) lack evidence and contradict medical and physical principles.2Nevertheless, in many western countries, homeopathy’s popularity as a safe, holistic and comparable alternative to modern medicine persists. A systematic review reported that up to 9.2% of adults in mostly western countries have relied on homeopathic remedies during the past 12 months.3 According to a market research report, the global homeopathic product market reached US$5.5 billion in 2018 with most sales in North America and Europe.4

For years, sceptics and homeopathic practitioners have engaged in a fierce debate on whether homeopathic treatments are more effective than placebo.5–11 Advocates of homeopathy often refer to two systematic reviews and meta-analyses by Mathie et al that reported statistically significant differences of homeopathic treatments compared with placebo.12 13 According to Google Scholar, these reviews have been cited up to 200 times in other research publications. Sceptics counter that homeopathy’s effectiveness is no different from a placebo’s effectiveness when effectiveness is based on methodologically sound studies.14 The Australian National Health and Medical Research Council concluded that according to an assessment of 57 systematic reviews on 68 conditions, ‘there are no health conditions for which there is reliable evidence that homeopathy is effective’.15 Institutions in the European Union16 and the UK17 have issued similar statements that support this stance.

An important factor often missing from the debate, however, is that published RCTs of homeopathic treatments might not represent the totality of conducted scientific studies but rather only a selected proportion with positive results. This phenomenon, known as reporting bias, occurs when the publication or non-publication of studies or outcomes depend on the nature and direction of results, with statistically significant findings having a higher likelihood of publication than non-significant findings.18 Because statistical methods to detect or correct for reporting bias have limitations, meta-analyses of published studies like the ones from Mathie et al12 13 can lead to inflated and misleading results because positive trials are overrepresented.

To reduce reporting bias, public trial registries such as ClinicalTrials.gov in the USA and ClinicalTrialsRegister.eu in the European Union have been founded to promote the prospective registration of all trials. Since an amendment of the Declaration of Helsinki in 2008, prospective trial registration and publication of results are regarded as an ethical obligation of investigators.19 In 2005, the International Committee of Medical Journal Editors (ICMJE) adopted a policy that the journals they oversee would only publish results of clinical trials which have been prospectively recorded in a public registry.20 These measures have led to an increase in the number of clinical trials that are prospectively registered21; however, researchers are not obligated to publish results of such trials22 and the proportion of non-publication remains high.

Because regulatory agencies do not require proof of effectiveness for homeopathic products, little attention has been paid to the non-publication of homeopathy trials and its consequences. An assessment by Thomas et al reported that up to the year 2013, only 46% of registered homeopathic trials (16/35) were published.23 When no information is publicly available about the majority of homeopathic trials, sound conclusions about the efficacy and the risks of using homeopathic medicinal products for treating health conditions are impossible.24 25

To conduct a comprehensive assessment of the state of registration and publication of trials on homeopathy, our study had four objectives: (1) to determine the proportion of registered trials assessing homeopathy that remains unpublished, (2) to examine whether registered primary outcomes are consistent with published primary outcomes, (3) to assess the proportion of published randomised controlled trials (RCTs) on homeopathy that have been registered in a public clinical trial registry and (4) to gauge the impact of reporting bias on evidence syntheses of homeopathy trials.

Methods

For the purpose of this project, we adopted the following definition of a clinical trial by the ICMJE: ‘Any research project that prospectively assigns human subjects to intervention and comparison groups to study the cause-and-effect relationship between a medical intervention and a health outcome’.20

Identification of registered trials and corresponding publications

To identify registered trials assessing homeopathy, we searched the following trial registries up to 19 April 2021: Clinicaltrials.gov (https://clinicaltrials.gov), EU Clinical Trials Register (www.clinicaltrialsregister.eu) and the International Clinical Trials Registry Platform (ICTRP) (https://ictrptest.azurewebsites.net) of the WHO. The ICTRP is a meta-register, which draws data from 17 national trial registries.26

We included registrations of all trials (randomised or non-randomised; controlled or uncontrolled) that assessed the effectiveness of individualised or non-individualised homeopathic treatments for any medical condition in humans. A recruitment status of ‘completed’ or ‘unknown’ was eligible for inclusion, trials classified as ‘terminated’ or ‘withdrawn’ were excluded from analyses. To ensure that investigators had at least 2 years to publish their research, we adopted a cut-off date of 19 April 2019. WHO recommends that clinical trials are published within 2 years of completion.27 To assess whether trials were registered prospectively, during enrolment, or retrospectively, we compared the date of first submission of the registration with the date of first enrolment based on the ICTRP.

To determine whether registered trials were published, we used publication information in registries (eg, links to publications) or searched PubMed, Ovid Allied and Complementary Medicine Database (AMED), Embase.com and Google Scholar using the unique trial registration number, the title of the registered trial or the name of the principal investigator to detect publications. Two persons independently conducted searches. We classified trials that we detected as published manuscripts in a searchable electronic database as ‘published’. If we could not detect a publication that corresponded to the registration, we contacted the investigator named as the person responsible for the registration per email. Registration records for which we could not detect publications were classified as ‘unpublished’. In case a published study was later retracted, we classified it as unpublished.

Assessment of the consistency of primary outcomes

For each study, we identified the registered primary outcome and its point of time of assessment as reported in the registry. We compared this information with the primary outcome and its point of time of assessment in the publication. If a publication did not explicitly state the primary outcome, we viewed the outcome most prominently reported in the abstract and the results of the manuscript as the primary outcome. Two persons assessed the consistency of outcomes.

Identification of published trials and corresponding registrations

We limited our searches for published trials to RCTs that were published in 2002 or later. We applied 2002 as a cut-off because the first clinical trials registry (ClinicalTrials.gov) was established in 2000.28 With this cut-off, we allowed for 2 years to register trials before publication.

To identify published RCTs, we reviewed reference lists of four landmark systematic reviews by Mathie et al that assessed homeopathic treatments for any medical condition.12 13 29 30 and nine Cochrane reviews on homeopathy for specific indications.31–38 In addition, we conducted systematic literature searches from January 2013 to July 2021 to update the systematic reviews by Mathie et al.12 13 An experienced information specialist searched PubMed, Cochrane Central Register of Controlled Trials (Cochrane Library/Wiley), Embase.com, Ovid AMED, CORE-Hom (https://www.carstens-stiftung.de/datenbanken-zur-integrativen-medizin.html#/) and Scopus.com. Online supplemental table 1 presents the search strategy. We limited searches to humans and RCTs. Two investigators independently screened abstracts and full-text articles of the literature searches for eligibility. We resolved conflicts by discussion and consensus.

Supplemental material

To determine whether published RCTs were registered, we checked full-text publications for information about registration or searched Clinicaltrials.gov, EU Clinical Trials Register and ICTRP to detect registration records of each published RCT. The first two registries are the most commonly used registries in North America and Europe and the third registry, ICTRP, is a meta-register that includes national registries from a more economically diverse group of countries throughout the world. We classified trials as registered if we were able to detect a registraiton record that was dated earlier than the publication of the manuscript. Ethical approval numbers did not qualify as registration numbers.

Data extraction and data management

We organised data in Microsoft Excel (Office 365). Extracted data from registries included the unique registration number, the status of the registered trial (ongoing, completed and terminated), whether the registration was prospective or retrospective, and characteristics of the registered trials (eg, individualised or non-individualised homeopathy, intended sample size, primary outcome and point of time of assessment, type of control intervention and others).

Extracted data from published trials included author and publication information (unique trial registration number, digital object identifier and PubMed identification number), medical condition of the population, intervention and control characteristics, sample size and study design.

Quantitative analyses

Based on the total number of registered trials, we calculated the proportion of trials that were published or remained unpublished. Likewise, based on the total number of published RCTs assessing homeopathy, we determined the proportion of registered and unregistered RCTs. We assessed time trends of registrations and publications using a logistic regression predicting whether studies were published or registered depending on the year of publication or registration.

For the meta-analytic example, we used data by Mathie et al.13 We re-analysed the meta-analysis using a random effects model (DerSimonian and Laird) and standardised mean differences (SMDs) as the outcome measure. For our analyses, we used the same outcomes as Mathie et al who selected outcomes in the following hierarchical order based on availability in the publications: mortality, morbidity, health impairment, limitation of activity, restriction of participation and surrogate outcomes. We stratified the meta-analysis by registration status (registered vs unregistered RCTs).39 40

We conducted all statistical analyses with R V.4.1.0 (R Core Team, 2021).

Results

In the following sections, we first summarise the publication status of registered homeopathy trials and present the degree of consistency of primary outcomes in registries and publications. We then present the registration status of published homeopathy trials. Finally, we present an example illustrating the potential impact of missing registration of RCTs on results of a meta-analysis of homeopathy trials.

Publication status of registered homeopathy trials

Of 116 registered homeopathy trials, 90 met our inclusion criteria. Figure 1 presents the deduplicated search results and the number of included and excluded records. Online supplemental table 2 summarises registration characteristics and, if available, publication identifiers of included trials. Most registered trials were RCTs (72%) or uncontrolled trials (27%). A single study was a non-randomised controlled trial. The first clinical trial on homeopathy was registered in July 2003. We contacted 25 investigators for further information about the publication status of their registered studies, of whom 5 responded to our query.

Figure 1

Disposition of search results in clinical trial registries (created by authors). ICTRP, International Clinical Trials Registry Platform; RCT, randomised clinical trial.

Fifty-five registered trials reported that non-individualised homeopathy was the intervention of choice, 34 used individualised homeopathy and one trial did not report the type of homeopathy. Intended sample sizes ranged from 1 (n-of-1 RCT) to 906 participants, with a median sample size of 60. Forty-three trials (47.8%) were registered before the start of the study or during recruitment, 47 trials (52.2%) were registered retrospectively. Most trials were registered in ClinicalTrials.gov (66%).

Overall, 57.8% (n=52) of registered trials were published in a journal that was listed in PubMed, Embase or Google Scholar, 4.4% (n=4) were published as grey literature and 37.8% (n=34) of registered homeopathy trials remained unpublished. The grey literature publications were master theses that were available through university servers and could be detected when searching Google Scholar for the registration number. Of trials registered in 2008 (when the Declaration of Helsinki started to require trial registration and publication) or later (n=78), only 24 trials (30.7%) were prospectively registered and had published results, and thus adhered to the Declaration of Helsinki19 on ethical principles of medical research.

Figure 2 depicts the proportions of published and unpublished trials per year and the trend for publication of registered trials over time. In this graph, we classified grey literature publications as ‘published’. Between 2003 and 2018 (we did not detect eligible registrations for 2019), the probability of completed and subsequently published trials did not show a statistically significant increase over time (p for trend=0.817).

Figure 2

Proportions of published and unpublished registered trials with trend by registration year (created by authors).

Consistency of Primary Outcomes

Of the 56 registered and published trials (including the four trials published in the grey literature), 25.0% (n=14) modified or switched the primary outcome measure or the point of time of assessment. Most commonly, outcomes that were not registered as the primary outcomes were reported as such in the study publications. In four instances, the point of time of assessment was not the same as pre-specified in the register.

Registration status of published homeopathy trials

Our literature searches rendered 193 published RCTs assessing the efficacy or effectiveness of homeopathic treatments between 2002 and 2021 (figure 1). Online supplemental table 3 presents publication characteristics and registration status of included RCTs. Of 193 RCTs, 49.7% (n=96) had been registered (prospectively or retrospectively), for the remaining 50.3% (n=97), no registration could be detected. Between 2002 and 2021, the proportion of registered RCTs statistically significantly increased over time (p<0.001 for trend). Figure 3 depicts the proportions of registered and unregistered RCTs by year of publication and the trend of registration over time.

Figure 3

Proportions of registered and unregistered trials with trend by publication year (created by authors).

The impact of missing trial registration and selective publication on results of a meta-analysis

To illustrate the potential bias introduced by missing registration, we used data from a meta-analysis by Mathie et al13 as an example. This meta-analysis pooled data from 54 double-blinded, low or unclear risk of bias RCTs that compared non-individualised homeopathy with placebo for any medical indication in humans. Included studies were published between 1976 and 2014. The original meta-analysis by Mathie et al yielded a statistically significantly greater efficacy of homeopathy than placebo (SMD: −0.33, 95% CI −0.44 to – 0.21).13

For our example, we included the 19 RCTs,41–59 which were published between 2002 and 2014. We chose 2002 as a threshold because ClinicalTrials.gov was introduced in 2000. By 2002, investigators would have had 2 years to register a trial. 2014 was the end of literature search of the Mathie et al’s review. In our example, we conducted a stratified meta-analysis of these 19 RCTs by registration status: 6 RCTs54–59 were registered and 13 were not.41–53

A meta-analysis of unregistered RCTs yielded a statistically significant treatment effect favouring homeopathy (SMD: −0.53, 95% CI −0.87 to −0.20). By contrast, a meta-analysis of registered RCTs did not show a statistically significant difference between homeopathy and placebo (SMD: −0.14, 95% CI −0.35 to 0.07). Meta-regression revealed that the difference in effect sizes between registered and unregistered studies did not reach statistical significance (difference in SMDs: 0.39, 95% CI −0.09 to 0.87). Figure 4 presents the meta-analyses of registered and unregistered RCTs.

Figure 4

Random effects meta-analyses stratified by registered and unregistered RCTs (created by authors). RCTs, randomised controlled trials; SMD, standardised mean difference.

Discussion

Our study revealed a poor adherence of investigators assessing homeopathic treatments to the Declaration of Helsinki, which requires prospective registration of trials and dissemination of results. Almost 38% of registered homeopathy trials remained unpublished, and 50% of published RCTs had not been registered. Retrospective registration was more common than prospective registration. Furthermore, 25% of primary outcomes were altered or changed compared with the registry. Although we could detect a statistically significant trend toward an increase of registrations of homeopathy trials, almost 30% of RCTs published during the past 5 years had not been registered.

Overall, these findings suggest a concerning lack of scientific and ethical standards in the field of homeopathy and a high risk for reporting bias. The high proportions of unregistered or retrospectively registered trials indicate that publication frequently depends on the nature of results. Presumably, the number of unknown cases of unregistered and unpublished homeopathy trials is high. Findings also indicate that journals publishing homeopathy trials do not adhere to policies by the ICMJE, which demand that only registered RCTs should be published.20

The non-publication of trial results and selective outcome reporting, however, is not a phenomenon that is limited to homeopathy.60 Over the past years, numerous studies reported that, despite registration, large proportions of completed trials remained unpublished.22 61–65 For example, an assessment of 2132 registered clinical trials in Germany between 2009 and 2013 showed that 33% remained unpublished after 5 years.62 Likewise, of 4347 clinical trials conducted in academic centres in the USA, 34% remained unpublished.61

Homeopathy is often viewed as a safe alternative to pharmaceutical treatments. For the approval of pharmaceutical interventions, however, the industry is required to submit all trial data to regulatory agencies, regardless of the publication in the medical literature. Homeopathy, however, is exempt from most regulatory requirements with the consequence that no independent regulatory agency reviews individual patient data of trials, assesses statistical analyses methods or requests long-term follow-up data. Any assessment of the effectiveness of homeopathic treatments, therefore, must rely only on published evidence.

To the best of our knowledge, Thomas et al were the first who assessed the extent of non-publication of trials in the homeopathic literature.23 Their assessment covered the trial literature from 2002 to 2014 and was less comprehensive than ours. Overall, however, they reported similar findings as ours. In their sample, 46% of registered trials remained unpublished and in 25% of published homeopathy trials, the primary outcome measures had been switched or modified.23 The consistency of these results indicates that over almost a decade, the situation has not substantially improved

Consequently, reporting biases are likely to have a substantial impact on the estimated treatment effect of homeopathy. Using data from a highly cited meta-analysis of homeopathy RCTs, our example showed that unregistered trials yielded substantially larger treatment effects than registered trials. This finding is consistent with a meta-epidemiological study of Cochrane reviews by Dechartres et al.66 Based on 32 meta-analyses of various medical topics, unregistered or retrospectively registered trials showed larger treatment effects than prospectively registered trials (ratio of ORs=0.81, 95 % CI 0.65 to 1.02).

Our study has several limitations. First, although our searches covered 17 trial registries, chances are high that we missed records because of limited recall (ie, the ability of a search strategy to identify relevant records) in these registries. Another reason we may have missed registrations of published trials concerns the registries themselves. Time lags often occur between national registry entries and exportation od the information to ICTRP. And registry information is occasionally incomplete or unclear. Likewise, despite the considerable effort of two investigators to detect publications of registered trials, we might have overlooked some, particularly if they were available only as grey literature. Second, our example is based on a single meta-analysis on non-individualised homeopathic treatments that might not be applicable to individualised homeopathy. We attempted to use data from a meta-analysis on individualised homeopathy, but in this body of evidence only a single RCT was registered, therefore, making it impossible to conduct any meaningful analysis. Third, we excluded registered trials from our analyses that were categorised as ‘terminated’ or ‘suspended’. One could argue that this excludes trials that were terminated early for either benefit or harm. We deemed it unlikely, however, that homeopathy trials were terminated for these reasons. A more common reason for termination is the lack of recruitment of participants. For these reasons, we chose to exclude terminated studies as not to bias results toward a lower publication rate. Finally, it is conceivable that a substantial number of unregistered (and thus untraceable) homeopathy trials remain unpublished. Unfortunately, due to a lack of registration requirement for homeopathic trials, such trials remain impossible for researchers like us to detect. Without a required registration, the actual magnitude of non-publication of homeopathic trials is impossible to fully assess. Our findings are, therefore, likely an underestimation of the true extent of non-publication of homeopathy trials.

Overall, it is paramount that the homeopathic research community commits to the adherence of scientific standards and requires the registration of all trials assessing homeopathic treatments. Furthermore, journals publishing homeopathic research must adhere to basic standards of quality by adopting or adhering to ICMJE policies and only publishing the results of clinical trials, which have been prospectively recorded in a public registry.

Conclusions

Registration of published trials was infrequent, many registered trials were not published and primary outcomes were often altered or changed. This likely affects the validity of the body of evidence of homeopathic literature and may substantially overestimate the true treatment effect of homeopathic remedies.

Data availability statement

Data are available upon reasonable request. Data available upon reasonable request.

Ethics statements

Patient consent for publication

Ethics approval

This study does not involve human participants.

Acknowledgments

We would like to thank Liliya Ziganshina from Cochrane Russia for providing input on a Russian publication. We are also grateful to Petra Wellemsen for administrative support and to Dawn Gartlehner for editing and improving the readability of our manuscript.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Correction notice This article has been corrected since it first published. We were contacted by a reader who identified a problem in Supplementary Table 3. We contacted the authors who reported that, in error, they had uploaded the wrong version of the table. In checking the intended version of the table carefully, the authors found that they had misclassified six published trials as unregistered (when they were registered). Upon correction, this changes the total number of unregistered trials from 103 to 97 of the 193 trials. This changes the result of the proportion of unregistered studies from 53.4% to 50.3%; this does not materially alter the conclusions of the study. In the course of investigating the reader’s query BMJ also asked the authors to make clearer to readers how they defined registration of the study and this has been added to the text.

  • Contributors GG and AID planned the study and were responsible for quality assurance. MH, FLJ, JEG and JNN searched registries and extracted data. IK conducted literature searches. RE ran statistical analyses. GG drafted the manuscript and acts as the guarantor. All authors commented on and revised the draft manuscript.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.