The article by Gartlehner et al. [1] is interesting because it allows the homeopathic community to elaborate on potential publication bias in clinical trials of homeopathy. There are, however, several questionable elements: in the article, and in the announcement made on the BMJ web, it is concluded that there was a high proportion of trials not preregistered, but at the same time Gartlehner acknowledges in the press that over time there has been a substantial improvement in the preregistration of trials [2]; it is mentioned that homeopaths must improve, but at the same time it is implied that "homeopathy cannot work".
On the second point, it is worth mentioning that in the article Gartlehner et al cite two trials, one by Grimes [3] and the other by Grams [4]. These essays are based on a biased selection of literature and have elementary errors. For example, Grimes says that Jacques Benveniste's famous study was published in "1987" and that Madaleine Ennis' work was negative when in fact it was positive [5]. Grimes bases his conclusions on theoretical claims (a simple calculation of Avogadro's constant) and not on experimental studies that at the time were available (e.g. [6]). Grams, on the other hand, only cites some old articles from 1992 and 1993 without mentioning more recent studies (e.g. [7]).
References:
1. Gartlehner G, Emprechtinger E, Hackl M, Gartlehner J, Nonninger J, et al. (2022). Assessing the magnitude...
The article by Gartlehner et al. [1] is interesting because it allows the homeopathic community to elaborate on potential publication bias in clinical trials of homeopathy. There are, however, several questionable elements: in the article, and in the announcement made on the BMJ web, it is concluded that there was a high proportion of trials not preregistered, but at the same time Gartlehner acknowledges in the press that over time there has been a substantial improvement in the preregistration of trials [2]; it is mentioned that homeopaths must improve, but at the same time it is implied that "homeopathy cannot work".
On the second point, it is worth mentioning that in the article Gartlehner et al cite two trials, one by Grimes [3] and the other by Grams [4]. These essays are based on a biased selection of literature and have elementary errors. For example, Grimes says that Jacques Benveniste's famous study was published in "1987" and that Madaleine Ennis' work was negative when in fact it was positive [5]. Grimes bases his conclusions on theoretical claims (a simple calculation of Avogadro's constant) and not on experimental studies that at the time were available (e.g. [6]). Grams, on the other hand, only cites some old articles from 1992 and 1993 without mentioning more recent studies (e.g. [7]).
References:
1. Gartlehner G, Emprechtinger E, Hackl M, Gartlehner J, Nonninger J, et al. (2022). Assessing the magnitude of reporting bias in trials of homeopathy: a cross-sectional study and meta-analysis. BMJ Evidence-Based Medicine; 27(1): 1-20.
2. Doheny K. (2022). Homeopathy: do ‘cherry-picked?’ studies exaggerate benefits? WebMD. Available in: https://www.webmd.com/balance/news/20220401/homeopathy-benefits-may-be-e...
3. Grimes R. (2012). Proposed mechanisms for homeopathy are physically impossible. Focus on Alternative and Complementary Therapies; 17(3): 149-155.
4. Grams N. (2019). Homeopathy – where is the science? A current inventory on a pre-scientific artifact. EMBO Reports; 20(3): 1-50.
5. Belon P, Cumps J, Ennis M, Mannaioni P, Sainte J, et al. (1999). Inhibition of human basophil degranulation by successive histamine dilutions: results of a European multicentre trial. Inflammation Research; 48 (S1): 17–18.
6. Demangeat L. (2009). NMR water proton relaxation in unheated and heated ultrahigh aqueous dilutions of histamine: evidence for an air-dependent supramolecular organization of water. Journal of Molecular Liquids; 144(1): 32-39.
7. Laudy S & Belon P. (2009). Inhibition of basophil activation by histamine: a sensitive and reproducible model for the study of the biological activity of high dilutions. Homeopathy; 98(4): 186-197.
Conflict of Interest: None. I am not a homeopath, but I have researched the subject of homeopathy from a social and historical point of view.
Gartlehner et al (1) concluded that the effects of homeopathic clinical trials may be overestimated due to publication bias. Such conclusions are inaccurate based on their own statement and their evaluation of the data they investigated. The authors asserted, “the difference in effect sizes between registered and unregistered studies did not reach statistical significance.” Despite this clear statement of what their data showed, the researchers instead came to a different conclusion that sought to question the integrity of research results with homeopathy.
To their credit, these authors acknowledge that the problem of “non-publication of trial results and selective outcome reporting …. is not a phenomenon that is limited to homeopathy.” And yet, they purposefully chose to not reference any literature that evaluated this problem in publication bias from clinical trails testing conventional medicine. A simple review of the literature would find that conventional medical trials have at least the same rate of publication bias as those reported upon that tested homeopathic medicines (2), to reviews of research that showed a much higher level of publication bias when reporting on conventional medical treatments (3).
The fact is that several media (4)(5) that have reported on this study have come to the mistaken conclusion that the results of homeopathic clinical trials are not to be trusted, and this biased conclusion stems from the Gartlehner articl...
Gartlehner et al (1) concluded that the effects of homeopathic clinical trials may be overestimated due to publication bias. Such conclusions are inaccurate based on their own statement and their evaluation of the data they investigated. The authors asserted, “the difference in effect sizes between registered and unregistered studies did not reach statistical significance.” Despite this clear statement of what their data showed, the researchers instead came to a different conclusion that sought to question the integrity of research results with homeopathy.
To their credit, these authors acknowledge that the problem of “non-publication of trial results and selective outcome reporting …. is not a phenomenon that is limited to homeopathy.” And yet, they purposefully chose to not reference any literature that evaluated this problem in publication bias from clinical trails testing conventional medicine. A simple review of the literature would find that conventional medical trials have at least the same rate of publication bias as those reported upon that tested homeopathic medicines (2), to reviews of research that showed a much higher level of publication bias when reporting on conventional medical treatments (3).
The fact is that several media (4)(5) that have reported on this study have come to the mistaken conclusion that the results of homeopathic clinical trials are not to be trusted, and this biased conclusion stems from the Gartlehner article that made conclusions that didn’t arise from their data.
It seems that these authors themselves are showing evidence of their own reporting bias and of their own avoidance of reporting on evidence that shows that conventional medical research may be as or more guilty of this problem. By maintaining conclusions that are not consistent with their finding that the effect size between registered and non-registered studies did not achieve statistical significance, and then deciding to avoid referencing how common this problem is in conventional medical research, these researchers seem to be showing “bad faith,” in other words, a deliberate intent to mislead.
(1) Gartlehner G, Emprechtinger R, Hackl M, et al. Assessing the magnitude of reporting bias in trials of
homeopathy: a cross-sectional study and meta-analysis BMJ Evidence-Based Medicine. Epub ahead of print: 15 March 2022. doi:10.1136/bmjebm-2021-111846.
(2) Kicinski M, Springate DA, Kontopantelis E. Publication bias in meta-analyses from the Cochrane Database of Systematic Reviews. Stat Med. 2015 Sep 10;34(20):2781-93. doi: 10.1002/sim.6525. Epub 2015 May 18. PMID: 25988604. https://pubmed.ncbi.nlm.nih.gov/25988604/
(3) Decullier E, Lhéritier V, Chapuis F. Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ. 2005;331(7507):19. doi:10.1136/bmj.38488.385995.8F https://www.ncbi.nlm.nih.gov/pmc/articles/PMC558532/
The new study by Gartlehner et al. (1) claims that the benefits of homeopathy may have been over-estimated due to high levels of reporting bias. However, as this problem is well-known to affect all areas of medical research, context is everything.
Although the authors state that, “non-publication of trial results and selective outcome reporting …. is not a phenomenon that is limited to homeopathy”, they failed to provide adequate context for their results by making any direct comparison to other areas of clinical research. Homeopathy is arguably out-performing conventional medicine, or, at the very least, has comparable levels of reporting bias. Using representative examples of high-impact studies on reporting bias across all medical fields, when compared with the data presented by Gartlehner et al.(1) it is clear that:
1) half of all registered clinical trials (2) in conventional medicine fail to report their results within 12 months; whereas 62% of all registered homeopathy trials reach publication, and
2) inconsistencies in reporting of primary outcome (3) occur in 43% of conventional medical studies; whilst this happens in only 25% of published homeopathy trials.
The potential impact of unregistered/unpublished results on estimates of treatment effects is well known (4), yet for homeopathy, according to Gartlehner et al.(1), the impact may be minimal, or nothing at all: “the difference in effect sizes between registered and unregistered stud...
The new study by Gartlehner et al. (1) claims that the benefits of homeopathy may have been over-estimated due to high levels of reporting bias. However, as this problem is well-known to affect all areas of medical research, context is everything.
Although the authors state that, “non-publication of trial results and selective outcome reporting …. is not a phenomenon that is limited to homeopathy”, they failed to provide adequate context for their results by making any direct comparison to other areas of clinical research. Homeopathy is arguably out-performing conventional medicine, or, at the very least, has comparable levels of reporting bias. Using representative examples of high-impact studies on reporting bias across all medical fields, when compared with the data presented by Gartlehner et al.(1) it is clear that:
1) half of all registered clinical trials (2) in conventional medicine fail to report their results within 12 months; whereas 62% of all registered homeopathy trials reach publication, and
2) inconsistencies in reporting of primary outcome (3) occur in 43% of conventional medical studies; whilst this happens in only 25% of published homeopathy trials.
The potential impact of unregistered/unpublished results on estimates of treatment effects is well known (4), yet for homeopathy, according to Gartlehner et al.(1), the impact may be minimal, or nothing at all: “the difference in effect sizes between registered and unregistered studies did not reach statistical significance”. Therefore, it is surprising that the authors claim that Dr Mathie’s “landmark meta-analyses”, used as the starting point for their analysis, “might substantially overestimate the true treatment effect of homeopathic remedies and need to be interpreted cautiously”. A thorough examination of their study reveals that their data do not support this claim.
While attempts have been made to use this new study to undermine the evidence base in homeopathy, claiming “poor research practice” (5), such claims are entirely unfounded. Reporting bias occurs in all areas of medical research, so, unsurprisingly, it occurs in homeopathy research too. Contrary to these authors’ claims, the clinical evidence base in homeopathy does not need more “cautious interpretation” than any other scientific evidence.
1. Gartlehner G et al. Assessing the magnitude of reporting bias in trials of homeopathy: a cross-sectional study and meta-analysis. BMJ Evidence-Based Medicine, 2022; eFirst
2. Goldacre B et al. Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource. BMJ, 2018;362:k3218
3. Shah K et al. Outcome reporting bias in Cochrane systematic reviews: a cross-sectional analysis. BMJ Open, 2020;16;10:e032497.
4. Chen T et al. Comparison of clinical trial changes in primary outcome and reported intervention effect size between trial registration and publication. JAMA, 2019; 2(7):e197242
5. https://www.bmj.com/company/newsroom/poor-research-practice-suggests-tru...
In their recent paper, Gartlehner et al [1] reached the headline conclusion that ‘effect estimates of meta-analyses of homeopathy trials might substantially overestimate the true treatment effect of homeopathic remedies’. Their conclusion is based on having re-analysed one of the systematic review papers’ data published by Mathie et al [2] by taking into account the possible impact of a trial’s registration status. Gartlehner et al analysed a sub-set of 19 trials of non-individualised homeopathic treatment, comparing 6 trials that were registered with 13 trials that were not registered. They observed a statistically significant difference between homeopathy and placebo only for the non-registered trials; however, the difference in effect sizes between registered and non-registered trials did not reach statistical significance.
In conducting their re-analysis, Gartlehner et al have failed to recognise that the meta-analysis by Mathie et al [2] was primarily based on a sensitivity analysis of trials that comprised reliable evidence (effectively, low risk of bias): the effect-size estimate collectively for those 3 trials yielded a statistically non-significant result. Those 3 trials are amongst the 6 registered trials in Gartlehner’s re-analysis, and so it is no surprise that they contributed to a non-significant pooled effect size. A majority of the other 13 trials, now defined as non-registered [1], had previously been categorised by Mathie et al as high risk of bias...
In their recent paper, Gartlehner et al [1] reached the headline conclusion that ‘effect estimates of meta-analyses of homeopathy trials might substantially overestimate the true treatment effect of homeopathic remedies’. Their conclusion is based on having re-analysed one of the systematic review papers’ data published by Mathie et al [2] by taking into account the possible impact of a trial’s registration status. Gartlehner et al analysed a sub-set of 19 trials of non-individualised homeopathic treatment, comparing 6 trials that were registered with 13 trials that were not registered. They observed a statistically significant difference between homeopathy and placebo only for the non-registered trials; however, the difference in effect sizes between registered and non-registered trials did not reach statistical significance.
In conducting their re-analysis, Gartlehner et al have failed to recognise that the meta-analysis by Mathie et al [2] was primarily based on a sensitivity analysis of trials that comprised reliable evidence (effectively, low risk of bias): the effect-size estimate collectively for those 3 trials yielded a statistically non-significant result. Those 3 trials are amongst the 6 registered trials in Gartlehner’s re-analysis, and so it is no surprise that they contributed to a non-significant pooled effect size. A majority of the other 13 trials, now defined as non-registered [1], had previously been categorised by Mathie et al as high risk of bias [2]. If non-registration contributes to higher risk of bias, producing an overestimated effect size, then Gartlehner et al have discovered nothing new, and it is misleading that they have portrayed the Mathie et al paper [2] as one that could ‘substantially overestimate the true treatment effect of homeopathic remedies’ because it did not reflect the registration status of its analysed trials.
Moreover, the findings of another systematic review by Mathie et al (on clinical trials of individualised homeopathic treatment [3]) also remain unaffected by the Gartlehner et al paper. For the three trials with reliable evidence in that review, sensitivity analysis revealed a small but statistically significant effect size favouring homeopathy [3]. Gartlehner et al were unable to re-analyse these data effectively because of a paucity of registered trials – an observation that is also not surprising, since only 3 of 22 of those homeopathy trials were published after 2007, a year in which fewer than 40% of published trials in conventional medicine were registered [4]. Thus the headline findings from the two systematic reviews and meta-analyses published by Mathie et al [2, 3] stand intact.
References
1. Gartlehner G, Emprechtinger R, Hackl M, et al. Assessing the magnitude of reporting bias in trials of
homeopathy: a cross-sectional study and meta-analysis BMJ Evidence-Based Medicine. Epub ahead of print: 15 March 2022. doi:10.1136/bmjebm-2021-111846.
2. Mathie RT, Ramparsad N, Legg LA, et al. Randomised, double-blind, placebo-controlled trials of non-individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev 2017;6:63.
3. Mathie RT, Lloyd SM, Legg LA, et al. Randomised placebo-controlled trials of individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev 2014;3:142.
4. Wong EKC, Lachance CC, Page J, et al. Selective reporting bias in randomised controlled trials from two network meta-analyses: comparison of clinical trial registrations and their respective publications. BMJ Open 2019;9:e031138.
Thank you for your Letter and concerns raised about delays to registration and failure to identify similar systematic review protocols in PROSPERO. PROSPERO, which provides registration free of charge, now receives over 47,000 submissions annually including both new submissions and resubmissions (from authors whose original submission required revision owing to low quality or incomplete information). With only 1.7 FTE administrative staff funded to manage PROSPERO, demand for registration outstrips capacity of staff to process requests.
Previous appeals to PROSPERO users to improve the quality of submission and thus reduce the need for query, feedback and resubmission and consequently demand on the register were unsuccessful. With the intention of reducing further delays, in 2020, a decision was made to automatically publish records waiting for more than 30 days from submission, providing they pass a series of automatic checks (assessing whether a record is submitted in English language and includes information about review methodology). This allowed us to focus efforts on supporting the research endeavour surrounding the global COVID-19 pandemic. We aim to publish reviews related to covid by the end of the next working day. As this has reduced the time to registration and been met with approval by the review community, we have decided to continue automatic publishing system. Future development plans involve exploring how a more detailed automatic checking process...
Thank you for your Letter and concerns raised about delays to registration and failure to identify similar systematic review protocols in PROSPERO. PROSPERO, which provides registration free of charge, now receives over 47,000 submissions annually including both new submissions and resubmissions (from authors whose original submission required revision owing to low quality or incomplete information). With only 1.7 FTE administrative staff funded to manage PROSPERO, demand for registration outstrips capacity of staff to process requests.
Previous appeals to PROSPERO users to improve the quality of submission and thus reduce the need for query, feedback and resubmission and consequently demand on the register were unsuccessful. With the intention of reducing further delays, in 2020, a decision was made to automatically publish records waiting for more than 30 days from submission, providing they pass a series of automatic checks (assessing whether a record is submitted in English language and includes information about review methodology). This allowed us to focus efforts on supporting the research endeavour surrounding the global COVID-19 pandemic. We aim to publish reviews related to covid by the end of the next working day. As this has reduced the time to registration and been met with approval by the review community, we have decided to continue automatic publishing system. Future development plans involve exploring how a more detailed automatic checking process can be implemented. Furthermore , we intend to reduce the time from receipt to automatic publication from 30 days to 10 days.
To maximise effective searching and prevent similar records being missed, PROSPERO is also exploring automatic identification of similar reviews pre-submission and a requirement for authors to explain how their proposed review is different and/or needed. This may help to reduce duplication and consequently, research waste. We hope these developments will also improve user experience and the quality of our service.
Yours sincerely,
Ruth Walker and Connor Evans on behalf of the PROSPERO team.
We fully agree that „non-publication of trial results and selective outcome reporting…is not a phenomenon that is limited to homeopathy.”
Previous reviews in conventional medicine, such as the study by Kosa et al. in 2018, report „…substantive disagreement in reporting between publications and current clinical trial registry, which were associated with several study characteristics”.[1]
In 2019 The Lancet commented on the reporting of clinical trial data for 30 European universities that sponsor the largest number of trials governed by EU clinical trials regulation: “The report shows that 778 (83%) of 940 clinical trials sponsored by these universities due to post their results on the EU Clinical trials Register (EudraCT) had not done so”.[2]
The International Committee of Medical Journal Editors (ICMJE) announced in 2005 that “… trials that begin enrolment of patients after 1 July 2005 must register in a public trials registry at or before the onset of enrolment to be considered for publication …”.[3] EU rules took effect in 2014, which require all clinical trials registered in EudraCT to post summary results within 12 months of study completion.[2] Hence, the inclusion of studies on homeopathy published before and in 2005 by Gartlehner et al. 2022 does not seem reasonable respectively of those published before and in 2014 is debatable.
Notwithstanding the above, precise information on sub-groups of studies was not given by Gartlehner et al. 202...
We fully agree that „non-publication of trial results and selective outcome reporting…is not a phenomenon that is limited to homeopathy.”
Previous reviews in conventional medicine, such as the study by Kosa et al. in 2018, report „…substantive disagreement in reporting between publications and current clinical trial registry, which were associated with several study characteristics”.[1]
In 2019 The Lancet commented on the reporting of clinical trial data for 30 European universities that sponsor the largest number of trials governed by EU clinical trials regulation: “The report shows that 778 (83%) of 940 clinical trials sponsored by these universities due to post their results on the EU Clinical trials Register (EudraCT) had not done so”.[2]
The International Committee of Medical Journal Editors (ICMJE) announced in 2005 that “… trials that begin enrolment of patients after 1 July 2005 must register in a public trials registry at or before the onset of enrolment to be considered for publication …”.[3] EU rules took effect in 2014, which require all clinical trials registered in EudraCT to post summary results within 12 months of study completion.[2] Hence, the inclusion of studies on homeopathy published before and in 2005 by Gartlehner et al. 2022 does not seem reasonable respectively of those published before and in 2014 is debatable.
Notwithstanding the above, precise information on sub-groups of studies was not given by Gartlehner et al. 2022. Hence, the conclusion, “This likely affects the validity of the body of evidence of homeopathic literature and may overestimate the true treatment effect of homeopathic remedies”, based on those studies with modified or switched primary outcome measure or the point of time of assessment, is not adequately justified.
Previous studies published in the BMJ which looked at reporting bias in all medical fields showed that (a) half of all registered clinical trials in conventional medicine fail to report their results within a 12 month period, whereas according to Gartlehner et al. 2022 62% of all registered homeopathy trials reach publication,[4] and that (b) inconsistencies in reporting of primary outcome occur in 43% of conventional medical studies, whilst according to Gartlehner et al. 2022 this occurs in only 25% of published homeopathy trials.[5]
Hence, the most interesting finding is that homeopathy is out-performing conventional medicine in this respect, with lower levels of reporting bias.
References:
1. Kosa SD, Mbuagbaw L, Debono VB, et al. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methological review. Contemp Clin Trials 2018;65:144-150. https://pubmed.ncbi.nlm.nih.gov/29287666/
2. Clinical trial registry reporting: a transparent solution needed. Lancet Oncol 2019; 20:741. https://www.thelancet.com/action/showPdf?pii=S1470-2045%2819%2930350-X
3. Kamran A. Compulsory registration of clinical trials. BMJ 2004;329:637. https://www.bmj.com/content/329/7467/637
4. Goldacre B, DeVito NJ, Heneghan C, et al. Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource. BMJ 2018;362:k3218. https://pubmed.ncbi.nlm.nih.gov/30209058/
5. Shah K, Egan G, Huan LN, et al. Outcome reporting bias in Cochrane systematic reviews: a cross-sectional analysis. BMJ Open 2020;10(3):e032497. https://pubmed.ncbi.nlm.nih.gov/32184303/
Michael Frass, M.D.
Em. Professor of Medicine
Medical University of Vienna,
Vienna, Austria
Menachem Oberbaum, MD, FFHom (Lond.)
The Center for Integrative Complementary Medicine
Shaare Zedek Medical Center
Jerusalem, Israel
Rapid Response, BMJ Evidence-Based Medicine
Re: Popp M, Kranke P, Meybohm P, et al. Evidence on the efficacy of ivermectin for covid-19: another story of apples and oranges. BMJ Evidence-Based Medicine Published Online First: 20 August 2021. doi: 10.1136/bmjebm-2021-111791
Ivermectin in Covid-19
Andrew Bryant MSc
Population Health Sciences Institute, Newcastle University
Baddiley-Clark Building, Richardson Road, Newcastle upon Tyne NE2 4AX, UK
Email: andy.bryant@ncl.ac.uk
Theresa A Lawrie MBBCh PhD
Edmund J Fordham PhD FInstP
EbMCsquared, a Community Interest Company
Northgate House, Upper Borough Walls, Bath BA1 1RG, UK
Scott Mitchell MBChB MRCS
Emergency Department, Princess Elizabeth Hospital, Guernsey
To the Editor
The sole substantive critique in this Letter[1] is the description of Bryant[2] et al. (hereafter “Bryant”) as a “bowl of colourful fruit salad”[1], because of the pre-specified comparison of “ivermectin” vs “no ivermectin”. Trials with (potentially) active comparators were indeed included. Reflection should show that any bias is conservatively against ivermectin. Helpful control interventions would dilute the apparent benefit of ivermectin, relative to inactive comparators exclusively. Efficacy will be understated, not overstated, with respect to controls.
Unsuspected active agents in ivermectin combination therapies might...
Rapid Response, BMJ Evidence-Based Medicine
Re: Popp M, Kranke P, Meybohm P, et al. Evidence on the efficacy of ivermectin for covid-19: another story of apples and oranges. BMJ Evidence-Based Medicine Published Online First: 20 August 2021. doi: 10.1136/bmjebm-2021-111791
Ivermectin in Covid-19
Andrew Bryant MSc
Population Health Sciences Institute, Newcastle University
Baddiley-Clark Building, Richardson Road, Newcastle upon Tyne NE2 4AX, UK
Email: andy.bryant@ncl.ac.uk
Theresa A Lawrie MBBCh PhD
Edmund J Fordham PhD FInstP
EbMCsquared, a Community Interest Company
Northgate House, Upper Borough Walls, Bath BA1 1RG, UK
Scott Mitchell MBChB MRCS
Emergency Department, Princess Elizabeth Hospital, Guernsey
To the Editor
The sole substantive critique in this Letter[1] is the description of Bryant[2] et al. (hereafter “Bryant”) as a “bowl of colourful fruit salad”[1], because of the pre-specified comparison of “ivermectin” vs “no ivermectin”. Trials with (potentially) active comparators were indeed included. Reflection should show that any bias is conservatively against ivermectin. Helpful control interventions would dilute the apparent benefit of ivermectin, relative to inactive comparators exclusively. Efficacy will be understated, not overstated, with respect to controls.
Unsuspected active agents in ivermectin combination therapies might contribute bias the other way, but could only bias a meta-analysis toward an illusory conclusion of ivermectin efficacy if the same adjunct were dominant among studies or patients. An effective “Adjunct X” (ivermectin presumed ineffective) and synergy between the two would not be resolved, but either way an effective therapy would have been demonstrated.
Specifically, in Bryant[2], the only candidate (cited in Popp et al.[3] – hereafter “Popp” – as grounds for exclusion) is doxycycline, which contributed just three included studies[4,5,6] coming nowhere near dominance in Bryant[2].
The principal flaw in Popp[3] is the exclusion of all prior data not conforming to their own post hoc specification, by which device most available evidence is simply disregarded. Tidy experimental design is no doubt to be welcomed, but wilful disregard of available but untidy evidence, merely because some thought may be required in interpretation, is intellectual laziness.
The insinuation[1] that Bryant did not perform “careful grading of the certainty of evidence” is baseless. The review team included three highly experienced systematic reviewers two of whom are guideline methodologists. GRADE criteria[7], and WHO guidance[8], were used to rate evidence certainty.
The further claims[1] that Bryant[2] “misuse[s] established evidence assessment tools as a guise for quality of evidence synthesis”[1] and tried “to create pseudo-trustworthiness” are unsupported by any evidence. Misleading information certainly abounds in social media[1] as well as journal opinion pieces[9,10]. However, Bryant[2] is a non-commissioned research paper that followed PRISMA[11] systematic review guidelines and has no motives other than disinterested humanitarian ones. In a learned journal, unsupported assertions of insincere motives should answer themselves.
However, Bryant[2] is a non-commissioned research paper that followed PRISMA[11] systematic review guidelines and has no motives other than disinterested humanitarian ones, following other reviews of the large body of evidence of ivermectin’s utility in Covid-19[12]. In a learned journal, unsupported assertions of insincere motives are best left to answer themselves.
Funding: None.
Conflicts of interest: AB TAL EJF and SM are co-authors of Ref [2] attacked in the Letter [1]. They were members of the British Ivermectin Recommendation Development (BiRD) panel at the “Evidence to Decision” event convened on 20 February 2021. TAL and AB were members of the steering group and did not vote. EJF and SM were ordinary members of the panel. BiRD is a public information activity managed by EbMCsquared, a non-profit Community Interest Company funded by public donations. EJF and SM are unpaid volunteers entirely without financial interest. EJF is a member of the Health Advisory and Recovery Team (HART), an unincorporated membership association with no financial or material interests in ivermectin or any other medical product. This work, and Ref [2], are not projects of HART, and are not funded or influenced in any way by them
References
[1] Popp M, Kranke P, Meybohm P, et al. Evidence on the efficacy of ivermectin for covid-19: another story of apples and oranges. BMJ Evidence-Based Medicine Published Online First: 20 August 2021. doi: 10.1136/bmjebm-2021-111791
[2] Bryant A, Lawrie TA, Dowswell T, et al. Ivermectin for prevention and treatment of covid-19 infection: a systematic review, meta-analysis, and trial sequential analysis to inform clinical guidelines. Am J Therap 2021; 28, e434–460. doi:10.1097/MJT.0000000000001402
[3] Popp M, Stegemann M, Metzendorf M-I, et al. Ivermectin for preventing and treating covid-19. Cochrane Database Syst Rev 2021;7:CD015017. doi: 10.1002/14651858.CD015017.pub2
pmid: http://www.ncbi.nlm.nih.gov/pubmed/34318930
[4] Hashim, H. A., Maulood, M. F., Rasheed, A. M., Fatak, D. F., Kabah, K. K. & Abdulamir, A. S. (2020). Controlled randomized clinical trial on using Ivermectin with Doxycycline for treating COVID-19 patients in Baghdad, Iraq. medRxiv preprint doi: 10.1101/2020.10.26.20219345
[5] Mahmud, R., Rahman, M. M., Alam, I., Ahmed, K. G. U., Kabir, A. H., Sayeed, S. J. B., et al. (2021). Ivermectin in combination with doxycycline for treating COVID-19 symptoms: a randomized trial. Journal of International Medical Research, 49, 030006052110135. doi: 10.1177/03000605211013550
[6] Chowdhury, A. T. M. M., Shahbaz, M., Karim, M. R., Islam, J., Guo, D. & He, S. (2020). A Randomized Trial of Ivermectin-Doxycycline and Hydroxychloroquine-Azithromycin therapy on COVID19 patients. Research Square preprint, doi: 10.21203/rs.3.rs-38896/v1
[7] GRADE Working Group 2020). GRADE 2020: Grading of Recommendations Assessment, Development and Evaluation (GRADE) https://www.gradeworkinggroup.org
[8] World Health Organization (2014). WHO handbook for guideline development. https://apps.who.int/iris/handle/10665/145714
[9] Parrish, A. G., Blockman, M., Cohen, K., Dawood, H., de Waal, R., Gray, A. L., Kredo, T., Leong, T. D., Nel, J., Rees, H. & Reubenson, G. (2021). Meta-analytic magic, ivermectin, and socially responsible reporting. South Africa Medical Journal, http://www.samj.org.za/index.php/samj/article/view/13373
[10] Garegnani, L. I., Madrid, E. & Meza, N. (2021). Misleading clinical evidence and systematic reviews on ivermectin for COVID-19. BMJ Evidence-Based Medicine, doi: 10.1136/bmjebm-2021-111678
[11] Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372. doi: 10.1136/bmj.n71. Accessed 22 July 2021.
[12] Kory, P., Meduri, G. U., Varon, J., Iglesias, J. & Marik, P. E. (2021). Review of the Emerging Evidence Demonstrating the Efficacy of Ivermectin in the Prophylaxis and Treatment of COVID-19. American Journal of Therapeutics, 28, e299-e318. doi: 10.1097/MJT.0000000000001377
There is a hypothesis that nicotine may have a protective effect in COVID-19 (1). I present two recent reports on the association between smoking and COVID-19 infection/progression, and then presented reports regarding another mechanism of the association.
Paleiron et al. conducted a cross-sectional study to investigate the association between smoking and COVID-19 (2). The adjusted odds ratios (ORs) (95% confidence intervals [CIs]) of current smokers and subjects aged over 50 years for the risk of developing Covid-19 were 0.64 (0.49-0.84) and 2.6 (1.2-6.9), respectively. Smoking presented a protective effect on the developing COVID-19.
Farsalinos et al. conducted a meta-analysis to examine the effects of current smoking on adverse outcomes among hospitalized COVID-19 patients (3). Pooled OR (95% CI) of current smokers against non-current smokers and against former smokers for adverse outcomes was 1.53 (1.06-2.20) and 0.42 (0.27-0.74), respectively. Smoking relates to the progression of clinical outcomes in hospitalized COVID-19 patients, although the reason of poor clinical outcomes in former smokers should be explored by further studies.
There is another hypothesis that lithium will limit SARS-CoV2 infections. Rudd presented a hypothesis that the repurposing of low-cost inhibitors of glycogen synthase kinase-3 (GSK-3) such as lithium will limit SARS-CoV2 infections by both reducing viral replication and potentiating the immune response against the vi...
There is a hypothesis that nicotine may have a protective effect in COVID-19 (1). I present two recent reports on the association between smoking and COVID-19 infection/progression, and then presented reports regarding another mechanism of the association.
Paleiron et al. conducted a cross-sectional study to investigate the association between smoking and COVID-19 (2). The adjusted odds ratios (ORs) (95% confidence intervals [CIs]) of current smokers and subjects aged over 50 years for the risk of developing Covid-19 were 0.64 (0.49-0.84) and 2.6 (1.2-6.9), respectively. Smoking presented a protective effect on the developing COVID-19.
Farsalinos et al. conducted a meta-analysis to examine the effects of current smoking on adverse outcomes among hospitalized COVID-19 patients (3). Pooled OR (95% CI) of current smokers against non-current smokers and against former smokers for adverse outcomes was 1.53 (1.06-2.20) and 0.42 (0.27-0.74), respectively. Smoking relates to the progression of clinical outcomes in hospitalized COVID-19 patients, although the reason of poor clinical outcomes in former smokers should be explored by further studies.
There is another hypothesis that lithium will limit SARS-CoV2 infections. Rudd presented a hypothesis that the repurposing of low-cost inhibitors of glycogen synthase kinase-3 (GSK-3) such as lithium will limit SARS-CoV2 infections by both reducing viral replication and potentiating the immune response against the virus (4). Murru et al. mentioned that lithium presented a clear antiviral activity demonstrated at preclinical level, which would remain the antiviral activity in clinical settings (5). Snitow et al. described that the molecular targets of lithium included the signaling kinase GSK-3 (6), and the therapeutic use of lithium by focusing on GSK-3 might be useful in patients with COVID-19 infections. In any case, clinical trials are needed for verifying the efficacy and safety of applying lithium.
References
1. Usman MS, Siddiqi TJ, Khan MS, Patel UK, Shahid I, Ahmed J, Kalra A, Michos ED. Is there a smoker's paradox in COVID-19? BMJ Evid Based Med. 2020 doi: 10.1136/bmjebm-2020-111492. [Epub ahead of print]
2. Paleiron N, Mayet A, Marbac V, Perisse A, Barazzutti H, Brocq FX, Janvier F, Bertrand D, Bylicki O. Impact of tobacco smoking on the risk of COVID-19.A large scale retrospective cohort study. Nicotine Tob Res 2021. doi: 10.1093/ntr/ntab004. [Epub ahead of print]
3. Farsalinos K, Barbouni A, Poulas K, Polosa R, Caponnetto P, Niaura R. Current smoking, former smoking, and adverse outcome among hospitalized COVID-19 patients: a systematic review and meta-analysis. Ther Adv Chronic Dis 2020;11:2040622320935765.
4. Rudd CE. GSK-3 Inhibition as a therapeutic approach against SARs CoV2: Dual benefit of inhibiting viral replication while potentiating the immune response. Front Immunol 2020;11:1638.
5. Murru A, Manchia M, Hajek T, Nielsen RE, Rybakowski JK, Sani G, Schulze TG, Tondo L, Bauer M; International Group for The Study of Lithium Treated Patients (IGSLi). Lithium's antiviral effects: a potential drug for CoViD-19 disease? Int J Bipolar Disord 2020;8(1):21.
6. Snitow ME, Bhansali RS, Klein PS. Lithium and therapeutic targeting of GSK-3. Cells 2021;10(2):255.
The article by Gartlehner et al. [1] is interesting because it allows the homeopathic community to elaborate on potential publication bias in clinical trials of homeopathy. There are, however, several questionable elements: in the article, and in the announcement made on the BMJ web, it is concluded that there was a high proportion of trials not preregistered, but at the same time Gartlehner acknowledges in the press that over time there has been a substantial improvement in the preregistration of trials [2]; it is mentioned that homeopaths must improve, but at the same time it is implied that "homeopathy cannot work".
On the second point, it is worth mentioning that in the article Gartlehner et al cite two trials, one by Grimes [3] and the other by Grams [4]. These essays are based on a biased selection of literature and have elementary errors. For example, Grimes says that Jacques Benveniste's famous study was published in "1987" and that Madaleine Ennis' work was negative when in fact it was positive [5]. Grimes bases his conclusions on theoretical claims (a simple calculation of Avogadro's constant) and not on experimental studies that at the time were available (e.g. [6]). Grams, on the other hand, only cites some old articles from 1992 and 1993 without mentioning more recent studies (e.g. [7]).
References:
Show More1. Gartlehner G, Emprechtinger E, Hackl M, Gartlehner J, Nonninger J, et al. (2022). Assessing the magnitude...
Gartlehner et al (1) concluded that the effects of homeopathic clinical trials may be overestimated due to publication bias. Such conclusions are inaccurate based on their own statement and their evaluation of the data they investigated. The authors asserted, “the difference in effect sizes between registered and unregistered studies did not reach statistical significance.” Despite this clear statement of what their data showed, the researchers instead came to a different conclusion that sought to question the integrity of research results with homeopathy.
To their credit, these authors acknowledge that the problem of “non-publication of trial results and selective outcome reporting …. is not a phenomenon that is limited to homeopathy.” And yet, they purposefully chose to not reference any literature that evaluated this problem in publication bias from clinical trails testing conventional medicine. A simple review of the literature would find that conventional medical trials have at least the same rate of publication bias as those reported upon that tested homeopathic medicines (2), to reviews of research that showed a much higher level of publication bias when reporting on conventional medical treatments (3).
The fact is that several media (4)(5) that have reported on this study have come to the mistaken conclusion that the results of homeopathic clinical trials are not to be trusted, and this biased conclusion stems from the Gartlehner articl...
Show MoreThe new study by Gartlehner et al. (1) claims that the benefits of homeopathy may have been over-estimated due to high levels of reporting bias. However, as this problem is well-known to affect all areas of medical research, context is everything.
Although the authors state that, “non-publication of trial results and selective outcome reporting …. is not a phenomenon that is limited to homeopathy”, they failed to provide adequate context for their results by making any direct comparison to other areas of clinical research. Homeopathy is arguably out-performing conventional medicine, or, at the very least, has comparable levels of reporting bias. Using representative examples of high-impact studies on reporting bias across all medical fields, when compared with the data presented by Gartlehner et al.(1) it is clear that:
1) half of all registered clinical trials (2) in conventional medicine fail to report their results within 12 months; whereas 62% of all registered homeopathy trials reach publication, and
2) inconsistencies in reporting of primary outcome (3) occur in 43% of conventional medical studies; whilst this happens in only 25% of published homeopathy trials.
The potential impact of unregistered/unpublished results on estimates of treatment effects is well known (4), yet for homeopathy, according to Gartlehner et al.(1), the impact may be minimal, or nothing at all: “the difference in effect sizes between registered and unregistered stud...
Show MoreIn their recent paper, Gartlehner et al [1] reached the headline conclusion that ‘effect estimates of meta-analyses of homeopathy trials might substantially overestimate the true treatment effect of homeopathic remedies’. Their conclusion is based on having re-analysed one of the systematic review papers’ data published by Mathie et al [2] by taking into account the possible impact of a trial’s registration status. Gartlehner et al analysed a sub-set of 19 trials of non-individualised homeopathic treatment, comparing 6 trials that were registered with 13 trials that were not registered. They observed a statistically significant difference between homeopathy and placebo only for the non-registered trials; however, the difference in effect sizes between registered and non-registered trials did not reach statistical significance.
In conducting their re-analysis, Gartlehner et al have failed to recognise that the meta-analysis by Mathie et al [2] was primarily based on a sensitivity analysis of trials that comprised reliable evidence (effectively, low risk of bias): the effect-size estimate collectively for those 3 trials yielded a statistically non-significant result. Those 3 trials are amongst the 6 registered trials in Gartlehner’s re-analysis, and so it is no surprise that they contributed to a non-significant pooled effect size. A majority of the other 13 trials, now defined as non-registered [1], had previously been categorised by Mathie et al as high risk of bias...
Show MoreThank you for your Letter and concerns raised about delays to registration and failure to identify similar systematic review protocols in PROSPERO. PROSPERO, which provides registration free of charge, now receives over 47,000 submissions annually including both new submissions and resubmissions (from authors whose original submission required revision owing to low quality or incomplete information). With only 1.7 FTE administrative staff funded to manage PROSPERO, demand for registration outstrips capacity of staff to process requests.
Previous appeals to PROSPERO users to improve the quality of submission and thus reduce the need for query, feedback and resubmission and consequently demand on the register were unsuccessful. With the intention of reducing further delays, in 2020, a decision was made to automatically publish records waiting for more than 30 days from submission, providing they pass a series of automatic checks (assessing whether a record is submitted in English language and includes information about review methodology). This allowed us to focus efforts on supporting the research endeavour surrounding the global COVID-19 pandemic. We aim to publish reviews related to covid by the end of the next working day. As this has reduced the time to registration and been met with approval by the review community, we have decided to continue automatic publishing system. Future development plans involve exploring how a more detailed automatic checking process...
Show MoreWe fully agree that „non-publication of trial results and selective outcome reporting…is not a phenomenon that is limited to homeopathy.”
Previous reviews in conventional medicine, such as the study by Kosa et al. in 2018, report „…substantive disagreement in reporting between publications and current clinical trial registry, which were associated with several study characteristics”.[1]
In 2019 The Lancet commented on the reporting of clinical trial data for 30 European universities that sponsor the largest number of trials governed by EU clinical trials regulation: “The report shows that 778 (83%) of 940 clinical trials sponsored by these universities due to post their results on the EU Clinical trials Register (EudraCT) had not done so”.[2]
The International Committee of Medical Journal Editors (ICMJE) announced in 2005 that “… trials that begin enrolment of patients after 1 July 2005 must register in a public trials registry at or before the onset of enrolment to be considered for publication …”.[3] EU rules took effect in 2014, which require all clinical trials registered in EudraCT to post summary results within 12 months of study completion.[2] Hence, the inclusion of studies on homeopathy published before and in 2005 by Gartlehner et al. 2022 does not seem reasonable respectively of those published before and in 2014 is debatable.
Show MoreNotwithstanding the above, precise information on sub-groups of studies was not given by Gartlehner et al. 202...
Test
Rapid Response, BMJ Evidence-Based Medicine
Re: Popp M, Kranke P, Meybohm P, et al. Evidence on the efficacy of ivermectin for covid-19: another story of apples and oranges. BMJ Evidence-Based Medicine Published Online First: 20 August 2021. doi: 10.1136/bmjebm-2021-111791
Ivermectin in Covid-19
Andrew Bryant MSc
Population Health Sciences Institute, Newcastle University
Baddiley-Clark Building, Richardson Road, Newcastle upon Tyne NE2 4AX, UK
Email: andy.bryant@ncl.ac.uk
Theresa A Lawrie MBBCh PhD
Edmund J Fordham PhD FInstP
EbMCsquared, a Community Interest Company
Northgate House, Upper Borough Walls, Bath BA1 1RG, UK
Scott Mitchell MBChB MRCS
Emergency Department, Princess Elizabeth Hospital, Guernsey
To the Editor
Show MoreThe sole substantive critique in this Letter[1] is the description of Bryant[2] et al. (hereafter “Bryant”) as a “bowl of colourful fruit salad”[1], because of the pre-specified comparison of “ivermectin” vs “no ivermectin”. Trials with (potentially) active comparators were indeed included. Reflection should show that any bias is conservatively against ivermectin. Helpful control interventions would dilute the apparent benefit of ivermectin, relative to inactive comparators exclusively. Efficacy will be understated, not overstated, with respect to controls.
Unsuspected active agents in ivermectin combination therapies might...
TEST
There is a hypothesis that nicotine may have a protective effect in COVID-19 (1). I present two recent reports on the association between smoking and COVID-19 infection/progression, and then presented reports regarding another mechanism of the association.
Paleiron et al. conducted a cross-sectional study to investigate the association between smoking and COVID-19 (2). The adjusted odds ratios (ORs) (95% confidence intervals [CIs]) of current smokers and subjects aged over 50 years for the risk of developing Covid-19 were 0.64 (0.49-0.84) and 2.6 (1.2-6.9), respectively. Smoking presented a protective effect on the developing COVID-19.
Farsalinos et al. conducted a meta-analysis to examine the effects of current smoking on adverse outcomes among hospitalized COVID-19 patients (3). Pooled OR (95% CI) of current smokers against non-current smokers and against former smokers for adverse outcomes was 1.53 (1.06-2.20) and 0.42 (0.27-0.74), respectively. Smoking relates to the progression of clinical outcomes in hospitalized COVID-19 patients, although the reason of poor clinical outcomes in former smokers should be explored by further studies.
There is another hypothesis that lithium will limit SARS-CoV2 infections. Rudd presented a hypothesis that the repurposing of low-cost inhibitors of glycogen synthase kinase-3 (GSK-3) such as lithium will limit SARS-CoV2 infections by both reducing viral replication and potentiating the immune response against the vi...
Show MorePages