Multisystem failure: the story of anti-influenza drugs
BMJ 2014; 348 doi: https://doi.org/10.1136/bmj.g2263 (Published 10 April 2014) Cite this as: BMJ 2014;348:g2263
- 1Cochrane Acute Respiratory Infections Group, 00187, Roma, Italy
- 2Department of Pharmaceutical Health Services Research, University of Maryland School of Pharmacy, Baltimore, USA
- Correspondence to: T Jefferson Jefferson.tom{at}gmail.com
For the past decade decisions makers worldwide have endorsed the use of neuraminidase inhibitors. They spent billions of pounds stockpiling the two anti-influenza drugs oseltamivir and zanamivir from the mid-2000s as part of a global effort to be prepared for an influenza pandemic. When the H1N1 pandemic emerged in 2009 the drugs were rolled out around the globe for treatment and prevention of influenza and its complications. Under this spotlight, we were asked to conduct a systematic review for Cochrane to update evidence on their efficacy. What should have been a routine review got complicated as the validity of a key study that underpinned the evidence on efficacy was unclear. Our three and half year battle for data has resulted in the drug manufacturers providing us with full clinical study reports and unveiled a story in which no party has taken full responsibility for ensuring the validity of the evidence underlying its decisions. We hope that the publication of our systematic review of the trials, alongside all the source clinical study reports,1 will change the way such decisions are made.
Emergence of problems
Officials have not unambiguously documented their reasons for stockpiling oseltamivir and zanamivir, but the decision seems to be based on the assumptions that the drugs would reduce hospital admissions and serious complications of influenza such as pneumonia by half and slow down the spread of the virus.2 3 4 Some of these assumptions were supported by a peer reviewed pooled analysis of 10 randomised trials of oseltamivir published in the Archives of Internal Medicine in 2003 by Kaiser and colleagues.5 Although this analysis seemed to be high quality science and formed a powerful scientific rationale for stockpiling,6 during our review in 2009 it became apparent that the data underlying it were largely unpublished and inaccessible to independent scrutiny. Roche, the manufacturer of oseltamivir, funded the Kaiser review, employed some of its authors, and had also sponsored the 10 trials. But for three and half years it refused to release the full clinical study reports despite a public pledge to do so made during the H1N1 “swine flu” outbreak of 2009.7 8
Clinical study reports, which are used in regulatory submissions, are comprehensive structured reports of industry sponsored trials that can run to hundreds or thousands of pages.9 They include the study protocol, statistical analysis plan, blank case report forms, and other appendices that provide important contextual information, such as certificates of analysis describing the content and physical appearance of the intervention and placebo.
Research for our 2009 review also highlighted inconsistencies in decision making. The Food and Drug Administration (FDA), which had access to the full clinical study reports, concluded on the product label that “Tamiflu has not been shown to prevent such complications [serious bacterial infections].” The European Medicines Agency (EMA), which had only partial reports, and another prominent US agency, the Centers for Disease Control and Prevention (CDC), came to the exact opposite conclusion—all apparently based on the same trials.10
In 2013, following a long running campaign by the BMJ and ourselves, Roche released full clinical study reports for what we believe are close to all of the oseltamivir studies it sponsored without any limitations on their use or access. GlaxoSmithKline also released the reports for studies of zanamivir, and we have now published our analysis of them.1
Unimpressive results
The results of our systematic review challenge some of the assumptions about these drugs. Although prophylactic use does reduce the risk of developing symptomatic influenza, because virus culture was not performed on all trial participants it is not clear whether this is because participants were not infected or because they had an asymptomatic infection. This is important because it is thought that infection is also spread by people with asymptomatic infection (www.accessdata.fda.gov/drugsatfda_docs/nda/2000/21-087SE1-002_review.pdf).
There is no trial evidence to show that the drugs had any effect on mortality: only 13 deaths occurred among the over 24 000 participants enrolled in the clinical trials (about 0.05%). Nor was there evidence that treatment with oseltamivir reduced the risk of hospital admission in adults (relative risk 0.92, 95% confidence interval 0.57 to 1.50) or children (1.92, 0.70 to 5.23); no data were collected for zanamivir. Like mortality, hospital admission was a rare event, and no trial protocol included either as an outcome of interest.
Although the Kaiser analysis claimed that oseltamivir reduces the risk of complications, the trials do not settle this question completely, particularly for pneumonia. Pneumonia is difficult to diagnose clinically, and no oseltamivir study report included a definition of pneumonia or any other complication, meaning that we (and possibly the trialists themselves) could not verify that the records of pneumonia or bronchitis were accurate. Most trials relied on participant self reporting. The lack of a definition is unsurprising because complications were not a specified outcome in the protocol for half of the trials. For the other half, complications were a secondary or tertiary outcome. Our meta-analysis of unverified pneumonia events suggests that oseltamivir reduces the risk in adults (relative risk 0.55, 0.33 to 0.99 (number needed to treat (NNT)=100, 95% CI 67 to 451)) but not in children (relative risk 1.06, 0.62 to 1.83). But as there were only 66 cases of pneumonia among 4452 participants, the results are not robust. To gauge oseltamivir’s possible effect against serious unverified pneumonia and other complications, we carried out an additional analysis of all secondary illnesses that led to hospital admission or withdrawal from the trial. With even fewer events (24), the result was not significant (0.91, 0.4 to 2.06).
Similar problems were present in the zanamivir trials, but we found no significant effect on unverified pneumonia (0.90, 0.58 to 1.40). Zanamivir reduced the risk of unverified bronchitis in adults (NNT=56, 36 to 155) but the reduction with oseltamivir was non-significant (relative risk 0.75, 0.56 to 1.01). Neither drug had a significant benefit on bronchitis in children.
Both drugs reduce the time to first alleviation of influenza symptoms by around half a day, but this was not the reason that they were stockpiled. Furthermore, in some zanamivir trials, there was no analysis according to whether infected participants took other drugs to relieve symptoms (mainly paracetamol), and the data are not reported in detail in the oseltamivir clinical study reports. Therefore the relative contribution of neuraminidase inhibitors versus rescue medications in alleviation of symptoms is unclear.
In short, the benefits of both drugs appear modest, and these need to be weighed against possible harms.
Data on harms
Concerns about harms have largely been absent from public discussions on oseltamivir, and the Kaiser analysis did not report the risk of harms. Our results show no excess rates of observed harms in adults treated with zanamivir. However, when used to treat influenza, oseltamivir increased the risk of nausea (relative risk 1.57, 1.14 to 2.15; number needed to harm (NNH)=28, 14 to 112) and vomiting (2.43, 1.75 to 3.38; NNH=22, 14 to 42). Prophylactic use increased the risk of headache (1.18, 1.05 to 1.33; NNH=32, 18 to 115) and psychiatric adverse events over the duration of follow-up (1.80, 1.05 to 3.08; NNH=94, 36 to 1538).1
Like complications, adverse event outcomes were not defined in the trials, so the quality of the data is variable. However, the prophylaxis study population included people of all ages with no influenza-like symptoms, providing a far better testing ground for learning about harms than treatment trials.
Multiple failures
It seems that every major player could have acted differently to ensure that the true picture was available sooner. If evidence played a role in government decisions to stockpile these drugs—and we would like to think it did—we need a rapid accounting of the judgments, evidence, and considerations that underlie the original decision as well as the present continued stockpiling.
Regulatory questions
The FDA, which was the only body to re-run the analyses and thoroughly review some clinical study reports, considered that the benefits of these drugs were “modest” 15 years ago (this adjective appears six times in a four page oseltamivir medical officer review document11). However, despite the statements to this effect in the product labels, the drugs were adopted enthusiastically. Today, one US public health agency, CDC, continues to suggest that neuraminidase inhibitors “may reduce” complications and death,12 although the FDA still makes only conservative conclusions.13
We might also ask whether regulators should approve drugs that they conclude are only modestly effective against placebo. Our reading of the FDA’s files on zanamivir suggests that the drug was approved not because of its performance but because it offered “an alternative therapeutic approach for an important public health problem” in a market where “current influenza treatment options [were] limited.”14
The European Medicines Agency approved oseltamivir without the the full dataset from Roche, which is not required as part of the European licensing process. The EMA also uses external experts to assess submissions for market approval, and the 2009 BMJ investigation found that two of the experts the EMA consulted for advice were featured in Roche promotional material. It is not known whether they disclosed this to the EMA.15 Competing interests should always be declared, but such problems could be avoided if the EMA was funded sufficiently to allow it to evaluate the evidence internally without relying on external experts.
Research methods and funding
Our investigation calls into question whether credible evidence synthesis should rely on peer reviewed publications. Current timelines and funding structures do not give sufficient resources for systematic reviewers to analyse detailed clinical study reports rather than short journal articles. We were fortunate to have received funding for our four year review, but there are many other therapeutic areas where in-depth, credible evidence synthesis is needed. Given the increasing public availability of clinical study reports,9 funders need to reconsider how they allocate resources towards evidence based medicine.
In 2013, Roche began funding the Multiparty Group for Advice on Science (MUGAS) to reanalyse the oseltamivir dataset, but industry funding raises questions about objectivity. At the first MUGAS meeting, Roche downplayed the importance of its trial data in settling the question about complications: “We didn’t ask physicians to actively look for complications … They simply reported them if they thought patients had, for example, sinusitis, otitis media, bronchitis, pneumonia, or other chest infections. … To be honest, we weren’t that stringent at the time.”16 Yet the Roche funded Kaiser analysis concluded, without qualification, that oseltamivir reduces such complications.
Lastly, the journals that published the studies have yet to correct the reporting biases that we have documented,17 and the editorial world has yet to tackle the failure of peer review to detect them.
Final word on influenza antivirals?
We hope public health bodies such as CDC and the World Health Organisation will consider our review’s conclusions and revise their recommendations. CDC’s endorsement of neuraminidase inhibitors is now based on analyses that include retrospective observational studies, many of which have substantial amounts of missing data and did not adjust for survivor bias (where those who die earlier have less opportunity to receive treatment).18 WHO considers oseltamivir important enough to place it on the list of essential medicines that should be universally available,19 20 and the many unproved assumptions about antiviral performance that supported pandemic plans largely remain unchanged. We need to act to make sure that future decisions are not made on incomplete data.
Story of neuraminidase inhibitors
Mid-late 1990s: Around 40 Roche sponsored randomised clinical trials of oseltamivir and 25 Glaxo-Wellcome sponsored randomised trials of zanamivir are completed
July 1999: FDA approves zanamivir for treatment of influenza
October 1999: FDA approves oseltamivir for treatment of influenza
November 2000: FDA approves oseltamivir for prophylaxis of influenza
June 2002: EMA approves oseltamivir for prophylaxis and treatment
November 2000: FDA tells Roche that its promotional material claiming a 45% reduction in secondary infections is “false or misleading”
2003: Kaiser and colleagues’ pooled analysis of 10 randomised clinical trials conducted before licensing in 1999 is published in Archives of Internal Medicine and concludes that oseltamivir reduces the risk of lower respiratory tract infections resulting in antibiotic use (55%; 4.6% v 10.3% for placebo) and hospital admissions (59% 1.7% v 0.7%) in adults aged 13 to 97 years5
2004-5: Governments around the world begin stockpiling oseltamivir spurred by fears of avian influenza H5N1
2005: UK and US pandemic contingency plans highlight the importance of antivirals in reducing the impact of a pandemic: Both state treatment will reduce hospital admissions by about 50% and decrease mortality, based on the Kaiser analysis2 3
January 2006: Cochrane review concludes that oseltamivir reduces complications such as pneumonia. The Kaiser 2003 paper drove the result in meta-analysis
March 2006: FDA approves zanamivir for prophylaxis of influenza
2009: Novel A/H1N1 influenza virus discovered to be spreading in North America. In June 2009 WHO declares A/H1N1 influenza a “pandemic”21
2009: Australian and UK governments commission rapid update of Cochrane reviews related to influenza
2009: Tip from a reader alerts the Cochrane team that only two of the 10 trials in the Kaiser analysis were published
2009: We request data from the authors of the Kaiser pooled analysis and the oseltamivir “pivotal” studies.22 23 The authors respond that they do not have the data and refer us to Roche
December 2009: Unable to (and unwilling to sign a confidentiality agreement with a secrecy clause necessary to) obtain the 10 trials’ raw data, we conclude that we no longer are sure whether oseltamivir reduces complications of influenza.24 A joint investigation by BMJ and Channel 4 News shows that one of the published trials had been ghostwritten and that the largest treatment trial of oseltamivir conducted (M76001) was presented as a conference abstract carrying the name of a professor who did not recollect being involved; it was never published in full15
December 2009: Roche promises to release full clinical study reports to legitimate investigators “within the coming days.”25 At the end of the month it releases 3195 pages of study report but none are complete
2010-2012: Cochrane team repeatedly requests the full clinical study reports
2011: A freedom of information request to the European Medicines Agency provides the Cochrane team with over 20 000 pages from 16 Roche oseltamivir clinical study reports. EMA has no data on zanamivir (the drug was approved at the national regulatory agency level). All but one oseltamivir report was incomplete
2012: We publish the interim version of our Cochrane review based on EMA’s incomplete clinical study reports and regulatory comments from the FDA. Our conclusions that there is no evidence the drugs reduce hospital admission and the evidence for or against a possible effect on complications is insufficient are dismissed by the CDC, WHO, and European Centre for Disease Prevention
February 2012: WHO refuses to answer our questions on the review process that led to the inclusion of oseltamivir on the essential medicines list (www.bmj.com/tamiflu/who)
February 2012: CDC refuses to answer our requests for clarification on what data its continued promotion of oseltamivir is based on (www.bmj.com/tamiflu/cdc)
October 2012: BMJ begins publishing Cochrane correspondence with Roche, EMA, CDC, and WHO at bmj.com/tamiflu as part of its Open Data campaign
2013: GSK suddenly decides that a contract on data use is no longer necessary to access the zanamivir studies and send the full reports for the 30 trials we requested
2013: Roche subsequently releases 77 full clinical study reports of Roche oseltamivir trials
2013: Roche begins funding the Multiparty Group for Advice on Science (MUGAS) to reanalyse the oseltamivir dataset
April 2014: We publish our updated review based solely on full clinical study reports and regulatory documents
Notes
Cite this as: BMJ 2014;348:g2263
Footnotes
Competing interests: Both authors have read and understood the BMJ Group policy on declaration of interests and declare that they have applied for and received competitive research grants. Both authors are co-recipients of the NIHR grant to carry out the linked Cochrane review. TJ receives royalties from his books published by Blackwells and Il Pensiero Scientifico Editore and is occasionally interviewed by market research companies about phase 1 or 2 pharmaceutical products. In 2011-2013 TJ was an expert witness in a litigation case related to oseltamivir and in a labour case on influenza vaccines in health care workers in Canada. He has acted as consultant for Roche, GSK, and Sanofi-Synthelabo and was a consultant for IMS Health in 2013. PD received €1500 from the European Respiratory Society in support of his travel to the society’s September 2012 annual congress in Vienna, where he gave an invited talk on oseltamivir. He is an associate editor of the BMJ.
Provenance and peer review: Commissioned; externally peer reviewed.