The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias: Response to the Cochrane editors

Lars Jørgensen, LJ (lj@cochrane.dk), 1
Peter C. Gøtzsche, PCG (pcg@cochrane.dk), 1
Tom Jefferson, TJ (tj@cochrane.dk), 1

1Nordic Cochrane Centre, Rigshospitalet 7811, Tagensvej 21, 2100 Copenhagen, Denmark.

Summary

In a report uploaded on the Cochrane.org website on 3 September 2018 (1), Cochrane’s Editor in Chief and Deputy Editor in Chief responded to our analysis published in BMJ Evidence-Based Medicine on 27 July 2018 (2) of the Cochrane review of the HPV vaccines published on 9 May 2018 (3).

The Cochrane editors acknowledge (1) that our analysis (2) addresses the importance of the selection of data sources for reviews, and we hope that Cochrane will take the threat posed by reporting bias (4) more seriously by using clinical study reports, rather than journal publications.

The Cochrane editors claimed that we had “substantially overstated” our criticisms and they concluded that “Jørgensen et al made allegations that are not warranted and provided an inaccurate and sensationalized report of their analysis” (1).

Here we address the Cochrane editors’ findings and present our further assessment and additional findings.

In summary, we found that our analysis (2) was appropriate and that the Cochrane editors substantially ignored several of our criticisms (1):

1) The Cochrane editors’ cross referencing (1) with our HPV vaccine study index (5) showed that the Cochrane HPV review was incomplete and resulted in 8% additional eligible female participants (6,191/73,428). Due to the discrepancy with our analysis (2), we assessed our index again and found that the Cochrane HPV review should have included at least 35% (25,550/73,428) additional eligible females in its meta-analyses;
2) The Cochrane editors’ considerations on harms ignored several of our criticisms including the incomplete reporting of serious adverse events in several of the Cochrane HPV review’s included studies;
3) The Cochrane editors’ considerations of the trials’ adjuvant and vaccine comparators was ambiguous, opaque, inaccurate and ignored the fact that the studies only tested the vaccine antigens—not the vaccines;
4) The Cochrane editors’ response on the Cochrane HPV review’s included composite surrogate outcomes was superficial and did not consider the substantial bias and confounding that these outcomes involve;
5) The Cochrane editors’ assessment of the Cochrane HPV review authors’ conflicts of interest was incomplete and ignored several additional important conflicts of interest;
6) The Cochrane editors’ considerations on the media coverage did not recognize that it should be balanced and free from financial conflicts of interest;
7) The Cochrane editors’ response appeared to advocate scientific censorship, which we do not approve of;
8) In conclusion, our analysis (2) was appropriate, the Cochrane editors substantially ignored several of our criticisms (1) and the Cochrane review is still incomplete and ignores important evidence of bias.

1) The Cochrane editors’ cross referencing with our HPV vaccine study index

We used our index (5) to identify additional eligible studies for the Cochrane HPV vaccine review that included 73,428 women from 26 studies (3).

From our index (5), the Cochrane editors (1) identified:
1. “five [i.e., 5/26 = 19%] eligible completed studies with available data representing 5267 women [i.e., 5,267/73,428 = 7%, that] may have been missed from the Cochrane Review, as a consequence of the search being based on bibliographic databases rather than trials registers.”
2. One “Additional 9-valent [Gardasil 9] study NCT01047345,” adding 924 women to the numerator: 6,191/73,428 = 8% additional women.
3. “Five studies awaiting classification (not recruiting, but no results available) potentially relevant for the current Cochrane Review” that included 4,044 participants.
4. “Eight ongoing studies (actively recruiting, no results available) potentially relevant for the current Cochrane Review” that included 121,531 participants.

The Cochrane editors’ analysis (1) shows that the Cochrane HPV review was incomplete. In their “Appendix A”, the Cochrane editors updated 20 of the Cochrane HPV review’s meta-analyses (20/66, 30%) with the additional data and added seven meta-analyses of the HPV vaccine Gardasil 9 (1), but as of September 14 2018 they have not updated the Cochrane review itself with the additional data (3).

It is not clear why the Cochrane editors thought that our study index (5) “did not appear to identify any important eligible studies” (6). Our index was sent to the editors on 19 January 2018. The Cochrane review was published 110 days later on 9 May 2018 (3), but it seemingly took the editors only 25 days from launching their “investigation” on 9 August 2018 (6) to updating their Cochrane review on 3 September 2018 (7) with the missing studies.

Our initial assessment of the Cochrane HPV review’s included studies:
Initially, we had cross-referenced the study IDs from our index with the 26 included study IDs in the Cochrane HPV review’s “Appendix 6.1.1. Published reports included in the Cochrane review,” and found 20 studies not included in the review. For example, we did not find any of the 20 studies included in the Cochrane HPV review’s two serious adverse events analyses: “Figure 10” of journal publication data and “Analysis 7.6” of data that the Cochrane authors “considered to represent the most complete follow-up” (3). This led us to believe that the studies were not included in the Cochrane review. When we checked again, we found some of the studies in the review’s reference list (3). The Cochrane HPV reviewers chose to use idiosyncratic referencing with study IDs such as “Phase 2 trial (ph2,2v)”, “Immunobridging (ph3,2v)” and “CVT (ph3,2v), which made the study assessment complicated. For numbers of participants, we did not subtract the male participants that were included in three of the studies, as we should have done.

As stated in our index paper (5), our detective work involved a degree of uncertainty, as we did not want to dismiss any possibly eligible studies. Therefore, the index included a “possibly exist” category for studies for which we only had one verification source. Four studies in our analysis (2) had no numbers for randomised participants; these were “probably exist” studies. We have obtained additional information for three of these studies, but we are still not sure that the fourth study exists in clinical study report form, as the manufacturer (Merck) did not answer our request for this information (2,5).

In Table 1, we list the 20 studies (plus the additional one that the Cochrane editors identified: NCT01489527) that we identified as eligible but not included in the Cochrane HPV review.

Table 1: Our reassessment of the studies we had identified as additionally eligible for the Cochrane HPV vaccine review [*Key: RCT = randomised clinical trial]

See: https://blogs.bmj.com/bmjebmspotlight/files/2018/09/Cochrane-HPV-vaccine...

Our analysis reinforces the view that Cochrane HPV review is incomplete. We found an additional 25,550 females (and possibly up to 30,195 for the Cochrane HPV review’s serious adverse events meta-analyses) that are eligible for the Cochrane HPV review’s meta-analyses. Furthermore, we found freely available clinical study reports for 6 of the 21 studies on GlaxoSmithKline’s trial register, which the Cochrane authors used data from. Clinical study reports are far more reliable than published reports (4), in particular in relation to possible serious harms. It is therefore not merely the studies the authors of the Cochrane HPV review missed; they also missed benefits and harms data from the studies they included. In addition, the criteria that the Cochrane authors used for inclusion of data in their primary serious adverse events analysis (Analysis 7.6) are not clear: “The primary analysis for these outcomes included data that we considered to represent the most complete follow-up” (3).

2) The Cochrane editors’ considerations on harms
The Cochrane HPV review’s harms analyses ("Comparison 7”) include seven meta-analyses (four of which report injection site harms) where the three most clinically important ones—deaths (Analysis 7.7), serious adverse events (Analysis 7.6) and systemic adverse events (Analysis 7.5)—contain errors or are incomplete.

Deaths
The Cochrane HPV review’s authors found 5% (90 vs. 86) more deaths in “Analysis 7.7” (journal publication and registry data) vs. “Figure 11 (only journal publication data). We found that this discrepancy reflects the difference between the study FUTURE III’s (V501-019) journal publication and registry entry (8 vs. 12 deaths). We also found that the Cochrane HPV review gave an incorrect number of deaths for the VIVIANE study (HPV-015): 13 deaths in the HPV vaccine group and 5 deaths in the AlOH3 group; according to VIVIANE’s journal publication (8), there were 14 deaths in the HPV vaccine group and 3 deaths in the Al(OH)3 group.

The Cochrane authors state that “The deaths reported in the trials had an identified cause, and none were assessed to be due to vaccination” (3), but such judgements are biased, particularly in industry sponsored trials, and the analysis of deaths should be based on all events (4).
Serious adverse events

There are 11% (4,758/4,291) more serious adverse events in “Analysis 7.6” (journal publication and register data) compared to “Figure 10” (only journal publication data). Since there were more serious adverse events and more deaths in the register entry data, we wonder why the Cochrane authors did not include register entry data for all their outcomes.

We also wonder why three (possibly four) studies were not included in the Cochrane HPV review’s serious adverse events meta-analyses (see Table 1). The Cochrane editors write that we “claim that the [Cochrane] review authors made an error in their reporting of serious adverse events in relation to the PATRICIA [HPV-008] study. This is not the case” (1). We stated that “the Cochrane authors did not explain what the serious adverse events consisted of or whether some of them were more common in the HPV vaccine groups,” (2) and gave the example that “the PATRICIA trial publication only included two thirds (1400/2028) of the serious adverse events listed on ClinicalTrials.Gov” (2). The 2,028 individual serious adverse events listed in ClinicalTrials.Gov are listed with the total denominators of randomised women for PATRICIA, for example, “Headache: participants affected/at risk: 5/9319 [in the HPV vaccine group] (0.05%) vs. 1/9325 [in the hepatitis a vaccine group] (0.01%)” (9), suggesting that the numbers represent participants with serious adverse events.

The Cochrane editors did not consider our highly relevant observations about the incomplete reporting of serious harms (1). For example, we wrote that “FUTURE I, FUTURE II and FUTURE III, which in total included 21 441 women with up to 4 years follow-up, only reported serious adverse events occurring within 14 days post-vaccination” (2). The editors did not comment on how such reporting of serious adverse events for only about 3% of the trial periods (FUTURE I, II and III: [(14 days*3 vaccinations)/(365 days*4 years)]) resulted in the Cochrane authors’ judgements of “low risk of bias” for reporting bias (1).

Both the 1st Cochrane HPV review protocol from 2011 (10) and the 2nd protocol from 2013 (11) list the primary outcome of "serious adverse events observed after four weeks of administration of the vaccine during the trial" (emphasis added), i.e., an incomplete reporting of serious adverse events was already a criterion at protocol stage.
Systemic adverse events

The Cochrane editors (1) did not comment on our criticism of the lack of studies (2)—including PATRICIA (HPV-008)—from the Cochrane HPV reviews’ “Analysis 7.5: systemic adverse events” (3). Analysis 7.5 is incomplete—in particular, for Cervarix studies where the Cochrane review only included numbers for “solicited general adverse events” for two studies (3): HPV-009 and HPV-015, although data for such events are eligible from several additional studies, for example, HPV-001, HPV-008, HPV-013, HPV-029, HPV-030, HPV-033, HPV-035, HPV-038 and HPV-058. The inclusion of these studies could change the Cochrane review’s conclusion that “Systemic events with general mild symptoms were similarly frequent in vaccinated recipients and placebo or control vaccine recipients” (3). As we wrote, “On ClinicalTrials.gov, PATRICIA has 7129 vs 6557 systemic events listed under ‘Results: Other Adverse Events (General disorders)’, which in itself is a significantly increased risk: RR 1.09 (95% CI 1.07 to 1.11)” (2).

Furthermore, the Cochrane authors did not address that “solicited general adverse events [Cervarix]” only were reported 7-days post-vaccination and “systemic adverse events [Gardasil]” only 14-days post-vaccination (3).
Assessment of safety signals

The Cochrane editors did not comment on our safety signal section (1). Some potential HPV vaccine-related harms to the nervous system—or “autonomic dysfunction syndromes,” as the Cochrane editors described them (1)—have been reported (2). The Cochrane authors should have used trial register data to investigate such safety signals; for example, if they had summarised the nervous system disorders from PATRICIA’s (HPV-008) ClinicalTrials.gov list of serious adverse events (9), they would have found more serious nervous system disorders in the HPV vaccine arm: 39/9,319 vs. 25/9,325 in the hepatitis A vaccine arm, risk ratio 1.56 (95% CI 0.95 to 2.58). The Cochrane authors write that “All estimates of adverse effects in our review were restricted to those reported from randomised trials and therefore could not detect rare events, for which post-marketing surveillance, pharmacovigilance activities and linkage studies, joining vaccine and morbidity registries, are needed” (3), but the review’s results might have been different had the authors included serious adverse events on both an individual and organ system level.

Additional points on the Cochrane editors’ harms assessment
The Cochrane HPV review (3) did not include the following harms categories that were reported in the eligible studies’ clinical study reports and in some journal publications: “unsolicited adverse events” (Cervarix), “medically significant conditions” (Cervarix), “new onset chronic/auto-immune disease” (Cervarix) and “new medical history” (Gardasil), even though the Cochrane authors mention the two first categories (reported in “Angelo 2014”) (3).

3) The editors’ considerations of the trials’ adjuvant and vaccine comparators

The Cochrane editors stated that “The trials comparators [sic] were unambiguously, transparently, and accurately described” (1), but in the Cochrane HPV review’s “Plain language summary” intended for lay readers, the review authors state that “The risk of serious adverse events is similar in HPV and control vaccines (placebo or vaccine against another infection than HPV” (emphasis added) (3), and the word “placebo” is repeated throughout the review and all its meta-analyses, which make the review ambiguous, opaque and inaccurate, as no included trial in the review used a placebo comparator.

The WHO states that using adjuvant or another vaccine as comparators instead of placebo makes it difficult to assess the harms of a vaccine (12). The HPV vaccine trials’ adjuvant comparators—Merck’s amorphous aluminium hydroxyphosphate sulphate (AAHS) and GlaxoSmithKline’s aluminium hydroxide (Al[OH]3)—have not been tested against an inert comparator in human trials. The adjuvants’ clinical properties are largely unknown; they are not regulated on their own, as regulators do not regard them as “active ingredients” (13). For example, Merck’s AAHS has a confidential formula and its properties are variable from batch to batch and even within batches (14). Because the HPV vaccines and their adjuvants had similar harms profiles, the manufacturers and the regulators concluded that the HPV vaccines are safe. However, this is like saying that cigarettes and cigars must be safe because they have similar harms profiles.

In addition, in those trials with a non-HPV vaccine comparator, the HPV vaccine aluminium adjuvant was used in nearly all the non-HPV comparator vaccines; for example, PATRICIA’s hepatitis A (Havrix) comparator contains Al(OH)3 (only the studies HPV-032 and HPV-063 used a non-aluminium containing comparator: the hepatitis a vaccine Aimmugen). Thus, the presence of AAHS or Al(OH)3 in nearly all arms of the studies thwarted the harms assessment. The studies tested the vaccine antigens—not the vaccines.

The exclusion criteria of the Cochrane HPV review’s included trials

The Cochrane editors (1) did not consider our point that many of the Cochrane HPV review’s included studies had excluded female participants “if they had received the [aluminium] adjuvants before or had a history of immunological or nervous system disorders; for example, in the PATRICIA trial with 18 644 women and the FUTURE II trial with 12 167 women” (2). These exclusion criteria lower the external validity of the studies and suggest that there were concerns about harms caused in such participants by the adjuvants.

4) The Cochrane editors’ response on the Cochrane HPV review’s included outcomes

The Cochrane editors state that “The selection of outcomes for benefits was appropriate and was consistent with World Health Organization [WHO] guidance” (1).

In 2004, the WHO recommended the use of cervical intraepithelial neoplasia or worse: CIN2+, as the primary outcome (15). CIN2+ is a composite surrogate outcome for cervical cancer and includes CIN2, CIN3, AIS and cervical cancer. In 2014, the WHO recommended persistent HPV infection instead of CIN2+ (16). The WHO’s CIN2+ and persistent HPV infection recommendations were approved to “accelerate vaccine development and evaluation” (16). Since 2014, HPV vaccines have only been required to show benefits against persistent HPV infection for getting regulatory approval as a vaccine against HPV related cancer (16).

According to the 2004 WHO recommendations, “Representatives of industry did NOT participate in the drafting of recommendations” of the use of CIN2+ (15), but researchers with conflicts of interest did participate in the recommendations, for example, Ian Frazer—the co-inventor of the HPV vaccine who “receives royalties from sales of HPV prophylactic vaccines, and is a consultant for Merck, [and] GlaxoSmithKline” (17). In 2014, all 17 members of the WHO group that recommended persistent HPV infection instead of CIN2+ as the primary outcome had financial ties with the HPV vaccine manufacturers. For example, the group included two patent-holders of the HPV vaccines antigens (or “virus-like particles”), who are entitled “to a limited share of royalties [that] the NIH [National Institutes of Health] receives for these technologies” (16).

Outcomes such as CIN2+ can be difficult to interpret, and significant clinical differences can be hidden in the Cochrane HPV review’s meta-analyses (3). For example, as an extreme example, if there were 5 participants with CIN2+ in the HPV vaccine group and 10 in the comparator group, the 5 participants in the HPV vaccine group could theoretically all have cervical cancer while the 10 in the comparator group could have CIN2 lesions that often regress (18,19).

The Cochrane editors (1) did not address our point that the VIVIANE study (HPV-015) included in its register entry “one case of ‘Adenocarcinoma of the cervix’ and one case of ‘Cervix cancer metastatic’ … in the HPV vaccine group” (2), and that the Cochrane HPV review includes a death caused by “Cervix cancer metastatic” in the HPV vaccine group, which was not mentioned in the main text (3). The Cochrane editors (1) did not address our point that the “Cochrane review’s 26 trials mainly included women below age 30 and used frequent cervical screening (often every 6 months) that did not reflect real-life practice (often every 3–5 years),” which also lower the external validity of the studies (2).

The Cochrane HPV review’s primary analysis—Analysis 1.1 that includes four trials (CVT, FUTURE I, II and PATRICIA)—was of “High‐grade cervical lesions in hr[high risk]HPV DNA negative women at baseline: CIN2+ associated with HPV 16/18 [HPV types 16 and 18 are targeted by the HPV vaccines]”. Analysis 1.1 is affected by selection bias. Up to 15% of cervical cancers may not contain HPV (20), and many cervical cancers are infected with more than one HPV type. For example, in the clinical study report that we received for PATRICIA (HPV-008), 63 of the 102 of CIN2+ cases were co-infected with two or more HPV types. In PATRICIA, if an HPV vaccine and a comparator participant were both diagnosed with CIN2+ and positive for HPV types 31 and 33, and HPV 16/18, 31 and 33, respectively, the HPV vaccine CIN2+ case would be assigned as caused by non-vaccine types (31 and 33) and excluded from the analyses (such as Analysis 1.1), while the comparator case would be caused by HPV 16/18 (HPV vaccine types) and included in the analyses; even though HPV types 31 and 33 could have caused the CIN2+ lesions in both participants. The Cochrane HPV review’s analyses of HPV infection (“Comparison 4, 5 and 6”) include 21 meta-analyses that all analyse infection of HPV vaccine types (i.e., HPV types 6, 11, 16 and 18)—not infection of any HPV type, which would decrease confounding by HPV co-infection.

Another issue with Analysis 1.1 (3) is the large proportion of excluded females: 23,676 participants were included, but the included four studies (CVT, FUTURE I, II and PATRICIA) randomised 43,732 participants, so 46% females were excluded. The Cochrane authors did not mention that FUTURE I, II and PATRICIA—that contained 49% (36,266/73,428) of the Cochrane review’s sample (3)—were stopped early when HPV 16/18-related CIN2+ was significantly reduced for the HPV vaccine populations. Trials stopped early for benefits are known to exaggerate the effects by 29% on average compared to completed trials of the same intervention (21).

The majority (24 of 31) of the Cochrane HPV review’s meta-analyses of histological outcomes (“Comparison 1, 2 and 3”) consider cervical lesions associated with HPV vaccine types (3). A less biased meta-analysis of cervical lesions is “Analysis 3.7: High‐grade cervical lesions in women regardless of baseline HPV DNA status: Any CIN2+ irrespective of HPV types, at least 1 dose.” The Cochrane HPV review’s primary analysis—Analysis 1.1—is much more statistically significant than Analysis 3.7 (3): risk ratio 0.01 (95% CI 0.01 to 0.05) vs. 0.79 (95% CI 0.65 to 0.97), i.e., a ratio of relative risks of 0.01 (95% CI 0.006 to 0.03), which may reflect the selection bias in Analysis 1.1.

Both the 1st Cochrane HPV review protocol from 2011 (10) and the 2nd protocol from 2013 (11) list "Invasive cervical cancer" as a primary outcome. The protocols state that the Cochrane authors “will contact study authors or data owners to request data on the outcomes that were not reported” (11), which they did not do for invasive cervical cancer. Also, “If data are reported for grouped end points, we will contact trial authors or data owners to request data on the separated outcomes” (11), which the authors did not do for CIN2+ and CIN3+. But the Cochrane authors could have looked in the journal publications; for example, CIN3 irrespective of HPV type in intention-to-treat populations was reported in FUTURE I (22) (“79/2723 [in the HPV vaccine group] vs. 72/2732 [in the AAHS group]”) and FUTURE II (23) (“127/6087 [in the HPV vaccine group] vs. 161/6080 [in the AAHS group]”: in total 206/8,810 vs. 233/8,812; risk ratio 0.91 [95% CI 0.66 to 1.27]).

Furthermore, the Cochrane authors write that “No results were found for the outcomes any [sic] CIN3+ or AIS+ irrespective of HPV type” (3). If the Cochrane authors had looked in the freely available clinical study reports on GlaxoSmithKline’s trial register that the authors assessed, they would have found the outcomes CIN3+ irrespective of HPV type for PATRICIA (86/9,319 vs. 158/9,325) and HPV-032/063 combined (9/464 vs. 14/463).

5) The Cochrane editors’ assessment of the Cochrane authors’ conflicts of interest

The Cochrane editors state that “The review was compliant with Cochrane’s current conflict of interest policy” (1). If that is the case, we believe Cochrane should reconsider its policy.

The Cochrane HPV vaccines are expensive blockbuster vaccines generating billions of dollars of revenue (24), and the Cochrane review ought, therefore, to have been independent of any financial conflicts of interests.

The Cochrane editors are confident that the Cochrane authors have no relevant conflicts of interest (2). We do not agree. For example, the Cochrane HPV review’s first author, Professor Marc Arbyn—who, according to the Cochrane review (3), only “received travel grants from MSD‐Sanofi‐Pasteur and GSK, (ceased in 2008)”—was until 2008 on GlaxoSmithKline’s advisory board: “Marc Arbyn (GSK advisory Board (interrupted in 2008))”) (25); in 2011, “EUROGIN covered his [Marc Arbyn’s] travel and lodging expenses … EUROGIN conferences are financially supported by a range of pharmaceutical companies with an interest in cervical cancer” (26); in 2014, “Marc Arbyn's research unit at The Scientific Institute of Public Health received research support not exceeding 48,000 Euros from MSD-Sanofi Pasteur [co-manufacturer of Gardasil] for a surveillance study of the effects of HPV vaccination in Belgium (SEHIB study) ... His [Marc Arbyn’s] unit has also received research support from BD, Bio-Greiner, Abbot, and Cepheid for validation studies of HPV genotyping tests (through the VALGENT studies, valued at 21,000, 21,000 & 38,000€ respectively)” (27); and in 2018, Marc Arbyn is on the EUROGIN programme committee where Merck is a platinum sponsor (28). The Cochrane review’s last author, Dr. Markowitz is sponsored by Merck via Medscape (“sponsored by the manufacturer of the quadrivalent vaccine (“supported by an independent educational grant from Merck”) (29).

The Cochrane editors (1) do not think that the Costa Rica trial (“CVT”, aka HPV-009) was industry funded, and they refer to its publication in JAMA that states that the trial was "funded by the NCI (grant N01-CP-11005).” The editors write, with reference to JAMA, that “Vaccine was provided for our trial by GSK [GlaxoSmithKline] Biologicals, under a Clinical Trials Agreement with the NCI" (1). GlaxoSmithKline also provided support for aspects of the trial associated with regulatory submissions under “FDA BB-IND 7920” (30). We consider this industry funding.

6) The Cochrane editors’ assessment of the media coverage

The Cochrane editors (1) did not comment on our note that “Two of the experts had financial conflicts of interest with the HPV vaccine manufactures … [and that] No expert criticised the review” (2).

The editors (1) write that “press coverage could be made more explicit on our organizational websites and other communications, essentially noting that these opinions represent personal perspectives from a range of contributors and do not reflect the views or policies of Cochrane” (1). We agree, but stress that Cochrane’s press officer ought to only include researchers with no financial conflicts of interest.

7) The Cochrane editors appear to advocate scientific censorship

The Cochrane editors wrote that “Scientific debate is to be welcomed, and differences of opinion between different Cochrane 'voices' is not unexpected. However, public confidence may be undermined, unnecessary anxiety caused, and public health put at risk if that debate is not undertaken in an appropriate way. This is especially true when such debates take place in public. There is already a formidable and growing anti-vaccination lobby. If the result of this controversy is reduced uptake of the vaccine among young women, this has the potential to lead to women suffering and dying unnecessarily from cervical cancer.” We believe that our criticism of the Cochrane HPV review is appropriate and has general interest. We believe that providing an assessment of all the evidence reduces uncertainty and allows the public to make informed decisions based on the benefits and the harms of HPV vaccines. Debates over sources of evidence must take place in public, especially when public health interventions are at stake.

8) Conclusion

We did not “substantially overstate” (1) our criticisms of the Cochrane HPV vaccine review (2). The Cochrane editors substantially ignored several of our criticisms. The Cochrane HPV review is still incomplete and ignores important evidence of bias.

The Cochrane editors stated that “Some of the criticisms will inform the next version of this Cochrane Review and the planned review of comparative studies of HPV vaccines,” and that the editors “recognize public concerns about the aluminium-based adjuvants” (1).

The editors also stated that “reliance on the published reports in scientific journals may introduce bias due to incomplete and selective reporting” (1). We agree and remind the Cochrane editors that the Cochrane review on neuraminidase inhibitors substantially changed its conclusions after it got updated and became based on clinical study reports instead of journal publications (31).

With our analysis (2), we have contributed to a scientific debate in an area that is complex and biased. The Cochrane authors stated that they will make a “Request for non‐published available data” such as clinical study reports that “will be integrated in future updates of the review” (3). We can offer them these data, which we have used for our own systematic review that we have submitted for publication.

Article info

Conflicts of interest: LJ and PCG have no conflicts of interest to declare. TJ was a recipient of a UK National Institute for Health Research grant for a Cochrane review of neuraminidase inhibitors for influenza. In addition, TJ receives royalties from his books published by Il Pensiero Scientifico Editore, Rome and Blackwells. TJ is occasionally interviewed by market research companies about phase I or II pharmaceutical products. In 2011-13, TJ acted as an expert witness in litigation related to the antiviral oseltamivir, in two litigation cases on potential vaccine-related damage and in a labour case on influenza vaccines in healthcare workers in Canada. He has acted as a consultant for Roche (1997-99), GSK (2001-2), Sanofi-Synthelabo (2003), and IMS Health (2013).In 2014 he was retained as a scientific adviser to a legal team acting on oseltamivir. TJ has a potential financial conflict of interest in the drug oseltamivir. In 2014-16, TJ was a member of three advisory boards for Boerhinger Ingelheim. TJ was holder of a Cochrane Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. TJ was a member of an independent data monitoring committee for a Sanofi Pasteur clinical trial on an influenza vaccine. Between 1994 and 2013, TJ was the coordinator of the Cochrane Vaccines Field. TJ was a co-signatory of the Nordic Cochrane Centre Complaint to the European Medicines Agency (EMA) over maladministration at the EMA in relation to the investigation of alleged harms of HPV vaccines and consequent complaints to the European Ombudsman. TJ is co-holder of a John and Laura Arnold Foundation grant for development of a RIAT support centre (2017-2020) and Jean Monnet Network Grant, 2017-2020 for The Jean Monnet Health Law and Policy Network. TJ is an unpaid collaborator to the project Beyond Transparency in Pharmaceutical Research and Regulation led by Dalhousie University and funded by the Canadian Institutes of Health Research (2018-2022).

Authors' experience: PCG has co-authored 17 Cochrane reviews, including several reviews based on clinical study reports. TJ has co-authored 17 Cochrane reviews including the first Cochrane review that used clinical study reports. LJ has co-authored several articles on the HPV vaccines.

Authors' contributions: LJ wrote the first draft. LJ, PCG and TJ contributed to the conception, drafting,
critical revision for important intellectual content and the final approval of the article.

Acknowledgements: We would like to thank the Cochrane editors for taking our analysis into consideration and the BMJ-EBM editors for allowing us to respond via their journal.

Provenance and peer review: Commissioned; not externally peer-reviewed.

Copyright information: © Article author(s) (or their employer(s) unless otherwise stated in the text of the
article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

References
1. Tovey D, Soares-Weiser K. Cochrane’s Editor in Chief responds to BMJ EBM article criticizing HPV review. 2018; 3 Sept. https://www.cochrane.org/news/cochranes-editor-chief-responds-bmj-ebm-ar....
2. Jørgensen L, Gøtzsche PC, Jefferson T. The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evidence-Based Medicine. 2018 Jul 27;bmjebm-2018-111012.
3. Arbyn M, Xu L, Simoens C, Martin‐Hirsch PP. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. John Wiley & Sons, Ltd; 2018. Available from: http://cochranelibrary-wiley.com/doi/10.1002/14651858.CD009069.pub3/full
4. Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review. PLOS Medicine. 2016 Sep 20;13(9):e1002127.
5. Jørgensen L, Gøtzsche PC, Jefferson T. Index of the human papillomavirus (HPV) vaccine industry clinical study programmes and non-industry funded studies: a necessary basis to address reporting bias in a systematic review. Systematic Reviews. 2018 Jan 18;7:8.
6. Hawkes N. HPV vaccine safety: Cochrane launches urgent investigation into review after criticisms. BMJ. 2018 Aug 9;362:k3472.
7. Cochrane’s Editor in Chief responds to BMJ EBM article criticizing HPV review. Available from: /news/cochranes-editor-chief-responds-bmj-ebm-article-criticizing-hpv-review
8. Skinner SR, Szarewski A, Romanowski B, Garland SM, Lazcano-Ponce E, Salmerón J, et al. Efficacy, safety, and immunogenicity of the human papillomavirus 16/18 AS04-adjuvanted vaccine in women older than 25 years: 4-year interim follow-up of the phase 3, double-blind, randomised controlled VIVIANE study. Lancet. 2014 Dec 20;384(9961):2213–27.
9. Human Papilloma Virus (HPV) Vaccine Efficacy Trial Against Cervical Pre-cancer in Young Adults With GlaxoSmithKline (GSK) Biologicals HPV-16/18 - Study Results - ClinicalTrials.gov. Available from: https://clinicaltrials.gov/ct2/show/results/NCT00122681
10. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors - Arbyn, M - 2011 | Cochrane Library. Available from: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD009069/full
11. Arbyn M, Bryant A, Martin-Hirsch PP, Xu L, Simoens C, Markowitz L. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd; 2013. Available from: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD009069.pub2/abstract
12. Expert consultation on the use of placebos in vaccine trials. World Health Organization. 2013. Available from: apps.who.int/iris/bitstream/handle/10665/94056/9789241506250_eng.pdf?sequence=1
13. Krause P. Update on Vaccine Regulation: Expediting vaccine development. FDA/CBER/OVRR. Available from: https://c.ymcdn.com/sites/www.casss.org/resource/resmgr/CMC_Euro_Speaker...
14. Thiriot DS, Ahl PL, Cannon J, Lobel GM. Method for preparation of aluminum hydroxyphosphate adjuvant. WO2013078102A1, 2013. Available from: https://patents.google.com/patent/WO2013078102A1/en
15. Pagliusi SR, Teresa Aguado M. Efficacy and other milestones for human papillomavirus vaccine introduction. Vaccine. 2004 Dec 16;23(5):569–78.
16. IARC HPV Working Group. Primary End-points for Prophylactic HPV Vaccine Trials [Internet]. Lyon (FR): International Agency for Research on Cancer; 2014. Available from: http://www.ncbi.nlm.nih.gov/books/NBK304971/
17. Trimble CL, Frazer IH. Development of therapeutic HPV vaccines. Lancet Oncol. 2009 Oct;10(10):975–80.
18. Motamedi M, Böhmer G, Neumann HH, von Wasielewski R. CIN III lesions and regression: retrospective analysis of 635 cases. BMC Infect Dis. 2015 Nov 21;15. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4654894/
19. Moscicki A-B, Ma Y, Wibbelsman C, Darragh TM, Powers A, Farhat S, et al. Rate of and Risks for Regression of CIN-2 in adolescents and young women. Obstet Gynecol. 2010 Dec;116(6):1373–80.
20. de Sanjose S, Quint WG, Alemany L, Geraets DT, Klaustermeier JE, Lloveras B, et al. Human papillomavirus genotype attribution in invasive cervical cancer: a retrospective cross-sectional worldwide study. Lancet Oncol. 2010;11:1048–56.
21. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010 Mar 24;303(12):1180–7.
22. Garland SM, Hernandez-Avila M, Wheeler CM, Perez G, Harper DM, Leodolter S, et al. Quadrivalent vaccine against human papillomavirus to prevent anogenital diseases. N Engl J Med. 2007 May 10;356(19):1928–43.
23. FUTURE II Study Group. Quadrivalent vaccine against human papillomavirus to prevent high-grade cervical lesions. N Engl J Med. 2007 May 10;356(19):1915–27.
24. Clendinen C, Zhang Y, Warburton RN, Light DW. Manufacturing costs of HPV vaccines for developing countries. Vaccine. 2016 Nov 21;34(48):5984–9.
25. De Belgische Kamer van volksvertegenwoordigers. Available from: https://www.dekamer.be/kvvcr/showpage.cfm?section=qrva&language=nl&cfm=q...
26. Information NC for B, Pike USNL of M 8600 R, MD B, Usa 20894. Declarations of interest. World Health Organization; 2013. Available from: https://www.ncbi.nlm.nih.gov/books/NBK195238/
27. Cancer Screening in the European Union (2017) Report on the implementation of the Council Recommendation on cancer screening. Health and Food Safety. Available from: https://docplayer.net/43107833-Cancer-screening-in-the-european-union-20...
28. EUROGIN 2018. Available from: https://www.eurogin.com/2018/341-program-committee.html
29. Conflicts of interests. Available from: http://www.nogracias.eu/wp-content/uploads/2013/05/vacc-HPV-Cochrane_HPV...
30. Herrero R. Human Papillomavirus (HPV) Vaccines: Limited Cross-Protection against Additional HPV Types. J Infect Dis. 2009 Apr 1;199(7):919–22.
31. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, et al. Neuraminidase inhibitors for preventing and treating influenza in adults and children. The Cochrane Library. John Wiley & Sons, Ltd; 2014. Available from: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD008965.pub4/full

The Cochrane Review of the HPV vaccine states :

"In older women, vaccinated between 25 to 45 years of age, the effects of HPV vaccine on precancer are smaller, which may be due to previous exposure to HPV. The risk of precancer associated with HPV16/18 is probably reduced from 145/10,000 in unvaccinated women to 107/10,000 women following HPV vaccination (moderate certainty). The risk of any precancer is probably similar between unvaccinated and vaccinated women (343 versus 356/10,000, moderate certainty)."

Cochrane estimates a significant effect of the vaccine on HPV 16/18 associated precancer in the 25-45 age group, but that effect disappears when risk of precancer of all (HPV) types is calculated. It is a matter of simple logic that for the positive (desired) effect on type 16/18 precancer to be offset in this way, the vaccine must have a corresponding negative (undesired) effect on precancer associated with other HPV types.

The observation holds for all age groups, but is more pronounced in the older group. It is reasonable to assume that if one took, say, a 35 to 45 years old group, the total undesired result of the HPV vaccination could be statistically significant.

If my reading of these numbers is too simplistic, it would be good to get a clarification, since the quoted passage appears right in the Summary with no further comments.

In Europe, the vaccine is routinely recommended to women of all ages.

References
[1] Arbyn M, Xu L, Simoens C, Martin-Hirsch PPL. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database Syst Rev. 2018; (5): CD009069.