Dear Dr. David Tovey,

I have read your response [1,2] to the paper by Jøorgensen et al. published in BMJ Evidence-Based Medicine [3] relating to the recently published Cochrane Review on HPV vaccines [4], and would like to give my feedback on this issue.
The key findings of your investigations are as follows:

1. The Cochrane Review did not miss "nearly half of the eligible trials". A small number of studies were missed due to the primary focus on peer-reviewed reports in scientific journals, but addition of these data makes little or no difference to the results of the review for the main outcomes;
2. The trials comparators were unambiguously, transparently, and accurately described;
3. The selection of outcomes for benefits was appropriate and was consistent with World Health
Organization guidance;
4. The review included published and unpublished data on serious harms, and the findings on
mortality were reported transparently and responsibly;
5. The review was compliant with Cochrane’s current conflict of interest policy;
6. Cochrane’s media coverage was cautious and balanced, but we recognize that there could be
improvements in relation to transparency where external experts are quoted;
7. The BMJ Evidence-Based Medicine article substantially overstated its criticisms.

I would like to comment on your findings 1, 2, 4 and lastly, I added comments as 8. Most observational studies neglect “healthy vaccine effect/healthy user bias”.

1. The Cochrane Review did not miss "nearly half of the eligible trials".

You explained “A small number of studies were missed due to the primary focus on peer-reviewed reports in scientific journals.” However, the abstract of the Cochrane review reports: “We searched MEDLINE ----- for reports on effects from trials. We searched trial registries and company results’ registers to identify unpublished data for mortality and serious adverse events.”

For the serious adverse events, the Cochran review described in the sensitivity analysis as follows: “We assessed the robustness of data collected for serious adverse events, all-cause mortality and pregnancy outcomes based on the source of data. The primary analysis for these outcomes included data that we considered to represent the most complete follow-up. As a sensitivity analysis we used data for these same outcomes that had only been reported in the published trial reports.”

It is evident that the Cochrane review recognized the importance of clinical study reports (CSRs) and partially gathered the unpublished data for mortality and serious adverse events.

Jørgensen et al. pointed out that there were many missed trials which Cochrane review did not included because they were not published.

As for the analysis of oseltamivir, we, the Cochrane neuraminidase inhibitor team, used all CSRs and found significantly increased psychiatric events in the 4 CSRs for prophylaxis of influenza. However, if we restricted to use peer reviewed published journals, we would not have detected the psychiatric harm even if CSRs for published trials were used. This is the most important point that Jørgensen et al. emphasized.

The review should be revised by including all CSRs identified and available. Before the revision is completed, the Cochrane review should be suspended.

2. Description of the comparators and the unavoidable toxicity of adjuvants

2-1. The safety of adjuvants is not established.

You wrote “The trials comparators were unambiguously, transparently, and accurately described”. However, the problem is not the descriptions but the real toxicity of adjuvants.

It is true that the Cochrane review by Arbyn et al. described the comparators unambiguously, transparently, and accurately. However, the problem is not the accuracy and transparency of the description of the comparators. It is that adjuvants as the comparators conceal the true harm of HPV vaccines because the safety of the vaccine adjuvant has never been established clinically [5] nor non-clinically. Instead, harm of adjuvant is rather unavoidable as indicated by laboratory tests and toxicity tests shown in the followings.

2-2. Non-clinical tests strongly suggest the harm of adjuvant.

It is revealed that true adjuvant is the DNA of the recipient which is produced by tissue damage by the administered adjuvant, such as alum adjuvant. This suggests that the bigger the damage, the stronger the stimulation of innate immunity and work as adjuvant [6,7], and autoimmune diseases [8], including those in the central nervous system [9] may be induced. GlaxoSmithKline conducted several animal toxicity tests, although they did not fulfil the standard toxicity testing method for the ordinary pharmaceutical products.

In the third toxicity test, single or repeated i.m., Cervarix, and AS04 adjuvant induced item-related local degeneration, necrosis, and regeneration of muscle fibers with hemorrhage and mild-to-moderate subacute inflammation at 4 days and 5 months after the first inoculation, unlike saline [10]. The extent and proportion of animals with these lesions were the same between Cervarix and AS04 group and immediately after the single dose but more prominent in the Cervarix group than AS04 adjuvant group after 4 repeated doses [10].

In a randomized controlled trial, saline control is appropriate as the comparator for the analysis of both efficacy and harm. For efficacy analysis, active control such as adjuvant or vaccines with/without adjuvant may not be necessarily inappropriate as the comparator. However, these active controls are definitely inappropriate for the harm analysis. The other approach should be applied for the harm analysis, if no saline control trials are available.

4. The review included published and unpublished data on serious harms, and the findings on mortality were reported transparently and responsibly;

4-1. Meta-analysis of mortality in the trials targeting mid-adults women
You wrote “The review included published and unpublished data on serious harms, and the findings on mortality were reported transparently and responsibly.” According to my meta-analysis using the extracted data from the reference peer reviewed papers in which subjects’ ages were 25 (or 26) to 45 year or older [11-14], the pooled odd ratio was 5.00 (95%CI: 1.71, 14.65), P = 0.0022 (I2 = 0%). This indicates that HPV vaccines increase mortality in woman aged 25 or older by 5 times within 4 years after the first injection.

• For the VIVIANE trial [11,12], I used 13 deaths in Cervarix group and 3 deaths in adjuvant group within 4 years, because after that period there was no difference between these groups.
• For the FUTURE III trial [13], the numbers of deaths described in the published paper were 7 in the Gardasil group and 1 in the adjuvant group as shown in its sensitivity analysis of death. However, in the Cochrane review, the numbers are 8 and 4, respectively, presumably based on the data including after 4 years from unpublished CSR.
• For the Chinese trial [14], my extraction was HPV vaccine 0 and adjuvant 0 while the Cochrane review data were 1 and 0, respectively, also in the sensitivity analysis.

Moreover, you described that after adding the mortality data from newly included trial NCT00834106 (targeted 20 to 45 years of age), RR of death increased from 1.54 (95% CI 0.73 to 3.23) to 1.65 (95% CI 0.80 to 3.38). It is highly probable that substantial difference of death risk may be reported in this added study. Please clarify the number of deaths in both groups (Data from Clinicaltrial.gov suggest that the number of deaths are estimated 2/1499 and 0/1498 for Gardasil and adjuvant group respectively).
Risk of death in women aged 25 to 45 years may be further robust by adding this trial.

Serious adverse event and mortality should be separately analysed for different age groups: namely under the age of 25 years and mid-adults (approximately 25 to 45 years old) in addition to the analysis as the whole ages.

4-2. Fluctuation of mortality and adverse events

If a vaccine does not cause harmful effects, the incidence rate of adverse events may stay at the baseline or increase only slightly as age increases. However, if the incidence rate of adverse events fluctuates substantially, then it may be the result of the harmful effect of the vaccine. Several patterns of theoretical trend of incidence rate are shown (Figure not shown).

In the pivotal RCT of Cervarix (PATRICIA), no difference in the adverse events between Cervarix group and adjuvanted HA vaccine group was observed. Hence, I calculated the overall (both groups) trend of incidence rates of adverse events, including chronic diseases (CD), autoimmune diseases (AD) and death (D) dividing 3 periods: 1: 0-1.2 year, 2: 1.2-3.4 year, 3:3.4-3.65 year). Incidence rate (/100,000 person-year) of CD, AD and D fluctuate as follows: CD: 129-55-164, AD: 25-24-88, D:22-30-135 respectively. These data show that the incidence rates or mortality rates greatly fluctuate and increase even after about 3.5 years from the first inoculation (Figure not shown).

For Gardasil studies, I calculated the incidence rates (/100,000 person-years) of various autoimmune-related adverse events (AE) among participants from both Gardasil and control (almost all received alum-adjuvant) groups (Figures not shown). Incidence rates (/100,000 person-years) of total autoimmune related adverse events are 2441 at the first period (day 1 to 6 months) and 625 at the second period (7-24 months) and that of inflammatory bowel disease are 77 and 28 respectively.

These also indicate that the incidence rates of various autoimmune diseases greatly fluctuate over time and suggest Gardasil affected these fluctuations (included studies were all phase II and III trials targeted women aged 9 to 24).

VIVIANE trial shows high mortality rate during the first 4 years and low mortality rate after 4 years in the Cervarix group, but this was not evident in the control adjuvant group (Figure not shown). These data show fluctuations are greater in the HPV vaccine group.

If the numbers of deaths for FUTURE III trial are 8 for Gardasil group and 4 for adjuvant group, 1 and 3 deaths may be observed after 4 years, respectively. Person-years at risk is not estimated, but 7 in the first 4 years and 1 thereafter indicate similar fluctuation, and less fluctuation is observed in the adjuvant group. This may be a very similar phenomenon as shown in the VIVIANE trial.

Fluctuation of serious adverse events including death should be considered in the RCTs using adjuvant as comparator. Analysis of high risk and low risk periods together may conceal true harm of HPV vaccines. Analysis should be conducted separately for the different risk periods.

4-3. Comparison of incidence rates of specific autoimmune diseases between RCT and general population of women of similar age group

The incidences rates of some autoimmune diseases examined, such as multiple sclerosis (MS), systemic lupus erythematosus (SLE), and inflammatory bowel diseases (IBD), calculated from the data in the “SBA” of Gardasil (age ranged 9–26 years, mainly 16–23 years) were all higher than those in the general female population of similar age (15 to 25 years old).

For example, the incidence rate (per 100,000 person-year) of MS in Gardasil RCTs (14.7) was about 3–15 times higher than that of general population in Italy (4.2–4.7), the United Kingdom (3.4), and France (1.0). Notably, MS incidence reported in Gardasil RCTs was even higher than the highest reported in the general population, namely, in north Sweden (8.4) and Iceland (12.5) (Figure not shown).

Incidence rates of IBD in the RCTs are also higher than that of the general female population of similar age, though the age range is a little broader (0 to 39 years old).

Annual incidence rates for typical adverse events such as specific autoimmune diseases in the RCTs should be compared with that of general population of similar age.

8. Most observational studies neglect “healthy vaccine effect/healthy user bias”

In the discussion section, the Cochrane review discussed the Adverse effects of HPV vaccines citing many observational studies.

However, most observational studies have fundamental flaws:
• Negligence of healthy vaccine effect (healthy user bias),

• Negligence of time dependent bias in the self-control case-series method and
• Confusion of incidence with prevalence.

Here I only discuss the Negligence of healthy vaccine effect (healthy user bias) which most observational studies neglected, affect the results seriously and sometimes reverse the association.

Negligence of healthy vaccinee effect (healthy user bias)

i) Theoretical basis
It is important that confounding bias is avoided in epidemiologic studies. In particular, confounding bias from “healthy vaccinee effect” always affects results in favour of an intervention, leading to overestimation of its effectiveness and safety.

People who have any diseases and/or are prone to have fever/infection tend to avoid vaccination. Autoimmune diseases often follow infection. Unless unvaccinated people who are sicklier or frailer than vaccinated people at baseline were not adequately adjusted, the results of the high incidence rate of autoimmune diseases in the vaccinated people may be offset by the disease incidence in the sickly unvaccinated people. This is “healthy-vaccinee effect,” “frailty selection bias,” or “frailty exclusion confounding bias.” [15]

Bias from “healthy-vaccinee effect” becomes more impactful as the coverage of vaccination becomes higher, namely 80% or higher [15]. Theoretically in the unvaccinated, odds ratio for having symptoms at a certain vaccine coverage (%) compared with the lowest coverage group increases as the percent coverage increases.

ii) Nagoya study
Nagoya study is a questionnaire survey on symptoms after HPV vaccination, involving about 70,000 girls (born between 2 April 1994 and 1 April 2001 (approximately aged 14 to 21 in September 2015) living in Nagoya City Japan. Of these, about 30,000 girls responded: 20,748 girls were vaccinated and 9,098 girls were unvaccinated [16].

Suzuki et al. [16] concluded “the results suggested no causal association between the HPV vaccines and their alleged harmful symptoms”.

However, according to my logistic regression analysis using the disclosed PDF data, odds ratios (ORs) of positive symptoms before vaccination were significantly (p<0.05) lower than 1.0 in 15 of 24 symptoms ranging from 0.16 (Dependent on stick or wheel chair) to 0.86 (Irregular menstrual cycle). ORs for symptoms leading to hospital visit were significantly lower than 1.0 in 7 of 24 symptoms, especially in severe symptoms, such as “Unable to walk normally” (0.22), “Dependent on stick or wheel chair” (0.21), “Sudden loss of strength” (0.28) and “Weak in the extremities” (0.30). OR for “Difficulties in calculation” is not significant (P=0.148), but point estimate of OR was very low (0.25, p=0.15).

The trends of the proportion of the frail, by coverage both in the unvaccinated and the vaccinated groups fit well to the theoretical trends expected in each group.

On the other hand, odd ratios of positive symptoms after vaccination leading to hospital visit were generally higher than 1.0 (no symptom was less than 1.0) and significantly higher than 1.0 in 11 of 24 symptoms: for example, “Weak in the extremities” (2.76, p=0.014) and “Difficult to remember Chinese characters” (8.46, p=0.047). “Dependent on stick or wheel chair leading to hospital visit” were reported in 13 girls in the vaccinated group, while none in the unvaccinated group: odds ratio = 9.61 (95%CI: 1.21- infinity, p=0.027) by the exact-like logistic regression using “R” software.

Considering the low odds ratio of health status before inoculation, each odds ratio after inoculation should be divided by the corresponding odds ratio before inoculation. Hence point odds ratio after vaccination of HPV vaccine for the symptoms leading to hospital visit are estimated as follows (those above 5.0 are shown): Dependent on stick or wheel chair: 46.7, Difficult to remember Chinese character: 24.8, Difficult to calculate: 15.5, Unable to walk normally: 11.0, Weak in the extremities: 9.2, Sudden loss of strength: 8.6 and Involuntary movement: 5.7

iii) French study: a rare study incorporating prior health status as confounder shows harm of autoimmune diseases

Among many observational studies, French study [17] is a rare study which considers the health status before vaccination, though partially. It is a retrospective cohort study utilizing database of national health system. Total 2.25 million girls aged 13 to 16 years between 2008 and 2012 were included: 0.84 million (37.3%) were exposed to HPV vaccine and 1.41 million were not exposed.

Outcome events were 14 autoimmune diseases (AD), including central nervous system (CNS) demyelinating diseases (MS: multiple sclerosis etc.), Guillain-Barres syndrome (GBS), and inflammatory bowel diseases (IBD). Hazard ratios of any of the14 ADs or of individual AD were estimated by Cox regression analysis adjusting age, year of vaccination as time dependent variable. Geometric data of residence, other diseases, level of care before inclusion, vaccination other than HPV vaccine were also adjusted. Sensitivity analysis and subgroup analysis were performed to make sure of the robustness of the results.

The adjusted factor “Level of care before inclusion” seems to be “at least one visit for medical care prior to inclusion”. Hence it is unclear how it influenced the results, and the adjustment of bias from healthy vaccinee effect may be partial.

However, a statistically significant association with HPV vaccine was shown in 2 autoimmune diseases: adjusted HR of inflammatory bowel diseases (IBD) was 1.19 (95%CI: 1.02-1.39) and that of Guillain-Barré syndrome (GBS) was 4.00 (95%CI: 1.84-8.69). The sensitivity analysis showed that the association of both autoimmune diseases were the strongest for the period from 0 to 3 months and tended to decrease subsequently without significance for IBD, but it remained significant for GBS. Risk of GBS is robust as suggested by the sensitivity and subgroup analysis. These suggest that the incidence rate of both symptoms may have fluctuated.

Moreover, using 4.7 million as the person-year (py) of unvaccinated people, incidence rates per 100,000 py are greatly different from those of the unvaccinated in the table reported by the French Agency: especially in CNS demyelinating diseases (4.7 by recalculation vs 5.8 by original report), inflammatory bowel diseases (IBD) (13.8 vs 16.9) and those having at least one autoimmune disease (AD) (63.4 vs 66.8). If our estimate of incidence rate in the unvaccinated group is correct, odds of CNS demyelinating diseases in the HPV vaccinated girls increased non-significantly (P=0.07). Odds of IBD increased more with highly significant OR up to 1.53 (95%CI: 1.33, 1.76, P<0.001), and increased odds of having at least one autoimmune disease was also significant: OR = 1.13 ((95%CI: 1.05, 1.21, P<0.001).

Moreover, incidence rate (/105 py) of CNS demyelinating diseases in the unvaccinated group (4.7 or 5.8) is very high compared with that of multiple sclerosis (0.99 [95% CI: 0.94–1.04]) in the general female population of the same age group (aged 15 to 24 years) in France between 2001 and 2007 [18]. Incidence rate (/105py) of CNS demyelinating diseases in the HPV vaccinated group in the French pharmacovigilance study [17] is far higher than that in the general population.

These suggest that unvaccinated group had included frail girls at the time of inclusion. Therefore, it should be considered that “healthy vaccinee effect” or “frailty exclusion bias” may not be completely excluded even in this French pharmacovigilance study [17].

Moreover, significantly high incidence of events after inclusion such as frequent consultation (≧4/year): Odds ratio (OR) = 2.81 (2.80,2.83) and at least one hospitalization: OR = 3.54 (3.52, 3.56) (based on person-year) should be taken into account. This is because high odds ratio needing frequent medical care and/or hospitalisation (9% of the girls per year need additional hospitalization compared with prior vaccination and unvaccinated girls) may be related to some adverse outcomes of HPV vaccination other than autoimmune diseases.
For the discussion of harm comparing the results with those from observational studies, it is essential to consider the “healthy vaccinee effect”. Unless the observational studies adjust the health status before vaccination (pseudo vaccination date for unvaccinated group), the findings from such studies should not be used as evidence of safety.

Conclusion

The review should be revised by including all identified and available data. Before the revision is completed, I consider the Cochrane review should be suspended.

In a randomized controlled trial, saline control is appropriate as the comparator for the analysis of both efficacy and harm. For efficacy analysis, active control such as adjuvant or vaccines with/without adjuvant may not be necessarily inappropriate as the comparator. However, these active controls are definitely inappropriate for the harm analysis. The other approach should be applied to the harm analysis, if no saline control trials are available.

Serious adverse event and mortality should be separately analysed for different age groups: under the age of 25 years and mid-adults (approximately 25 to 45 years old) in addition to the analysis as the whole ages.

Fluctuation of serious adverse events including death should be considered in the RCTs using adjuvant as comparator. Analysis of high risk and low risk periods together may conceal true harm of HPV vaccines. Analysis should be conducted separately for the different risk periods.

Annual incidence rate for typical adverse events such as specific autoimmune diseases in the RCTs should be compared with that of general population of similar age.

For the discussion of harm comparing the results with those from observational studies, it is essential to consider the “healthy vaccinee effect”. Unless the observational studies adjust the health status before vaccination (pseudo vaccination date for unvaccinated group), the findings from such studies should not be used as evidence of safety.

It is my sincere hope that you would consider my feedback seriously in order to further improve the quality of the Cochrane review. Should you have any questions, please feel free to contact me.

Sincerely yours,
Rokuro Hama M.D.
Chairperson, NOP Japan Institute of Pharmacovigilance

Conflict of interest

Rokuro Hama (RH) has been a member of the Cochrane neuraminidase inhibitor team headed by Tom Jefferson since January 2010. RH has met Dr. David Tovey (DT) and discussed the methods for conducting the systematic review of neuraminidase inhibitors [19] at the Oxford meeting in April 2011. RH explained DT and other members of the Cochrane neuraminidase inhibitor team how Tamiflu lowered the antibody production in the treatment trial and that ITTI population was inappropriate for the assessment of efficacy. Instead, RH proposed that ITT population should be used to assess efficacy and harm of Tamiflu. The method using ITT population was agreed by the Cochrane neuraminidase inhibitor team at the meeting in April 2011.

References
1. Cochrane’s Editor in Chief responds to BMJ EBM article criticizing HPV review
https://www.cochrane.org/news/cochranes-editor-chief-responds-bmj-ebm-ar...
2. Tovey D and Soares-Weiser K. Cochrane’s Editor in Chief responds to a BMJ Evidence-Based Medicine article criticizing the Cochrane Review of HPV vaccines.
https://www.cochrane.org/sites/default/files/public/uploads/cochrane_hpv...
3. Jørgensen L, Gøtzsche PC, Jefferson T. The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evidence-Based Medicine 2018 July 27 https://doi.org/10.1136/bmjebm-2018-111012 2.
4. Arbyn M, Xu L, Simoens C, Martin-Hirsch PPL. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database of Systematic Reviews 2018;(5):CD009069. https://doi.org/10.1002/14651858.CD009069.pub3
5. Jefferson T, Rudin M and Pietrantonj CD, Adverse events after immunization with aluminium-containing DTP vaccines: systematic review of the evidence, Lancet Infect Dis 4, 2004, 84–90.
6. Marichal T, K. Ohata K, Bedoret D, Mesnil C, Sabatel C, Ishii KJ et al., DNA released from dying host cells mediates aluminum adjuvant activity, Nat Med 17, 2011, 996–1002.
7. Sloane JA, Blitz D, Margolin Z and Vartanian T, A clear and present danger: endogenous ligands of Toll-like receptors, Neuromolecular Med 12, 2010, 149–163.
8. Tsumiyama K, Miyazaki Y and Shiozawa S, Self-organized criticality theory of autoimmunity, PLoS One. 4, 2009, e8382.
9. Lampron A, Elali A and Rivest S, Innate immunity in the CNS: redefining the relationship between the CNS and its environment, Neuron 24 (78), 2013, 214–232.
10. GlaxoSmithKline Biologicals. Japanese Summary Basis of Approval of Cervarix. http://www.pmda.go.jp/drugs/2009/P200900052/index.html ; [accessed 18.9.4].
11. Skinner SR, Szarewski A, Romanowski B, Garland SM, Lazcano-Ponce E, Salmeron J. Efficacy, safety, and immunogenicity of the human papillomavirus 16/18 AS04- adjuvanted vaccine in women older than 25 years: 4-year interim follow-up of the phase 3, double-blind, randomised controlled VIVIANE study. Lancet 2014;384(9961): 2213–27.
12. Wheeler CM, Skinner SR, Del Rosario-Raymundo MR, Garland SM, Chatterjee A, Lazcano-Ponce E, et al. Efficacy, safety, and immunogenicity of the human papillomavirus 16/18 AS04-adjuvanted vaccine in women older than 25 years: 7-year interim follow-up of the phase 3, doubleblind, randomised controlled VIVIANE study. Lancet Infectious Diseases 2016;16(10):1154-68.
13. Castellsagué X, Munoz N, Pitisuttithum P, Ferris D, Monsonego J, Ault K. End-of-study safety, immunogenicity, and efficacy of quadrivalent HPV (types 6, 11, 16, 18) recombinant vaccine in adult women 24 to 45 years of age. British Journal of Cancer 2011;105(1):28–37.
14. Zhu FC, Li CG, Pan HX, Zhang YJ, Bi D, Tang HW. Safety and immunogenicity of human papillomavirus-16/ 18 AS04-adjuvanted vaccine in healthy Chinese females aged 15 to 45 years: a phase I trial. Chinese Journal of Cancer 2011;30(8):559–64.
15. Hama R. Symptoms after HPV vaccine: typical "frailty exclusion bias" in Nagoya City study.
MedCheckTIP 2016: 2(No5); 18-21 available at
https://npojip.org/english/MedCheck/Med%20Check-TIP%2005-08-08.pdf
16. Suzuki S, Hosono A. No Association between HPV Vaccine and Reported Post-Vaccination Symptoms in Japanese Young Women: Results of the Nagoya Study. Papillomavirus Res. 2018 Feb 23. pii: S2405-8521(17)30070-8
17. French National Agency for Medicines and Health Products Safety. Vaccins anti-HPV et risque de maladies autoimmunes: étude pharmacoépidémiologique Rapport final Septembre 2015
http://ansm.sante.fr/content/download/80841/1023043/version/1/file/Ansm_...
18. Fromont A, Binquet C, Sauleau E, Fournel I, Despalins R, Moreau T et al. National estimate of multiple sclerosis incidence in France (2001-2007). Mult Scler. 2012;18:1108-15.
19. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2014;4:CD008965.
http://cochranelibrary-wiley.com/doi/10.1002/14651858.CD008965.pub4/full

The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias: Response to the Cochrane editors

Lars Jørgensen, LJ (lj@cochrane.dk), 1
Peter C. Gøtzsche, PCG (pcg@cochrane.dk), 1
Tom Jefferson, TJ (tj@cochrane.dk), 1

1Nordic Cochrane Centre, Rigshospitalet 7811, Tagensvej 21, 2100 Copenhagen, Denmark.

Summary

In a report uploaded on the Cochrane.org website on 3 September 2018 (1), Cochrane’s Editor in Chief and Deputy Editor in Chief responded to our analysis published in BMJ Evidence-Based Medicine on 27 July 2018 (2) of the Cochrane review of the HPV vaccines published on 9 May 2018 (3).

The Cochrane editors acknowledge (1) that our analysis (2) addresses the importance of the selection of data sources for reviews, and we hope that Cochrane will take the threat posed by reporting bias (4) more seriously by using clinical study reports, rather than journal publications.

The Cochrane editors claimed that we had “substantially overstated” our criticisms and they concluded that “Jørgensen et al made allegations that are not warranted and provided an inaccurate and sensationalized report of their analysis” (1).

Here we address the Cochrane editors’ findings and present our further assessment and additional findings.

In summary, we found that our analysis (2) was appropriate and that the Cochrane editors substantially ignored several of our criticisms (1):

1) The Cochrane editors’ cross referencing (1) with our HPV vaccine study index (5) showed that the Cochrane HPV review was incomplete and resulted in 8% additional eligible female participants (6,191/73,428). Due to the discrepancy with our analysis (2), we assessed our index again and found that the Cochrane HPV review should have included at least 35% (25,550/73,428) additional eligible females in its meta-analyses;
2) The Cochrane editors’ considerations on harms ignored several of our criticisms including the incomplete reporting of serious adverse events in several of the Cochrane HPV review’s included studies;
3) The Cochrane editors’ considerations of the trials’ adjuvant and vaccine comparators was ambiguous, opaque, inaccurate and ignored the fact that the studies only tested the vaccine antigens—not the vaccines;
4) The Cochrane editors’ response on the Cochrane HPV review’s included composite surrogate outcomes was superficial and did not consider the substantial bias and confounding that these outcomes involve;
5) The Cochrane editors’ assessment of the Cochrane HPV review authors’ conflicts of interest was incomplete and ignored several additional important conflicts of interest;
6) The Cochrane editors’ considerations on the media coverage did not recognize that it should be balanced and free from financial conflicts of interest;
7) The Cochrane editors’ response appeared to advocate scientific censorship, which we do not approve of;
8) In conclusion, our analysis (2) was appropriate, the Cochrane editors substantially ignored several of our criticisms (1) and the Cochrane review is still incomplete and ignores important evidence of bias.

1) The Cochrane editors’ cross referencing with our HPV vaccine study index

We used our index (5) to identify additional eligible studies for the Cochrane HPV vaccine review that included 73,428 women from 26 studies (3).

From our index (5), the Cochrane editors (1) identified:
1. “five [i.e., 5/26 = 19%] eligible completed studies with available data representing 5267 women [i.e., 5,267/73,428 = 7%, that] may have been missed from the Cochrane Review, as a consequence of the search being based on bibliographic databases rather than trials registers.”
2. One “Additional 9-valent [Gardasil 9] study NCT01047345,” adding 924 women to the numerator: 6,191/73,428 = 8% additional women.
3. “Five studies awaiting classification (not recruiting, but no results available) potentially relevant for the current Cochrane Review” that included 4,044 participants.
4. “Eight ongoing studies (actively recruiting, no results available) potentially relevant for the current Cochrane Review” that included 121,531 participants.

The Cochrane editors’ analysis (1) shows that the Cochrane HPV review was incomplete. In their “Appendix A”, the Cochrane editors updated 20 of the Cochrane HPV review’s meta-analyses (20/66, 30%) with the additional data and added seven meta-analyses of the HPV vaccine Gardasil 9 (1), but as of September 14 2018 they have not updated the Cochrane review itself with the additional data (3).

It is not clear why the Cochrane editors thought that our study index (5) “did not appear to identify any important eligible studies” (6). Our index was sent to the editors on 19 January 2018. The Cochrane review was published 110 days later on 9 May 2018 (3), but it seemingly took the editors only 25 days from launching their “investigation” on 9 August 2018 (6) to updating their Cochrane review on 3 September 2018 (7) with the missing studies.

Our initial assessment of the Cochrane HPV review’s included studies:
Initially, we had cross-referenced the study IDs from our index with the 26 included study IDs in the Cochrane HPV review’s “Appendix 6.1.1. Published reports included in the Cochrane review,” and found 20 studies not included in the review. For example, we did not find any of the 20 studies included in the Cochrane HPV review’s two serious adverse events analyses: “Figure 10” of journal publication data and “Analysis 7.6” of data that the Cochrane authors “considered to represent the most complete follow-up” (3). This led us to believe that the studies were not included in the Cochrane review. When we checked again, we found some of the studies in the review’s reference list (3). The Cochrane HPV reviewers chose to use idiosyncratic referencing with study IDs such as “Phase 2 trial (ph2,2v)”, “Immunobridging (ph3,2v)” and “CVT (ph3,2v), which made the study assessment complicated. For numbers of participants, we did not subtract the male participants that were included in three of the studies, as we should have done.

As stated in our index paper (5), our detective work involved a degree of uncertainty, as we did not want to dismiss any possibly eligible studies. Therefore, the index included a “possibly exist” category for studies for which we only had one verification source. Four studies in our analysis (2) had no numbers for randomised participants; these were “probably exist” studies. We have obtained additional information for three of these studies, but we are still not sure that the fourth study exists in clinical study report form, as the manufacturer (Merck) did not answer our request for this information (2,5).

In Table 1, we list the 20 studies (plus the additional one that the Cochrane editors identified: NCT01489527) that we identified as eligible but not included in the Cochrane HPV review.

Table 1: Our reassessment of the studies we had identified as additionally eligible for the Cochrane HPV vaccine review [*Key: RCT = randomised clinical trial]

See: https://blogs.bmj.com/bmjebmspotlight/files/2018/09/Cochrane-HPV-vaccine...

Our analysis reinforces the view that Cochrane HPV review is incomplete. We found an additional 25,550 females (and possibly up to 30,195 for the Cochrane HPV review’s serious adverse events meta-analyses) that are eligible for the Cochrane HPV review’s meta-analyses. Furthermore, we found freely available clinical study reports for 6 of the 21 studies on GlaxoSmithKline’s trial register, which the Cochrane authors used data from. Clinical study reports are far more reliable than published reports (4), in particular in relation to possible serious harms. It is therefore not merely the studies the authors of the Cochrane HPV review missed; they also missed benefits and harms data from the studies they included. In addition, the criteria that the Cochrane authors used for inclusion of data in their primary serious adverse events analysis (Analysis 7.6) are not clear: “The primary analysis for these outcomes included data that we considered to represent the most complete follow-up” (3).

2) The Cochrane editors’ considerations on harms
The Cochrane HPV review’s harms analyses ("Comparison 7”) include seven meta-analyses (four of which report injection site harms) where the three most clinically important ones—deaths (Analysis 7.7), serious adverse events (Analysis 7.6) and systemic adverse events (Analysis 7.5)—contain errors or are incomplete.

Deaths
The Cochrane HPV review’s authors found 5% (90 vs. 86) more deaths in “Analysis 7.7” (journal publication and registry data) vs. “Figure 11 (only journal publication data). We found that this discrepancy reflects the difference between the study FUTURE III’s (V501-019) journal publication and registry entry (8 vs. 12 deaths). We also found that the Cochrane HPV review gave an incorrect number of deaths for the VIVIANE study (HPV-015): 13 deaths in the HPV vaccine group and 5 deaths in the AlOH3 group; according to VIVIANE’s journal publication (8), there were 14 deaths in the HPV vaccine group and 3 deaths in the Al(OH)3 group.

The Cochrane authors state that “The deaths reported in the trials had an identified cause, and none were assessed to be due to vaccination” (3), but such judgements are biased, particularly in industry sponsored trials, and the analysis of deaths should be based on all events (4).
Serious adverse events

There are 11% (4,758/4,291) more serious adverse events in “Analysis 7.6” (journal publication and register data) compared to “Figure 10” (only journal publication data). Since there were more serious adverse events and more deaths in the register entry data, we wonder why the Cochrane authors did not include register entry data for all their outcomes.

We also wonder why three (possibly four) studies were not included in the Cochrane HPV review’s serious adverse events meta-analyses (see Table 1). The Cochrane editors write that we “claim that the [Cochrane] review authors made an error in their reporting of serious adverse events in relation to the PATRICIA [HPV-008] study. This is not the case” (1). We stated that “the Cochrane authors did not explain what the serious adverse events consisted of or whether some of them were more common in the HPV vaccine groups,” (2) and gave the example that “the PATRICIA trial publication only included two thirds (1400/2028) of the serious adverse events listed on ClinicalTrials.Gov” (2). The 2,028 individual serious adverse events listed in ClinicalTrials.Gov are listed with the total denominators of randomised women for PATRICIA, for example, “Headache: participants affected/at risk: 5/9319 [in the HPV vaccine group] (0.05%) vs. 1/9325 [in the hepatitis a vaccine group] (0.01%)” (9), suggesting that the numbers represent participants with serious adverse events.

The Cochrane editors did not consider our highly relevant observations about the incomplete reporting of serious harms (1). For example, we wrote that “FUTURE I, FUTURE II and FUTURE III, which in total included 21 441 women with up to 4 years follow-up, only reported serious adverse events occurring within 14 days post-vaccination” (2). The editors did not comment on how such reporting of serious adverse events for only about 3% of the trial periods (FUTURE I, II and III: [(14 days*3 vaccinations)/(365 days*4 years)]) resulted in the Cochrane authors’ judgements of “low risk of bias” for reporting bias (1).

Both the 1st Cochrane HPV review protocol from 2011 (10) and the 2nd protocol from 2013 (11) list the primary outcome of "serious adverse events observed after four weeks of administration of the vaccine during the trial" (emphasis added), i.e., an incomplete reporting of serious adverse events was already a criterion at protocol stage.
Systemic adverse events

The Cochrane editors (1) did not comment on our criticism of the lack of studies (2)—including PATRICIA (HPV-008)—from the Cochrane HPV reviews’ “Analysis 7.5: systemic adverse events” (3). Analysis 7.5 is incomplete—in particular, for Cervarix studies where the Cochrane review only included numbers for “solicited general adverse events” for two studies (3): HPV-009 and HPV-015, although data for such events are eligible from several additional studies, for example, HPV-001, HPV-008, HPV-013, HPV-029, HPV-030, HPV-033, HPV-035, HPV-038 and HPV-058. The inclusion of these studies could change the Cochrane review’s conclusion that “Systemic events with general mild symptoms were similarly frequent in vaccinated recipients and placebo or control vaccine recipients” (3). As we wrote, “On ClinicalTrials.gov, PATRICIA has 7129 vs 6557 systemic events listed under ‘Results: Other Adverse Events (General disorders)’, which in itself is a significantly increased risk: RR 1.09 (95% CI 1.07 to 1.11)” (2).

Furthermore, the Cochrane authors did not address that “solicited general adverse events [Cervarix]” only were reported 7-days post-vaccination and “systemic adverse events [Gardasil]” only 14-days post-vaccination (3).
Assessment of safety signals

The Cochrane editors did not comment on our safety signal section (1). Some potential HPV vaccine-related harms to the nervous system—or “autonomic dysfunction syndromes,” as the Cochrane editors described them (1)—have been reported (2). The Cochrane authors should have used trial register data to investigate such safety signals; for example, if they had summarised the nervous system disorders from PATRICIA’s (HPV-008) ClinicalTrials.gov list of serious adverse events (9), they would have found more serious nervous system disorders in the HPV vaccine arm: 39/9,319 vs. 25/9,325 in the hepatitis A vaccine arm, risk ratio 1.56 (95% CI 0.95 to 2.58). The Cochrane authors write that “All estimates of adverse effects in our review were restricted to those reported from randomised trials and therefore could not detect rare events, for which post-marketing surveillance, pharmacovigilance activities and linkage studies, joining vaccine and morbidity registries, are needed” (3), but the review’s results might have been different had the authors included serious adverse events on both an individual and organ system level.

Additional points on the Cochrane editors’ harms assessment
The Cochrane HPV review (3) did not include the following harms categories that were reported in the eligible studies’ clinical study reports and in some journal publications: “unsolicited adverse events” (Cervarix), “medically significant conditions” (Cervarix), “new onset chronic/auto-immune disease” (Cervarix) and “new medical history” (Gardasil), even though the Cochrane authors mention the two first categories (reported in “Angelo 2014”) (3).

3) The editors’ considerations of the trials’ adjuvant and vaccine comparators

The Cochrane editors stated that “The trials comparators [sic] were unambiguously, transparently, and accurately described” (1), but in the Cochrane HPV review’s “Plain language summary” intended for lay readers, the review authors state that “The risk of serious adverse events is similar in HPV and control vaccines (placebo or vaccine against another infection than HPV” (emphasis added) (3), and the word “placebo” is repeated throughout the review and all its meta-analyses, which make the review ambiguous, opaque and inaccurate, as no included trial in the review used a placebo comparator.

The WHO states that using adjuvant or another vaccine as comparators instead of placebo makes it difficult to assess the harms of a vaccine (12). The HPV vaccine trials’ adjuvant comparators—Merck’s amorphous aluminium hydroxyphosphate sulphate (AAHS) and GlaxoSmithKline’s aluminium hydroxide (Al[OH]3)—have not been tested against an inert comparator in human trials. The adjuvants’ clinical properties are largely unknown; they are not regulated on their own, as regulators do not regard them as “active ingredients” (13). For example, Merck’s AAHS has a confidential formula and its properties are variable from batch to batch and even within batches (14). Because the HPV vaccines and their adjuvants had similar harms profiles, the manufacturers and the regulators concluded that the HPV vaccines are safe. However, this is like saying that cigarettes and cigars must be safe because they have similar harms profiles.

In addition, in those trials with a non-HPV vaccine comparator, the HPV vaccine aluminium adjuvant was used in nearly all the non-HPV comparator vaccines; for example, PATRICIA’s hepatitis A (Havrix) comparator contains Al(OH)3 (only the studies HPV-032 and HPV-063 used a non-aluminium containing comparator: the hepatitis a vaccine Aimmugen). Thus, the presence of AAHS or Al(OH)3 in nearly all arms of the studies thwarted the harms assessment. The studies tested the vaccine antigens—not the vaccines.

The exclusion criteria of the Cochrane HPV review’s included trials

The Cochrane editors (1) did not consider our point that many of the Cochrane HPV review’s included studies had excluded female participants “if they had received the [aluminium] adjuvants before or had a history of immunological or nervous system disorders; for example, in the PATRICIA trial with 18 644 women and the FUTURE II trial with 12 167 women” (2). These exclusion criteria lower the external validity of the studies and suggest that there were concerns about harms caused in such participants by the adjuvants.

4) The Cochrane editors’ response on the Cochrane HPV review’s included outcomes

The Cochrane editors state that “The selection of outcomes for benefits was appropriate and was consistent with World Health Organization [WHO] guidance” (1).

In 2004, the WHO recommended the use of cervical intraepithelial neoplasia or worse: CIN2+, as the primary outcome (15). CIN2+ is a composite surrogate outcome for cervical cancer and includes CIN2, CIN3, AIS and cervical cancer. In 2014, the WHO recommended persistent HPV infection instead of CIN2+ (16). The WHO’s CIN2+ and persistent HPV infection recommendations were approved to “accelerate vaccine development and evaluation” (16). Since 2014, HPV vaccines have only been required to show benefits against persistent HPV infection for getting regulatory approval as a vaccine against HPV related cancer (16).

According to the 2004 WHO recommendations, “Representatives of industry did NOT participate in the drafting of recommendations” of the use of CIN2+ (15), but researchers with conflicts of interest did participate in the recommendations, for example, Ian Frazer—the co-inventor of the HPV vaccine who “receives royalties from sales of HPV prophylactic vaccines, and is a consultant for Merck, [and] GlaxoSmithKline” (17). In 2014, all 17 members of the WHO group that recommended persistent HPV infection instead of CIN2+ as the primary outcome had financial ties with the HPV vaccine manufacturers. For example, the group included two patent-holders of the HPV vaccines antigens (or “virus-like particles”), who are entitled “to a limited share of royalties [that] the NIH [National Institutes of Health] receives for these technologies” (16).

Outcomes such as CIN2+ can be difficult to interpret, and significant clinical differences can be hidden in the Cochrane HPV review’s meta-analyses (3). For example, as an extreme example, if there were 5 participants with CIN2+ in the HPV vaccine group and 10 in the comparator group, the 5 participants in the HPV vaccine group could theoretically all have cervical cancer while the 10 in the comparator group could have CIN2 lesions that often regress (18,19).

The Cochrane editors (1) did not address our point that the VIVIANE study (HPV-015) included in its register entry “one case of ‘Adenocarcinoma of the cervix’ and one case of ‘Cervix cancer metastatic’ … in the HPV vaccine group” (2), and that the Cochrane HPV review includes a death caused by “Cervix cancer metastatic” in the HPV vaccine group, which was not mentioned in the main text (3). The Cochrane editors (1) did not address our point that the “Cochrane review’s 26 trials mainly included women below age 30 and used frequent cervical screening (often every 6 months) that did not reflect real-life practice (often every 3–5 years),” which also lower the external validity of the studies (2).

The Cochrane HPV review’s primary analysis—Analysis 1.1 that includes four trials (CVT, FUTURE I, II and PATRICIA)—was of “High‐grade cervical lesions in hr[high risk]HPV DNA negative women at baseline: CIN2+ associated with HPV 16/18 [HPV types 16 and 18 are targeted by the HPV vaccines]”. Analysis 1.1 is affected by selection bias. Up to 15% of cervical cancers may not contain HPV (20), and many cervical cancers are infected with more than one HPV type. For example, in the clinical study report that we received for PATRICIA (HPV-008), 63 of the 102 of CIN2+ cases were co-infected with two or more HPV types. In PATRICIA, if an HPV vaccine and a comparator participant were both diagnosed with CIN2+ and positive for HPV types 31 and 33, and HPV 16/18, 31 and 33, respectively, the HPV vaccine CIN2+ case would be assigned as caused by non-vaccine types (31 and 33) and excluded from the analyses (such as Analysis 1.1), while the comparator case would be caused by HPV 16/18 (HPV vaccine types) and included in the analyses; even though HPV types 31 and 33 could have caused the CIN2+ lesions in both participants. The Cochrane HPV review’s analyses of HPV infection (“Comparison 4, 5 and 6”) include 21 meta-analyses that all analyse infection of HPV vaccine types (i.e., HPV types 6, 11, 16 and 18)—not infection of any HPV type, which would decrease confounding by HPV co-infection.

Another issue with Analysis 1.1 (3) is the large proportion of excluded females: 23,676 participants were included, but the included four studies (CVT, FUTURE I, II and PATRICIA) randomised 43,732 participants, so 46% females were excluded. The Cochrane authors did not mention that FUTURE I, II and PATRICIA—that contained 49% (36,266/73,428) of the Cochrane review’s sample (3)—were stopped early when HPV 16/18-related CIN2+ was significantly reduced for the HPV vaccine populations. Trials stopped early for benefits are known to exaggerate the effects by 29% on average compared to completed trials of the same intervention (21).

The majority (24 of 31) of the Cochrane HPV review’s meta-analyses of histological outcomes (“Comparison 1, 2 and 3”) consider cervical lesions associated with HPV vaccine types (3). A less biased meta-analysis of cervical lesions is “Analysis 3.7: High‐grade cervical lesions in women regardless of baseline HPV DNA status: Any CIN2+ irrespective of HPV types, at least 1 dose.” The Cochrane HPV review’s primary analysis—Analysis 1.1—is much more statistically significant than Analysis 3.7 (3): risk ratio 0.01 (95% CI 0.01 to 0.05) vs. 0.79 (95% CI 0.65 to 0.97), i.e., a ratio of relative risks of 0.01 (95% CI 0.006 to 0.03), which may reflect the selection bias in Analysis 1.1.

Both the 1st Cochrane HPV review protocol from 2011 (10) and the 2nd protocol from 2013 (11) list "Invasive cervical cancer" as a primary outcome. The protocols state that the Cochrane authors “will contact study authors or data owners to request data on the outcomes that were not reported” (11), which they did not do for invasive cervical cancer. Also, “If data are reported for grouped end points, we will contact trial authors or data owners to request data on the separated outcomes” (11), which the authors did not do for CIN2+ and CIN3+. But the Cochrane authors could have looked in the journal publications; for example, CIN3 irrespective of HPV type in intention-to-treat populations was reported in FUTURE I (22) (“79/2723 [in the HPV vaccine group] vs. 72/2732 [in the AAHS group]”) and FUTURE II (23) (“127/6087 [in the HPV vaccine group] vs. 161/6080 [in the AAHS group]”: in total 206/8,810 vs. 233/8,812; risk ratio 0.91 [95% CI 0.66 to 1.27]).

Furthermore, the Cochrane authors write that “No results were found for the outcomes any [sic] CIN3+ or AIS+ irrespective of HPV type” (3). If the Cochrane authors had looked in the freely available clinical study reports on GlaxoSmithKline’s trial register that the authors assessed, they would have found the outcomes CIN3+ irrespective of HPV type for PATRICIA (86/9,319 vs. 158/9,325) and HPV-032/063 combined (9/464 vs. 14/463).

5) The Cochrane editors’ assessment of the Cochrane authors’ conflicts of interest

The Cochrane editors state that “The review was compliant with Cochrane’s current conflict of interest policy” (1). If that is the case, we believe Cochrane should reconsider its policy.

The Cochrane HPV vaccines are expensive blockbuster vaccines generating billions of dollars of revenue (24), and the Cochrane review ought, therefore, to have been independent of any financial conflicts of interests.

The Cochrane editors are confident that the Cochrane authors have no relevant conflicts of interest (2). We do not agree. For example, the Cochrane HPV review’s first author, Professor Marc Arbyn—who, according to the Cochrane review (3), only “received travel grants from MSD‐Sanofi‐Pasteur and GSK, (ceased in 2008)”—was until 2008 on GlaxoSmithKline’s advisory board: “Marc Arbyn (GSK advisory Board (interrupted in 2008))”) (25); in 2011, “EUROGIN covered his [Marc Arbyn’s] travel and lodging expenses … EUROGIN conferences are financially supported by a range of pharmaceutical companies with an interest in cervical cancer” (26); in 2014, “Marc Arbyn's research unit at The Scientific Institute of Public Health received research support not exceeding 48,000 Euros from MSD-Sanofi Pasteur [co-manufacturer of Gardasil] for a surveillance study of the effects of HPV vaccination in Belgium (SEHIB study) ... His [Marc Arbyn’s] unit has also received research support from BD, Bio-Greiner, Abbot, and Cepheid for validation studies of HPV genotyping tests (through the VALGENT studies, valued at 21,000, 21,000 & 38,000€ respectively)” (27); and in 2018, Marc Arbyn is on the EUROGIN programme committee where Merck is a platinum sponsor (28). The Cochrane review’s last author, Dr. Markowitz is sponsored by Merck via Medscape (“sponsored by the manufacturer of the quadrivalent vaccine (“supported by an independent educational grant from Merck”) (29).

The Cochrane editors (1) do not think that the Costa Rica trial (“CVT”, aka HPV-009) was industry funded, and they refer to its publication in JAMA that states that the trial was "funded by the NCI (grant N01-CP-11005).” The editors write, with reference to JAMA, that “Vaccine was provided for our trial by GSK [GlaxoSmithKline] Biologicals, under a Clinical Trials Agreement with the NCI" (1). GlaxoSmithKline also provided support for aspects of the trial associated with regulatory submissions under “FDA BB-IND 7920” (30). We consider this industry funding.

6) The Cochrane editors’ assessment of the media coverage

The Cochrane editors (1) did not comment on our note that “Two of the experts had financial conflicts of interest with the HPV vaccine manufactures … [and that] No expert criticised the review” (2).

The editors (1) write that “press coverage could be made more explicit on our organizational websites and other communications, essentially noting that these opinions represent personal perspectives from a range of contributors and do not reflect the views or policies of Cochrane” (1). We agree, but stress that Cochrane’s press officer ought to only include researchers with no financial conflicts of interest.

7) The Cochrane editors appear to advocate scientific censorship

The Cochrane editors wrote that “Scientific debate is to be welcomed, and differences of opinion between different Cochrane 'voices' is not unexpected. However, public confidence may be undermined, unnecessary anxiety caused, and public health put at risk if that debate is not undertaken in an appropriate way. This is especially true when such debates take place in public. There is already a formidable and growing anti-vaccination lobby. If the result of this controversy is reduced uptake of the vaccine among young women, this has the potential to lead to women suffering and dying unnecessarily from cervical cancer.” We believe that our criticism of the Cochrane HPV review is appropriate and has general interest. We believe that providing an assessment of all the evidence reduces uncertainty and allows the public to make informed decisions based on the benefits and the harms of HPV vaccines. Debates over sources of evidence must take place in public, especially when public health interventions are at stake.

8) Conclusion

We did not “substantially overstate” (1) our criticisms of the Cochrane HPV vaccine review (2). The Cochrane editors substantially ignored several of our criticisms. The Cochrane HPV review is still incomplete and ignores important evidence of bias.

The Cochrane editors stated that “Some of the criticisms will inform the next version of this Cochrane Review and the planned review of comparative studies of HPV vaccines,” and that the editors “recognize public concerns about the aluminium-based adjuvants” (1).

The editors also stated that “reliance on the published reports in scientific journals may introduce bias due to incomplete and selective reporting” (1). We agree and remind the Cochrane editors that the Cochrane review on neuraminidase inhibitors substantially changed its conclusions after it got updated and became based on clinical study reports instead of journal publications (31).

With our analysis (2), we have contributed to a scientific debate in an area that is complex and biased. The Cochrane authors stated that they will make a “Request for non‐published available data” such as clinical study reports that “will be integrated in future updates of the review” (3). We can offer them these data, which we have used for our own systematic review that we have submitted for publication.

Article info

Conflicts of interest: LJ and PCG have no conflicts of interest to declare. TJ was a recipient of a UK National Institute for Health Research grant for a Cochrane review of neuraminidase inhibitors for influenza. In addition, TJ receives royalties from his books published by Il Pensiero Scientifico Editore, Rome and Blackwells. TJ is occasionally interviewed by market research companies about phase I or II pharmaceutical products. In 2011-13, TJ acted as an expert witness in litigation related to the antiviral oseltamivir, in two litigation cases on potential vaccine-related damage and in a labour case on influenza vaccines in healthcare workers in Canada. He has acted as a consultant for Roche (1997-99), GSK (2001-2), Sanofi-Synthelabo (2003), and IMS Health (2013).In 2014 he was retained as a scientific adviser to a legal team acting on oseltamivir. TJ has a potential financial conflict of interest in the drug oseltamivir. In 2014-16, TJ was a member of three advisory boards for Boerhinger Ingelheim. TJ was holder of a Cochrane Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. TJ was a member of an independent data monitoring committee for a Sanofi Pasteur clinical trial on an influenza vaccine. Between 1994 and 2013, TJ was the coordinator of the Cochrane Vaccines Field. TJ was a co-signatory of the Nordic Cochrane Centre Complaint to the European Medicines Agency (EMA) over maladministration at the EMA in relation to the investigation of alleged harms of HPV vaccines and consequent complaints to the European Ombudsman. TJ is co-holder of a John and Laura Arnold Foundation grant for development of a RIAT support centre (2017-2020) and Jean Monnet Network Grant, 2017-2020 for The Jean Monnet Health Law and Policy Network. TJ is an unpaid collaborator to the project Beyond Transparency in Pharmaceutical Research and Regulation led by Dalhousie University and funded by the Canadian Institutes of Health Research (2018-2022).

Authors' experience: PCG has co-authored 17 Cochrane reviews, including several reviews based on clinical study reports. TJ has co-authored 17 Cochrane reviews including the first Cochrane review that used clinical study reports. LJ has co-authored several articles on the HPV vaccines.

Authors' contributions: LJ wrote the first draft. LJ, PCG and TJ contributed to the conception, drafting,
critical revision for important intellectual content and the final approval of the article.

Acknowledgements: We would like to thank the Cochrane editors for taking our analysis into consideration and the BMJ-EBM editors for allowing us to respond via their journal.

Provenance and peer review: Commissioned; not externally peer-reviewed.

Copyright information: © Article author(s) (or their employer(s) unless otherwise stated in the text of the
article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

References
1. Tovey D, Soares-Weiser K. Cochrane’s Editor in Chief responds to BMJ EBM article criticizing HPV review. 2018; 3 Sept. https://www.cochrane.org/news/cochranes-editor-chief-responds-bmj-ebm-ar....
2. Jørgensen L, Gøtzsche PC, Jefferson T. The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evidence-Based Medicine. 2018 Jul 27;bmjebm-2018-111012.
3. Arbyn M, Xu L, Simoens C, Martin‐Hirsch PP. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. John Wiley & Sons, Ltd; 2018. Available from: http://cochranelibrary-wiley.com/doi/10.1002/14651858.CD009069.pub3/full
4. Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review. PLOS Medicine. 2016 Sep 20;13(9):e1002127.
5. Jørgensen L, Gøtzsche PC, Jefferson T. Index of the human papillomavirus (HPV) vaccine industry clinical study programmes and non-industry funded studies: a necessary basis to address reporting bias in a systematic review. Systematic Reviews. 2018 Jan 18;7:8.
6. Hawkes N. HPV vaccine safety: Cochrane launches urgent investigation into review after criticisms. BMJ. 2018 Aug 9;362:k3472.
7. Cochrane’s Editor in Chief responds to BMJ EBM article criticizing HPV review. Available from: /news/cochranes-editor-chief-responds-bmj-ebm-article-criticizing-hpv-review
8. Skinner SR, Szarewski A, Romanowski B, Garland SM, Lazcano-Ponce E, Salmerón J, et al. Efficacy, safety, and immunogenicity of the human papillomavirus 16/18 AS04-adjuvanted vaccine in women older than 25 years: 4-year interim follow-up of the phase 3, double-blind, randomised controlled VIVIANE study. Lancet. 2014 Dec 20;384(9961):2213–27.
9. Human Papilloma Virus (HPV) Vaccine Efficacy Trial Against Cervical Pre-cancer in Young Adults With GlaxoSmithKline (GSK) Biologicals HPV-16/18 - Study Results - ClinicalTrials.gov. Available from: https://clinicaltrials.gov/ct2/show/results/NCT00122681
10. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors - Arbyn, M - 2011 | Cochrane Library. Available from: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD009069/full
11. Arbyn M, Bryant A, Martin-Hirsch PP, Xu L, Simoens C, Markowitz L. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database of Systematic Reviews. John Wiley & Sons, Ltd; 2013. Available from: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD009069.pub2/abstract
12. Expert consultation on the use of placebos in vaccine trials. World Health Organization. 2013. Available from: apps.who.int/iris/bitstream/handle/10665/94056/9789241506250_eng.pdf?sequence=1
13. Krause P. Update on Vaccine Regulation: Expediting vaccine development. FDA/CBER/OVRR. Available from: https://c.ymcdn.com/sites/www.casss.org/resource/resmgr/CMC_Euro_Speaker...
14. Thiriot DS, Ahl PL, Cannon J, Lobel GM. Method for preparation of aluminum hydroxyphosphate adjuvant. WO2013078102A1, 2013. Available from: https://patents.google.com/patent/WO2013078102A1/en
15. Pagliusi SR, Teresa Aguado M. Efficacy and other milestones for human papillomavirus vaccine introduction. Vaccine. 2004 Dec 16;23(5):569–78.
16. IARC HPV Working Group. Primary End-points for Prophylactic HPV Vaccine Trials [Internet]. Lyon (FR): International Agency for Research on Cancer; 2014. Available from: http://www.ncbi.nlm.nih.gov/books/NBK304971/
17. Trimble CL, Frazer IH. Development of therapeutic HPV vaccines. Lancet Oncol. 2009 Oct;10(10):975–80.
18. Motamedi M, Böhmer G, Neumann HH, von Wasielewski R. CIN III lesions and regression: retrospective analysis of 635 cases. BMC Infect Dis. 2015 Nov 21;15. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4654894/
19. Moscicki A-B, Ma Y, Wibbelsman C, Darragh TM, Powers A, Farhat S, et al. Rate of and Risks for Regression of CIN-2 in adolescents and young women. Obstet Gynecol. 2010 Dec;116(6):1373–80.
20. de Sanjose S, Quint WG, Alemany L, Geraets DT, Klaustermeier JE, Lloveras B, et al. Human papillomavirus genotype attribution in invasive cervical cancer: a retrospective cross-sectional worldwide study. Lancet Oncol. 2010;11:1048–56.
21. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010 Mar 24;303(12):1180–7.
22. Garland SM, Hernandez-Avila M, Wheeler CM, Perez G, Harper DM, Leodolter S, et al. Quadrivalent vaccine against human papillomavirus to prevent anogenital diseases. N Engl J Med. 2007 May 10;356(19):1928–43.
23. FUTURE II Study Group. Quadrivalent vaccine against human papillomavirus to prevent high-grade cervical lesions. N Engl J Med. 2007 May 10;356(19):1915–27.
24. Clendinen C, Zhang Y, Warburton RN, Light DW. Manufacturing costs of HPV vaccines for developing countries. Vaccine. 2016 Nov 21;34(48):5984–9.
25. De Belgische Kamer van volksvertegenwoordigers. Available from: https://www.dekamer.be/kvvcr/showpage.cfm?section=qrva&language=nl&cfm=q...
26. Information NC for B, Pike USNL of M 8600 R, MD B, Usa 20894. Declarations of interest. World Health Organization; 2013. Available from: https://www.ncbi.nlm.nih.gov/books/NBK195238/
27. Cancer Screening in the European Union (2017) Report on the implementation of the Council Recommendation on cancer screening. Health and Food Safety. Available from: https://docplayer.net/43107833-Cancer-screening-in-the-european-union-20...
28. EUROGIN 2018. Available from: https://www.eurogin.com/2018/341-program-committee.html
29. Conflicts of interests. Available from: http://www.nogracias.eu/wp-content/uploads/2013/05/vacc-HPV-Cochrane_HPV...
30. Herrero R. Human Papillomavirus (HPV) Vaccines: Limited Cross-Protection against Additional HPV Types. J Infect Dis. 2009 Apr 1;199(7):919–22.
31. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJ, et al. Neuraminidase inhibitors for preventing and treating influenza in adults and children. The Cochrane Library. John Wiley & Sons, Ltd; 2014. Available from: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD008965.pub4/full

The Cochrane Review of the HPV vaccine states :

"In older women, vaccinated between 25 to 45 years of age, the effects of HPV vaccine on precancer are smaller, which may be due to previous exposure to HPV. The risk of precancer associated with HPV16/18 is probably reduced from 145/10,000 in unvaccinated women to 107/10,000 women following HPV vaccination (moderate certainty). The risk of any precancer is probably similar between unvaccinated and vaccinated women (343 versus 356/10,000, moderate certainty)."

Cochrane estimates a significant effect of the vaccine on HPV 16/18 associated precancer in the 25-45 age group, but that effect disappears when risk of precancer of all (HPV) types is calculated. It is a matter of simple logic that for the positive (desired) effect on type 16/18 precancer to be offset in this way, the vaccine must have a corresponding negative (undesired) effect on precancer associated with other HPV types.

The observation holds for all age groups, but is more pronounced in the older group. It is reasonable to assume that if one took, say, a 35 to 45 years old group, the total undesired result of the HPV vaccination could be statistically significant.

If my reading of these numbers is too simplistic, it would be good to get a clarification, since the quoted passage appears right in the Summary with no further comments.

In Europe, the vaccine is routinely recommended to women of all ages.

References
[1] Arbyn M, Xu L, Simoens C, Martin-Hirsch PPL. Prophylactic vaccination against human papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database Syst Rev. 2018; (5): CD009069.