Table 1

Peer review comments and responses: The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias - 3 July 2018: accessed from scholar one.

Serial	Peer-reviewer comments	Authors’ comment	Authors’ action
1	Peer-reviewer no. 1: The paper is well written and makes some important points:	Thank you very much.	None.
2	there is one issue to think about which is the structure of how the biases are presented. 1) Therefore the article would benefit from a summary box or powerpoint figure that summarises the main biases - could the authors consider this as a teaching aid when discussing the major biases in the review, and consider the structure of the major headings	Thank you for the suggestion. We tried a box and a figure, but we did not find them to fit or add much to the paper, since we list the major biases in the subheaders.	We inserted three key findings in the beginning of of the paper.
3	the Cochrane authors stated that ‘nearly all end-of-study reports have been published in the peer-reviewed literature.’ it is not clear to me where this was stated	The Cochrane authors stated the sentence under the sub-header ‘Differences between protocol and review.’	We have clarified that this was stated in the Cochrane review.
4	It is noteworthy that many females were only included in the trials -It is not clear to me in the next sentence whether these women were therefore excluded - consider separation of the issue of adjuvants from external validity. I understand these are interlinked, but they are two separate limitations - reduced reporting of harms and reduced external validity	We agree that this is unclear.	We changed the sentence to "It is noteworthy that many females were excluded from the trials if they had received the adjuvants before and had a history of immunological or nervous system disorders…" We did not separate the criteria, as they all may lessen external validity and decrease harms differences.
5	regarding the composite outcome would the header be better as the ’wrong outcome or ’clinically irrelevant outcome' and then there are three issues to discuss the surrogate and the composite and the lack of a clinically relevant issue	Outcomes such as CIN2+are considered the ‘right clinically relevant’ outcomes, as these outcomes were approved by WHO (see Pagliusi 2004).	Our sub-header now points more to the limitations of these outcomes.
6	The Assessment of harms section might also benefit from a table of the author’s major findings and conclusions and the comments. This would allow the text in this section to be tightened.	Thank you for the suggestion.	We have expanded our sub-headers to tighten the text. We have included ‘Key findings’ that we submitted in the revised manuscript.
7	In the conclusion, there is no mention of what could, or should, be done differently. I consider this is is an oversight. For example, should the review be retracted? A summary of recommendations would help guide what Cochrane reviews might do differently to avoid all of these issues.	We agree that a recommendation is a good idea.	We have inserted our recommendation in the conclusion.
8	Peer-reviewer no. 2: The analysis conducted in the manuscript The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias is interesting and contributes to the debate on quality of systematic review and problems of transparency and financial conflict of interests. It brings the internal debate within Cochrane groups or collaborators to public domain, and this certainly add to transparency, but it also highlights difficulties in solving these problems with processes codified by the Cochrane Collaboration itself. It could be worth to comment a little bit on this, for readers that are not familiar with the Cochrane Collaboration organisation and dynamics.	Thank you. We do not think that a comment of this kind is relevant for our analysis. We are not even sure that the problems with the Cochrane review reflects any particular difficulties in the setup of Cochrane. This could happen for any medical journal.	None.
9	Few other aspects need to be clarified: missing trials: this is the core of the discussion. Many trials were not included in the Cochrane review, although that was suggested from the authors of the analysis. How the inclusion of other trials would have changed the estimate of HPV vaccine efficacy? Please comment on it.	We do not know how the inclusion of unpublished trials would affect the Cochrane review’s results. However, we are aware that unpublished trials 10have a tendency to have less positive results than published trials.	None.
10	The Cochrane review authors stated that they searched journal publication databases, trial registers, reference list, experts in the field, and contacted WHO, ECDC, CDC, IARC (page 11 of the Cochrane review): which of the six steps that you described in the paper Index of the human papillomavirus (HPV) vaccine industry clinical study programmes and non-industry funded studies: a necessary basis to address reporting bias in a systematic review would have contributed more to fill the gap? Please, comment on it.	We agree.	We inserted our suggestion in the text.
11	Please, also comment on feasibility of following such a procedure in every systematic review. It seems that the process suggested is the optimum way to find all relevant data but, being demanding and time consuming, it should be suggested when it is really needed. It is easy to agree that for an expensive intervention involving such a large part of a healthy population, as it is the case for HPV vaccination campaign, it should be needed. But maybe this is not always the case. Please, elaborate on this aspect.	Cochrane reviews are by default obliged to try to identify all eligible trials. Our aim was to analyse this particular Cochrane review—not to make recommendations for all Cochrane reviews. The new version of the Cochrane Handbook (that likely will be published within 2018) will hopefully make the necessary recommendations for all Cochrane reviews.	We have inserted our recommendations in the paper.
12	2. composite surrogate outcomes: Line 40 page 3. While CIN2+is a surrogate outcome not valid, CIN3+can be considered more reliable, as CIN3 shows a lower rate of spontaneous regression. Moreover, time interval from immunisation and detection of lesions was not long enough to detect cancers in most of the published trials. Please, differentiate between the two outcomes.	CIN3+is more reliable, but CIN2+was used as the primary outcome since WHO recommended it in 2004 (Pagliusi et al.).	We included CIN3+.
13	In your paper you do not discuss the doubtful correspondence between naïve girls included in the vaccine trials and the 12 years old girls targeted by the immunisation programme. In the trials girls that were naïve to HPV viruses were not necessarily girls without contact with the viruses; based on data provided they often had more than one sexual partner and were regular users of contraception. They were maybe different, more resistant, or maybe it was just a matter of different sensitivity of tests used in different trials to assess naïve status (see Di Mario S, et al. Are the Two Human Papillomavirus Vaccines Really Similar? A Systematic Review of Available Evidence: Efficacy of the Two Vaccines against HPV. J Immunol Res. 2015;2015:435141. doi:10.1155/2015/435141). Lack of correspondence between naïve status and 12 years target can explain the lower vaccine efficacy detected in field studies (see Lehtinen M, et al. Ten-year follow-up of human papillomavirus vaccine efficacy against the most stringent cervical neoplasia end-point-registry-based follow-up of three cohorts from randomised trials. BMJ Open. 2017 Aug18;7 (8):e015867. doi: 10.1136/bmjopen-2017–0 15 867 and VänskäS, et al. Estimating effectiveness of HPV vaccination against HPV infection from post-vaccination data in the absence of baseline data. Vaccine. 2018 May	It is an important, but the real-life efficacy is not part of the Cochrane review. We believe we discuss the major limitations in the trials.	None.
14	3. assessment of harms: Lines 10–12 page 4: please, check that numbers are correctly reported. It seems to me that they should be the other way around, but maybe I am wrong.	The numbers are correct.	None.
15	Line 26 page 4: the problem of reporting only serious adverse event occurring within 14 days from immunisation is not a problem of the Cochrane review, but it is a more general one: why were those papers published in eminent international journals in the first place, without asking more data? This point could require a line of comment.	Although an interesting point, it is not the aim of our analysis to question editorial practices in journals.	None.
16	Lines 39–40 page 4: about death, is seems not enough to affirm that as data come from RCT the only possible reason for higher number of deaths is the vaccine. If careful assessment of causes of deaths does not link the events with the vaccination other explanations are possible (being death a rare event). Please articulate the comment at this regard.	Randomisation protects against confounding when counting the number of deaths in the two groups. Deaths could be caused by vaccines although being coded differently (eg, a ‘traumatic brain injury’ or ‘drowning’ could have been caused by a ‘syncope’).	We have expanded on this point.
17	4. other bias: Lines 18–19 page 6: the meta-regression was meaningless, which it would also have been with only one non-manufacturer sponsored trial. Could you please explain the reasons of this statement for not expert readers?	Agree.	We deleted ‘which it would also have been with only one nonmanufacturer sponsored trial’ to avoid confusion.
18	5. conclusions: Lines 14–17 page 7. Conclusions are clear. The Cochrane systematic review is biased and incomplete, probably due to financial conflict of interest. But, based on your indexing complete HPV vaccine study programmes, how do you think that including other trials would have changed the Cochrane review conclusions in term of efficacy of HPV vaccines? Please, add. This is relevant for citizens to make informed decisions.	We have not studied this and it was not the aim of our paper. We have criticised a Cochrane review and we would rather urge the Cochrane authors to include the missing trials.	We have inserted recommendations by the end of our paper.