A Cochrane systematic review of human papillomavirus (HPV) vaccine (hereafter referred to as the Cochrane HPV Review) was published on 9th May 2018.

The article, ‘The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias’ was submitted to BMJ EBM on 24th May 2018 for the ‘Debate, analysis and opinion’ section of the journal. The handling editor, Dr Igho Onakpoya (Research Editor, BMJ EBM) sent it for external peer review to an expert in HPV vaccines and for internal peer review to Professor Carl Heneghan, Editor in Chief of BMJ EBM. Peer reviewers’ reports were returned by 19th June and were sent to the authors, who were invited to make revisions. The revised version was submitted on third July, accepted by the handling editor on seventh July and published online on 27th July 2018.¹

The analysis of the Cochrane HPV review stated there were missing eligible trials, reporting bias, and biased trial designs and conflicts of interest.^{2 3}

Cochrane initiated an investigation in response to the criticism and published a response to the article on third September authored by Cochrane’s then Editor in Chief (EiC) David Tovey and deputy EiC Karla Soares-Weiser. This response defended the Cochrane HPV Review and outlined key findings from the Cochrane investigation:

• The Cochrane Review did not miss “nearly half of the eligible trials”. A small number of studies were missed due to the primary focus on peer-reviewed reports in scientific journals, but the addition of these data makes little or no difference to the results of the review for the main outcomes;

• The trials comparators were unambiguously, transparently, and accurately described;

• The selection of outcomes for benefits was appropriate and was consistent with WHO guidance;

• The review included published and unpublished data on serious harms, and the findings on mortality were reported transparently and responsibly;

• The review was compliant with Cochrane’s current conflict of interest policy;

• Cochrane’s media coverage was cautious and balanced, but we recognise that there could be improvements in relation to transparency where external experts are quoted;

• The BMJ Evidence-Based Medicine article “substantially overstated its criticisms”⁴

The response also criticised the peer review process of the journal and whether the conclusions were justified and proportionate. When these criticisms were raised with the journal, we embarked on a lengthy clarification process with the authors and Cochrane.

September 2018

We wrote to the authors of the BMJ EBM analysis article and received a response that asserted ‘(their) analysis was appropriate and that the Cochrane editors substantially ignored several of (their) criticisms’.⁵ We also contacted Cochrane’s EiC and the Cochrane corresponding author to ask for further details of their criticisms of the BMJ EBM peer review process and how the article was overstated.

We set out the contentious questions arising from the Cochrane Editors’ response and considered whether the BMJ EBM analysis⁶ required further editorial comments or corrections. We sent a copy of this review to all parties concerned and posted it on BMJ EBM Spotlight on 16^th of October 2018. Our review concluded that the BMJ EBM analysis required several corrections and that the overall article did not overstate its claims, nor did it warrant retraction.

The authors of the BMJ EBM analysis article responded to the conclusions of the Cochrane investigation in a rapid response to the original article. They acknowledged and approved the conclusions of the BMJ EBM Editors’ Review and provided further details and clarification. Jørgensen et al. 2018

October 2018

The Cochrane HPV review corresponding author contacted The BMJ to enquire whether The BMJ might provide a forum to reply. The BMJ EiC suggested sending a detailed response to the BMJ EBM publication, a rapid response to The BMJ and potentially an opinion piece about the broader issues.

November 2018

Cochrane EiC emailed the BMJ EBM EiC to say that there remained uncertainties with identifying studies for inclusion. Six studies had now been identified and will be added to the Cochrane HPV Review. Their re-analysis of the data incorporating the missing data suggested that their results did not change with the inclusion of this data. The email also reiterated concerns with the BMJ EBM peer review process for that article, specifically that there was only one external peer reviewer.

2019

Since then BMJ EBM has updated its editorial policy to ensure EBM Analysis pieces are reviewed by two external peer reviewers at a minimum. Debate, Insights and opinion pieces are sent to external or internal review, and EBM verdicts as commentaries are subject to internal review. The journal’s peer review policy is stated on the BMJ EBM website to increase transparency.

We have published a correction to the BMJ EBM HPV analysis article,¹ alongside this linked Editor’s Note.

The BMJ EBM analysis article by Jørgensen, Gøtzsche and Jefferson defined methods for securing the available evidence and we consider the article title, and conclusions, are justified. Until the updated review is published we remain unclear about the actual number of trials that require inclusion in the Cochrane Review, and whether the missing trials impact on the Cochrane results.

Competing interests

CH has co-authored 16 peer-reviewed articles with Tom Jefferson (two of which are Cochrane reviews) and holds grant funding jointly with Tom Jefferson from Cochrane on methods for deciding when to prioritise the use of clinical study reports in systematic reviews. CH is a member of Cochrane and a contact editor for the Cochrane Acute Respiratory Infection group and has been an author 21 Cochrane reviews including updates. He is a founder of AllTrials and an advisor to the WHO International Clinical Trial Registry Platform. and He has received expenses and fees for his media work (including payments from BBC Radio 4 Inside Health). He has received expenses from the WHO and holds grant funding from the NIHR, the NIHR School of Primary Care Research, NIHR BRC Oxford and Cochrane. He has received financial remuneration from an asbestos case and given free legal advice on mesh cases. He has also received income from the publication of a series of toolkit books published by Blackwells. On occasion, he receives expenses for teaching EBM and is also paid for his NHS GP work in urgent care (contract with Oxford Health NHS Foundation Trust). He is Director of CEBM at the University of Oxford, which jointly runs the EvidenceLive Conference with the BMJ and the Overdiagnosis Conference with international partners, based on a non-profit making model. He is Editor in Chief of BMJ Evidence-Based Medicine. IO is Research Fellow in Evidence Synthesis at the CEBM, University of Oxford, a clinician who works across several NHS Trusts and is funded by the NIHR School of Primary Care Research. IO has co-authored eight peer-reviewed articles jointly with Tom Jefferson (one of these is a Cochrane review).