Complaint to Chief Editor of Cochrane

The Cochrane vertebroplasty review (CVR) of April 20181 was updated in November 20182 to address complaints to the Chief Editor of Cochrane3 about errors in the report. The review does not accurately report the evidence for vertebroplasty in patients with severe symptoms and early fractures. The VAPOUR trial is the only blinded trial to specifically assess this patient group and found vertebroplasty more effective than placebo in alleviating severe pain. Readers of the review would be unable to discern this information from the way that the review has been presented. There is open randomised trial evidence supporting the findings of the VAPOUR trial that vertebroplasty is an effective remedy for severe pain when the intervention is performed early, mostly within the first 3 weeks. We are authors of the VAPOUR trial.

New randomised evidence

Three randomised trials comparing vertebroplasty with placebo4 5 or usual care6 have been published since the 2015 CVR.

The VAPOUR trial4 enrolled patients referred for vertebroplasty with severe, uncontrolled pain despite opiate analgesia. Eligible patients had fractures for a <6-week duration causing severe pain, uncontrolled by medical therapy including opiates. Vertebroplasty was offered without further delay. 79% of patients in the VAPOUR trial had the fracture duration ≤3 weeks at the time of intervention. This is the only blinded trial to enrol hospitalised inpatients. All patients required severe pain for enrolment, defined as the numeric rated score (NRS) of 7/10 or more. The primary outcome measured the proportion of patients who converted to a mild pain score (NRS <4/10) at 14 days. This favoured vertebroplasty over placebo at 14 days and at every other time point to 6 months (figure 1). Vertebroplasty reduced hospital stays and provided clinically significant reductions in the Roland Morris Disability score at 1, 3 and 6 months. Mean fracture duration at time of vertebroplasty was 2.8 weeks.

• Download figure
• Open in new tab
• Download powerpoint

Figure1

Primary outcome of the VAPOUR trial. Primary outcome is the proportion of patients in each group with mild pain (NRS<4/10) having enrolled with severe pain at baseline. NRS ≥7/10 was an inclusion requirement. Lines above each column represent 95% CIs. The positive primary outcome was rejected by the CVR. CVR, Cochrane vertebroplasty review; NRS, numeric rated score.

VERTOS47 recruited outpatients referred for radiography, not for vertebroplasty. Protocol7 states ‘all patients, 50 years of age or older, referred for an X-ray of the thoracic and/or lumbar spine, receive a short clinical questionnaire’. Patients who had a fracture, VAS pain ≥5/10 and pain duration ≤9 weeks (at the time of radiograph), were invited to provisionally enrol. The amended time of radiography is accessible with trial publication5 via ‘peer review’ tab and clicking on ‘protocol’, which lists amendments. Consenting patients were then referred for physician assessment and MRI scanning, causing additional 1–3 week delay after radiography, so that fracture duration would extend to 12 weeks at the time of vertebroplasty although the full range is not reported. The IQR of fracture duration was 4.1–7.4 weeks, so <20% of patients had fracture duration ≤3 weeks, compared with 79% in VAPOUR. Primary outcome, mean pain, was equivalent in placebo and vertebroplasty groups at all time points. Mean fracture duration at time of vertebroplasty was 6.1 weeks.

A study by Yang et al 6 is an open randomised trial which recruited patients with acute, severe pain and fracture duration <3 weeks duration at the time of the procedure. Vertebroplasty was immediately offered as acute pain treatment without further delay. A total of 135 patients were randomised to vertebroplasty or conservative care. The primary outcome, mean pain, was lower in the vertebroplasty group at all time points to 12 months. Mean fracture duration was 1.1 weeks.

The VOPE trial is unpublished, but its authors have kindly shared trial data and manuscript. Forty-six outpatients from an orthopaedic outpatient clinic with vertebral fracture and back pain ≤8 weeks duration were randomised to vertebroplasty (22) or placebo (24). The placebo involved a bone biopsy of the vertebral body with an 11G needle. Vertebroplasty injected 2–4cc PMMA, half that of the VAPOUR trial. Patients were younger (mean age 70 vs 80), with mild osteoporosis (T score −2.4 vs −4.3) and less severe pain (7.5/10 vs 8.6/10) than patients in the VAPOUR trial. There is no mention of opiate use. The authors report ‘we found in our study a significant higher VAS in the SHAM group throughout the follow-up period (p=0.001) when applying ANOVA statistical model on our data’, but there was no significant difference between groups at any single time point, although the trial was not powered to detect this. The unpublished VOPE manuscript concludes ‘When reviewing the evidence at hand, we believe PVP should be offered to patients with acute VCFs in severe pain and immobilised after optimal conservative treatment including pain medication’. This concurs with the findings of the VAPOUR trial. Mean/median fracture duration not available.

Clinical differences between VAPOUR and the other blinded trials

VAPOUR adopted a different clinical approach to other blinded trials. Rather than waiting for the pain to abate before offering vertebroplasty, VAPOUR offered earlier intervention in patients whose pain remained severe despite the use of opiate analgesia. VAPOUR patients were older, had higher pain score at entry and a substantial proportion had been hospitalised before enrolment, compared with other blinded trials which enrolled younger patients with less comorbidities and less severe symptoms, none of whom were hospitalised (table 1).

VAPOUR provides the only blinded evidence of vertebroplasty for hospitalised patients with severely painful vertebral fractures. Hospitalisation usually occurs due to uncontrolled pain and loss of independence within the first 3 weeks of fracture, when pain is maximal. Kallmes et al 8 excluded inpatients by the protocol. VERTOS4 excluded inpatients, confirmed by private author correspondence. VOPE exclusively recruited from an outpatient clinic. Buchbinder et al 9 did not report any hospital inpatients and this is consistent with the methods of the trial; trial design assessed outcome via posted questionnaires with no provision for inpatient follow-up, small enrolment (38 vertebroplasty patients) and few patients with fractures ≤3 weeks duration (fracture duration IQR 4–13 weeks). VAPOUR found a 6-day reduction in hospitalisation in the vertebroplasty group and all attending physicians were blinded, so patient discharge related to improved pain and functional status.

VAPOUR performed vertebroplasty earlier than other blinded trials. The mean fracture duration at the time of vertebroplasty in VAPOUR, VERTOS4, Buchbinder et al and Kallmes et al was 2.8, 6.1, 12 and 23 weeks, respectively. 79% of patients in VAPOUR had intervention within 3 weeks compared with <20% in other blinded trials (table 1). VAPOUR subgroup analysis by fracture duration suggested better vertebroplasty outcome in the <3-week group although there were insufficient patients in the 4–6 week group for statistical significance. All patients in Yang et al had a fracture duration <3 weeks and the results strongly favoured vertebroplasty.

VAPOUR used different vertebroplasty techniques to trials8 9 by authors of this CVR. The ‘vertebral fill technique’ illustrated in trial publication4 braced the whole vertebral body, not just the fracture line, requiring three times the PMMA volume of the 2009 trials (table 1). Dismissing technical differences between VAPOUR and their trials,3 CVR authors reference level 4 evidence10 from chronic fracture treatment to assert that PMMA volume does not affect the outcome. Patients with early fractures and uncontrolled pain are likely to have fracture instability and plasticity. The ‘vertebral fill’ technique is designed for this patient group to prevent further collapse and restore vertebral height. Calibrated radiographs at 6 months demonstrated 36% vertebral height improvement with vertebroplasty compared with placebo. This is an entirely different vertebroplasty approach to the trials of the CVR authors.

The positive findings of VAPOUR are limited to patients with severe pain caused by fractures <6 weeks duration and are not generalisable to patients with older fractures or non-severe symptoms.

Trials of vertebroplasty for fractures <6 weeks duration

One blinded4 and three open6 11–13 randomised trials for fractures <6 weeks duration are published. VAPOUR, VERTOS2 and Yang et al showed clear benefits from vertebroplasty in pain reduction.

Rousing et al 12 13 failed to meet its primary endpoint (pain reduction at 3 months), but was underpowered to detect this (47 patients randomised). Baseline pain differed between groups (p=0.02), and 28% of patients did not have baseline pain measured. Vertebroplasty significantly reduced NRS pain at 1 day and 1 month and shortened duration of hospitalisation.

Meta-analysis14 of 25 patients from twin non-acute blinded trials8 9 who underwent vertebroplasty within 6 weeks of pain onset, found no benefit for vertebroplasty. The analysis was underpowered, and the parent trials had fundamental clinical differences to VAPOUR (table 1).

There is no analysis of outcomes from the subgroup of patients in VERTOS4 with fractures <6 weeks duration.

Protocol breaches

CVR protocol15 states ‘Prior to meta‐analysis, studies will be assessed for clinical homogeneity with respect to study population, type of therapy, …. Clinically heterogeneous studies will not be combined for analysis but will be individually described’.

Important clinical and technical differences between the VAPOUR trial and the other blinded trials are listed in table 1. Different degrees of pain, osteoporosis, hospitalisation status, earlier intervention and technical differences in PMMA administration are key contrasts between the trials. Despite this heterogeneity, VAPOUR has been combined for analysis with these trials, breaching Cochrane protocol. This error in analysis renders the conclusions of the CVR unsupported.

CVR (April 2018) Analysis 8.1/2, not specified by protocol, wrongly included all VERTOS4 patients in subgroup of fractures <6 weeks duration despite half having fractures >6 weeks, rendering the analysis nonsense. Rather than removing incorrect data from the subgroup, the November version opted to retain these data but to change the designation of the subgroup.

The subgroup previously defined as ‘fracture duration ≤6/52’ was replaced by a subgroup of ‘acute fractures’ defined as any blinded trial using the word ‘acute’ in publication. ‘Acute’ fractures were extended to 12 weeks rather than 6 as previously. Manipulation of a subgroup analysis to arbitrarily fit the data, for the purpose of the authors, is data dredging.

The VERTOS4 trial, initially reported on April analysis 8.1/8.2 as ≤6 weeks, is now reported on November analysis 8.1/8.2 as ≤9 weeks which is still incorrect. Nine weeks is the duration limit for radiography, not for vertebroplasty which was undertaken up to 12 weeks after fracture.

Misreporting of VAPOUR and VERTOS4

VAPOUR published its primary outcome online16 before the trial commenced and this has never been altered despite the assertions of the CVR. The advantage of a pre-specified clinical improvement over mean group scores was described by Farrar et al 17 who wrote ‘differences between groups, as summarised by a change in mean values over time, can be difficult to apply to clinical care. … group mean differences could reflect large changes in a few patients, small changes in many patients, or any combination of these outcomes. Determination of the proportion of patients who have a clinically important improvement in their pain would provide a more interpretable result with direct clinical implications’.

CVR rejects Farrar et al’s advice, dismissing the primary outcome to focus on mean NRS. Re-defining minimally important difference (MID) NRS as 1.5, (again breaching CVR protocol15 which defined it as 1), the authors describe NRS reduction as clinically insignificant when the between group difference is <1.5. Katz18 describes this methodology of applying MID to assess group differences as invalid.

MID is not measured from groups but from individual patients and should be applied as such. In order to use MID to compare group outcomes, it should be measured from individual patients. The proportion of patients in each group who achieve it can then be compared. Its manner of use to directly interpret group outcomes in CVR creates non-sensical conclusions. For example, 31% more patients in the vertebroplasty group than the placebo group achieved clinically important pain reduction at 1 month and mean NRS difference between groups of 1.4 favoured vertebroplasty (table 2). CVR authors describe4 this outcome as clinically irrelevant and consistent with the negative results of their trials. This assertion is false.

The treatment effect is directly measured by the primary outcome of the VAPOUR trial. It is likely to be larger in clinical practice where conservatively managed patients do not benefit from the placebo effect. Patients who were 80-year-olds, who have severe pain despite opiate analgesia are difficult to manage and have no other treatment option except increased opiate dose with all the attendant negative side-effects in this age group. More than half of the placebo group of the VAPOUR trial reported moderate or severe pain at 6 months and 76% were still using analgesic medication.

Mean VERTOS4 fracture duration at the time of vertebroplasty was 43 days, as reported in the conference proceedings used by CVR authors, not 29 as listed in the CVR. The fracture duration would extend to 12 weeks, not 9, as reported in CVR. Early intervention is a critical factor in the outcome, so these reporting errors are important. Despite advice in our letter of complaint to Cochrane,3 they remain uncorrected in the updated November version of the review.

Undisclosed conflict of interest

There is ongoing disagreement19–22 between the authors of the VAPOUR trial and the CVR. It is unlikely in this context for the CVR author group to provide an independent assessment of the VAPOUR trial.

The review coincided precisely with Medicare vertebroplasty funding application23 based on identical clinical inclusion criteria to the VAPOUR trial. CVR’s first author belongs to the Medicare Services Advisory Committee (MSAC) which determines application outcomes. This author was excluded from committee involvement due to conflict of interest, but not from the Cochrane review. CVR became a late-breaking centrepiece of evidence before the MSAC committee. Medicare funding was previously removed based on trials8 9 by authors of this CVR and provides part of the legacy of their trials.

Responding to our complaints, the Cochrane Chief Editor advised that ‘Cochrane does not have a non-financial conflict of interest policy’, an apparent weakness in the Cochrane model.

The author group of CVR includes first authors of two key trials, one of whom is co-ordinating editor of the Cochrane musculoskeletal section charged with editing the CVR. Cochrane author advice recommends that editors can write reviews but should clarify the editorial pathway to overcome this potential conflict of interest. This did not occur in the CVR. The safeguard against potential author bias is ensuring strict adherence to protocol, which has been breached in this review.

Faulty risk of bias assessment

Risk of bias is judged more harshly for VAPOUR than other blinded trials despite independent randomisation, data collection and analysis. Invalid blinding is inferred from more vertebroplasty patients correctly guessing their intervention than placebo patients, although they overwhelmingly nominated pain relief, not lack of blinding, as a reason for their guess.

Cochrane’s tool for assessing risk of bias24 stipulates: ‘Evidence of correct guesses exceeding 50% ….can simply reflect the patients’ experiences in the trial: a good outcome, or a marked side effect, will tend to be more often attributed to an active treatment, and a poor outcome to a placebo’, as in the VAPOUR trial. Judgements of reporting bias and placebo bias are other examples of disproportionate use of risk of bias assessment against VAPOUR compared with other blinded trials by CVR.

Conclusion

The Cochrane review ignores positive evidence supporting the use of early vertebroplasty to manage the most severely affected patients. It uses inappropriate meta-analysis technique and fails to address the differences between trials, the role of the different type of procedure used and the clear evidence of vertebroplasty superiority in patients with fracture <3 weeks including hospitalised patients with severe pain.

To deny this advantage of vertebroplasty sidelines a procedure that can provide better health outcomes to a specific group of patients and permits publication of guidelines from a biased review, by authors with undisclosed conflicts of interest.