Background

Attention deficit hyperactivity disorder (ADHD) in adults is one of the most controversial diagnoses in medicine,1 2 and the use of methylphenidate is sharply rising.3 The Cochrane review by Epstein and colleagues about immediate-release methylphenidate for adults with ADHD from September 20144 was therefore long awaited, also because it took 9 years to appear after the protocol was published. However, four research groups independently submitted extensive criticisms of the review during the period October 2014 to November 2015. Three of the four groups coauthored the present paper and were in frequent correspondence with the editors and eventually Cochrane's editor-in-chief until the Cochrane editorial group decided to withdraw the review in May 2016.5 They issued a note outlining the reasons for their retraction.

"The Editorial Base of the DPLPG (Developmental, Psychosocial, and Learning Problems Group) appreciate the feedback received from the contributors named below, and their patience with our endeavours to secure a response from the author team. These endeavours have been persistent, but we were, until recently, unable to secure a response from the author team, by which time we had received guidance from Cochrane's Funding Arbiter as to how to manage their conflict of interest. Subsequently, one of the authors has expressed a wish to have his name removed from the review, and we have still to hear from a second author. The third author wishes to remain an author but given these, and subsequent criticisms, together with an unsatisfactory response to those criticisms from the author team, we have decided to withdraw this review. To make public comments for the benefits of readers, we will delay withdrawing the review until 26 May 2016". 4

The criticisms focused mainly on the review's flaws and misleading conclusions that gave a false sense of certainty of the benefits and the absence of harms, when in fact this could not be concluded. Fundamental methodological problems in the included trials that were not addressed in the review were also highlighted in the criticisms.4 The withdrawn Cochrane review reflects in many ways a general inadequacy of clinical ADHD research,6–8 but the review authors failed to point out important flaws in the included trials. We provide here a summary of the criticism and discuss the consequences of the withdrawal.

The confidence in the evidence

Using the Grades of Recommendation Assessment Development and Evaluation (GRADE) approach,9 the review authors expressed ‘high’ confidence in the evidence for most outcomes, despite the fact that most domains in Cochrane's Risk of Bias assessment for the included trials were labelled as ‘unclear’. Furthermore, there was very high heterogeneity between results from the individual trials for ADHD symptoms (between 76% and 88%; the maximum is 100%), and the estimates were very imprecise. The confidence in the evidence should therefore have been ‘very low’. The authors did not explore the reasons for the heterogeneity, except for a subgroup analysis that compared low versus high dosage methylphenidate, arguing that they had too few studies for meta-regression. However, reasons for substantial heterogeneity should always be explored.

Cross-over studies

Nine of the included 11 trials had a cross-over design. The cross-over design is questionable to use in trials of central stimulants (such as methylphenidate or amphetamine) due to the risk of carry-over effects.10 It also increases the risk of unblinding and is therefore more susceptible to expectancy effects (see also Lack of blinding). According to the Cochrane Handbook, 11 cross-over studies may be used for ‘stable’ conditions where ‘long-term follow-up is not required’, neither of which is the case for ADHD treatment. A previous meta-analysis that included trials of stimulants for adults with ADHD showed larger effects in cross-over studies than in parallel-group trials.12 The failures to consider such fundamental elements of the trial design and explore its importance in sensitivity analyses raise major concern.

Psychiatric comorbidity

Most adults with ADHD suffer from psychiatric comorbidity, primarily anxiety and depression,13 and the exclusion of such participants from the trials greatly impairs their external validity.14 Only 2 of the 11 trials explicitly did not exclude participants with psychiatric comorbidity; 5 trials excluded them, and it was unclear in 4 trials. The review paid no attention to this, which should have led to a downgrading for ‘indirectness’ according to GRADE.9 Downgrading for indirectness should be applied when the trial conditions (population, intervention, outcome measures) differ considerably from clinical practice.

‘Functional’ outcomes

The authors acknowledged the negative impact of ADHD on ‘functional outcomes’, such as frequency of job change, work performance, traffic accidents, marital status and academic achievements. Yet, the treatment effect on functional outcomes was not quantified in the review, although these outcomes are arguably the most important ones. None of the included trials reported data on these outcomes specifically, but this should have been highlighted in the review as an important limitation.

Trial duration

The trials lasted only 1–7 weeks, and the long-term beneficial and harmful effects of methylphenidate are therefore unknown. Most adults with ADHD receive methylphenidate for substantially longer periods,15 and the beneficial effects may furthermore diminish over time. This was observed in the large Multimodal Treatment of Attention Deficit Hyperactivity Disorder Study in children.7 16–18 The short trial duration should therefore have led to additional downgrading for indirectness according to GRADE.9

Small trials

The total sample size in the review was only 474 participants, and the trial median was only 30. Small trials are underpowered to detect clinically relevant effects, and their results are generally misleading,19 particularly when they are biased, as is the case here. Furthermore, small trials cannot detect rare harms of drugs, a very important issue that was not mentioned in the review.4

Lack of blinding

Maintaining blinding in trials of central stimulant drugs with characteristic detectable (and for some people) pleasurable effects and also characteristic adverse effects is very unlikely without the use of an active placebo. This important issue was not considered in the review. Furthermore, 9 of the 11 studies used a titration scheme to achieve the highest tolerated dose, which virtually excludes the possibility that blinding was effective. Still, the trials were described as ‘double-blind’, and this was accepted by the review authors, the peer reviewers and the editors. The included trials did not formally assess whether the blinding was effective, but one trial20 reported that all eight participants were able to guess their allocation. The inherent problem of maintaining blinded conditions in trials of central stimulants for ADHD was highlighted in the Cochrane reviews of methylphenidate for children with ADHD7 and of amphetamines for adults with ADHD.6

Pooling investigator-rated and self-rated ADHD symptom scores

The review reported the combined effect size for self-rated and investigator-rated ADHD symptom scores, where the latter was preferred when both were reported in the same study. However, previous meta-analyses have demonstrated substantial differences, with investigators estimating higher effects.12 21 Obviously, the participants’ perception of the effect is more important than that of the investigator. It has been argued that the investigator is more ‘useful’ to detect changes in ADHD behaviour,12 but investigators may simply be more biased than participants in trials that have not been adequately blinded even though the detectable effects of central stimulants may bias the participants’ assessments. It is essential that both participant and investigator ratings are analysed and reported separately.

Outcome reporting bias

The authors of the retracted review wrote that prestated outcomes were reported in most studies and therefore judged the risk of outcome reporting bias as ‘low’.4 However, in direct contradiction to this, they wrote in the abstract that “We were unable to determine whether adverse effects were not discussed by study authors because none occurred or because no data on adverse effects were collected”.4 They also noted that this prevented them from conducting meta-analyses of adverse events and concluded that “Trial data suggest that adverse effects from immediate-release methylphenidate for adults with ADHD are not of serious clinical significance, although this conclusion may be limited, certainly in the case of weight loss, by the short duration of published studies”.4 It is misleading to conclude that trial data suggest that adverse effects are not of ‘serious clinical significance’ when important data are not available and follow-up is very short. The risk of outcome reporting bias in the included trials, most evident for the adverse events data, should have been stated clearly in the review.

Outcome reporting bias is not limited to old trials of immediate-release methylphenidate. We have pointed out22 that the most recent trial of extended-release methylphenidate from 201523 failed to report and specify crucial outcomes such as adverse events and quality of life.22 In addition, ‘absence from work’ and long-term outcomes after 2.5 years were omitted from the publication.

Cardiovascular and psychiatric harms

The cardiovascular harms of stimulants have long been in focus.24 The review authors narratively described the cardiovascular effects in five studies, but they did not conduct meta-analyses for pulse or blood pressure changes, which are the only meaningful cardiovascular surrogate outcomes in small, short-term trials. Three of those five trials reported data for such an analysis. Stimulants increase blood pressure and pulse in adults25 and may increase the risk of ischaemic heart disease beyond the short follow-up period of the trials, particularly in adults, where many have additional risk factors for cardiovascular disease. In children and adolescents with ADHD, there is an increased risk of arrhythmias and myocardial infarction during the initial phase of methylphenidate treatment,26 and the risk of cardiovascular events with different levels of severity is doubled in stimulant users compared with non-users.27 The risk of developing psychiatric adverse events such as psychosis, mania and aggression was not mentioned in the review,4 although a large Canadian population-based study28 and an FDA review29 30 found that ADHD drug use, including methylphenidate, was associated with such events in children and adolescents.

Comparison with other research

The review authors compared their results with those of three meta-analyses but not with three newer systematic reviews,12 31 32 nor with a comprehensive drug class review.33 These omissions are problematic as these reviews reported different effect sizes, considered several of the methodological limitations that we have outlined above and conducted subgroup and meta-regression analyses that Epstein and colleagues did not.4 The drug class review33 even abstained from doing meta-analyses as the authors considered the trials to be too unreliable.

Conflicts of interests

We believe that the trialists’ conflicts of interests should have been discussed in the review. Only 2 of the 11 trials were explicitly funded by the drug industry, whereas 2 were unclear about this. Yet, two trials34 35 (one publicly funded35 and one with no declaration of funding34) were authored by three psychiatrists, which were investigated by US Senator Grassley in 2008 for financial ties to the pharmaceutical industry.36 The investigation revealed that each researcher had received millions of dollars from the drug industry without disclosing it.36 Two of the three authors of the Cochrane review declared financial conflicts of interests.4 It is very likely that authors of systematic reviews with conflicts of interests are influenced in the same way as trialists are.11 We believe that the Cochrane Collaboration should introduce a policy that does not allow any financial conflicts of interest for review authors.