Inappropriate selection of studies for inclusion

Banaschewski et al suggest that in our analyses we included trials that should have been excluded. This is not correct. They highlight three trials, which they cite as having used ‘active controls’ whereas these are actually cointerventions used in the methylphenidate group and in the control group. Such trials are eligible for inclusion in accordance with our protocol because they still compare ‘methylphenidate’ with ‘no methylphenidate’.2 The fact that the ‘methylphenidate’ and ‘no methylphenidate’ arms receive an identical cointervention does not invalidate the direct comparison of ‘methylphenidate’ with ‘no methylphenidate’. Anyway, excluding these trials from our review would have produced only a negligible change in our results (a difference of 0.06 points in the standardised mean difference).

Banaschewski et al also claim that we included a trial that was not randomised.5 This is not correct, as we received information by email from the study author on 17 September 2013, clarifying that this trial was randomised (email correspondence available on request).

The Multimodal Treatment of Attention Deficit Hyperactivity Disorder (MTA) trial is correctly included in accordance with our protocol.2 We raise doubts about the long-term benefit of methylphenidate. The MTA trial, which reported that beneficial effects may diminish over time,6 ,7 was the only trial that met our ‘long duration’ inclusion criteria and was therefore included. Our subgroup analysis of teacher-rated ADHD symptoms in ‘short duration’ trials (up to 6 months) compared to ‘long duration’ trials (over 6 months) showed a significant difference. The fact that there was only one trial included simply reflects the dearth of long-term trials of methylphenidate. In real-world setting, however, most patients receive methylphenidate for substantially longer periods than the length of existing trials.

Randomised discontinuation trials are useful where the drug effect is known to be beneficial. However, we cannot yet be sure that this is the case for methylphenidate as the true estimate of its efficacy is still uncertain. The correct design for further trials would be to undertake randomised controlled trials using active placebo (ie, nocebo).8

Assessment of study quality

Our assessment of the evidence as being ‘very low quality’ is not only based on the assessment of risk of bias in the included trials, but on additional factors encompassing heterogeneity, imprecision, indirectness of the evidence and publication bias.2 ,9 We comment on this in detail in our review.2 ,3

We downgraded the quality of the included trials in the teacher-reported ADHD symptom meta-analysis to ‘very low quality’ because of high risk of bias and moderate heterogeneity (I²). We considered I² values between 30% to 60% as a moderate level of heterogeneity.2 The downgrading for heterogeneity might be considered debatable, but we chose to downgrade because we think that an I² of 37% may affect the findings. Had we not done so, the downgrading would have become ‘low quality’, still signalling that one ought not to have too much confidence in the findings. The short trial duration could be the basis for further downgrading for indirectness according to GRADE.9 We did not downgrade for this, but arguably could have done so.

For many years, there has been intense discussion within The Cochrane Collaboration, about the risk of bias due to vested interests.10–12 Lundh et al13 have shown that sponsorship and conflict of interest affect outcomes through many subtle mechanisms. They also demonstrated that vested interests per se were enough to lead to overestimation of benefits and underestimation of harms, even when all other bias domains were assessed as being at low risk of bias.13

Banaschewski et al state that our vested interest domain was inconsistently rated across the included trials and show some examples of this in their webappendix.1 Had there been inconsistencies regarding one domain of bias in a few trials, this would not change the fact that these trials, overall, should still be considered as being at high risk of bias. Regarding the three trials by Barkley (1991), Brown (1985) and Jensen (1999),1 we erroneously rated them as being of low risk of bias. They should have been rated as ‘unclear’ because of the vested interest bias. We will amend this in the next update of our review (again, it will not materially change our results or conclusions).

It is not incorrect for us to state that none of the trials funded by the pharmaceutical industry showed a low risk of bias in all other areas, as we considered all the trials at high risk of bias on the domain of blinding because the common adverse effects of sleep difficulties and appetite suppression are easily recognisable by outcome assessors.

We contacted the authors of 161 trials twice for supplemental information, including information about vested interests. Only 92 responded. On the basis of the available evidence, we assessed 71 trials as having a high risk of bias in the vested interest domain as they were funded by the industry and/or the authors were affiliated with the industry. We contacted Ashare et al by email in 2013, but did not receive a response. However, considering that this trial was funded by National Institute of Mental Health and from the National Institute on Drug Abuse, we rated it as low risk of bias on the vested interest domain.

In our BMJ article,3 we reported that we had undertaken the risk of bias subgroup analysis, but did not report the results there as we considered all trials to be high risk of bias trials. As our BMJ article cross-references our full Cochrane review,2 the readership will locate this information in this publication, should they wish to.

In placebo-controlled trials, it is possible to discriminate between drugs and placebo on the basis of the reported adverse events alone. This fact raises questions about the true blindness of such trials.14 Adverse events such as decreased appetite and sleep difficulties (and subsequent tiredness) are unsubtle and obvious, so easy detection by teachers is likely.

On the issue of nocebo, we acknowledge that there are substantial ethical dilemmas around their use. Several authors have underlined the importance of the use of ‘active placebo’ (nocebo) in clinical trials.14 This is a methodological issue and we would like to stress that nocebos would need to first be shown to be safe in adults, and methylphenidate versus nocebo trials also shown to favour methylphenidate in adults.2 ,3 Only then would nocebo controlled trials be ethically defensible in children.

Serious adverse effects of methylphenidate

We agree with Banaschewski et al that there is certainly a weakness of the available evidence of serious adverse events from randomised clinical trials. On the basis of our other protocol on methylphenidate for ADHD, we are presently examining the reporting of adverse events in observational studies.15 This work is not yet complete.

Effect sizes and clinical effectiveness

The problem is that no one knows the true magnitude of the effect size for methylphenidate, due to the very low quality of supporting evidence. Therefore, it is difficult to state whether the effect size of this drug should be favourably viewed.

Errors in our primary analyses of the teacher-rated ADHD symptoms outcome

As reported, all 19 trials were includable according to our protocol.2 Banaschewski et al state that they found errors in the imputation of data and/or sample size in seven trials. We have now checked these trials for errors and found that we included incorrect values in the placebo group of the Butter 1993 trial. We will correct this in our next review update (even if this will not materially change our results or conclusions).

When reporting end-of-period data in crossover trials in meta-analyses, it is correct to count the data from the pre-crossover period as well as the post-crossover if such data are available. This is not ‘double counting’ as separate data exist for exposure of the participants to placebo and active drug.16 However, we did erroneously count participants twice in two crossover trials (Moshe (2012) and Taylor (1987)) in which we had data only from the first period. We re-analysed the data accordingly and the standardised mean difference is now −0.78, 95% CI −0.92 to −0.64 (rather than −0.77, −0.90 to −0.64, as originally reported); test for subgroup differences: χ²=0.01, df=1 (p=0.91), I²=0%. None of the corrections leads to any noticeable changes in our results or conclusions. Anyway, we will correct these in an update of our review.

Regarding the Findling (2006) trial, the authors state that: “The primary efficacy population was the per-protocol (PP) population defined as those subjects who received study treatment and had at least one efficacy measurement after the first dose, with no major efficacy protocol deviations.”17 Yes, we used the per-protocol population, because—as stated in the review—we conducted the analyses using available data.2

Many of the trials had several publications. In some of these, there were participant numbers that differed, depending on which outcome was being reported. Moreover, in some trials we had to compute SDs from SE values and in one trial we had to calculate the mean difference and SDs from the total mean difference and p value. This was due to data not being presented in the original publications and authors not providing these data. We received data for the first period of the crossover trial by Moshe 2012. These data are not reported in the published articles, so it is impossible to find them in any published report. For transparency and replicability, we reported in the full review all the data we used for analyses.

We appreciate these small errors being highlighted as it has given us the opportunity to reflect on their meaning and evaluate their impact; however, they do not lead to noticeable changes in our results or conclusions. Please do let us know if more errors are identified and we shall correct any mistakes in future updates.