Selective reporting

The NICE reviews on efficacy and adverse events are similar to regular systematic reviews and they include protocols to outline study objectives (Appendices A).2 3 These protocols were not published a priori or registered on Preferred Reporting Items for Systematic Reviews and Meta- Analyses (the international prospective register of systematic reviews).17 All systematic review authors are strongly advised against this because it limits consistency, accountability and transparency18 and inflates the risk of a range of biases, including selective reporting bias.19 20 The Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols list several critical items for protocol development, including a priori registration.20 Considering how rigorous and evidence-based guidelines minimise harms,15 we invite NICE to adhere to such standards.

NICE excluded unpublished data and open-label trials from the reviews.2 3 When systematic reviews exclude unpublished data, valuable clinical information is often lost and meta-analytic estimates become more prone to systematic errors and misleading effect estimates.21 NICE also included ’double-blind' clinical trials only, but this does not guarantee unbiased blinding procedures. Adverse events and observable behavioural effects in clinical trials on ADHD are likely more frequent in drug treatments than placebo comparators,8 10 11 22 which increases the risk of unblinding in participants, caregivers and outcome assessors.23 In turn, unblinding could well have impacted the subjective efficacy outcomes reported by NICE, namely quality of life and ADHD symptom severity.2

NICE’s selection of efficacy outcomes should itself be scrutinised. Outcomes assessing symptom severity and quality of life are frequently reported in clinical trials, and regulatory agencies require them in clinical investigations of drug treatments for ADHD.24 But mere reporting of the two lack comprehensiveness and objectivity, and outcomes related to substance-use, accidents, academic functioning and suicidal behaviour6 are often more meaningful to patients and clinicians. Also, under-reporting of such outcomes in randomised clinical trials does not justify their exclusion from the guideline development process. Observational evidence should instead be assessed in combination with randomised data to ensure accuracy and comprehensiveness.

Cochrane reviews have frequently been cited by NICE guidelines in the past,25 and evaluating them is a distinct step in the NICE review process (p. 8).4 However, NICE did not identify the two largest Cochrane reviews on the efficacy and adverse events of methylphenidate and amphetamines from 2015 and 2016.10 22 Both of these included markedly more studies than the NICE reviews for the respective interventions and populations. To illustrate this, our review on methylphenidate versus placebo for children and adolescents aggregated data from 175 randomised trials.10 NICE included 16 trials only on immediate and osmotic-release methylphenidate in children and adolescents.2 3 The Cochrane review on amphetamines versus placebo for children and adolescents included 23 trials.22 NICE included one trial for the same population.2 3 If the NICE committee was concerned about bias and inconsistencies in the excluded data, they should have favoured using appropriate subgroup and sensitivity analyses instead of dismissing potentially valuable information from their evaluations.

Multiple comparisons

The NICE review on efficacy reported on 17 primary and 6 secondary outcomes and conducted 309 head-to-head and placebo-controlled meta-analyses on randomised trials.2 The systematic review on adverse events reported on 16 primary outcomes (which were further stratified on trial duration) and conducted 174 meta-analyses.3 Apparently, no adjustment strategies for multiple comparisons were carried out, and with this many outcomes and comparisons, the likelihood of finding false positive results at p<0.05 (type I error) is dramatically increased.26 Also, a large number of the meta-analyses retrieved data from only one clinical trial with few participants (Appendices E).2 3 This increases the risk of imprecise estimates with wide CIs due to insufficient statistical power to detect true differences (type II errors), and the rates of random errors and false positives (type I errors) due to unequal distributions of prognostic factors despite randomisation procedures.26 27

NICE and the National Guideline Centre could have combined the 15 out of 17 outcomes on ADHD symptomology in their efficacy review into one meta-analytic estimate to prevent this. They could subsequently have conducted subgroup and sensitivity analyses on the variations in ADHD outcomes, such as type of outcome assessor or ADHD symptoms. This would have satisfied their review objectives while avoiding multiplicity and power issues in the main meta-analytic comparisons. NICE might also have adjusted the threshold for statistical significance in their reviews according to the number of primary outcomes, calculated required information sizes (ie, the meta-analytic sample size) for each outcome and applied sequential hypothesis testing techniques to their comparisons.26 27 We hope that NICE will consider such solutions in the future.

Generalisability

As a neurodevelopmental condition, ADHD frequently affects people through childhood and into adulthood.7 However, the clinical trials in the NICE reviews lasted typically for under 12 weeks,2 3 which undermines the generalisability of the data for any long-term treatment. The short duration was a major reason for not recommending pharmacological treatments as first-line treatments to children under five (p. 42).1 However, the data from all other age groups in the NICE evidence base also arose from short-term trials.2 3 It appears inconsistent that data from short-term trials for children over five, adolescents and adults are considered acceptable in terms of clinical relevance, while the data from those under five are not. Moreover, the choice to stratify patients into age groups of those above and below five seems arbitrary and not evidence based. So does the recommendation for clinicians to review ADHD medication at least once a year and typically every 6 months or more (recommendations under 1.8),1 given the short trial duration.

Risk of bias and quality

For drug efficacy,2 NICE assessed the GRADE evidence quality at moderate to low for all age groups (section 1.3.1.2), and for the clinical trials on harms, the quality was low to very low (section 1.9.1.2).3 In comparison, the previously published Cochrane reviews on methylphenidate and amphetamines assessed the evidence quality at low to very low.8–10 22 NICE consistently graded studies with low sample sizes as having no serious imprecision (imprecision meaning studies with sufficiently wide CIs or few participants).10 22 28 This led many studies to be judged at moderate instead of low evidence quality. According to GRADE, downgrading the quality of critically imprecise estimates is crucial if study participation is sufficiently low,28 and we invite NICE to adopt such standards in the future, or even to reinforce imprecision assessments by using sequential hypothesis testing techniques.29 30

Regardless of these differences, both NICE and the Cochrane reviews found widespread low-quality evidence, which systematically reduced the interpretability of the study findings. The majority of the clinical trials in the NICE reviews were subject to high risk of bias, with frequent methodological flaws such as incomplete outcome data, issues with blinding and outcome reporting bias,18 and the low quality affected both the clinical and economic evidence.2 3 Most of the included studies were also industry funded, which has been shown to lead to more favourable results and conclusions in systematic reviews,31 and despite stating so in consultation with stakeholders (p. 263),32 we see little evidence of it being weighted when evaluating quality or generating recommendations. In the efficacy review, the NICE committee acknowledges the low quality of evidence (p. 157),2 but strong recommendations for methylphenidate and lisdexamfetamine are still offered in the guideline.1 This is puzzling and contradictory.

Adverse events

When assessing adverse events, non-randomised studies are often better than randomised trials because the former usually involve more participants, reflect clinical practice more accurately, include longer follow-up periods, and cost less.9 10 33 Under-reporting and neglect of adverse events in clinical trials on ADHD drugs is also commonplace,9 and issues with statistical power can lead to inflated type II errors and failure to detect harms.6 9 In the review on harms,3 NICE largely evaluated randomised studies, and the search for non-randomised studies lacked comprehensiveness, with 14 studies on all drug treatments and patient populations. In comparison, our Cochrane review on the adverse events of methylphenidate for children and adolescents with ADHD alone included 260 non-randomised studies.9 This review also documented significantly higher rates of serious adverse events, psychotic disorder and arrhythmia in patients relative to non-responders, not otherwise found in the NICE review. Overall, NICE judged the reporting of harms in the literature as inconsistent and emphasised how studies used conflicting methods to assess adverse events (section 1.9.1.2).3 From a methodological and quantitative perspective, the NICE assessment of adverse events seems less than ideal.

Both NICE and the Cochrane reviews did nevertheless find prevalent adverse events for methylphenidate and amphetamines, with sleep disturbance, decreased appetite and weight changes being more commonly reported on.2 3 9–11 22 NICE did not measure all-cause treatment discontinuation in ADHD participants (a measure of acceptability that weighs symptom improvement against safety9 10), but Cochrane evidence and regular systematic reviews have found decreased acceptability rates for methylphenidate9 10 34 and lisdexamfetamine8 11 relative to placebo, with potential variations between age groups.35 Considering how the NICE committee reported having problems interpreting the numerous pairwise comparisons it was presented with (p.157),2 along with the risks of frequent non-serious and potential serious adverse events, accurate evaluations of the ratio between the benefits and harms of ADHD drug treatments seem troublesome.

Bridging the evidence–recommendation gap

The responsibility of NICE is both simple and noble: to provide evidence-based guidance on optimal health and social care.13 Much of the work is commendable, and several of the individual NICE recommendations on ADHD management are useful, including the emphasis on thorough baseline assessments, qualified practitioners for prescribing medications and social support for patients.1 The heterogeneous nature and unclear aetiological pathways of ADHD are also mentioned in the guideline, and recommendations for research are given, which is constructive. Moreover, the American Academy of Pediatrics has not updated its guideline on ADHD management since 201136—a serious problem with guideline development on the whole37—and we applaud NICE for leading the way here. Compared with more recent practice guidelines on ADHD pharmacological management from The Canadian ADHD Practice Guidelines (2018)38 and The British Association for Psychopharmacology (2014),39 the 2018 NICE update also employs a more comprehensive appraisal of the clinical evidence, which should be commended.

The 2018 NICE guidance on pharmacological treatments for ADHD1 is nevertheless informed by systematic reviews with serious methodological limitations and low-quality studies. Although the guideline document elaborates on the rationale behind medication decisions, the low quality of evidence is never addressed to readers. It seems unlikely that patients, clinicians and families will have the time or energy to properly evaluate the lengthy NICE reviews, deeming efforts to determine the qualitative rationale behind the strong practice recommendations favouring methylphenidate and lisdexamfetamine, more difficult than necessary. Importantly, NICE reserves the right to recommend treatments based on scarce evidence and the experience of committee members alone.14 But should not readers at least be made explicitly aware of such justifications?

The GRADE Working Group offers symbolic representations for rating evidence quality to organisations (see figure 1).16 Such visual cues, or other easily interpretable symbols,40 could serve as accessible and intuitive indications of evidence strength and certainty next to individual NICE recommendations (ie, recommendations 1.7 for ADHD drug management1). This would make care providers and receivers more cognisant of the evidence strength behind each piece of guidance they read or employ in practice, while preserving NICE’s right to integrate different types of data in its decision-making process.13 It would also likely decrease the rates of strong practice recommendations based on low-quality evidence, which seem to affect other prominent guideline institutions as well, like WHO.41 Conclusively, we hope that NICE and The National Guideline Centre will recognise the suggested limitations with their systematic reviews on pharmacological treatments for ADHD. Close cooperation between NICE and The Cochrane Collaboration would benefit both institutions reciprocally, and ultimately facilitate improved clinical decision-making.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Representation of Grading of Recommendations, Assessment, Development and Evaluations-inspired symbols for evaluating evidence quality. Inspired by Guyatt and colleagues.16