Statistics from Altmetric.com
In March 2018, the National Institute for Health and Care Excellence (NICE) updated its guideline (NG87) on diagnosing and managing attention-deficit hyperactivity disorder (ADHD),1 and its in-house systematic reviews on the efficacy and adverse events of pharmacological treatments for ADHD.2 3 The guideline recommends methylphenidate as the first-line pharmacological treatment for children over five, adolescents and adults with ADHD and lisdexamfetamine for adults only (recommendations under section 1.7).1 An appointed NICE committee formulated the recommendations based on clinical experience, drug licensing regulations and the systematic review evidence (developed in concert with the National Guideline Centre).4 The NICE committee conclude that methylphenidate and lisdexamfetamine provide clinically important benefits to patients with ADHD compared with placebo and other drugs (p. 47).1
ADHD is a neurodevelopmental disorder characterised by excessive hyperactivity, impulsivity and inattention.5 The condition is associated with increased risks of substance-use disorders, accidents, academic failure, diabetes, suicidal behaviour and other adverse health outcomes.6 There is some, but arguably not much, high-quality evidence that current behavioural and pharmacological interventions are effective, and the genetic and environmental causes of ADHD remain largely unknown.7 We previously authored four Cochrane systematic reviews on the clinical efficacy and adverse events of methylphenidate and amphetamines for children, adolescents and adults with ADHD.8–11 All published reviews concluded that the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) evidence quality (ie, the certainty in the treatment estimates)12 was low to very low, urging readers to cautiously interpret the results. The updated NICE systematic reviews and recommendations contrast markedly from the previous Cochrane findings.
As stated in the NICE manual,13 recommendations should be informed by the best available evidence. However, if confronted with unsubstantial or biased evidence, committee members often depend on subjective value judgements when developing recommendations.14 This illustrates why clinical practice guidelines can be controversial,15 and arguably why patients, clinicians and decision-makers should be explicitly informed on the certainty behind each recommendation they read.16 Considering the influence and credibility of NICE both in the UK and globally, critical evaluations of its reporting standard and methods for evidence synthesis should be encouraged. Here we discuss the updated 2018 NICE guidance and evidence base on ADHD, with a particular focus on pharmacological management, and propose steps to advance the way NICE synthesises and communicates evidence to its readership.
The NICE reviews on efficacy and adverse events are similar to regular systematic reviews and they include protocols to outline study objectives (Appendices A).2 3 These protocols were not published a priori or registered on Preferred Reporting Items for Systematic Reviews and Meta- Analyses (the international prospective register of systematic reviews).17 All systematic review authors are strongly advised against this because it limits consistency, accountability and transparency18 and inflates the risk of a range of biases, including selective reporting bias.19 20 The Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols list several critical items for protocol development, including a priori registration.20 Considering how rigorous and evidence-based guidelines minimise harms,15 we invite NICE to adhere to such standards.
NICE excluded unpublished data and open-label trials from the reviews.2 3 When systematic reviews exclude unpublished data, valuable clinical information is often lost and meta-analytic estimates become more prone to systematic errors and misleading effect estimates.21 NICE also included ’double-blind' clinical trials only, but this does not guarantee unbiased blinding procedures. Adverse events and observable behavioural effects in clinical trials on ADHD are likely more frequent in drug treatments than placebo comparators,8 10 11 22 which increases the risk of unblinding in participants, caregivers and outcome assessors.23 In turn, unblinding could well have impacted the subjective efficacy outcomes reported by NICE, namely quality of life and ADHD symptom severity.2
NICE’s selection of efficacy outcomes should itself be scrutinised. Outcomes assessing symptom severity and quality of life are frequently reported in clinical trials, and regulatory agencies require them in clinical investigations of drug treatments for ADHD.24 But mere reporting of the two lack comprehensiveness and objectivity, and outcomes related to substance-use, accidents, academic functioning and suicidal behaviour6 are often more meaningful to patients and clinicians. Also, under-reporting of such outcomes in randomised clinical trials does not justify their exclusion from the guideline development process. Observational evidence should instead be assessed in combination with randomised data to ensure accuracy and comprehensiveness.
Cochrane reviews have frequently been cited by NICE guidelines in the past,25 and evaluating them is a distinct step in the NICE review process (p. 8).4 However, NICE did not identify the two largest Cochrane reviews on the efficacy and adverse events of methylphenidate and amphetamines from 2015 and 2016.10 22 Both of these included markedly more studies than the NICE reviews for the respective interventions and populations. To illustrate this, our review on methylphenidate versus placebo for children and adolescents aggregated data from 175 randomised trials.10 NICE included 16 trials only on immediate and osmotic-release methylphenidate in children and adolescents.2 3 The Cochrane review on amphetamines versus placebo for children and adolescents included 23 trials.22 NICE included one trial for the same population.2 3 If the NICE committee was concerned about bias and inconsistencies in the excluded data, they should have favoured using appropriate subgroup and sensitivity analyses instead of dismissing potentially valuable information from their evaluations.
The NICE review on efficacy reported on 17 primary and 6 secondary outcomes and conducted 309 head-to-head and placebo-controlled meta-analyses on randomised trials.2 The systematic review on adverse events reported on 16 primary outcomes (which were further stratified on trial duration) and conducted 174 meta-analyses.3 Apparently, no adjustment strategies for multiple comparisons were carried out, and with this many outcomes and comparisons, the likelihood of finding false positive results at p<0.05 (type I error) is dramatically increased.26 Also, a large number of the meta-analyses retrieved data from only one clinical trial with few participants (Appendices E).2 3 This increases the risk of imprecise estimates with wide CIs due to insufficient statistical power to detect true differences (type II errors), and the rates of random errors and false positives (type I errors) due to unequal distributions of prognostic factors despite randomisation procedures.26 27
NICE and the National Guideline Centre could have combined the 15 out of 17 outcomes on ADHD symptomology in their efficacy review into one meta-analytic estimate to prevent this. They could subsequently have conducted subgroup and sensitivity analyses on the variations in ADHD outcomes, such as type of outcome assessor or ADHD symptoms. This would have satisfied their review objectives while avoiding multiplicity and power issues in the main meta-analytic comparisons. NICE might also have adjusted the threshold for statistical significance in their reviews according to the number of primary outcomes, calculated required information sizes (ie, the meta-analytic sample size) for each outcome and applied sequential hypothesis testing techniques to their comparisons.26 27 We hope that NICE will consider such solutions in the future.
As a neurodevelopmental condition, ADHD frequently affects people through childhood and into adulthood.7 However, the clinical trials in the NICE reviews lasted typically for under 12 weeks,2 3 which undermines the generalisability of the data for any long-term treatment. The short duration was a major reason for not recommending pharmacological treatments as first-line treatments to children under five (p. 42).1 However, the data from all other age groups in the NICE evidence base also arose from short-term trials.2 3 It appears inconsistent that data from short-term trials for children over five, adolescents and adults are considered acceptable in terms of clinical relevance, while the data from those under five are not. Moreover, the choice to stratify patients into age groups of those above and below five seems arbitrary and not evidence based. So does the recommendation for clinicians to review ADHD medication at least once a year and typically every 6 months or more (recommendations under 1.8),1 given the short trial duration.
Risk of bias and quality
For drug efficacy,2 NICE assessed the GRADE evidence quality at moderate to low for all age groups (section 220.127.116.11), and for the clinical trials on harms, the quality was low to very low (section 18.104.22.168).3 In comparison, the previously published Cochrane reviews on methylphenidate and amphetamines assessed the evidence quality at low to very low.8–10 22 NICE consistently graded studies with low sample sizes as having no serious imprecision (imprecision meaning studies with sufficiently wide CIs or few participants).10 22 28 This led many studies to be judged at moderate instead of low evidence quality. According to GRADE, downgrading the quality of critically imprecise estimates is crucial if study participation is sufficiently low,28 and we invite NICE to adopt such standards in the future, or even to reinforce imprecision assessments by using sequential hypothesis testing techniques.29 30
Regardless of these differences, both NICE and the Cochrane reviews found widespread low-quality evidence, which systematically reduced the interpretability of the study findings. The majority of the clinical trials in the NICE reviews were subject to high risk of bias, with frequent methodological flaws such as incomplete outcome data, issues with blinding and outcome reporting bias,18 and the low quality affected both the clinical and economic evidence.2 3 Most of the included studies were also industry funded, which has been shown to lead to more favourable results and conclusions in systematic reviews,31 and despite stating so in consultation with stakeholders (p. 263),32 we see little evidence of it being weighted when evaluating quality or generating recommendations. In the efficacy review, the NICE committee acknowledges the low quality of evidence (p. 157),2 but strong recommendations for methylphenidate and lisdexamfetamine are still offered in the guideline.1 This is puzzling and contradictory.
When assessing adverse events, non-randomised studies are often better than randomised trials because the former usually involve more participants, reflect clinical practice more accurately, include longer follow-up periods, and cost less.9 10 33 Under-reporting and neglect of adverse events in clinical trials on ADHD drugs is also commonplace,9 and issues with statistical power can lead to inflated type II errors and failure to detect harms.6 9 In the review on harms,3 NICE largely evaluated randomised studies, and the search for non-randomised studies lacked comprehensiveness, with 14 studies on all drug treatments and patient populations. In comparison, our Cochrane review on the adverse events of methylphenidate for children and adolescents with ADHD alone included 260 non-randomised studies.9 This review also documented significantly higher rates of serious adverse events, psychotic disorder and arrhythmia in patients relative to non-responders, not otherwise found in the NICE review. Overall, NICE judged the reporting of harms in the literature as inconsistent and emphasised how studies used conflicting methods to assess adverse events (section 22.214.171.124).3 From a methodological and quantitative perspective, the NICE assessment of adverse events seems less than ideal.
Both NICE and the Cochrane reviews did nevertheless find prevalent adverse events for methylphenidate and amphetamines, with sleep disturbance, decreased appetite and weight changes being more commonly reported on.2 3 9–11 22 NICE did not measure all-cause treatment discontinuation in ADHD participants (a measure of acceptability that weighs symptom improvement against safety9 10), but Cochrane evidence and regular systematic reviews have found decreased acceptability rates for methylphenidate9 10 34 and lisdexamfetamine8 11 relative to placebo, with potential variations between age groups.35 Considering how the NICE committee reported having problems interpreting the numerous pairwise comparisons it was presented with (p.157),2 along with the risks of frequent non-serious and potential serious adverse events, accurate evaluations of the ratio between the benefits and harms of ADHD drug treatments seem troublesome.
Bridging the evidence–recommendation gap
The responsibility of NICE is both simple and noble: to provide evidence-based guidance on optimal health and social care.13 Much of the work is commendable, and several of the individual NICE recommendations on ADHD management are useful, including the emphasis on thorough baseline assessments, qualified practitioners for prescribing medications and social support for patients.1 The heterogeneous nature and unclear aetiological pathways of ADHD are also mentioned in the guideline, and recommendations for research are given, which is constructive. Moreover, the American Academy of Pediatrics has not updated its guideline on ADHD management since 201136—a serious problem with guideline development on the whole37—and we applaud NICE for leading the way here. Compared with more recent practice guidelines on ADHD pharmacological management from The Canadian ADHD Practice Guidelines (2018)38 and The British Association for Psychopharmacology (2014),39 the 2018 NICE update also employs a more comprehensive appraisal of the clinical evidence, which should be commended.
The 2018 NICE guidance on pharmacological treatments for ADHD1 is nevertheless informed by systematic reviews with serious methodological limitations and low-quality studies. Although the guideline document elaborates on the rationale behind medication decisions, the low quality of evidence is never addressed to readers. It seems unlikely that patients, clinicians and families will have the time or energy to properly evaluate the lengthy NICE reviews, deeming efforts to determine the qualitative rationale behind the strong practice recommendations favouring methylphenidate and lisdexamfetamine, more difficult than necessary. Importantly, NICE reserves the right to recommend treatments based on scarce evidence and the experience of committee members alone.14 But should not readers at least be made explicitly aware of such justifications?
The GRADE Working Group offers symbolic representations for rating evidence quality to organisations (see figure 1).16 Such visual cues, or other easily interpretable symbols,40 could serve as accessible and intuitive indications of evidence strength and certainty next to individual NICE recommendations (ie, recommendations 1.7 for ADHD drug management1). This would make care providers and receivers more cognisant of the evidence strength behind each piece of guidance they read or employ in practice, while preserving NICE’s right to integrate different types of data in its decision-making process.13 It would also likely decrease the rates of strong practice recommendations based on low-quality evidence, which seem to affect other prominent guideline institutions as well, like WHO.41 Conclusively, we hope that NICE and The National Guideline Centre will recognise the suggested limitations with their systematic reviews on pharmacological treatments for ADHD. Close cooperation between NICE and The Cochrane Collaboration would benefit both institutions reciprocally, and ultimately facilitate improved clinical decision-making.
Contributors CG and OJS had the initial idea for the article. EF retrieved relevant literature and wrote multiple drafts including the final version. All authors commented, edited, made suggestions and approved of the final version.
Competing interests CG, OJS, MZ and ES have authored two of the Cochrane reviews on methylphenidate for ADHD, and XC has authored two of the Cochrane reviews on amphetamines for ADHD. MZ is a clinician working with patients with ADHD and prescribes stimulant medications regularly.
Patient consent Not required.
Provenance and peer review Not commissioned; internally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.