Background Evidence-based medicine (EBM) has no doubt resulted in great improvements in the practice of medicine. However, there are problems with overly zealous application of EBM, that for some amounts to religious practice. When good evidence exists, it should guide therapeutic and diagnostic choices. However, when evidence is lacking for a given patient, medicine is best practised by extrapolation from available evidence, interpreted in the light of the pathophysiology of the condition under consideration, and effects of various therapies in relation to that pathophysiology.
Objective To assess ways in which the unthinking application of EBM can go wrong; these include withholding therapy in patients whose subgroup was excluded from clinical trials, blind acceptance of the numbers, reliance on studies with crucial design flaws and reliance on intention-to-treat analysis when it is not appropriate.
Study selection Examples assessed included withholding cholesterol-lowering therapy in the elderly, not using B-vitamin therapy for stroke prevention, not using revascularisation for true renovascular hypertension and avoiding statin therapy for fear of intracerebral haemorrhage.
Findings Zealous application of EBM is often inappropriate.
Conclusions In some instances, when there is a lack of evidence, or faulty interpretation of the evidence, clinical judgement should inform the application of EBM.
- lipid disorders
- geriatric medicine
- stroke medicine
Statistics from Altmetric.com
Take home messages
Sometimes zealous application of evidence-based medicine (EBM) is inappropriate.
Examples include withholding lipid lowering in the elderly, not using B vitamins for stroke prevention, not using renal revascularisation for true renovascular hypertension and avoiding statins for fear of intracranial haemorrhage.
Clinical judgement is needed in the application of EBM.
There can be no doubt that when evidence exists, it is better to practise medicine on the basis of evidence, rather than on the bases of hunches, guesses, personal preference and other imperfect drivers of medical decisions. One of the greatest advances in medicine was Cochrane’s concept of combining results of clinical trials to better assess the evidence. The Cochrane Reviews, and the contributions of the late Dr David Sackett (perhaps the father of evidence-based medicine (EBM)) have added much to EBM. However, EBM and the guidance emanating from it were never intended to replace clinical judgement. The idea was that it provided a better platform for making clinical decisions, not that it subverted the decision-making process. There are problems with how EBM is sometimes applied unthinkingly. In the extreme, it amounts for some to a religion, and would result in the following scenario:
A patient comes to the physician with a medical problem, seeking advice. The physician’s reply is: ‘Sorry, the randomised clinical trial (RCT) has not been done yet for this condition in people exactly like you, so there’s nothing we can do. You’ll have to come back in a few years’. An example is found in a systematic review of RCTs for the use of parachutes to prevent death and trauma from gravitational challenges.1 As none could be found, the authors commented as follows: ‘Only two options exist. The first is that we accept that, under exceptional circumstances, common sense might be applied when considering the potential risks and benefits of interventions. The second is that we continue our quest for the holy grail of exclusively evidence based interventions and preclude parachute use outside the context of a properly conducted trial’.
Of course, the way medicine is actually best practised is on the basis of a combination of evidence when evidence exists, and extrapolation from available evidence, interpreted in the light of the pathophysiology of the condition under consideration, and effects of various therapies in relation to that pathophysiology. This is called clinical judgement. A recent commentary in the British Medical Journal argued that with regard to drug safety, clinical judgement trumps clinical trials.2 A number of previous papers have commented on the philosophical limitations of EBM, and the need for caution in how it is applied.3–7
There are perhaps four major ways in which religious application of EBM goes off the rails: inappropriately withholding therapy in patients whose subgroup was excluded from clinical trials, blind acceptance of the numbers, reliance on studies with crucial design flaws and reliance on intention-to-treat (ITT) analysis when it is not appropriate. In this essay, I will discuss examples of these problems.
Withholding therapy in patients whose subgroup was not included in RCTs
Some geriatricians argue that one should not prescribe statins to people over the age of 80 years, because they were not included in randomised trials of statins, so there is no evidence of benefit. However, there is also no evidence that they are not beneficial! This kind of unthinking was exemplified recently in comments in the Journal of the American Medical Association, objecting to new cholesterol-lowering guidelines that state that ‘it may be reasonable to prescribe statins in the elderly’. Skolnik8 emphasised adverse effects of statins as a reason not to prescribe them in the elderly, and suggested that it was unreasonable to prescribe statins in the elderly. An online comment by Budenholzer8 echoed Skolnik, bemoaning the lack of EBM in USA.
However, both authors overestimated the potential harm from statins, while underestimating the potential benefit. Most of the adverse effects of statins that are widely touted are mythical,9 10 based on observational studies with indication bias and confounding, or on inappropriate reliance on ITT analysis. As discussed below, statins do not cause intracerebral haemorrhage(ICH). They also do not cause hepatotoxicity, renal failure, cataracts11 or dementia.12 If anything, statins reduce the risk of dementia,12 13 as would be expected: cholesterol lowering prevents strokes, and preventing strokes is at present the best available way to prevent dementia.14 15 With the rare exception of autoimmune statin myopathy,16 probably the only valid causally related adverse effects are myopathy and a slight increase in the risk of diabetes,10 probably both due to impaired mitochondrial function resulting from depletion of ubiquinone.12
The reason that elderly patients should be treated with cholesterol-lowering agents is that they are at a much higher risk of cardiovascular events.17 In any medical condition, the absolute risk reduction is highest, and hence the number needed to treat (NNT)18 is lowest in persons with the highest risk. The elderly stand to have the greatest benefit of cholesterol lowering17 (figure 1). My main reason for prescribing cholesterol-lowering medication to healthy elderly patients at risk of stroke is not to prolong life, but to reduce the risk of stroke, and thereby reduce the risk of dementia and increase the likelihood of continued independence with a good quality of life.19
Indeed, there is good evidence that statins are beneficial in the elderly. In a meta-analysis of RCTs, the benefit of statins did not decline significantly with age.20 In a nationwide Korean population-based case-control study of primary prevention in persons age >75 years, 11 017 cases were matched to 55 085 control subjects, with a mean age at baseline of 83.7±3.2 years. Current use of statins was significantly associated with a reduced risk of composite outcome of myocardial infarction/stroke/all-cause death (adjusted OR (AOR) 0.77; 95% CI 0.71 to 0.84), compared with non-users. Current use of statin also reduced the risk of stroke (AOR 0.74; 95% CI 0.61 to 0.89) and of all-cause death (AOR 0.73; 95% CI 0.66 to 0.81).
In a propensity matching Korean study of 1278 primary prevention patients with a median age of 78 years, IQR (76–80), half were on statins at baseline. After 5 years of follow-up, the risk of Major Adverse Cardiovascular Events (vascular death, myocardial infarction, stroke/Transient Ischemic Attack, coronary revascularisation) was significantly reduced among those taking statins: HR 0.59, 95% CI 0.41 to 0.85, p=0.0050.21
A French population-based study analysed outcomes of stopping statins in all persons who turned 75 years in 2012–2014, had no history of cardiovascular disease (CVD) and had possessed >80% of prescribed statins in the previous 2 years: 102 969 continued statin and 17 204 stopped them. Stopping statins was associated with a 33% increase in hospitalisation for cardiovascular events.22
In an RCT in Japan, 3796 patients 75 years or older with LDL cholesterol levels ≥140 mg/dL (3.68 mmol/dL) and no history of CAD were randomised to receive dietary counselling with or without a daily dose of ezetimibe 10 mg. The patients were high risk, with a history of one or more of the following: diabetes mellitus, hypertension, low high-density lipoprotein cholesterol, hypertriglyceridaemia, smoking, peripheral vascular disease or a previous history of cerebral infarction. Mean age was 80.7 years; 74% of the population was female. Ezetimibe reduced the risk of the composite primary end point (sudden cardiac death, Myocardial Infarction, Percutaneous Coronary Intervention or Coronary Artery Bypass Graft, and/or stroke) by 34% over 5 years (HR 0.66; 95% CI 0.50 to 0.86). This was driven by a 40% reduction in cardiac events (HR 0.60; 95% CI 0.37 to 0.98).23
In my view, the emphasis should not be on statins, but on cholesterol-lowering therapy. Ezetimibe and statins are synergistic, because they affect different mechanisms: statins reduce synthesis of cholesterol, whereas ezetimibe blocks absorption of cholesterol; the combination more than doubles lowering of low-density lipoprotein cholesterol (LDL-C) compared with either alone. Adverse effects of statins can be minimised by adding ezetimibe to permit lower doses of statin, and perhaps by using higher doses of ubiquinone than were used in some randomised trials of ubiquinone for statin myalgias.12 In an RCT of ezetimibe versus placebo added to statin, ezetimibe reduced the risk of stroke among patients with coronary artery disease,24 and the NNT to prevent one cardiovascular event was much lower in older participants: 11 above age 75 years, versus 125 in younger participants.25 The 2019 European guideline on dyslipidaemia made a class I recommendation that ezetimibe should be added to statins in patients not achieving target LDL-C; for very high-risk patients the target was LDL-C <1.4 mmol/L (53 mg/dL).26
In view of the foregoing, it seems unreasonable not to offer at least a trial of cholesterol-lowering therapy to elderly patients at high risk of stroke and other cardiovascular events. Adverse effects of statins could be minimised by using a low dose of statin, combined with ezetimibe.27 Mortensen and Falk make a strong argument for lipid lowering in the elderly, even in primary prevention.17
Blind acceptance of the numbers
In recent years there have been suggestions that meta-analyses should be conducted only by methodologists with no content expertise. An example of the problem with that approach is the widespread belief that B vitamins for homocysteine lowering do not prevent stroke. That commonly held belief probably has its origins in the widely quoted statement that ‘Homocysteine is dead’, made by Dr Kaare Harald Bønaa in September 2005 at the European Society of Cardiology Congress in Stockholm.28 He was describing the results of the Norwegian Vitamin Trial (NORVIT),29 which was published in 2006 in the same issue of the New England Journal of Medicine as the results of the Heart Outcomes Prevention Evaluation (HOPE) 2 Trial30 and an accompanying editorial by Loscalzo.31 Those trials followed the Vitamin Intervention for Stroke Prevention (VISP) Trial, published in 2004, which showed no benefit of B vitamins.32 So in 2006 it seemed clear that B vitamins did not prevent stroke, and that conclusion was echoed by even the most recent Cochrane review of the topic, in 2017.33 However, that conclusion was biologically implausible, based on a lot of evidence of increased risk of CVD from elevated plasma total homocysteine (tHcy), by a number of biological mechanisms.34
It required considerable detective work based on content expertise, a huge Chinese trial of folic acid alone35 and 13 years, to unravel the complexity of the evidence.
While the VISP Trial showed no benefit of B vitamins, and the NORVIT trialshowed not only no benefit, but slight harm in the study arm that included cyanocobalamin, the Heart Outcomes Prevention Evaluation (HOPE) 2 Trial actually showed a 23% reduction of stroke. The authors, being cardiologists and therefore innocent of the cerebral circulation, concluded that since B vitamins did not reduce the risk of myocardial infarction, the reduction of stroke was a chance finding. However, stroke is not the same as myocardial infarction; it is much more likely to be embolic,34 and elevated tHcy quadruples the risk of stroke in patients with atrial fibrillation.36 Loscalzo concluded31 that it was clear that B vitamins did not reduce cardiovascular disease, and hypothesised that excess unmetabolised folic acid was the reason.
However, there were important reasons why there was no benefit overall in the VISP Trial: (1) We did not use a placebo control; to minimise the risk that patients would take vitamins in addition to study vitamins in case they were randomised to placebo, we compared high-dose B vitamins including 400 mcg daily of cyanocobalamin (B12) versus low-dose vitamins including the recommended daily intake of B12 (6 mcg). (2) Folate fortification of the grain supply was implemented in North America in 1989, the same year the study began, thus negating much of the benefit of folic acid supplements. (3) In both arms of the study, participants with a low serum B12 at baseline were given monthly injections of B12, for ethical reasons, thus negating the benefit of B12 in the very participants who would have benefited most.
To address those issues, we carried out an hypothesis-driven subgroup analysis excluding participants who received B12 outside the randomised assignment.37 For the wrong reason, we also excluded participants in the lowest 10% of estimated glomerular filtration rate (eGFR). The reason was that in dialysis patients tHcy was not lower with 5 mg of folic acid than with 1 mg daily,38 so I thought that B vitamins would not be beneficial in patients with impaired renal function. I never dreamed they would be harmful. The participants excluded because of low eGFR had a mean eGFR of 48 mL/min/1.73 m2. As shown in figure 2, participants with a baseline serum B12 above the median of 322 pmol/L (indicating that they could absorb B12 well) and had received high-dose vitamins had a 34% reduction of stroke/myocardial infarction/vascular death compared with those who had a baseline serum B12 below the median and received low-dose B vitamins.
Two further clues developed in 2010. The French SUpplementation with FOlate, vitamin B6 and B12 and/or OMega-3 fatty acids (Su.Fol.OM3) Trial39 reported a 43% reduction of stroke with B vitamins that included a much lower dose of cyanocobalamin (only 20 mcg daily vs 400–1000 mcg daily in the other trials) in participants with better renal function than in earlier studies (table 1), and a randomised trial in patients with diabetic nephropathy actually showed harm from high-dose B vitamin containing 1000 mcg daily of cyanocobalamin. High-dose B vitamins accelerated the rate of decline of renal function, and as shown in figure 3, doubled the risk of a composite of myocardial infarction, stroke, revascularisation and all-cause mortality.40 All the events occurred among participants with a GFR <50 mL/min/1.73 m2, tantalisingly close to the eGFR of 48 mL/min/1.73 m2 excluded from the VISP efficacy analysis.41 In 2011, I hypothesised with Stampfer42 that the reason for the null results of earlier trials was not excess folic acid, as hypothesised by Loscalzo,31 but harm from cyanocobalamin among persons with impaired renal function.
Then in 2015, the light went on. The China Stroke Primary Prevention Trial, in >20 000 patients with hypertension followed for 5 years, reported that folic acid significantly reduced ischaemic stroke by 24%, in a population with no folate fortification.35 Crucially, they found that even among participants with a baseline eGFR <60 mL/min/1.73 m2, folic acid slowed the decline of renal function and reduced a composite event that included renal failure and mortality.43
In 2017 it became apparent that harm from cyanocobalamin had obscured the benefit among study participants with renal failure. Table 1 shows the results of the trials ordered by improving renal function. A meta-analysis stratified by dose of cyanocobalamin and by renal function confirmed that B vitamins are beneficial in patients with good renal function, or when given with low doses of cyanocobalamin or no cyanocobalamin.44 In 2018 a meta-analysis also reported reduction of stroke with folic acid, and with B vitamin combinations including folic acid and B12.45 B vitamins do reduce the risk of stroke, but we should be using methylcobalamin, not cyanocobalamin, and even the vaunted Cochrane methodologists who blindly accepted the numbers got it wrong.
Reliance on inappropriate ITT analyses
One of the sacraments of the religion of EBM is ITT analysis. However, ITT is mainly directed at policy decisions. For the purpose of therapeutic decisions in individual patients, on-treatment analysis is more informative. Hernan and Robins discussed this issue,46 saying that ITT ‘may not be directly relevant for guiding decisions in clinical settings with different adherence patterns’. Sheiner and Rubin made the distinction between ‘use effectiveness’ (the result of prescribing a medication) and ‘method effectiveness’ (the result of taking a medication). They pointed out that for the purpose of treating individual patients, method effectiveness was a more useful pharmacological characteristic.
One example of the problem with ITT, as well as problems with study design, is renovascular hypertension. It is widely believed that revascularisation is not beneficial in this condition, but anyone who has managed hundreds of patients with true severe renovascular hypertension (as I have) absolutely knows that revascularisation (or nephrectomy of a small shrunken severely ischaemic kidney) is the only way to achieve blood pressure control. The kidney will simply do what it takes to force blood through the blocked artery.47 There are two key design flaws in RCTs in renovascular hypertension. The first is ignoring the key difference between true renovascular hypertension and hypertension with incidental renal artery stenosis; the second is waiting until the patient already has severe renal dysfunction before randomising the patients.48 When renovascular hypertension has been permitted to go on too long, the contralateral kidney is destroyed by hypertensive nephropathy; revascularisation needs to occur early enough to preserve both kidneys.
As I have said previously,48 ‘In the study by Wheatley et al,49 60% of the participants had a serum creatinine level of 150 mmol/L or higher at the time of randomization, so the intervention was too late to achieve a benefit with regard to renal function. However, there are other equally or more important limitations of the literature in this area. In the study by Wheatley et al,49 only 79% of the participants randomised to the intervention actually had revascularisation performed, and 6% of those randomised to medical therapy crossed over to the intervention. In the study by van Jaarsveld et al,50 44% of participants randomised to medical therapy crossed over to the intervention. Yet the main analysis in both studies was an intention-to-treat analysis’. This is ludicrous.
Another glaring example is the widespread belief that statins cause ICH. This belief arises mainly from the ITT analysis of the Stroke Prevention by Aggressive Reduction in Cholesterol Levels (SPARCL) Trial,51 in which participants randomised to 80 mg daily atorvastatin after an ischaemic stroke had an excess of ICH from statins compared with those randomised to placebo. Indeed, this belief is so widespread that in a lecture on stroke prevention at a World Stroke Organization (WSO) conference in 2018, the chair of the WSO education committee repeated it, and the National Institutes of Health is funding a randomised trial in which patients with ICH are being randomised to continuation or cessation of statin. This could be regarded as unethical, because it is clear that stopping statin in this high-risk group of patients will increase the risk of ischaemic stroke and myocardial infarction, without reducing the risk of ICH. It is not possible that atorvastatin caused ICH in the SPARCL Trial, since among participants in SPARCL who had >50% reduction of LDL-C, there was a significant reduction of stroke, but no increase in ICH.52 It is most likely that ICH occurred among patients randomised to atorvastatin in SPARCL because they stopped their statin because of adverse effects, and at the same time, as patients so often do, also stopped their antihypertensive medication. Those with ICH were more likely to have stage II hypertension.53 Meta-analyses confirm that statins and low LDL-C do not cause ICH,54 55 yet this belief remains widespread.
Notwithstanding the great benefits of EBM, unthinking application of it can do great harm. EBM needs to be tempered with scepticism, an understanding of the limitations of studies, a knowledge of the pathophysiology of disease and the effects of interventions in relation to that pathophysiology, and yes—clinical judgement.
Contributors JDS wrote this article.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests JDS is a consultant to Amgen and Orphan Technologies, an officer of Vascularis Inc., and has received lecture fees from Pfizer and Bristol Myers Squibb.
Patient and public involvement statement Not applicable
Patient consent for publication Not applicable.
Ethics approval Not applicable
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No data are available.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.