Critics of evidence-based medicine (EBM) often ask for randomised trial proof that ‘EBM’ works. The trial1 by Izcovich and colleagues is a laudable attempt to address this gap, but, to interpret it, we need to examine two things: the EBM processes used and the statistical power of the study.

EBM is an approach to clinical care and continuing medical education; it is not a single standard process. Different specialties and different individuals have adapted and adopted the principles of EBM to different degrees and in vastly different ways (to learn about some examples of this diversity of processes listen to the podcasts on http://www.cebm.net/index.aspx?o=4648). The EBM process of Izcovich and colleagues focused on a posthospital rounds literature search – an idea that evolved over several decades.2 It is useful to compare this process with the ‘evidence cart’ approach that Sackett used in Oxford,3 which has some similarities and also some important differences. In comparing the two studies (table 1), two important differences are found as follows: (1) Izcovich and colleagues answered fewer questions (one question per five patients compared with Sackett's two questions per three patients – a threefold difference) and (2) the search was done after rather than during rounds (when the decisions are being made) so there appeared to be no team discussion of the evidence, only a passive delivery. In Sackett's approach, searches altered about one-third of decisions made during rounds. The measures needed to assess any particular processes of EBM are similar to those of the steps of EBM: How many questions were asked (including those answered on rounds using other sources, eg, UpToDate – http://www.uptodate.com)? How often was good evidence found? How often were decisions changed? How often were those decisions implemented?

With any randomised controlled trial approach to assess any particular variety of EBM, a crucial problem is the statistical power of the study. If we require big studies to answer important treatment questions, we require even bigger studies to detect whether EBM – which results in an incremental use of evidence-based treatments – improves outcomes such as mortality. To illustrate this, imagine two wards with only patients with myocardial infarction: Ward A uses aspirin routinely while Ward B never uses it. To detect the expected 25% RR reduction would require thousands of patients enrolled. However, if Ward B – the less ‘EBM’ ward – uses it half the time, then it would take four times the sample size to detect a difference. And if one-third of such decision was changed by evidence, then we would need a study nine times larger than the primary study. With the Izcovich study, even if every search done had led to a change of decision (one in five patients as only 19% of patients generated searches), we would still need a trial 25 times larger than the primary studies used to answer the individual clinical questions – perhaps in the 100 000+ range.

So measurement of processes and adequate power is important to answer the question directly. However, we should also ask whether these are the right questions. Instead, we might ask “what is the best way to improve the use of good evidence in clinical practice?” The ‘evidence cart’ approach is one, the ‘supported search’ another, but there are still many other varieties of EBM implementation. For example, a cluster trial of intensive implementation of guidelines for the management of malaria showed a 50% RR reduction in in-hospital mortality.4 A randomised trial of inserting an evidence statement into hospital discharge letters5 showed an 11% absolute increase in general practitioner adherence to discharge medications – a ‘Number Needed to Write’ (NNW) of 9!

Comparative studies are needed to assess different approaches and need to account for the process measures mentioned, not only the immediate outcomes but also the learning that occurs long-term. Some years ago, Sharon Straus asked the challenging question “what is the E for EBM?”6 Although there are many good arguments and indirect evidence, there have been few direct attempts to answer this question using a randomised trial – the ‘gold standard’ for intervention evidence. Izcovich et al are to be applauded for that. However, as Straus concluded, “. . . it may be too soon to tell if evidence-based medicine changes clinical performance and outcomes because advocates think that it requires lifelong learning, and this is not something that can be measured over the short term.”