A graphic framework for teaching critical appraisal of randomised controlled trials
The critical appraisal of a randomised controlled trial (RCT) is one of the basic elements of clinical epidemiology and evidence-based medicine curricula. Various frameworks for critical appraisal have been proposed. The “Users' Guides” series in JAMA includes a framework for evaluating RCTs that consists of 12 questions grouped under 3 headings1, 2:
Are the results of the study valid?
What were the results?
Will the results help me in caring for my patients?
However, >20 published users' guides currently exist, and our experience shows that residents and clinicians attempting to pursue a practice that uses evidence-based medicine have 1 of 2 problems: either they try to memorise, without much success, all the questions from several of these guides, or they rely completely on small pocket cards that contain the pertinent questions and then find themselves lost if they misplace or forget those cards. Other groups have developed more general criteria. For example, the Newcastle Critical Appraisal Worksheet is a 3 × 10 grid that can be applied to any type of study, but it requires some basic knowledge of epidemiological terms and concepts.3
Over time, we've put together a flow diagram of RCTs and the points at which bias can creep in; this flow diagram serves as a memory aid and can act as a framework on which to “hang” whatever critical appraisal guide the user is most comfortable with. We have often found that the process of drawing this flow diagram for the particular study being appraised shows the sources of bias and allows the reader to track the patients and points at which patients are lost, even before referring to a set of critical appraisal questions. We offer this framework as a pedagogical aid when teaching clinicians and residents.
Figure 1 shows the flow of an RCT and can be drawn as the starting point for a critical appraisal exercise.
Each of the 5 numbered steps in the RCT flow diagram can act as the focal point for a discussion of methods.
Step 1: selection and sampling issues
When drawing the larger circle, one can ask such questions as “What sorts of patients were recruited, and where were they recruited from (ie, primary care or a referral centre)?” When drawing the arrow that indicates which patients are recruited into the study, one can ask whether the inclusion and exclusion criteria make sense and whether consecutive patients are being recruited; exclusions can be represented graphically by drawing a second arrow peeling off from the main one. This arrow also serves to indicate the total number of patients screened to obtain the enrolled sample. All these considerations affect the external validity and generalisability of the study, whereas subsequent steps affect internal validity. The graphical representation of this concept is the box drawn around the remainder of the study (fig 2).
Step 2: randomisation
The arrows representing the allocation of participants to the 2 groups serves to highlight graphically the process of randomisation, which evenly (or at least randomly) distributes known and unknown confounders. One can then discuss the principle of randomisation, including concealment (the person enrolling the patient in the study must not be able to predict to which group the patient will be randomly allocated), and less secure processes, such as quasirandomisation. One can then ask how to check the success of randomisation. Are the baseline characteristics of the study groups similar? Any imbalance in those characteristics can signal 1 of 2 things: randomisation has not been done properly (the participants were not truly randomly allocated) or randomisation was done properly but a discrepancy has arisen by chance (which is more likely when the number of patients is small). In the latter case, one can check to see whether the study authors statistically adjusted for the differences.
Step 3: follow up
A catchy mnemonic we have found helpful in highlighting the sources of bias at this step are the “5 Cs”: contamination, crossover, compliance, co-intervention, and count (ie, loss to follow up).
Contamination often occurs in a trial of an educational intervention where, for example, the control group may adopt such lifestyle changes as dietary modification or exercise intended for the intervention group only. Contamination can also happen when control and treated patients share their medications, as in certain AIDS trials.
Crossover occurs most often in open trials in which patients or clinicians know what medication the patient is receiving and also what the alternative treatment is: for example, in trials where there are medical and surgical arms or a conservative and aggressive treatment protocol, patients in the less aggressive arm who worsen may cross over to the more aggressive treatment arm. Crossovers may be represented graphically by arrows that cross between the 2 arms of the trial (fig 3).
Lack of compliance with the intervention can introduce bias, as can count or loss to follow up, if patients who drop out differ in their characteristics from those who remain. Both of these sources of bias may be represented by arrows that peel off from the 2 arms (fig 3).
Co-interventions also introduce bias if they are applied differentially to the 2 groups. Co-interventions can be represented by arrows that join the 2 arms (fig 3). Contamination, crossover, and lack of compliance will bias results toward the null (attenuate any effect), whereas co-intervention and count may bias results in either direction.
Step 4: outcomes
The boxes in the figures represent the outcomes in the 2 groups. Discussion can focus on whether the chosen outcomes were reasonable and whether all important outcomes were considered. Who judges or counts the outcome is of concern. Outcome assessment leads to a discussion of the importance of blinding, which can include blinding of the patient, the caregiver, the outcome assessor (particularly important when such patient-reported subjective measures as quality of life are used), and even the statistician doing the analysis. Having the assessors and patients guess their assignments at the end of the trial can give an indication of whether unblinding occurred.
Step 5: analysis
This step includes a discussion of intention to treat (ITT) analysis. ITT analysis is important because it preserves the benefit of randomisation. If those patients who dropped out, were non-compliant, or had intolerable adverse effects are not considered in the analysis, it is the equivalent of allowing participants to self select or opt out of the study. If these patients' characteristics differ from those of the rest of the group, the even distribution of confounders obtained through randomisation is lost.
Other issues of analysis include the magnitude of the effect (measured by relative and absolute risk differences, odds ratios, and numbers needed to treat), precision (confidence intervals), and subgroup analyses (were these preplanned or derived from a “fishing expedition” once the data became available?).
Students of evidence-based medicine often try unsuccessfully to commit to memory a particular critical appraisal framework (often lengthy), or they have to depend on pocket cards and are lost without them. We have described a pedagogic aid: a flow diagram of an RCT, which has been developed over years of teaching residents. This diagram focuses on the steps in an RCT, and by drawing arrows, it highlights the biases possible at each step. This diagram serves as a framework on which the list of critical appraisal questions can be hung and is easy to remember.