Statistics from Altmetric.com
We are surrounded by maelstroms of claims about the effects of treatments. Such claims may include strategies to prevent illness, such as changes in health behaviour or screening, therapeutic interventions, or public health and system interventions. Many causal claims are demonstrably wrong, and although some are well intentioned, others are deliberately misleading to serve interests other than the well-being of patients and the public.1 Learning how to judge which claims to believe is a core competence needed by effective practitioners of Evidence-Based Healthcare (EBHC).2
It is not only practitioners who need to be able to apply these skills. Patients and the public also need to be equipped to assess the trustworthiness of treatment claims. A survey of 2041 adults commissioned by the UK Academy of Medical Sciences reported only 37% of the public said they trusted evidence from medical research, compared with 65% who trusted the experiences of their friends and families.3
Unfortunately, skills to assess the trustworthiness of causal claims remain rare. Recognition of this ‘skill deficit’ was the motivation for the book entitled Testing Treatments: better research for better healthcare (http://www.testingtreatments.org/)4 The book draws attention to the importance of certain concepts when assessing treatment claims. For example, one chapter emphasises that ‘treatment given earlier is not necessarily better treatment’; and another shows that ‘more treatment is not necessarily better treatment’. Acquiring the ability to assess the trustworthiness of causal claims depends on being familiar with these and other Key Concepts (box).
Short titles for the IHC Key Concepts
Recognising an unreliable basis for a claim
Treatments can harm.*†
Anecdotes are unreliable evidence.*†
Association is not the same as causation.†
Common practice is not always evidence-based.*†
Newer is not necessarily better.*
Expert opinion is not always right.*†
Beware of conflicting interests.*
More is not necessarily better.
Earlier is not necessarily better.
Hope may lead to unrealistic expectations.
Explanations about how treatments work can be wrong.
Dramatic treatment effects are rare.
Understanding whether comparisons are fair and reliable
Comparisons are needed to identify treatment effects.*†
Comparison groups should be similar.*†
Peoples’ outcomes should be analysed in their original groups.
Comparison groups should be treated equally.
People should not know which treatment they get.*
Peoples’ outcomes should be assessed similarly.
All should be followed up.
Consider all the relevant fair comparisons.*†
Reviews of fair comparisons should be systematic.
Peer review and publication does not guarantee reliable information.
All fair comparisons and outcomes should be reported.
Subgroup analyses may be misleading.
Relative measures of effects can be misleading.
Average measures of effects can be misleading.
Fair comparisons with few people or outcome events can be misleading.*
Confidence intervals should be reported.
Do not confuse ‘statistical significance’ with ‘importance’.
Do not confuse ‘no evidence of a difference’ with ‘evidence of no difference’.
Making informed choices
Do the outcomes measured matter to you?
Are you very different from the people studied?
Are the treatments practical in your setting?
Do treatment comparisons reflect your circumstances?
How certain is the evidence?
Do the advantages outweigh the disadvantages?*†
*Concepts included in the IHC primary school resources (Nsangi et al, 2017).
†Concepts included in the IHC podcast (Semakula et al, 2017).
IHC, Informed Health Choices.
The concepts in the box were developed as the first step in the Informed Health Choices (IHC) project as a conceptual framework to guide the development and evaluation of learning resources for primary school children and their parents in Uganda.5 6 We believe this to be the first framework with this objective. This paper describes the current status of the IHC Key Concepts List7 and some of its uses.
What do we mean by ‘concepts’?
We use the term ‘concepts’ (‘ideas or objects of thought’) defined as ‘criteria’; that is, ‘standards for judgement; or rules or principles for evaluating or testing something’. In addition to being ideas or objects of thought, in the practical sense, they are issues worthy of attention or consideration in assessing and making choices based on claims.
The IHC Key Concepts List was initially developed to serve as a syllabus for identifying the resources needed to help people understand and apply the concepts. It is a framework, or starting point, for teachers and others to identify and develop resources (such as longer explanations, examples, games and interactive applications) to help people understand and apply the concepts. The IHC Key Concepts List currently includes 36 concepts (box), divided into three groups:
Recognising an unreliable basis for a claim.
Understanding whether treatment comparisons are fair and reliable.
Making informed choices.
The original List included 32 concepts divided into six groups.7 The List is reviewed and amended annually, and an up-to-date List and explanations for each concept can be found on the Testing Treatments interactive and Informed Health Choices websites.
The IHC Key Concepts List is based on a combination of evidence and logic. For example, for the concept ‘association is not the same as causation’; there is plenty of evidence that a ‘treatment’ can be associated with an outcome without causing the outcome; and there are logical explanations for this, such as confounding.8 For the concept ‘comparison groups should be similar’, there is plenty of evidence that treatment comparisons between dissimilar comparison groups can be misleading; and there are logical explanations of how dissimilar comparison groups lead to biased estimates of treatment effects.9
The concepts are intended to apply to decisions about any type of treatment (defined as any action to improve or maintain the health of individuals or communities). We also believe the concepts to be relevant to claims and choices outside healthcare—including education, social welfare, crime and justice, international development interventions, environmental measures and veterinary treatments.
Developing the IHC List of Key Concepts
We used explicit and pragmatic criteria to determine which concepts to include in the List.7 We used a systematic, transparent and iterative process, involving potential end users and experts within the field. The book Testing Treatments 4 was the initial source for creating a list of ‘candidate’ concepts. The List was supplemented by reviewing other books written for the general public10 11; checklists for the general public, journalists and health professionals12–17; and consideration of concepts related to assessing the certainty of evidence for treatment effects and making informed health choices.18
We also invite people to submit suggestions through the TTi website (http://en.testingtreatments.org/key-concepts-for-assessing-claims-about-treatment-effects/feedback-key-concepts/) or by submitting them directly (eg, when we present the Key Concepts at conferences) or through open workshops (eg, at the Global Evidence Summit in Cape Town in 2017).
Once a year, AA-D, AO and IC discuss each suggestion; decide by consensus how to respond; and record their response to each suggestion and their reasoning. This information is available on request and will be published in future updates.
The List includes several concepts relating to numerical or statistical information (Concepts 2.12–2.17). These concepts are included in the second group (understanding whether comparisons of treatments are fair and reliable) because they refer to ways in which people are commonly misled by statistical information. An explanation for each concept is provided in the full list. For example, there is an explanation of why the use of P values to indicate the probability of something having occurred by chance may be misleading and why confidence intervals are more informative (Concept 2.16). Statistical concepts that are relevant to treatment comparisons but which are not commonly misleading are outside the scope of the IHC Key Concepts List.
How does the framework provided by the IHC key concepts list differ from other frameworks?
One of the first lists of criteria for making causal inferences in healthcare was proposed in 1882 by Robert Koch for establishing the existence of an infectious pathogen. Subsequent lists of criteria for making causal inferences include those of Hill8 and Susser.19–21 More recently, a wide range of tip sheets and checklists have been published.14–16 22 23 Although all these lists address some of the IHC Key Concepts, the latter differs in two respects.
First, the IHC working group has used a systematic, transparent, and iterative process to develop the IHC Key Concepts List with the intention of helping the general public, children and healthcare practitioners to assess the trustworthiness of claims about the effects of treatments. Some checklists—for example, the Cochrane Risk of Bias tool24 and those for assessing the certainty of the evidence for estimates of effect (GRADE)18—have been developed systematically, transparently and iteratively, but they are intended primarily for those preparing systematic reviews or developing clinical practice guidelines.
Second, the IHC Key Concepts are not a guide for making causal inferences, a tip sheet, a checklist or a tool for those developing systematic reviews or guidelines. Rather, they have been developed to offer a coherent framework to serve as a starting point for teachers, learners and researchers to map out what people need to learn; determine how best to help people learn; and measure the extent to which they have learnt to assess treatment claims and to make informed health choices.
How can the IHC key concepts list be used?
The IHC learning resources and the Claim Evaluation Tools database
The aim of the IHC Project is to help people learn how to assess treatment claims and make informed choices. The project focused initially on primary school children in Uganda and their parents. Identification of the Key Concepts was the starting point for developing relevant resources and tests. Members of the IHC Project team developed a textbook, using a comic book story, to teach 12 of the key concepts to children (box). Selection of the 12 concepts was made in two stages. First, with input from primary school teachers, we selected the 24 concepts that teachers felt could be mastered by children aged 10–12 years. Second, because 24 concepts could not all be taught during the one term available for teaching, 12 concepts were selected after taking account importance and difficulty, as judged after pilot and user testing. A podcast was developed to teach nine of the concepts to parents of the primary school children (box).
Although there are over 400 studies of interventions and assessment tools that address at least one of the IHC Key Concepts, only four assessment tools included ≥10 Key Concepts.25 The Claim Evaluation Tools database of multiple-choice questions was developed to address this gap.26 The multiple-choice questions are designed to evaluate people’s ability to apply the IHC Key Concepts in an objective way. The database is open access to researchers or teachers. New questions for both new and old concepts will be added to the database as they become available. Questions from the database can be used by learners for self-assessment; by teachers to assess students using scenarios, and by researchers to evaluate interventions or map people’s abilities to apply the Key Concepts.
Questions from this database were used as the primary outcome measures in randomised trials of the primary school resources and the podcast.27 28 In the first trial, half of over 10 000 children in 120 schools used the primary school resources over a school term. The study showed convincingly that young children, including those with poor reading skills, could be taught to apply those 12 concepts.27 In the second trial, half of over 500 parents listened to the podcast, with stories explaining nine Key Concepts, and the other half listened to typical public service health announcements.28 This trial showed convincingly that parents, including those with poor reading skills, could learn those nine concepts.
The Critical thinking and Appraisal Resource Library
The Critical thinking and Appraisal Resource Library (CARL) is a database currently containing over 500 open-access learning resources in a variety of formats—text, audio, video, webpages, cartoons and lesson materials—which has built on efforts beginning in 2011 to establish such an inventory.29 After the initial IHC Key Concepts List had been published in 2015,7 learning resources in CARL were coded so that they could be retrieved using each of the IHC Key Concepts. Resources included in CARL are currently accessible through the Testing Treatments interactive website.
Students 4 Best Evidence
Students 4 Best Evidence (S4BE) is a website for students worldwide interested in evidence-based healthcare.30 The site features blogs written by over 400 students currently registered to blog. Blogs are being written on the IHC Key Concepts, each of them communicating the essential messages and providing illustrative examples (http://www.students4bestevidence.net/tag/keyconcepts/). Encouragingly, students are offering to translate the Key Concepts into other languages and also write brief, informal reviews of the learning resources in CARL, which are then posted both on the S4BE website and on Testing Treatments interactive.
The James Lind Library
The James Lind Library contains material illustrating the development of fair tests of treatments in healthcare. Material in the Library illustrating one or more of the IHC Key Concepts has been appropriately coded so that it can be retrieved through CARL.
Ensuring coverage of an international core curriculum for teaching EBHC to professional learners
The IHC Key Concepts have recently been considered by 140 clinicians and teachers developing an international core curriculum for teaching EBHC (Loai Albarqouni, personal communication, August 2017). The curriculum items are mostly expressed as essential competences for healthcare workers practising EBHC. The IHC Key Concepts List has been useful in identifying gaps in the EBHC curriculum and in ensuring that it covers relevant key concepts. The online supplementary appendix 1 shows how they relate to each other.
Supplementary file 1
The National Science Curriculum for schools in England
Key Stage 3 of the National Science Curriculum for schools in England has been developed for children between ages 11 and 14. It covers conceptual understanding, scientific enquiry, and the uses and implications of science. The online supplementary appendix 2 shows the relevance of some IHC Key Concepts to these themes, and how they can help learners to understand what working scientifically entails.
Supplementary file 2
Using Key Concepts and focusing on health is a good way to achieve these learning goals for three reasons. First, everyone is interested in health, so it makes these learning goals immediately relevant and engages learners. In addition, they focus on using science to inform decisions—something that is important to everyone—rather than on doing science. Second, they provide a coherent and sensible framework for identifying the key concepts that are most important for learners to understand and apply. Third, learning to apply the IHC Key Concepts can enable learners to recognise unreliable claims about treatments, to make better personal health choices and to participate in informed debate about health policies.
A spiral curriculum, other types of interventions and other frameworks for critical thinking
The IHC Key Concepts can be the starting point for developing a spiral curriculum (http://www.informedhealthchoices.org/spiral-curriculum/) and additional learning resources for primary and secondary school children. The aim of the spiral curriculum is to guide what to teach and when; introducing basic concepts first, repeating and reinforcing those in subsequent cycles; and introducing more difficult concepts later. It helps to avoid the trap of trying to teach too much at once and provides milestones for what should have been learnt and by when—teaching all 24 concepts at one time was overwhelming for teachers and children.5 31
Similarly, learning to apply the Key Concepts in the context of health could help improve critical thinking and decision making outside of health. It is uncertain to what extent this will be the case, but anecdotal evidence and some indirect evidence from pilot studies,5 6 randomised trials27 28 and process evaluations6 31 support this hypothesis. Data collected in a process evaluation of the IHC primary school resources show that children and teachers also reported applying the Key Concepts to decisions that were not necessarily about treatments, such as recognising that new is not necessarily better when buying shoes and recognising other unreliable claims when shopping.
Indirect evidence from randomised trials of other strategies to teach critical thinking have found that developing critical thinking skills can benefit academic learning outcomes as well as wider reasoning and problem-solving capabilities, and that this is of particular benefit to pupils from poorer families.32–34
The IHC Key Concepts List is a systematically developed collection of concepts to help people to assess whether treatment claims are trustworthy. The IHC Key Concepts continue to be developed, taking account of feedback and evaluation.
We have developed the IHC Key Concepts specifically for treatment claims and choices. However, criteria for making causal inferences about interventions and choices are similar for other types of interventions. Introducing the concepts to people at a young age not only helps to prepare patients and future health professionals to make informed choices in healthcare, but should also prepare them to make other types of informed choices as citizens, consumers and future policy-makers.
We invite all those interested in promoting critical thinking about causal claims—particularly about the effects of interventions—to consider applying the IHC Key Concepts List as a framework for conceptualising and promoting critical thinking, and for evaluating the effectiveness of alternative learning approaches.
The authors are grateful to Jeremy Howick for inviting IC, SR-V, SP, AA-D and ADO to present these ideas at the conference ‘Too much medicine: exploring the relevance of philosophy of medicine to medical research and practice’, 20 April 2017, Kellogg College, University of Oxford; and to Ruth Davis and David Spiegelhalter for comments on previous drafts of this manuscript.
Contributors IC suggested and organised the conference session on which the paper has been based, and coordinated the preparation of a first draft coauthored with ADO, AA-D, SR-V and SP. IC incorporated comments, and additional text contributed by other coauthors and redrafted the paper in the light of comments and suggestions from peer reviewers. CH prepared the final draft of the paper, which has been approved by all coauthors.
Funding IC and DB receive support through National Institute for Health Research funding for the James Lind Initiative. The IHC Key Concepts were developed as part of the Informed Health Choices Project, which is funded by the Research Council of Norway (project no: 220603/H10).
Competing interests DN has received expenses and fees for his media work. He holds grant funding from the NIHR School of Primary Care Research and the Royal College of General Practitioners. CH has received expenses and fees for his media work. He holds grant funding from the NIHR, the NIHR School of Primary Care Research, The Wellcome Trust and the WHO. IC declares no competing interest.
Provenance and peer review Commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.