You are here

Article Text

Article menu

Download PDFPDF

EBM analysis
Adapt or die: how the pandemic made the shift from EBM to EBM+ more urgent
  1. Trisha Greenhalgh1,
  2. David Fisman2,
  3. Danielle J Cane3,
  4. Matthew Oliver4,
  5. Chandini Raina Macintyre5
  1. 1 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
  2. 2 Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
  3. 3 Coalition for Healthcare Acquired Infection Reduction, Cambridge, Ontario, Canada
  4. 4 Association of Professional Engineers and Geoscientists, Edmonton, Alberta, Canada
  5. 5 The Biosecurity Program, Kirby Institute, University of New South Wales, Sydney, New South Wales, Australia
  1. Correspondence to Professor Trisha Greenhalgh, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford OX2 6GG, UK; trish.greenhalgh{at}phc.ox.ac.uk

Abstract

Evidence-based medicine (EBM’s) traditional methods, especially randomised controlled trials (RCTs) and meta-analyses, along with risk-of-bias tools and checklists, have contributed significantly to the science of COVID-19. But these methods and tools were designed primarily to answer simple, focused questions in a stable context where yesterday’s research can be mapped more or less unproblematically onto today’s clinical and policy questions. They have significant limitations when extended to complex questions about a novel pathogen causing chaos across multiple sectors in a fast-changing global context. Non-pharmaceutical interventions which combine material artefacts, human behaviour, organisational directives, occupational health and safety, and the built environment are a case in point: EBM’s experimental, intervention-focused, checklist-driven, effect-size-oriented and deductive approach has sometimes confused rather than informed debate. While RCTs are important, exclusion of other study designs and evidence sources has been particularly problematic in a context where rapid decision making is needed in order to save lives and protect health. It is time to bring in a wider range of evidence and a more pluralist approach to defining what counts as ‘high-quality’ evidence. We introduce some conceptual tools and quality frameworks from various fields involving what is known as mechanistic research, including complexity science, engineering and the social sciences. We propose that the tools and frameworks of mechanistic evidence, sometimes known as ‘EBM+’ when combined with traditional EBM, might be used to develop and evaluate the interdisciplinary evidence base needed to take us out of this protracted pandemic. Further articles in this series will apply pluralistic methods to specific research questions.

  • evidence-based practice
  • COVID-19
  • behavioral medicine
  • biomedical engineering
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjebm-2022-111952

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Two and a half years into the COVID-19 pandemic, it is time to take stock. What began as—we assumed—an acute respiratory illness rapidly sweeping the world has become a prolonged global crisis with medical, social, economic and political dimensions. The SARS-CoV-2 virus has proved tenacious and shifting. It causes a complex multisystem disease which disproportionately impacts minority ethnic groups and the sick, poor, old and disadvantaged. It produces significant mortality and—in some—prolonged sequelae. Effective and safe vaccines were produced rapidly, but uptake has been patchy and highly transmissible variants continue to spread and mutate. Coordinated disinformation campaigns have weakened the public health response.

    Despite a quarter of a million scientific papers on COVID-19, some basic issues remain contested. How exactly does the virus spread? How effective are non-pharmaceutical interventions—masks, distancing, closure of buildings, remote working and learning, lockdowns—in reducing transmission, and what are their trade-offs? How can we make schools, hospitals and other public buildings safe? How can we protect workers and the public without closing down the economy? How cn we reduce the shocking inequalities that have characterised this pandemic?

    This paper explains why we need to go beyond lip-service to evidential pluralism to fully answer those questions. While acknowledging the primacy of randomised controlled trials (RCTs) and meta-analyses of RCTs for estimating the efficacy of drugs and vaccines, we highlight and extend the arguments made by some scholars within the evidence-based medicine (EBM) movement for ‘EBM+’,1 2 defined as an approach which systematically considers mechanistic evidence (studies which aim to explain which factors and interactions are responsible for a phenomenon3) on a par with probabilistic clinical and epidemiological studies. Our central argument is that for some aspects of the pandemic, especially those characterised by a combination of complexity (multiple variables interacting dynamically with a high degree of uncertainty), urgency (decisions needed in days not years) and threat (the consequences of not acting could be catastrophic), mechanistic evidence has been mission-critical and RCTs difficult or impossible. Thousands of lives were likely lost as a result of what was incorrectly claimed to be an “evidence-based” approach—dismissing or downgrading mechanistic evidence, overvaluing findings from poorly designed or irrelevant RCTs, and advocating for inaction where RCT evidence was lacking. The pandemic is an epistemic opportunity for the EBM movement to come to better understand, debate and embrace EBM+.

    Evidence hierarchies: not fit for all purposes

    Every student of EBM is taught that the hierarchy of evidence—with meta-analyses of RCTs at the top and so-called ‘less trustworthy’ evidence (case studies, mechanistic evidence, expert opinion) at the bottom (figure 1A)4 —is a useful heuristic but does not apply in all circumstances. But guidance to apply the hierarchy flexibly to take account of—for example—the nature of the research question, the degree of complexity involved, whether RCTs of the necessary size and scope are actually feasible, and whether policy decisions can or should be put on hold pending definitive results—has been honoured more in the breach than the observance. From Guyatt et al’s5 paper announcing EBM as a ‘new paradigm’ in medicine to their 2008 Grading of Recommendations, Assessment, Development and Evaluation statement (which offers eight domains of strengths and limitations of the overall evidence base, including observational studies as well as RCTs)6 and the Oxford Centre for EBM 2011 Levels of Evidence Document,7 there remains within EBM orthodoxy an explicitly hierarchical relationship between probabilistic clinical and epidemiological evidence (invariably placed at the top of the hierarchy) and the kinds of evidence which help establish causal mechanisms (invariably placed at or near the bottom).

    Figure 1
    Figure 1

    One version of EBM’s hierarchy of evidence, showing evolution over time to encourage a more flexible approach. (Reproduced under CC-BY-4.0 Licence from Murad et al 4). EBM, evidence-based medicine.

    The cornerstone of EBM as taught to students—ask focused questions in population-intervention (or exposure)-comparison-outcome (PICO) format; search for RCTs and, if these are unavailable, work down the hierarchy of evidence; appraise studies for risk of bias; and combine them using meta-analysis—was developed for a stable context in which predictions based on yesterday’s research map unproblematically onto today’s policy questions (eg, what actions should we take to prevent and manage coronavirus?). Despite their theoretical strengths for some kinds of question, and despite heroic achievements in the pandemic with fast-track platform trials such as RECOVERY,8 PRINCIPLE9 and PANORAMIC,10 RCTs are not a panacea (table 1).11 Modifications of the hierarchy of evidence (eg, by separating the layers with dotted or wavy lines, as in figure 1B, and using systematic review as a lens through which primary evidence is interpreted, as in figure 1C 4 fall short of the epistemic shift that is now needed.

    View this table:
    Table 1

    Strengths and limitations of RCTs and meta-analyses for pandemic research

    Strengthening causal claims through mechanistic evidence

    Mechanistic evidence includes a wide range of designs, including in vitro experiments, biomedical imaging, autopsy, established theory, animal experiments, aerosol science, engineering research and simulations.3 Like all evidence, mechanistic studies can be of variable quality and definitiveness. Table 2 shows some preliminary criteria for grading mechanistic evidence, based partly on previous studies,12–14 which would benefit from further elaboration.

    View this table:
    Table 2

    A suggested hierarchy of evidence for mechanistic evidence (drawing partly on previous publications)12 14

    No one kind of evidence stands alone. Rather, as encapsulated in the Bradford Hill indicators15–17 (table 3), evidence of mechanisms must be combined with probabilistic evidence from clinical trials (and, where appropriate, evidence from non-randomised comparative and observational studies) to make a strong case for causality.13 This is because real-world circumstances often differ from the ones in which a probabilistic estimate (such as an effect size in an RCT) was demonstrated; evidence supporting a plausible mechanism affirms (or not) that the causal relationship is likely to be stable across settings.13 The presence of high-quality and consistent mechanistic evidence greatly increases the external validity of an RCT finding; absence of such evidence should make us question RCT findings.3 Public health interventions involve two kinds of mechanisms: the upstream causes of disease (for COVID-19 ‘mechanisms involving family structures and interaction patterns, occupational behaviour, urban density, housing occupation and overcrowding, workplace and retail environment structures and organisation, as well as local social, economic and cultural variation’—Aronson JK, page 6882 and the ‘causes’ through which an interventions might act (eg, attitudes, beliefs, capability, personal resources).2

    View this table:
    Table 3

    Bradford-Hill indicators of causality, showing the importance of both probabilistic evidence (eg, from RCTs) and mechanistic evidence (eg, from laboratory studies)

    Mechanistic evidence is inherently explanatory. It helps address the question ‘how might this effect be produced?’. In the early months of the pandemic, the grammar of our research was prematurely fixed and PICO-constrained (eg, ‘what is the effect size of masks in preventing respiratory infections?’). This narrowed our focus onto probabilistic studies of interventions (‘masks-on’), comparisons (‘masks-off’) and predefined outcomes (infections in the wearer) expressed in statistical terms (see masking example in box 1 18–28). We suppressed our scientific imagination. We failed sufficiently to wonder at the novelty of the disease and the significance of its unique patterns of spread such as super-spreading events, overdispersion, transmission in the absence of symptoms, and indoor predominance—all of which should have raised mechanistic hypotheses about a predominantly airborne mechanism of transmission.29

    Box 1

    Public masking—‘naked statistics’ or evidential pluralism?

    Mortality rates differed dramatically between Asian countries which introduced public masking very early in the pandemic and Western countries which did not.18 But was this relationship causal?

    The ‘naked statistics view of evidence-based medicine (EBM)’12 would answer this question exclusively from randomised controlled trial (RCT) evidence, rejecting all other study designs as less trustworthy. The DANMASK researchers, for example, randomised 6024 people to being advised to wear masks outside the home or not.19 The primary outcome (infection with SARS-CoV-2) occurred in 42 people (1.8%) of the intervention arm and 53 (2.1%) of the control arm—a difference that was not statistically significant. This finding was described by some in the EBM community thus: ‘now that we have properly rigorous scientific research we can rely on, the evidence shows that wearing masks in the community does not significantly reduce the rates of infection’.20

    DANMASK had numerous design flaws21: it was underpowered; it focused only on whether masks protect the wearer rather than whether they reduce overall community transmission; it occurred at a time of extremely low incidence of COVID-19; it had an intervention period of only 1 month; and—as the authors themselves observed19—the 95% CIs were compatible with a 46% reduction to a 23% increase in infection with masking (ie, it was inconclusive rather than negative). An editorial in the Cochrane Database of Systematic Reviews encouraged policy-makers to adopt the precautionary principle and ‘act on incomplete evidence’ for several reasons: the urgency of the situation (and the risks of doing nothing), the multiple complexities and complex chains of causation for this particular behavioural intervention, and the potential benefits, over time, of even a non-statistically significant reduction in transmission rate.22

    An EBM+ approach would consider the DANMASK trial and other RCTs (such as a large community masking study in Bangladesh villages23 on their merits, but also take account of studies which throw light on the mechanism by which masking might work—including.

    • Laboratory, real-world case studies and mathematical modelling studies supporting an airborne route of transmission.29

    • Engineering studies which established the filtration properties of different kinds of masks.24 25

    • Studies on the psychological, sociocultural and wider structural causes of whether people actually wear masks.26

    • Studies confirming that masking does not appear to have other unintended psychosocial consequences such as risk compensation27 or increased face touching.28

    When the pandemic began, we immediately sought ‘robust’ evidence from new RCTs.30 But overapplication of this heuristic led to narrowly focused reviews,31 excluded experts from other disciplines, and closed our minds to the mechanistic explanations we could have built from well-conducted laboratory studies,32 animal studies,33 modelling studies,34 engineering studies35 or careful analysis of real-world events36—not to mention prepandemic studies on the transmission of comparable respiratory viruses.37

    Dynamic mechanisms: complex interventions in complex systems

    A complex intervention (eg, facial protection with masks or respirators) consists of multiple components acting independently and inter-dependently at multiple levels (eg, material artefact, person, group, organisation, system), such that it is difficult to identify the ‘active ingredient’.38 Facial protection can be home-made or produced to formal technical standards; humans may agree—or not—to mask; masking may be socially expected, organisationally required or legally mandated; the threshold case rate for introducing or reintroducing masking may vary; global supply may be unreliable; masking is often paired with other interventions, introducing confounders.

    A complex system—such as the one for delivering public health (which includes government, health service providers and the public)—is characterised by a number of features:

    • Emergence (the system evolves over time, in ways that cannot be fully anticipated).

    • Adaptation (the system adjusts to accommodate change).

    • Feedback (either positive, where change in one factor reinforces another factor, or negative, where change in one factor diminishes another).

    • Self-organisation (eg, a public health department adapts a national programme to suit local constraints and priorities).39

    Particular outcomes emerge as a result of multiple interactions in the system.

    Whereas studies of drug and vaccine efficacy can be interpreted within the traditional biomedical paradigm (table 4, column 2),40 findings from such trials may not map to complex, real-world settings for multiple social and behavioural reasons. Studies how complex interventions generate outcomes in complex systems require a new paradigm (table 4, column 3) with designs that can capture dynamic change, accommodate non-linearity and embrace uncertainty.41

    View this table:
    Table 4

    Traditional biomedical versus complex systems paradigms (reproduced and adapted under CC-BY-4.0 licence from Greenhalgh and Papoutsi)41

    One approach to researching these generative mechanisms is the in-depth case study, a mixed-method design in which data collection, interpretation and synthesis are oriented to producing a rich description of a phenomenon in context.42 Hypotheses are explored and tested by constructing a plausible story (narrative synthesis) in which all data—qualitative and quantitative—are accounted for. Local public health interventions, even when part of national policy, are typically path-dependent (shaped and constrained by what has happened here previously) and emergent (they grow organically over time); they tend to be implemented and improved through testing and iteration in the real world.43 As Ogilvie et al 43 have pointed out, this pragmatic approach is not ‘weaker’ than an RCT but complementary to it (figure 2). Observations from real-world case studies inform hypothesis-driven questions about the impact of particular interventions under controlled conditions, which can be tested in RCTs. Conversely, RCTs may produce important findings which inform development and real-world testing of an intervention. These authors contrast the ‘brick wall’ of a meta-analysis (in which every primary study contributes a similar ‘brick’) with the ‘dry stone wall’ of a mixed-method narrative review44 (in which different kinds of primary evidence are combined to illuminate a problem from multiple angles).45

    Figure 2
    Figure 2

    Combining controlled trials with evaluation of real-world action via case studies. (Reproduced and adapted under CC-BY-4.0 Licence from Ogilvie et al 43). RCT, randomised controlled trial.

    Generative mechanisms in complex systems can also be studied quantitatively using modelling. Models (simplified versions of reality) can be used for a variety of purposes—to describe epidemic growth, estimate the likely impacts of interventions, simulate airflow inside buildings, and synthesise data from multiple sources in a manner analogous to systematic review and meta-analysis.46 Model projections can be useful for presenting plausible counterfactuals, and for quantifying the impact of uncertainty. Policy-makers’ use of modelling studies permit considering multiple ‘what if’ scenarios and sensitivity assessment as policy options develop. For communicable disease outbreaks, models can capture the non-independence of events that are the defining attribute of these diseases, and allow quantification of indirect effects on individuals who do not receive a given intervention46—for example creation of herd effects via immunisation.

    As with all research designs, real-world case studies42 47 and modelling46 can be done well or badly, and the findings of such studies can be more or less relevant to a policy decision that is being contemplated in a different context or setting.

    Mechanisms in the physical world: standards as evidence

    Engineers use knowledge of the natural world to craft designs which benefit and improve society. That design process takes account of physical and technical properties and capabilities along with social needs and assessment of impact. Central to the science of engineering is a high reliance on standards, which are continuously developed to improve on the state of the art.48 Indeed, quality in engineering evidence is defined in terms of relevant accepted standards and codes (eg, on indoor air quality controls for healthcare facilities).49

    New engineering problems are addressed largely by creatively applying and actively improving existing standards and models. Designing medical equipment such as ventilators or diagnostic imaging systems does not begin de novo but with a standard selection of component products built and tested to meet established standards. Such devices are not iteratively constructed and tested to confirm success. Most aspects are determined analytically using numerical models and design tools. Designs are constrained to meet established national standards that dictate performance and safety requirements. RCTs have no place in such design work.

    Engineering knowledge rarely generates a single ‘truth’. Rather, it tends to produce a range of options which vary in how well they meet social needs and satisfy design and resource constraints. There is usually more than one way of designing a medical device to meet safety and performance standards. The success of the engineering method is illustrated by the rarity of harmful failures of engineered systems. When rare failures occur, there follows a detailed investigation of causes, and application of that knowledge to future designs.50

    Building engineering contributes crucial evidence on how to protect people from airborne pathogens, since the use and impact of personal protective equipment is influenced by (among other things) the quality of indoor air.35 Furthermore, the mechanisms by which filtering facepiece respirators work are long-established and well-understood. Robust certification systems, standards and workplace usage protocols for respirators currently minimise exposure to occupational hazards for millions of workers worldwide. Nobody would propose an RCT comparing these products with less effective protection to ‘prove’ their value in protecting against chemical contamination in a lead smelter—yet such studies are currently being proposed for determining whether respirators protect against biological hazards in healthcare environments.

    Human mechanisms: social and behavioural evidence

    The human (or social) sciences seek to explain why people act in the way they do, taking account of wider influences such as cultural norms or economic constraints. The Medical Research Council’s framework on developing and evaluating complex interventions emphasises the need for developing a programme theory that takes account of the many influences on human (and organisational and system) behaviour.39

    One such framework is the capability-opportunity-motivation-behaviour model,51 whose lead developer, Professor Susan Michie, was an adviser to UK’s Scientific Advisory Group on Emergencies. In relation to wearing masks or respirators, people may not be physically able to do so, or they may lack key knowledge (eg, they may wrongly consider themselves at low risk, wrongly believe that cloth masks, surgical masks and respirators are equally effective, or underestimate the importance of a good fit); because they lack motivation (including conscious beliefs and subconscious emotional reactions), or because they lack opportunity (eg, their employer bans masks or social pressures convey negative norms).

    Other frameworks, such as those based on social ecology theory, place greater emphasis on the multiple layers of influence which shape and constrain human behaviour (including individual, interpersonal, organisational, community and public policy).52

    Without a careful exploration of these wider influences, experimental studies of complex interventions may produce misleading findings.

    Conclusion

    The pandemic brought complex and fast-changing problems. We have argued that EBM’s traditional methods and quality standards, which favour probabilistic evidence from clinical trials, must be extended to place greater emphasis on other sources of evidence. We are not, however, arguing for an anything-goes approach to evidence or what one reviewer of an earlier draft of this paper called ‘a throwback to the 1970s’—a time before much of the rigorous scholarship of EBM had been built.

    An urgent debate is needed within the EBM/evidence-based healthcare community on how, in what circumstances and to what extent EBM should evolve into EBM+. We hope to contribute to that debate with further articles on how an EBM +approach can enhance and extend EBM’s important contribution to pandemic science.

    Ethics statements

    Patient consent for publication

    Ethics approval

    Not applicable.

    Acknowledgments

    Thanks to Kevin Hedges, Leyla Asadi, and Simon Smith and a colleague who preferred to remain anonymous for helpful comments on previous versions of this paper.

    References

    1. ↵
      2. Tresker S
      . Treatment Effectiveness and the Russo–Williamson Thesis, EBM+, and Bradford Hill’s Viewpoints. Int Stud Philos Sci 2022;33:1–28.doi:10.1080/02698595.2022.2054396
      OpenUrl
    2. ↵
      2. Aronson JK ,
      3. Auker-Howlett D ,
      4. Ghiara V , et al
      . The use of mechanistic Reasoning in assessing coronavirus interventions. J Eval Clin Pract 2021;27:684–93.doi:10.1111/jep.13438 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32666676
      OpenUrlPubMed
    3. ↵
      2. Parkkinen V-P ,
      3. Wallmann C ,
      4. Wilde M
      . Evaluating evidence of mechanisms in medicine: principles and procedures. Springer Nature, 2018.
    4. ↵
      2. Murad MH ,
      3. Asi N ,
      4. Alsawas M , et al
      . New evidence pyramid. Evid Based Med 2016;21:125–7.doi:10.1136/ebmed-2016-110401 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27339128
      OpenUrlPubMed
    5. ↵
      2. Guyatt G ,
      3. Cairns J ,
      4. Churchill D , et al
      . Evidence-Based medicine. A new approach to teaching the practice of medicine. JAMA 1992;268:2420–5.doi:10.1001/jama.1992.03490170092032 pmid:http://www.ncbi.nlm.nih.gov/pubmed/1404801
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      2. Guyatt GH ,
      3. Oxman AD ,
      4. Vist GE , et al
      . GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6.doi:10.1136/bmj.39489.470347.AD pmid:http://www.ncbi.nlm.nih.gov/pubmed/18436948
      OpenUrlFREE Full Text
    7. ↵
      2. Howick J ,
      3. Chalmers I ,
      4. Glasziou P
      . Explanation of the 2011 Oxford centre for evidence-based medicine (OCEBM) levels of evidence (background document) (2011) Oxford Centre for Evidence-Based Medicine; 2018. http://www. cebm. net
    8. ↵
      1. RECOVERY Collaborative Group,
      2. Horby P ,
      3. Lim WS , et al
      . Dexamethasone in hospitalized patients with Covid-19. N Engl J Med 2021;384:693–704.doi:10.1056/NEJMoa2021436 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32678530
      OpenUrlCrossRefPubMed
    9. ↵
      2. Butler CC ,
      3. Dorward J ,
      4. Yu L-M , et al
      . Azithromycin for community treatment of suspected COVID-19 in people at increased risk of an adverse clinical course in the UK (PRINCIPLE): a randomised, controlled, open-label, adaptive platform trial. Lancet 2021;397:1063–74.doi:10.1016/S0140-6736(21)00461-X pmid:http://www.ncbi.nlm.nih.gov/pubmed/33676597
      OpenUrlCrossRefPubMed
    10. ↵
      1. PANORAMIC trial group
      . PANORAMIC: platform adaptive trial of novel aniviRals for eArly treatMent of COVID-19 in the Commmunity Oxford: Nuffield department of primary care health sciences. Available: https://www.panoramictrial.org/for-healthcare-professionals/documents 2022 [Accessed 19 Jun 2022].
    11. ↵
      2. Khalil H ,
      3. Tamara L ,
      4. Rada G , et al
      . Challenges of evidence synthesis during the 2020 COVID pandemic: a scoping review. J Clin Epidemiol 2022;142:10–18.doi:10.1016/j.jclinepi.2021.10.017 pmid:http://www.ncbi.nlm.nih.gov/pubmed/34718121
      OpenUrlPubMed
    12. ↵
      2. Clarke B ,
      3. Gillies D ,
      4. Illari P , et al
      . Mechanisms and the evidence hierarchy. Topoi 2014;33:339–60.doi:10.1007/s11245-013-9220-9
      OpenUrl
    13. ↵
      2. Williamson J
      . Establishing causal claims in medicine. International Studies in the Philosophy of Science 2019;32:33–61.doi:10.1080/02698595.2019.1630927
      OpenUrl
    14. ↵
      2. Williamson J
      . Establishing the teratogenicity of Zika and evaluating causal criteria. Synthese 2021;198:2505–18.doi:10.1007/s11229-018-1866-9
      OpenUrl
    15. ↵
      2. HILL AB ,
      3. Bradford Hill A
      . The environment and disease: association or causation? Proc R Soc Med 1965;58:295–300.doi:10.1177/003591576505800503 pmid:http://www.ncbi.nlm.nih.gov/pubmed/14283879
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      2. Russo F ,
      3. Williamson J
      . Interpreting causality in the health sciences. International Studies in the Philosophy of Science 2007;21:157–70.doi:10.1080/02698590701498084
      OpenUrlCrossRef
    17. ↵
      2. Fedak KM ,
      3. Bernal A ,
      4. Capshaw ZA , et al
      . Applying the Bradford Hill criteria in the 21st century: how data integration has changed causal inference in molecular epidemiology. Emerg Themes Epidemiol 2015;12:1–9.doi:10.1186/s12982-015-0037-4
      OpenUrlCrossRefPubMed
    18. ↵
      2. Leffler CT ,
      3. Ing E ,
      4. Lykins JD , et al
      . Association of country-wide coronavirus mortality with demographics, testing, lockdowns, and public wearing of masks. Am J Trop Med Hyg 2020;103:2400–11.doi:10.4269/ajtmh.20-1015 pmid:http://www.ncbi.nlm.nih.gov/pubmed/33124541
      OpenUrlCrossRefPubMed
    19. ↵
      2. Bundgaard H ,
      3. Bundgaard JS ,
      4. Raaschou-Pedersen DET , et al
      . Effectiveness of Adding a Mask Recommendation to Other Public Health Measures to Prevent SARS-CoV-2 Infection in Danish Mask Wearers : A Randomized Controlled Trial. Ann Intern Med 2021;174:335–43.doi:10.7326/M20-6817 pmid:http://www.ncbi.nlm.nih.gov/pubmed/33205991
      OpenUrlCrossRefPubMed
    20. ↵
      2. Heneghan C ,
      3. Jefferson T
      . Landmark Danish study finds no significant effect for facemask wearers. The Spectator 2020.
    21. ↵
      2. Frieden TR ,
      3. Cash-Goldwasser S
      . Of masks and methods. Ann Intern Med 2021;174:421–2.doi:10.7326/M20-7499 pmid:http://www.ncbi.nlm.nih.gov/pubmed/33205992
      OpenUrlPubMed
    22. ↵
      2. Soares-Weiser K ,
      3. Lasserson T ,
      4. Jorgensen KJ , et al
      . Policy makers must act on incomplete evidence in responding to COVID-19. Cochrane Database Syst Rev 2020;11:ED000149.doi:10.1002/14651858.ED000149 pmid:http://www.ncbi.nlm.nih.gov/pubmed/33215686
      OpenUrlPubMed
    23. ↵
      2. Abaluck J ,
      3. Kwong LH ,
      4. Styczynski A , et al
      . Impact of community masking on COVID-19: a cluster-randomized trial in Bangladesh. Science 2022;375:eabi9069.doi:10.1126/science.abi9069 pmid:http://www.ncbi.nlm.nih.gov/pubmed/34855513
      OpenUrlCrossRefPubMed
    24. ↵
      2. Konda A ,
      3. Prakash A ,
      4. Moss GA
      . Aerosol filtration efficiency of common fabrics used in respiratory cloth masks. ACS Nano 2020.
    25. ↵
      2. Hill WC ,
      3. Hull MS ,
      4. MacCuspie RI
      . Testing of commercial masks and respirators and cotton mask insert materials using SARS-CoV-2 virion-sized particulates: comparison of ideal aerosol filtration efficiency versus fitted filtration efficiency. Nano Lett 2020;20:7642–7.doi:10.1021/acs.nanolett.0c03182 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32986441
      OpenUrlPubMed
    26. ↵
      2. van der Westhuizen H-M ,
      3. Kotze K ,
      4. Tonkin-Crine S , et al
      . Face coverings for COVID-19: from medical intervention to social practice. BMJ 2020;370:e3020.doi:10.1136/bmj.m3021 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32816815
      OpenUrlPubMed
    27. ↵
      2. Mantzari E ,
      3. Rubin GJ ,
      4. Marteau TM
      . Is risk compensation threatening public health in the covid-19 pandemic? BMJ 2020;370:m2913.doi:10.1136/bmj.m2913 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32713835
      OpenUrlFREE Full Text
    28. ↵
      2. Chen Y-J ,
      3. Qin G ,
      4. Chen J , et al
      . Comparison of Face-Touching behaviors before and during the coronavirus disease 2019 pandemic. JAMA Netw Open 2020;3:e2016924–e24.doi:10.1001/jamanetworkopen.2020.16924 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32725247
      OpenUrlPubMed
    29. ↵
      2. Greenhalgh T ,
      3. Jimenez JL ,
      4. Prather KA , et al
      . Ten scientific reasons in support of airborne transmission of SARS-CoV-2. Lancet 2021;397:1603–5.doi:10.1016/S0140-6736(21)00869-2 pmid:http://www.ncbi.nlm.nih.gov/pubmed/33865497
      OpenUrlCrossRefPubMed
    30. ↵
      2. Djulbegovic B ,
      3. Guyatt G
      . Evidence-Based medicine in times of crisis. J Clin Epidemiol 2020;126:164–6.doi:10.1016/j.jclinepi.2020.07.002 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32659364
      OpenUrlPubMed
    31. ↵
      2. Jefferson T ,
      3. Del Mar CB ,
      4. Dooley L , et al
      . Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev 2020;11:CD006207.doi:10.1002/14651858.CD006207.pub5 pmid:http://www.ncbi.nlm.nih.gov/pubmed/33215698
      OpenUrlCrossRefPubMed
    32. ↵
      2. van Doremalen N ,
      3. Bushmaker T ,
      4. Morris DH , et al
      . Aerosol and surface stability of SARS-CoV-2 as compared with SARS-CoV-1. N Engl J Med 2020;382:1564–7.doi:10.1056/NEJMc2004973 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32182409
      OpenUrlCrossRefPubMed
    33. ↵
      2. Kutter JS ,
      3. de Meulder D ,
      4. Bestebroer TM , et al
      . SARS-CoV and SARS-CoV-2 are transmitted through the air between ferrets over more than one meter distance. Nat Commun 2021;12:1–8.doi:10.1038/s41467-021-21918-6
      OpenUrlCrossRefPubMed
    34. ↵
      2. Chen W ,
      3. Zhang N ,
      4. Wei J , et al
      . Short-Range airborne route dominates exposure of respiratory infection during close contact. Build Environ 2020;176:106859.doi:10.1016/j.buildenv.2020.106859
    35. ↵
      2. Morawska L ,
      3. Allen J ,
      4. Bahnfleth W , et al
      . A paradigm shift to combat indoor respiratory infection. Science 2021;372:689–91.doi:10.1126/science.abg2025 pmid:http://www.ncbi.nlm.nih.gov/pubmed/33986171
      OpenUrlAbstract/FREE Full Text
    36. ↵
      2. Miller SL ,
      3. Nazaroff WW ,
      4. Jimenez JL , et al
      . Transmission of SARS-CoV-2 by inhalation of respiratory aerosol in the Skagit Valley Chorale superspreading event. Indoor Air 2021;31:314–23.doi:10.1111/ina.12751 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32979298
      OpenUrlCrossRefPubMed
    37. ↵
      2. Wang CC ,
      3. Prather KA ,
      4. Sznitman J , et al
      . Airborne transmission of respiratory viruses. Science 2021;373:eabd9149.doi:10.1126/science.abd9149 pmid:http://www.ncbi.nlm.nih.gov/pubmed/34446582
      OpenUrlAbstract/FREE Full Text
    38. ↵
      2. Campbell M ,
      3. Fitzpatrick R ,
      4. Haines A , et al
      . Framework for design and evaluation of complex interventions to improve health. BMJ 2000;321:694–6.doi:10.1136/bmj.321.7262.694 pmid:http://www.ncbi.nlm.nih.gov/pubmed/10987780
      OpenUrlFREE Full Text
    39. ↵
      2. Skivington K ,
      3. Matthews L ,
      4. Simpson SA , et al
      . A new framework for developing and evaluating complex interventions: update of medical Research Council guidance. BMJ 2021;374:n2061.doi:10.1136/bmj.n2061 pmid:http://www.ncbi.nlm.nih.gov/pubmed/34593508
      OpenUrlFREE Full Text
    40. ↵
      2. Borgerson K
      . Valuing evidence: bias and the evidence hierarchy of evidence-based medicine. Perspect Biol Med 2009;52:218–33.doi:10.1353/pbm.0.0086 pmid:http://www.ncbi.nlm.nih.gov/pubmed/19395821
      OpenUrlPubMed
    41. ↵
      2. Greenhalgh T ,
      3. Papoutsi C
      . Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med 2018;16:95.doi:10.1186/s12916-018-1089-4 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29921272
      OpenUrlCrossRefPubMed
    42. ↵
      2. Paparini S ,
      3. Papoutsi C ,
      4. Murdoch J
      . Evaluating complex interventions in context: systematic, meta-narrative review of case study approaches. BMC Medical Research Methodology 2021;21:1–22.
      OpenUrl
    43. ↵
      2. Ogilvie D ,
      3. Adams J ,
      4. Bauman A , et al
      . Using natural experimental studies to guide public health action: turning the evidence-based medicine paradigm on its head. J Epidemiol Community Health 2020;74:203–8.doi:10.1136/jech-2019-213085 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31744848
      OpenUrlAbstract/FREE Full Text
    44. ↵
      2. Greenhalgh T ,
      3. Thorne S ,
      4. Malterud K
      . Time to challenge the spurious hierarchy of systematic over narrative reviews? Eur J Clin Invest 2018;48:e12931.doi:10.1111/eci.12931 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29578574
      OpenUrlCrossRefPubMed
    45. ↵
      2. Ogilvie D ,
      3. Bauman A ,
      4. Foley L , et al
      . Making sense of the evidence in population health intervention research: building a dry stone wall. BMJ Glob Health 2020;5:e004017.doi:10.1136/bmjgh-2020-004017 pmid:http://www.ncbi.nlm.nih.gov/pubmed/33298470
      OpenUrlAbstract/FREE Full Text
    46. ↵
      1. Pandemic Influenza Outbreak Research Modelling Team (Pan-InfORM),
      2. Fisman D
      . Modelling an influenza pandemic: a guide for the perplexed. CMAJ 2009;181:171–3.doi:10.1503/cmaj.090885 pmid:http://www.ncbi.nlm.nih.gov/pubmed/19620267
      OpenUrlFREE Full Text
    47. ↵
      2. Flyvbjerg B
      . Five misunderstandings about case-study research. Qualitative Inquiry 2006;12:219–45.doi:10.1177/1077800405284363
      OpenUrlCrossRefWeb of Science
    48. ↵
      2. Koen BV
      . Toward a definition of the engineering method. European Journal of Engineering Education 1988;13:307–15.doi:10.1080/03043798808939429
      OpenUrl
    49. ↵
      1. American Society of Heating R, and Air-Conditioning Engineers
      . ANSI/ASHRAE/ASHE standard 170-2017, ventilation of health care facilities. Washington, DC ASHRAE; 2022. https://www.ashrae.org/technical-resources/standards-and-guidelines/standards-addenda/ansi-ashrae-ashe-standard-170-2017-ventilation-of-health-care-facilities 2017 [Accessed 17 Feb 2022].
    50. ↵
      2. Englehardt E ,
      3. Werhane PH ,
      4. Newton LH
      . Leadership, engineering and ethical clashes at Boeing. Sci Eng Ethics 2021;27:1–17.doi:10.1007/s11948-021-00285-x
      OpenUrl
    51. ↵
      2. McDonagh LK ,
      3. Saunders JM ,
      4. Cassell J , et al
      . Application of the COM-B model to barriers and facilitators to Chlamydia testing in general practice for young people and primary care practitioners: a systematic review. Implementation Science 2018;13:1–19.doi:10.1186/s13012-018-0821-y
      OpenUrl
    52. ↵
      2. Glanz K ,
      3. Bishop DB
      . The role of behavioral science theory in development and implementation of public health interventions. Annu Rev Public Health 2010;31:399–418.doi:10.1146/annurev.publhealth.012809.103604 pmid:http://www.ncbi.nlm.nih.gov/pubmed/20070207
      OpenUrlCrossRefPubMedWeb of Science

    Footnotes

    • Twitter @trishgreenhalgh, @sameo416

    • Contributors TG wrote the original draft. All authors contributed to planning, discussing, drafting and refining the manuscript.

    • Funding TG’s work is partly supported by the NIHR Oxford Biomedical Research Centre (BRC-1215-20008). DC received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. MO received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. RM is funded by the National Health and Medical Research Council Principal Research Fellowship (grant number 1137582). DF is supported by Canadian Institutes for Health Research (2019 COVID-19 rapid researching funding OV4-170360).

    • Competing interests TG is a member of Independent SAGE and an unpaid adviser to Balvi, a philanthropic fund. RM in the last 5 years has received funding from Sanofi and Seqirus for investigator driven research on influenza vaccines. She has been on advisory boards or consulted for COVID-19 vaccines for Seqirus, Janssen and Astra-Zeneca. She has received funding for an industry-linkage grant scheme from mask manufacturer Detmold and consulted for mask manufacturer Ascend Performance Technologies. DF has served on advisory boards related to influenza and SARS-CoV-2 vaccines for Seqirus, Pfizer, Astrazeneca and Sanofi-Pasteur Vaccines, and has served as a legal expert on issues related to COVID-19 epidemiology for the Elementary Teachers Federation of Ontario and the Registered Nurses Association of Ontario. DJC worked as a Clinical Specialist for the Canadian PPE Manufacturers Association between November 2021 and February 2022. She now serves as an executive board member with the Coalition of Healthcare Associated Infection Reduction (CHAIR), a volunteer, not-for-profit group, and is a co-founder of the volunteer grassroots group Community Access to Ventilation Information (CAVI) supporting public library CO2 monitor loaning programmes.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    Linked Articles