A clash of cultures

Although the explosive growth in data, computational power, analytic methods and communication enables the potential for individualised health and learning healthcare systems, thus far, the field has been characterised by a clash of cultures, largely driven by the differences in perspectives that exist between the statistical, computer science/engineering and medical communities. The most basic simplification of the cultural divide can be explained in terms of emphasis on prediction or inference. The statistical community is often characterised by the data modelling culture where long-standing focus has been on inference through creation and fitting of problem-specific models based on prior knowledge and assumptions. In contrast, the machine learning community is often characterised by the algorithmic culture, where emphasis is on high predictive performance through the use of general-purpose learning algorithms that find patterns in data with few user-specified assumptions, often leading to models with impressive predictive ability but minimal interpretability and, thus, ‘black box’ predictions.8 Because of their high performance, there has already been widespread adoption of machine learning methods in areas such as weather forecasting, speech translation, fraud detection, spam filtering and self-driving cars.9 However, in medicine, where both high predictive performance and model transparency are essential for clinical care, challenges remain before these algorithms can reach their full impact to improve human health. With increasing emphasis on individualising healthcare to provide the best care at the right time and at lower costs through learning healthcare systems,10 there is a great need to bring about synergy between the statistical, computer science/machine learning and medical perspectives.

As shown in figure 1, there are currently numerous barriers to the realisation of learning healthcare systems, where data inform clinical practice, which generates additional data and new knowledge, further informing patient care. While data have the potential to inform predictions, difficulties exist when working with missing data and numerous predictors, as well as nonlinearities and interactions. Although sophisticated statistical and machine learning methods have been developed to address these challenges, these predictions are often made in ways that are difficult to interpret. The lack of transparency of ’black boxes’ hinders the generation of knowledge from these predictions and impede effective clinical translation.11–15

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

The synergistic Learning PIC framework for evidence-based learning healthcare systems: In a learning healthcare system, data would inform predictions that would allow for the generation of new knowledge to inform patient care. Then, data collected from patient care can be used to inform future decision-making in a cycle of continuous learning. Nevertheless, as shown in the red boxes, there are numerous challenges hindering the realisation of learning healthcare systems. Our synergistic Learning PIC framework is designed to address each of these barriers. 

Synergy to enable the common goal of learning

Although statistics and machine learning vary in their history, notation, emphasis and culture, both are essentially ‘the science of learning from data’.16 As the fields advance, there is a growing blurring between the distinctions between the disciplines. To emphasise the need to draw on the strengths of multiple domains and to focus on the common goal of learning from data, we use the term synergistic learning. In synergistic learning, scientific/clinical knowledge imposes structure to the learning models/algorithms in order to draw on the strengths of each discipline and to enable the realisation of individualised health and continuously learning healthcare systems. Specifically, to enable the effective delivery of evidence-based medicine, synergistic learning is learning that:

• synergistically incorporates the strengths of traditional statistical approaches, as well as the advantages of machine learning methods,

• is structured based on ultimate project objectives, expert domain knowledge and methodological requirements, and

• is scientifically/clinically motivated.

Key features of the Learning PIC

Here, we outline the key features of the Learning PIC framework. Table 1 provides example approaches and use cases for each aspect of the Learning PIC.

Prediction

The current advancements in biomedical and information technologies increase the potential to substantially improve clinical decision-making with large amounts of data. Furthermore, methods from machine learning and statistical prediction are increasing our capacity to learn from a wide variety of data sources.17 18 Nevertheless, the majority of approaches to clinical-support tools only employ a small fraction of the available data. Specifically, even when variables are repeatedly measured (eg, multiple laboratory blood tests on the same individual over time), oftentimes the patient’s risk score is based only on the last available measurement rather than the full history of the measurements. Additionally, to date, most risk models are based on regression approaches.19 For example, the Cox regression model was used to develop the Framingham Risk Score and the logistic regression model was used to develop the 30-day mortality risk prediction for patients with ST-elevation myocardial infarction.20 Nevertheless, these regression strategies suffer from a number of limitations. For instance, these methods can typically handle only a small number of predictors, disregard potential interactions with time and assume constant predictor effects throughout their entire range.

The challenges not well handled by typical regression modelling strategies include non-linearities, heterogeneity of effects (interactions) and consideration of many potential predictors. The basic assumption of a regression model is that there is a linear relationship between the risk factor and outcome. Although this can be an appropriate approximation for some risk factors, in many cases, predictors have non-linear relationships with the outcome. For example, the risk of death sharply rises with increasing age. In other cases, values both above and below the normal ranges are indicative of high risk (eg, hypoglycaemia, hyperglycaemia, and body mass index for underweight and obese individuals). Furthermore, a variable’s impact on the prediction can depend on the level of another variable (eg, gene–environment interactions and treatment–race interactions). In standard regression approaches, interactions need to be prespecified, requiring the individual developing the model to know a priori to consider the interaction term in the model. In addition, with a large number of potential predictor variables to consider, it can be challenging to determine which to include in the model, and strategies must also be taken to avoid overfitting.

When clinical decision-making requires the consideration of a large number of predictors, as well as interactions and non-linear predictor effects, moving beyond traditional regression approaches offers the potential to improve predictive performance.19 The increasing emergence of large, heterogeneous data sets, such as electronic health records, for risk prediction requires novel tools to support improved clinical decisions to inform evidence-based medicine. Further development of approaches to learn from large amounts of heterogeneous data has potential to accelerate the progress towards individualised health.19 In contrast to traditional regression strategies or parametric approaches that make assumptions about the underlying model, machine learning approaches allow the data to ‘speak for themselves’ and often result in impressive predictive ability.19 21 However, machine learning approaches are sometimes criticised as ‘data hungry’ (meaning that they require large amounts of data) and for their lack of concern for enabling understanding of the relationships between the inputs and outcomes.22 As a result, effective learning healthcare systems require a synergistic approach that draws on the strengths of traditional statistical approaches, as well as the advantages of machine learning methods combined with expert domain knowledge (table 1).

Interpretation/Inference

Despite their high predictive performance, the ability of machine learning algorithms to ‘learn by themselves’ often results in poor interpretability. Additionally, because these algorithms have traditionally been developed and evaluated primarily on predictive performance, moving beyond prediction may not result in valid inferences. For instance, an illustrative example is the following: ‘algorithms might select discolored teeth as a better predictor of lung cancer than self-reported smoking status, which might be correct in some groups but because it is not a causal factor, may be useless in others, such as populations with different dietary patterns or dental care.’23–25 Despite the promise of novel learning algorithms for improving healthcare, there remains enormous concern regarding the lack of transparency of these algorithms and the risk that their predictions may be based on noise rather than on clinically meaningful relationships (ie, ‘overfitting’; see table 1).11–15

The current high predictive ability of these learning algorithms is not sufficient to facilitate their adoption in healthcare. Overall, clinicians have expressed hesitation in integrating these black boxes into their practice without an understanding of how these algorithms are generating their predictions and how to communicate these predictions to their patients. With the goal of not only achieving high predictive performance but also generating an understanding of how this predictive ability is obtained, new methods for peeking into these black boxes are currently in development. To increase interpretability, methods have been developed to determine feature importance (eg, ranking of the feature’s importance to the prediction or performance of the model) and feature effect (eg, visualisation of how changes in a feature impact the prediction).26–29 Additionally, there is growing interest in incorporating causal inference tools to enable causal reasoning within machine learning to allow cause-and-effect relationships to be examined and questioned.23–25 For instance, a clinician may be interested in determining, prior to the initiation of treatment, what the patient’s expected outcome would be under treatment A versus treatment B. Learning algorithms can help clinicians reason about the best course of action tailored to the specific patient under consideration and provide transparency behind their predictions to help facilitate evidence-based individualised health. However, expanding the ability of these algorithms to enable interpretability and inference must also be considered in the context of clinical communication.

Communication

For learning algorithms to gain clinical utility, strategies must be developed not only for interpretability/inference but also for communication of these results to clinicians and patients. It is essential to provide communication tools that allow clinicians to gain understanding and trust in these predictions, as well as to gain confidence in their ability to explain these predictions to their patients. Developing visualisations that are easy to interpret and based on familiar ways clinicians understand algorithms or results can help in translating these predictive tools into clinical practice. Focus groups with clinicians and patients can provide important feedback on how these tools can be optimised to help inform clinical decision-making in ways that are clinician and patient friendly and tailored to the unique needs/preferences of the end user. Furthermore, discussions regarding the ethical considerations of integrating research and clinical practice are necessary (table 1). A possible guiding ethical framework is one that is founded on a ‘moral priority (of) learning’, specifically with regard to the obligation of health professionals, health care institutions and patients to engage in ‘ongoing learning that is integrated with … health care’.30 In order to achieve effective learning healthcare systems with novel learning algorithms, multidisciplinary teams, ranging from engineers and computer scientists to clinicians and patients, will be increasingly important to develop impactful tools that empower evidence-based medicine to learn from data and to continuously improve.