Introduction There is a proliferation of predictive algorithm development studies but a need for real- world effectiveness evaluation. Current predictive algorithm reporting guidelines and frameworks focus on technical performance. We propose the development of a Predictive algorithm Impact Evaluation (PIE) framework to address the impact of predictive algorithms in the clinical setting while prioritizing patient-centred outcomes and shared-decision making.

Methods We used a flexible approach to synthesize knowledge related to the evaluation of predictive algorithms and enable exploration and mapping of key evaluation criteria. The following steps were taken: 1) defining key concepts, 2) conducting a preliminary exploration of the literature, and 3) extracting and synthesising the data into themes. The framework was developed iteratively by a multi- disciplinary group of researchers (expertise in artificial intelligence in healthcare, implementation science, learning health systems, and patient-oriented research) and our Patient and Family Advisory Committee.

Results There are limited real-world evaluative studies, but a wide range of scope, approaches, and outcomes. Preliminary domains include: uses (e.g., risk stratification, risk communication); target population (e.g., patient-centred, health systems, clinical or clinician-centred); development stage (pilot, scaling); and, cross-cutting themes (e.g., equity, data privacy, bioethics, transparency).

Discussion Concepts of implementation science, learning health systems, patient-centred uses are not well-established in impact or real-world evaluative studies of predictive algorithms. The iterative, consensus driven nature of the process and involvement of diverse stakeholders, including patients and families, will help ensure a comprehensive and inclusive evaluation framework.

Conclusion The imperative for evaluating predictive algorithms in clinical practice is rooted in commitment to providing safe, effective, and patient-centered healthcare. A comprehensive evaluation framework can help ensure that these algorithms contribute positively to patient and clinical decision- making while upholding ethical standards and promoting equity in healthcare.