Article Text
PDF
Statistics from Altmetric.com
Background
Worldwide, around 800 000 people die each year from suicide,1 which is the leading cause of death in the UK in young adults.2 Prediction modelling studies have attempted to incorporate demographic, clinical and other factors to identify high-risk individuals so that appropriate interventions can be offered.3 4
This approach has a large literature but has not always been judged successful. In one review, all 35 suicide risk prediction studies assessed were classified as having high risk of bias or insufficient diagnostic accuracy, based on targets of 80% sensitivity and 50% specificity.5 Others have written of a performance ‘glass ceiling’ in suicide prediction, and even that ‘risk categorization of individual patients has no role to play in preventing the suicide of psychiatric inpatients’.6
In spite of the mortality data quoted previously, in most populations, risk of death from suicide is low, usually below 1%. This has led to claims about performance of prediction models that appear counterintuitive, such as ‘suicide prediction models produce accurate overall classification models, but their accuracy of predicting a future event is near 0’, while noting that even in high-risk populations, the positive predictive value (PPV) of many prediction rules may be less than 1%.3
We describe two principal reasons why the nature of developing clinical prediction rules in low prevalence scenarios will almost invariably result in concerns about performance, on standards by which this is usually judged. Although we focus primarily on suicide prediction, these points apply more generally to other low prevalence clinical areas, some examples of which are also discussed.
The effect of low prevalence on sample size
Low prevalence can have a prohibitive impact on sample size requirements because of the need to observe enough outcome events for model development, as also noted in diagnostic evaluation studies.7 For example, consider a single risk factor that …
Footnotes
Contributors Both authors contributed to the writing of the manuscript.
Funding This research was supported by the National Institute for Health Research (NIHR) Applied Research Collaboration Oxford and Thames Valley at Oxford Health NHS Foundation Trust. TRF also receives funding from the NIHR Community Healthcare MedTech and In Vitro Diagnostics Co-operative at Oxford Health NHS Foundation Trust (MIC-2016-018).
Disclaimer The views expressed in this publication are those of the authors and not necessarily those of the National Health Service, the NIHR or the Department of Health and Social Care. SF is additionally funded by the Wellcome Trust (202836/Z/16/Z).
Competing interests Both authors have previously published research relating to the development and validation of risk prediction tools, including a suicide risk prediction tool for individuals with severe mental illness (OxMIS).
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.