UMMS Affiliation

Center for Outcomes Research; Department of Surgery

Publication Date


Document Type



Clinical Epidemiology | Epidemiology | Health Services Administration | Health Services Research | Statistics and Probability


The ability to identify patients who are likely to have an adverse outcome is an essential component of good clinical care. Therefore, predictive risk stratification models play an important role in clinical decision making. Determining whether a given predictive model is suitable for clinical use usually involves evaluating the model's performance on large patient datasets using standard statistical measures of success (e.g., accuracy, discriminatory ability). However, as these metrics correspond to averages over patients who have a range of different characteristics, it is difficult to discern whether an individual prediction on a given patient should be trusted using these measures alone. In this paper, we introduce a new method for identifying patient subgroups where a predictive model is expected to be poor, thereby highlighting when a given prediction is misleading and should not be trusted. The resulting "unreliability score" can be computed for any clinical risk model and is suitable in the setting of large class imbalance, a situation often encountered in healthcare settings. Using data from more than 40,000 patients in the Global Registry of Acute Coronary Events (GRACE), we demonstrate that patients with high unreliability scores form a subgroup in which the predictive model has both decreased accuracy and decreased discriminatory ability.


Predictive markers, Prognostic markers, Risk factors

Rights and Permissions

Copyright © The Author(s) 2020. Open Access. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit

DOI of Published Version



Myers PD, Ng K, Severson K, Kartoun U, Dai W, Huang W, Anderson FA, Stultz CM. Identifying unreliable predictions in clinical risk models. NPJ Digit Med. 2020 Jan 23;3:8. doi: 10.1038/s41746-019-0209-7. PMID: 31993506; PMCID: PMC6978376. Link to article on publisher's site

Journal/Book/Conference Title

NPJ digital medicine

Related Resources

Link to Article in PubMed

PubMed ID


Creative Commons License

Creative Commons Attribution 4.0 License
This work is licensed under a Creative Commons Attribution 4.0 License.