UMMS Affiliation

Department of Medicine

Publication Date


Document Type



Artificial Intelligence and Robotics | Critical Care | Data Science | Health Information Technology


Objective: Predictive models for critical events in the intensive care unit (ICU) might help providers anticipate patient deterioration. At the heart of predictive model development lies the ability to accurately label significant events, thereby facilitating the use of machine learning and similar strategies. We conducted this study to establish the validity of an automated system for tagging respiratory and hemodynamic deterioration by comparing automatic tags to tagging by expert reviewers.

Methods: This retrospective cohort study included 72,650 unique patient stays collected from Electronic Medical Records of the University of Massachusetts' eICU. An enriched subgroup of stays was manually tagged by expert reviewers. The tags generated by the reviewers were compared to those generated by an automated system.

RESULTS: The automated system was able to rapidly and efficiently tag the complete database utilizing available clinical data. The overall agreement rate between the automated system and the clinicians for respiratory and hemodynamic deterioration tags was 89.4% and 87.1%, respectively. The automatic system did not add substantial variability beyond that seen among the reviewers.

Conclusions: We demonstrated that a simple rule-based tagging system could provide a rapid and accurate tool for mass tagging of a compound database. These types of tagging systems may replace human reviewers and save considerable resources when trying to create a validated, labeled database used to train artificial intelligence algorithms. The ability to harness the power of artificial intelligence depends on efficient clinical validation of targeted conditions; hence, these systems and the methodology used to validate them are crucial.


Artificial Intelligence, Big Data, Clinical Deterioration, Critical Care, Respiratory Insufficiency

Rights and Permissions

Copyright © 2021 The Korean Society of Medical Informatics. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License ( which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

DOI of Published Version



Jeddah D, Chen O, Lipsky AM, Forgacs A, Celniker G, Lilly CM, Pessach IM. Validation of an Automatic Tagging System for Identifying Respiratory and Hemodynamic Deterioration Events in the Intensive Care Unit. Healthc Inform Res. 2021 Jul;27(3):241-248. doi: 10.4258/hir.2021.27.3.241. Epub 2021 Jul 31. PMID: 34384206; PMCID: PMC8369051. Link to article on publisher's site

Journal/Book/Conference Title

Healthcare informatics research

Related Resources

Link to Article in PubMed

PubMed ID


Creative Commons License

Creative Commons Attribution-Noncommercial 4.0 License
This work is licensed under a Creative Commons Attribution-Noncommercial 4.0 License