Real-time machine learning model to predict short-term mortality in critically ill patients: development and international validation.

BACKGROUND: A real-time model for predicting short-term mortality in critically ill patients is needed to identify patients at imminent risk. However, the performance of the model needs to be validated in various clinical settings and ethnicities before its clinical application. In this study, we aim to develop an ensemble machine learning model using routinely measured clinical variables at a single academic institution in South Korea. METHODS: We developed an ensemble model using deep learning and light gradient boosting machine models. Internal validation was performed using the last two years of the internal cohort dataset, collected from a single academic hospital in South Korea between 2007 and 2021. External validation was performed using the full Medical Information Mart for Intensive Care (MIMIC), eICU-Collaborative Research Database (eICU-CRD), and Amsterdam University Medical Center database (AmsterdamUMCdb) data. The area under the receiver operating characteristic curve (AUROC) was calculated and compared to that for the National Early Warning Score (NEWS). RESULTS: The developed model (iMORS) demonstrated high predictive performance with an internal AUROC of 0.964 (95% confidence interval [CI] 0.963-0.965) and external AUROCs of 0.890 (95% CI 0.889-0.891) for MIMIC, 0.886 (95% CI 0.885-0.887) for eICU-CRD, and 0.870 (95% CI 0.868-0.873) for AmsterdamUMCdb. The model outperformed the NEWS with higher AUROCs in the internal and external validation (0.866 for the internal, 0.746 for MIMIC, 0.798 for eICU-CRD, and 0.819 for AmsterdamUMCdb; p < 0.001). CONCLUSIONS: Our real-time machine learning model to predict short-term mortality in critically ill patients showed excellent performance in both internal and external validations. This model could be a useful decision-support tool in the intensive care units to assist clinicians.

External Validation of Deep Learning-Based Cardiac Arrest Risk Management System for Predicting In-Hospital Cardiac Arrest in Patients Admitted to General Wards Based on Rapid Response System Operating and Nonoperating Periods: A Single-Center Study.

OBJECTIVES: The limitations of current early warning scores have prompted the development of deep learning-based systems, such as deep learning-based cardiac arrest risk management systems (DeepCARS). Unfortunately, in South Korea, only two institutions operate 24-hour Rapid Response System (RRS), whereas most hospitals have part-time or no RRS coverage at all. This study validated the predictive performance of DeepCARS during RRS operation and nonoperation periods and explored its potential beyond RRS operating hours. DESIGN: Retrospective cohort study. SETTING: In this 1-year retrospective study conducted at Yonsei University Health System Severance Hospital in South Korea, DeepCARS was compared with conventional early warning systems for predicting in-hospital cardiac arrest (IHCA). The study focused on adult patients admitted to the general ward, with the primary outcome being IHCA-prediction performance within 24 hours of the alarm. PATIENTS: We analyzed the data records of adult patients admitted to a general ward from September 1, 2019, to August 31, 2020. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Performance evaluation was conducted separately for the operational and nonoperational periods of the RRS, using the area under the receiver operating characteristic curve (AUROC) as the metric. DeepCARS demonstrated a superior AUROC as compared with the Modified Early Warning Score (MEWS) and the National Early Warning Score (NEWS), both during RRS operating and nonoperating hours. Although the MEWS and NEWS exhibited varying performance across the two periods, DeepCARS showed consistent performance. CONCLUSIONS: The accuracy and efficiency for predicting IHCA of DeepCARS were superior to that of conventional methods, regardless of whether the RRS was in operation. These findings emphasize that DeepCARS is an effective screening tool suitable for hospitals with full-time RRS, part-time RRS, and even those without any RRS.

Development and Validation of Deep-Learning-Based Sepsis and Septic Shock Early Prediction System (DeepSEPS) Using Real-World ICU Data.

BACKGROUND: Successful sepsis treatment depends on early diagnosis. We aimed to develop and validate a system to predict sepsis and septic shock in real time using deep learning. METHODS: Clinical data were retrospectively collected from electronic medical records (EMRs). Data from 2010 to 2019 were used as development data, and data from 2020 to 2021 were used as validation data. The collected EMRs consisted of eight vital signs, 13 laboratory data points, and three demographic information items. We validated the deep-learning-based sepsis and septic shock early prediction system (DeepSEPS) using the validation datasets and compared our system with other traditional early warning scoring systems, such as the national early warning score, sequential organ failure assessment (SOFA), and quick sequential organ failure assessment. RESULTS: DeepSEPS achieved even higher area under receiver operating characteristic curve (AUROC) values (0.7888 and 0.8494 for sepsis and septic shock, respectively) than SOFA. The prediction performance of traditional scoring systems was enhanced because the early prediction time point was close to the onset time of sepsis; however, the DeepSEPS scoring system consistently outperformed all conventional scoring systems at all time points. Furthermore, at the time of onset of sepsis and septic shock, DeepSEPS showed the highest AUROC (0.9346). CONCLUSIONS: The sepsis and septic shock early warning system developed in this study exhibited a performance that is worth considering when predicting sepsis and septic shock compared to other traditional early warning scoring systems. DeepSEPS showed better performance than existing sepsis prediction programs. This novel real-time system that simultaneously predicts sepsis and septic shock requires further validation.

Early prediction of need for invasive mechanical ventilation in the neonatal intensive care unit using artificial intelligence and electronic health records: a clinical study.

BACKGROUND: Respiratory support is crucial for newborns with underdeveloped lung. The clinical outcomes of patients depend on the clinician's ability to recognize the status underlying the presented symptoms and signs. With the increasing number of high-risk infants, artificial intelligence (AI) should be considered as a tool for personalized neonatal care. Continuous monitoring of vital signs is essential in cardiorespiratory care. In this study, we developed deep learning (DL) prediction models for rapid and accurate detection of mechanical ventilation requirements in neonates using electronic health records (EHR). METHODS: We utilized data from the neonatal intensive care unit in a single center, collected between March 3, 2012, and March 4, 2022, including 1,394 patient records used for model development, consisting of 505 and 889 patients with and without invasive mechanical ventilation (IMV) support, respectively. The proposed model architecture includes feature embedding using feature-wise fully connected (FC) layers, followed by three bidirectional long short-term memory (LSTM) layers. RESULTS: A mean gestational age (GA) was 36.61 ± 3.25 weeks, and the mean birth weight was 2,734.01 ± 784.98 g. The IMV group had lower GA, birth weight, and longer hospitalization duration than the non-IMV group (P < 0.05). Our proposed model, tested on a dataset from March 4, 2019, to March 4, 2022. The mean AUROC of our proposed model for IMV support prediction performance demonstrated 0.861 (95%CI, 0.853-0.869). It is superior to conventional approaches, such as newborn early warning score systems (NEWS), Random Forest, and eXtreme gradient boosting (XGBoost) with 0.611 (95%CI, 0.600-0.622), 0.837 (95%CI, 0.828-0.845), and 0.0.831 (95%CI, 0.821-0.845), respectively. The highest AUPRC value is shown in the proposed model at 0.327 (95%CI, 0.308-0.347). The proposed model performed more accurate predictions as gestational age decreased. Additionally, the model exhibited the lowest alarm rate while maintaining the same sensitivity level. CONCLUSION: Deep learning approaches can help accurately standardize the prediction of invasive mechanical ventilation for neonatal patients and facilitate advanced neonatal care. The results of predictive, recall, and alarm performances of the proposed model outperformed the other models.

Prospective, multicenter validation of the deep learning-based cardiac arrest risk management system for predicting in-hospital cardiac arrest or unplanned intensive care unit transfer in patients admitted to general wards.

BACKGROUND: Retrospective studies have demonstrated that the deep learning-based cardiac arrest risk management system (DeepCARS™) is superior to the conventional methods in predicting in-hospital cardiac arrest (IHCA). This prospective study aimed to investigate the predictive accuracy of the DeepCARS™ for IHCA or unplanned intensive care unit transfer (UIT) among general ward patients, compared with that of conventional methods in real-world practice. METHODS: This prospective, multicenter cohort study was conducted at four teaching hospitals in South Korea. All adult patients admitted to general wards during the 3-month study period were included. The primary outcome was predictive accuracy for the occurrence of IHCA or UIT within 24 h of the alarm being triggered. Area under the receiver operating characteristic curve (AUROC) values were used to compare the DeepCARS™ with the modified early warning score (MEWS), national early warning Score (NEWS), and single-parameter track-and-trigger systems. RESULTS: Among 55,083 patients, the incidence rates of IHCA and UIT were 0.90 and 6.44 per 1,000 admissions, respectively. In terms of the composite outcome, the AUROC for the DeepCARS™ was superior to those for the MEWS and NEWS (0.869 vs. 0.756/0.767). At the same sensitivity level of the cutoff values, the mean alarm counts per day per 1,000 beds were significantly reduced for the DeepCARS™, and the rate of appropriate alarms was higher when using the DeepCARS™ than when using conventional systems. CONCLUSION: The DeepCARS™ predicts IHCA and UIT more accurately and efficiently than conventional methods. Thus, the DeepCARS™ may be an effective screening tool for detecting clinical deterioration in real-world clinical practice. Trial registration This study was registered at ClinicalTrials.gov ( NCT04951973 ) on June 30, 2021.

Multicenter validation of a deep-learning-based pediatric early-warning system for prediction of deterioration events.

BACKGROUND: Early recognition of deterioration events is crucial to improve clinical outcomes. For this purpose, we developed a deep-learning-based pediatric early-warning system (pDEWS) and aimed to validate its clinical performance. METHODS: This is a retrospective multicenter cohort study including five tertiary-care academic children's hospitals. All pediatric patients younger than 19 years admitted to the general ward from January 2019 to December 2019 were included. Using patient electronic medical records, we evaluated the clinical performance of the pDEWS for identifying deterioration events defined as in-hospital cardiac arrest (IHCA) and unexpected general ward-to-pediatric intensive care unit transfer (UIT) within 24 hours before event occurrence. We also compared pDEWS performance to those of the modified pediatric early-warning score (PEWS) and prediction models using logistic regression (LR) and random forest (RF). RESULTS: The study population consisted of 28,758 patients with 34 cases of IHCA and 291 cases of UIT. pDEWS showed better performance for predicting deterioration events with a larger area under the receiver operating characteristic curve, fewer false alarms, a lower mean alarm count per day, and a smaller number of cases needed to examine than the modified PEWS, LR, or RF models regardless of site, event occurrence time, age group, or sex. CONCLUSIONS: The pDEWS outperformed modified PEWS, LR, and RF models for early and accurate prediction of deterioration events regardless of clinical situation. This study demonstrated the potential of pDEWS as an efficient screening tool for efferent operation of rapid response teams.

Quick Sequential Organ Failure Assessment Score and the Modified Early Warning Score for Predicting Clinical Deterioration in General Ward Patients Regardless of Suspected Infection.

BACKGROUND: The quick sequential organ failure assessment (qSOFA) score is suggested to use for screening patients with a high risk of clinical deterioration in the general wards, which could simply be regarded as a general early warning score. However, comparison of unselected admissions to highlight the benefits of introducing qSOFA in hospitals already using Modified Early Warning Score (MEWS) remains unclear. We sought to compare qSOFA with MEWS for predicting clinical deterioration in general ward patients regardless of suspected infection. METHODS: The predictive performance of qSOFA and MEWS for in-hospital cardiac arrest (IHCA) or unexpected intensive care unit (ICU) transfer was compared with the areas under the receiver operating characteristic curve (AUC) analysis using the databases of vital signs collected from consecutive hospitalized adult patients over 12 months in five participating hospitals in Korea. RESULTS: Of 173,057 hospitalized patients included for analysis, 668 (0.39%) experienced the composite outcome. The discrimination for the composite outcome for MEWS (AUC, 0.777; 95% confidence interval [CI], 0.770-0.781) was higher than that for qSOFA (AUC, 0.684; 95% CI, 0.676-0.686; P < 0.001). In addition, MEWS was better for prediction of IHCA (AUC, 0.792; 95% CI, 0.781-0.795 vs. AUC, 0.640; 95% CI, 0.625-0.645; P < 0.001) and unexpected ICU transfer (AUC, 0.767; 95% CI, 0.760-0.773 vs. AUC, 0.716; 95% CI, 0.707-0.718; P < 0.001) than qSOFA. Using the MEWS at a cutoff of ≥ 5 would correctly reclassify 3.7% of patients from qSOFA score ≥ 2. Most patients met MEWS ≥ 5 criteria 13 hours before the composite outcome compared with 11 hours for qSOFA score ≥ 2. CONCLUSION: MEWS is more accurate that qSOFA score for predicting IHCA or unexpected ICU transfer in patients outside the ICU. Our study suggests that qSOFA should not replace MEWS for identifying patients in the general wards at risk of poor outcome.

Development and validation of a deep-learning-based pediatric early warning system: A single-center study.

BACKGROUND: Early detection and prompt intervention for clinically deteriorating events are needed to improve clinical outcomes. There have been several attempts at this, including the introduction of rapid response teams (RRTs) with early warning scores. We developed a deep-learning-based pediatric early warning system (pDEWS) and validated its performance. METHODS: This single-center retrospective observational cohort study reviewed, 50,019 pediatric patients admitted to the general ward in a tertiary-care academic children's hospital from January 2012 to December 2018. They were split by admission date into a derivation and a validation cohort. We developed a pDEWS for the early prediction of cardiopulmonary arrest and unexpected ward-to-pediatric intensive care unit (PICU) transfer. Then, we validated this system by comparing modified pediatric early warning score (PEWS), random forest (RF); an ensemble model of multiple decision trees and logistic regression (LR); a statistical model that uses a logistic function. RESULTS: For predicting cardiopulmonary arrest, the pDEWS (area under the receiver operating characteristic curve (AUROC), 0.923) outperformed modified PEWS (AUROC, 0.769) and reduced the mean alarm count per day (MACPD) and number needed to examine (NNE) by 82.0% (from 46.7 to 8.4 MACPD) and 89.5% (from 0.303 to 0.807), respectively. Furthermore, for predicting unexpected ward-to-PICU transfer pDEWS also showed superior performance compared to existing methods. CONCLUSION: Our study showed that pDEWS was superior to the modified PEWS and prediction models using RF and LR. This study demonstrates that the integration of the pDEWS into RRTs could increase operational efficiency and improve clinical outcomes.

A multicentre validation study of the deep learning-based early warning score for predicting in-hospital cardiac arrest in patients admitted to general wards.

BACKGROUND: The recently developed deep learning (DL)-based early warning score (DEWS) has shown potential in predicting deteriorating patients. We aimed to validate DEWS in multiple centres and compare the prediction, alarming and timeliness performance with the modified early warning score (MEWS) to identify patients at risk for in-hospital cardiac arrest (IHCA). METHOD/RESEARCH DESIGN: This retrospective cohort study included adult patients admitted to the general wards of five hospitals during a 12-month period. The occurrence of IHCA within 24 h of vital sign observation was the outcome of interest. We assessed the discrimination using the area under the receiver operating characteristic curve (AUROC). RESULTS: The study population consists of 173,368 patients (224 IHCAs). The predictive performance of DEWS was superior to that of MEWS in both the internal (AUROC: 0.860 vs. 0.754, respectively) and external (AUROC: 0.905 vs. 0.785, respectively) validation cohorts. At the same specificity, DEWS had a higher sensitivity than MEWS, and at the same sensitivity, DEWS reduced the mean alarm count by nearly half of MEWS. Additionally, DEWS was able to predict more IHCA patients in the 24-0.5 h before the outcome, and DEWS was reasonably calibrated. CONCLUSION: Our study showed that DEWS was superior to MEWS in three key aspects (IHCA predictive, alarming, and timeliness performance). This study demonstrates the potential of DEWS as an effective, efficient screening tool in rapid response systems (RRSs) to identify high-risk patients.

Graph Convolutional Networks-Based Noisy Data Imputation in Electronic Health Record.

OBJECTIVES: A deep learning-based early warning system is proposed to predict sepsis prior to its onset. DESIGN: A novel algorithm was devised to detect sepsis 6 hours prior to its onset based on electronic medical records. SETTING: Retrospective cohorts from three separate hospitals are used in this study. Sepsis onset was defined based on Sepsis-3. Algorithms are evaluated based on the score function used in the Physionet Challenge 2019. PATIENTS: Over 60,000 ICU patients with 40 clinical variables (vital signs, laboratory results) for each hour of a patient's ICU stay were used. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: The proposed algorithm predicted the onset of sepsis in the preceding n hours (where n = 4, 6, 8, or 12). Furthermore, the proposed method compared how many sepsis patients can be predicted in a short time with other methods. To interpret a given result in a clinical perspective, the relationship between input variables and the probability of the proposed method were presented. The proposed method achieved superior results (area under the receiver operating characteristic curve, area under the precision-recall curve, and score) and predicted more sepsis patients in advance. In official phase, the proposed method showed the utility score of -0.101, area under the receiver operating characteristic curve 0.782, area under the precision-recall curve 0.041, accuracy 0.786, and F-measure 0.046. CONCLUSIONS: Using Physionet Challenge 2019, the proposed method can accurately and early predict the onset of sepsis. The proposed method can be a practical early warning system in the environment of real hospitals.

Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services.

BACKGROUND: In emergency medical services (EMSs), accurately predicting the severity of a patient's medical condition is important for the early identification of those who are vulnerable and at high-risk. In this study, we developed and validated an artificial intelligence (AI) algorithm based on deep learning to predict the need for critical care during EMS. METHODS: We conducted a retrospective observation cohort study. The algorithm was established using development data from the Korean national emergency department information system, which were collected during visits in real time from 151 emergency departments (EDs). We validated the algorithm using EMS run sheets from two EDs. The study subjects comprised adult patients who visited EDs. The endpoint was critical care, and we used age, sex, chief complaint, symptom onset to arrival time, trauma, and initial vital signs as the predicted variables. RESULTS: The number of patients in the development data was 8,981,181, and the validation data comprised 2604 EMS run sheets from two hospitals. The area under the receiver operating characteristic curve of the algorithm to predict the critical care was 0.867 (95% confidence interval, [0.864-0.871]). This result outperformed the Emergency Severity Index (0.839 [0.831-0.846]), Korean Triage and Acuity System (0.824 [0.815-0.832]), National Early Warning Score (0.741 [0.734-0.748]), and Modified Early Warning Score (0.696 [0.691-0.699]). CONCLUSIONS: The AI algorithm accurately predicted the need for the critical care of patients using information during EMS and outperformed the conventional triage tools and early warning scores.

Detecting Patient Deterioration Using Artificial Intelligence in a Rapid Response System.

OBJECTIVES: As the performance of a conventional track and trigger system in a rapid response system has been unsatisfactory, we developed and implemented an artificial intelligence for predicting in-hospital cardiac arrest, denoted the deep learning-based early warning system. The purpose of this study was to compare the performance of an artificial intelligence-based early warning system with that of conventional methods in a real hospital situation. DESIGN: Retrospective cohort study. SETTING: This study was conducted at a hospital in which deep learning-based early warning system was implemented. PATIENTS: We reviewed the records of adult patients who were admitted to the general ward of our hospital from April 2018 to March 2019. INTERVENTIONS: The study population included 8,039 adult patients. A total 83 events of deterioration occurred during the study period. The outcome was events of deterioration, defined as cardiac arrest and unexpected ICU admission. We defined a true alarm as an alarm occurring within 0.5-24 hours before a deteriorating event. MEASUREMENTS AND MAIN RESULTS: We used the area under the receiver operating characteristic curve, area under the precision-recall curve, number needed to examine, and mean alarm count per day as comparative measures. The deep learning-based early warning system (area under the receiver operating characteristic curve, 0.865; area under the precision-recall curve, 0.066) outperformed the modified early warning score (area under the receiver operating characteristic curve, 0.682; area under the precision-recall curve, 0.010) and reduced the number needed to examine and mean alarm count per day by 69.2% and 59.6%, respectively. At the same specificity, deep learning-based early warning system had up to 257% higher sensitivity than conventional methods. CONCLUSIONS: The developed artificial intelligence based on deep-learning, deep learning-based early warning system, accurately predicted deterioration of patients in a general ward and outperformed conventional methods. This study showed the potential and effectiveness of artificial intelligence in an rapid response system, which can be applied together with electronic health records. This will be a useful method to identify patients with deterioration and help with precise decision-making in daily practice.