Clinical Implementation of an AI Early Warning System Algorithm: Lessons Learned.

The Deterioration Index (DI) is an automatic early warning system that utilizes a machine learning algorithm integrated into the electronic health record and was implemented to improve risk stratification of inpatients. Our pilot implementation showed superior diagnostic accuracy than standard care. A score >60 had a specificity of 88.5% and a sensitivity of 59.8% (PPV 0.1758, NPP 0.9817). However, acceptance in the clinical workflow was divided; nurses preferred standard care, while providers found it helpful.

A Clinician's Guide to Running Custom Machine-Learning Models in an Electronic Health Record Environment.

We recently brought an internally developed machine-learning model for predicting which patients in the emergency department would require hospital admission into the live electronic health record environment. Doing so involved navigating several engineering challenges that required the expertise of multiple parties across our institution. Our team of physician data scientists developed, validated, and implemented the model. We recognize a broad interest and need to adopt machine-learning models into clinical practice and seek to share our experience to enable other clinician-led initiatives. This Brief Report covers the entire model deployment process, starting once a team has trained and validated a model they wish to deploy in live clinical operations.

Novel machine learning model to improve performance of an early warning system in hospitalized patients: a retrospective multisite cross-validation study.

Background: Threshold-based early warning systems (EWS) are used to predict adverse events (Aes). Machine learning (ML) algorithms that incorporate all EWS scores prior to an event may perform better in hospitalized patients. Methods: The deterioration index (DI) is a proprietary EWS. A threshold of DI >60 is used to predict a composite AE: all-cause mortality, cardiac arrest, transfer to intensive care, and evaluation by the rapid response team in practice. The DI scores were collected for adult patients (≥18 y-o) hospitalized on medical or surgical services during 8-23-2021 to 3-31-2022 from four different Mayo Clinic sites in the United States. A novel ML model was developed and trained on a retrospective cohort of hospital encounters. DI scores were represented in a high-dimensional space using random convolution kernels to facilitate training of a classifier and the area under the receiver operator characteristics curve (AUC) was calculated. Multiple time intervals prior to an AE were analyzed. A leave-one-out cross-validation protocol was used to evaluate performance across separate clinic sites. Findings: Three different classifiers were trained on 59,617 encounter-derived DI scores in high-dimensional feature space and the AUCs were compared to two threshold models. All three tested classifiers improved the AUC over the threshold approaches from 0.56 and 0.57 to 0.76, 0.85 and 0.94. Time interval analysis of the top performing classifier showed best accuracy in the hour before an event occurred (AUC 0.91), but prediction held up even in the 12 h before an AE (AUC 0.80 at minus 12 h, 0.81 at minus 9 h, 0.85 at minus 6 h, and 0.88 at minus 3 h before an AE). Multisite cross-validation using leave-one-out approach on data from four different clinical sites showed broad generalization performance of the top performing ML model with AUC of 0.91, 0.91, 0.95, and 0.91. Interpretation: A novel ML model that incorporates all the longitudinal DI scores prior to an AE in a hospitalized patient performs better at outcome prediction than the currently used threshold model. The use of clinical data, a generalized ML technique, and successful multisite cross-validation demonstrate the feasibility of our model in clinical implementation. Funding: No funding to report.

Impact of the Performance Gap Between Interactive Alerts and Quality Metrics.

Interactive alerts are used to enhance compliance with primary prevention and have been shown to improve quality metrics. However, the degree of impact of these alerts is controversial and there is concern with excessive alerting. Our objective is to develop reliable processes to assess the direct impact of interactive alerts on clinical performance. Here we present preliminary finding related to the evaluation of the performance gaps between alerts and clinical practice.