Voice-Enabled Response Analysis Agent (VERAA): Leveraging Large Language Models to Map Voice Responses in SDoH Survey.

Social Determinants of Health (SDoH) have been shown to have profound impacts on health-related outcomes, yet this data suffers from high rates of missingness in electronic health records (EHR). Moreover, limited English proficiency in the United States can be a barrier to communication with health care providers. In this study, we have designed a multilingual conversational agent capable of conducting SDoH surveys for use in healthcare environments. The agent asks questions in the patient's native language, translates responses into English, and subsequently maps these responses via a large language model (LLM) to structured options in a SDoH survey. This tool can be extended to a variety of survey instruments in either hospital or home settings, enabling the extraction of structured insights from free-text answers. The proposed approach heralds a shift towards more inclusive and insightful data collection, marking a significant stride in SDoH data enrichment for optimizing health outcome predictions and interventions.

Prediction of Readmission Following Sepsis Using Social Determinants of Health.

OBJECTIVES: To determine the predictive value of social determinants of health (SDoH) variables on 30-day readmission following a sepsis hospitalization as compared with traditional clinical variables. DESIGN: Multicenter retrospective cohort study using patient-level data, including demographic, clinical, and survey data. SETTINGS: Thirty-five hospitals across the United States from 2017 to 2021. PATIENTS: Two hundred seventy-one thousand four hundred twenty-eight individuals in the AllofUs initiative, of which 8909 had an index sepsis hospitalization. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Unplanned 30-day readmission to the hospital. Multinomial logistic regression models were constructed to account for survival in determination of variables associate with 30-day readmission and are presented as adjusted odds rations (aORs). Of the 8909 sepsis patients in our cohort, 21% had an unplanned hospital readmission within 30 days. Median age (interquartile range) was 54 years (41-65 yr), 4762 (53.4%) were female, and there were self-reported 1612 (18.09%) Black, 2271 (25.49%) Hispanic, and 4642 (52.1%) White individuals. In multinomial logistic regression models accounting for survival, we identified that change to nonphysician provider type due to economic reasons (aOR, 2.55 [2.35-2.74]), delay of receiving medical care due to lack of transportation (aOR, 1.68 [1.62-1.74]), and inability to afford flow-up care (aOR, 1.59 [1.52-1.66]) were strongly and independently associated with a 30-day readmission when adjusting for survival. Patients who lived in a ZIP code with a high percentage of patients in poverty and without health insurance were also more likely to be readmitted within 30 days (aOR, 1.26 [1.22-1.29] and aOR, 1.28 [1.26-1.29], respectively). Finally, we found that having a primary care provider and health insurance were associated with low odds of an unplanned 30-day readmission. CONCLUSIONS: In this multicenter retrospective cohort, several SDoH variables were strongly associated with unplanned 30-day readmission. Models predicting readmission following sepsis hospitalization may benefit from the addition of SDoH factors to traditional clinical variables.

Impact of a deep learning sepsis prediction model on quality of care and survival.

Sepsis remains a major cause of mortality and morbidity worldwide. Algorithms that assist with the early recognition of sepsis may improve outcomes, but relatively few studies have examined their impact on real-world patient outcomes. Our objective was to assess the impact of a deep-learning model (COMPOSER) for the early prediction of sepsis on patient outcomes. We completed a before-and-after quasi-experimental study at two distinct Emergency Departments (EDs) within the UC San Diego Health System. We included 6217 adult septic patients from 1/1/2021 through 4/30/2023. The exposure tested was a nurse-facing Best Practice Advisory (BPA) triggered by COMPOSER. In-hospital mortality, sepsis bundle compliance, 72-h change in sequential organ failure assessment (SOFA) score following sepsis onset, ICU-free days, and the number of ICU encounters were evaluated in the pre-intervention period (705 days) and the post-intervention period (145 days). The causal impact analysis was performed using a Bayesian structural time-series approach with confounder adjustments to assess the significance of the exposure at the 95% confidence level. The deployment of COMPOSER was significantly associated with a 1.9% absolute reduction (17% relative decrease) in in-hospital sepsis mortality (95% CI, 0.3%-3.5%), a 5.0% absolute increase (10% relative increase) in sepsis bundle compliance (95% CI, 2.4%-8.0%), and a 4% (95% CI, 1.1%-7.1%) reduction in 72-h SOFA change after sepsis onset in causal inference analysis. This study suggests that the deployment of COMPOSER for early prediction of sepsis was associated with a significant reduction in mortality and a significant increase in sepsis bundle compliance.

Unsupervised Detection and Correction of Model Calibration Shift at Test-Time.

The wide adoption of predictive models into clinical practice require generalizability across hospitals and maintenance of consistent performance across time. Model calibration shift, caused by factors such as changes in prevalence rates or data distribution shift, can affect the generalizability of such models. In this work, we propose a model calibration detection and correction (CaDC) method, specifically designed to utilize only unlabeled data at a target hospital. The proposed method is very flexible and can be used alongside any deep learning-based clinical predictive model. As a case study, we focus on the problem of detecting and correcting model calibration shift in the context of early prediction of sepsis. Three patient cohorts consisting of 545,089 adult patients admitted to the emergency departments at three geographically diverse healthcare systems in the United States were used to train and externally validate the proposed method. We successfully show that utilizing the CaDC model can help assist the sepsis prediction model in achieving a predefined positive predictive value (PPV). For instance, when trained to achieve a PPV of 20%, the performance of the sepsis prediction model with and without the calibration shift estimation model was 18.0% vs 12.9% and 23.1% vs 13.4% at the two external validation cohorts, respectively. As such, the proposed CaDC method has potential applications in maintaining performance claims of predictive models deployed across hospital systems.Clinical relevance- Model generalizability is a requirement of wider adoption of clinical predictive models.

Development & Deployment of a Real-time Healthcare Predictive Analytics Platform.

The deployment of predictive analytic algorithms that can safely and seamlessly integrate into existing healthcare workflows remains a significant challenge. Here, we present a scalable, cloud-based, fault-tolerant platform that is capable of extracting and processing electronic health record (EHR) data for any patient at any time following admission and transferring results back into the EHR. This platform has been successfully deployed within the UC San Diego Health system and utilizes interoperable data standards to enable portability.Clinical relevance- This platform is currently hosting a deep learning model for the early prediction of sepsis that is operational in two emergency departments.

Predicting Hospital Readmission among Patients with Sepsis Using Clinical and Wearable Data.

Sepsis is a life-threatening condition that occurs due to a dysregulated host response to infection. Recent data demonstrate that patients with sepsis have a significantly higher readmission risk than other common conditions, such as heart failure, pneumonia and myocardial infarction and associated economic burden. Prior studies have demonstrated an association between a patient's physical activity levels and readmission risk. In this study, we show that distribution of activity level prior and post-discharge among patients with sepsis are predictive of unplanned rehospitalization in 90 days (P-value<1e-3). Our preliminary results indicate that integrating Fitbit data with clinical measurements may improve model performance on predicting 90 days readmission.Clinical relevance Sepsis, Activity level, Hospital readmission, Wearable data.

Voice-Enabled Response Analysis Agent (VERAA): Leveraging Large Language Models to Map Voice Responses in SDoH Survey.

Social Determinants of Health (SDoH) have been shown to have profound impacts on health-related outcomes, yet this data suffers from high rates of missingness in electronic health records (EHR). Moreover, limited English proficiency in the United States can be a barrier to communication with health care providers. In this study, we have designed a multilingual conversational agent capable of conducting SDoH surveys for use in healthcare environments. The agent asks questions in the patient's native language, translates responses into English, and subsequently maps these responses via a large language model (LLM) to structured options in a SDoH survey. This tool can be extended to a variety of survey instruments in either hospital or home settings, enabling the extraction of structured insights from free-text answers. The proposed approach heralds a shift towards more inclusive and insightful data collection, marking a significant stride in SDoH data enrichment for optimizing health outcome predictions and interventions.

Making the Improbable Possible: Generalizing Models Designed for a Syndrome-Based, Heterogeneous Patient Landscape.

Syndromic conditions, such as sepsis, are commonly encountered in the intensive care unit. Although these conditions are easy for clinicians to grasp, these conditions may limit the performance of machine-learning algorithms. Individual hospital practice patterns may limit external generalizability. Data missingness is another barrier to optimal algorithm performance and various strategies exist to mitigate this. Recent advances in data science, such as transfer learning, conformal prediction, and continual learning, may improve generalizability of machine-learning algorithms in critically ill patients. Randomized trials with these approaches are indicated to demonstrate improvements in patient-centered outcomes at this point.

Representation Learning and Spectral Clustering for the Development and External Validation of Dynamic Sepsis Phenotypes: Observational Cohort Study.

BACKGROUND: Recent attempts at clinical phenotyping for sepsis have shown promise in identifying groups of patients with distinct treatment responses. Nonetheless, the replicability and actionability of these phenotypes remain an issue because the patient trajectory is a function of both the patient's physiological state and the interventions they receive. OBJECTIVE: We aimed to develop a novel approach for deriving clinical phenotypes using unsupervised learning and transition modeling. METHODS: Forty commonly used clinical variables from the electronic health record were used as inputs to a feed-forward neural network trained to predict the onset of sepsis. Using spectral clustering on the representations from this network, we derived and validated consistent phenotypes across a diverse cohort of patients with sepsis. We modeled phenotype dynamics as a Markov decision process with transitions as a function of the patient's current state and the interventions they received. RESULTS: Four consistent and distinct phenotypes were derived from over 11,500 adult patients who were admitted from the University of California, San Diego emergency department (ED) with sepsis between January 1, 2016, and January 31, 2020. Over 2000 adult patients admitted from the University of California, Irvine ED with sepsis between November 4, 2017, and August 4, 2022, were involved in the external validation. We demonstrate that sepsis phenotypes are not static and evolve in response to physiological factors and based on interventions. We show that roughly 45% of patients change phenotype membership within the first 6 hours of ED arrival. We observed consistent trends in patient dynamics as a function of interventions including early administration of antibiotics. CONCLUSIONS: We derived and describe 4 sepsis phenotypes present within 6 hours of triage in the ED. We observe that the administration of a 30 mL/kg fluid bolus may be associated with worse outcomes in certain phenotypes, whereas prompt antimicrobial therapy is associated with improved outcomes.

Development & Deployment of a Real-time Healthcare Predictive Analytics Platform.

The deployment of predictive analytic algorithms that can safely and seamlessly integrate into existing healthcare workflows remains a significant challenge. Here, we present a scalable, cloud-based, fault-tolerant platform that is capable of extracting and processing electronic health record (EHR) data for any patient at any time following admission and transferring results back into the EHR. This platform has been successfully deployed within the UC San Diego Health system and utilizes interoperable data standards to enable portability.

Predicting Hospital Readmission among Patients with Sepsis using Clinical and Wearable Data.

Sepsis is a life-threatening condition that occurs due to a dysregulated host response to infection. Recent data demonstrate that patients with sepsis have a significantly higher readmission risk than other common conditions, such as heart failure, pneumonia and myocardial infarction and associated economic burden. Prior studies have demonstrated an association between a patient's physical activity levels and readmission risk. In this study, we show that distribution of activity level prior and post-discharge among patients with sepsis are predictive of unplanned rehospitalization in 90 days (P-value<1e-3). Our preliminary results indicate that integrating Fitbit data with clinical measurements may improve model performance on predicting 90 days readmission.

Prediction of Unplanned Hospital Readmission using Clinical and Longitudinal Wearable Sensor Features.

Predictive models have been suggested as potential tools for identifying highest risk patients for hospital readmissions, in order to improve care coordination and ultimately long-term patient outcomes. However, the accuracy of current predictive models for readmission prediction is still moderate and further data enrichment is needed to identify at risk patients. This paper describes models to predict 90-day readmission, focusing on testing the predictive performance of wearable sensor features generated using multiscale entropy techniques and clinical features. Our study explores ways to incorporate pre-discharge and post-discharge wearable sensor features to make robust patient predictions. Data were used from participants enrolled in the AllofUs Research program. We extracted the inpatient cohort of patients and integrated clinical data from the electronic health records (EHR) and Fitbit sensor measurements. Entropy features were calculated from the longitudinal wearable sensor data, such as heart rate and mobility-related measurements, in order to characterize time series variability and complexity. Our best performing model acheived an AUC of 83%, and at 80% sensitivity acheived 75% specificity and 57% positive predictive value. Our results indicate that it would be possible to improve the ability to predict unplanned hospital readmissions by considering pre-discharge and post-discharge wearable features.

Optimizing the Implementation of Clinical Predictive Models to Minimize National Costs: Sepsis Case Study.

BACKGROUND: Sepsis costs and incidence vary dramatically across diagnostic categories, warranting a customized approach for implementing predictive models. OBJECTIVE: The aim of this study was to optimize the parameters of a sepsis prediction model within distinct patient groups to minimize the excess cost of sepsis care and analyze the potential effect of factors contributing to end-user response to sepsis alerts on overall model utility. METHODS: We calculated the excess costs of sepsis to the Centers for Medicare and Medicaid Services (CMS) by comparing patients with and without a secondary sepsis diagnosis but with the same primary diagnosis and baseline comorbidities. We optimized the parameters of a sepsis prediction algorithm across different diagnostic categories to minimize these excess costs. At the optima, we evaluated diagnostic odds ratios and analyzed the impact of compliance factors such as noncompliance, treatment efficacy, and tolerance for false alarms on the net benefit of triggering sepsis alerts. RESULTS: Compliance factors significantly contributed to the net benefit of triggering a sepsis alert. However, a customized deployment policy can achieve a significantly higher diagnostic odds ratio and reduced costs of sepsis care. Implementing our optimization routine with powerful predictive models could result in US $4.6 billion in excess cost savings for CMS. CONCLUSIONS: We designed a framework for customizing sepsis alert protocols within different diagnostic categories to minimize excess costs and analyzed model performance as a function of false alarm tolerance and compliance with model recommendations. We provide a framework that CMS policymakers could use to recommend minimum adherence rates to the early recognition and appropriate care of sepsis that is sensitive to hospital department-level incidence rates and national excess costs. Customizing the implementation of clinical predictive models by accounting for various behavioral and economic factors may improve the practical benefit of predictive models.

Inclusion of social determinants of health improves sepsis readmission prediction models.

OBJECTIVE: Sepsis has a high rate of 30-day unplanned readmissions. Predictive modeling has been suggested as a tool to identify high-risk patients. However, existing sepsis readmission models have low predictive value and most predictive factors in such models are not actionable. MATERIALS AND METHODS: Data from patients enrolled in the AllofUs Research Program cohort from 35 hospitals were used to develop a multicenter validated sepsis-related unplanned readmission model that incorporates clinical and social determinants of health (SDH) to predict 30-day unplanned readmissions. Sepsis cases were identified using concepts represented in the Observational Medical Outcomes Partnership. The dataset included over 60 clinical/laboratory features and over 100 SDH features. RESULTS: Incorporation of SDH factors into our model of clinical and demographic features improves model area under the receiver operating characteristic curve (AUC) significantly (from 0.75 to 0.80; P < .001). Model-agnostic interpretability techniques revealed demographics, economic stability, and delay in getting medical care as important SDH predictive features of unplanned hospital readmissions. DISCUSSION: This work represents one of the largest studies of sepsis readmissions using objective clinical data to date (8935 septic index encounters). SDH are important to determine which sepsis patients are more likely to have an unplanned 30-day readmission. The AllofUS dataset provides granular data from a diverse set of individuals, making this model potentially more generalizable than prior models. CONCLUSION: Use of SDH improves predictive performance of a model to identify which sepsis patients are at high risk of an unplanned 30-day readmission.

Leveraging clinical data across healthcare institutions for continual learning of predictive risk models.

The inherent flexibility of machine learning-based clinical predictive models to learn from episodes of patient care at a new institution (site-specific training) comes at the cost of performance degradation when applied to external patient cohorts. To exploit the full potential of cross-institutional clinical big data, machine learning systems must gain the ability to transfer their knowledge across institutional boundaries and learn from new episodes of patient care without forgetting previously learned patterns. In this work, we developed a privacy-preserving learning algorithm named WUPERR (Weight Uncertainty Propagation and Episodic Representation Replay) and validated the algorithm in the context of early prediction of sepsis using data from over 104,000 patients across four distinct healthcare systems. We tested the hypothesis, that the proposed continual learning algorithm can maintain higher predictive performance than competing methods on previous cohorts once it has been trained on a new patient cohort. In the sepsis prediction task, after incremental training of a deep learning model across four hospital systems (namely hospitals H-A, H-B, H-C, and H-D), WUPERR maintained the highest positive predictive value across the first three hospitals compared to a baseline transfer learning approach (H-A: 39.27% vs. 31.27%, H-B: 25.34% vs. 22.34%, H-C: 30.33% vs. 28.33%). The proposed approach has the potential to construct more generalizable models that can learn from cross-institutional clinical big data in a privacy-preserving manner.

Artificial intelligence sepsis prediction algorithm learns to say "I don't know".

Sepsis is a leading cause of morbidity and mortality worldwide. Early identification of sepsis is important as it allows timely administration of potentially life-saving resuscitation and antimicrobial therapy. We present COMPOSER (COnformal Multidimensional Prediction Of SEpsis Risk), a deep learning model for the early prediction of sepsis, specifically designed to reduce false alarms by detecting unfamiliar patients/situations arising from erroneous data, missingness, distributional shift and data drifts. COMPOSER flags these unfamiliar cases as indeterminate rather than making spurious predictions. Six patient cohorts (515,720 patients) curated from two healthcare systems in the United States across intensive care units (ICU) and emergency departments (ED) were used to train and externally and temporally validate this model. In a sequential prediction setting, COMPOSER achieved a consistently high area under the curve (AUC) (ICU: 0.925-0.953; ED: 0.938-0.945). Out of over 6 million prediction windows roughly 20% and 8% were identified as indeterminate amongst non-septic and septic patients, respectively. COMPOSER provided early warning within a clinically actionable timeframe (ICU: 12.2 [3.2 22.8] and ED: 2.1 [0.8 4.5] hours prior to first antibiotics order) across all six cohorts, thus allowing for identification and prioritization of patients at high risk for sepsis.

A Locally Optimized Data-Driven Tool to Predict Sepsis-Associated Vasopressor Use in the ICU.

OBJECTIVES: To train a model to predict vasopressor use in ICU patients with sepsis and optimize external performance across hospital systems using domain adaptation, a transfer learning approach. DESIGN: Observational cohort study. SETTING: Two academic medical centers from January 2014 to June 2017. PATIENTS: Data were analyzed from 14,512 patients (9,423 at the development site and 5,089 at the validation site) who were admitted to an ICU and met Center for Medicare and Medicaid Services definition of severe sepsis either before or during the ICU stay. Patients were excluded if they never developed sepsis, if the ICU length of stay was less than 8 hours or more than 20 days or if they developed shock up to the first 4 hours of ICU admission. MEASUREMENTS AND MAIN RESULTS: Forty retrospectively collected features from the electronic medical records of adult ICU patients at the development site (four hospitals) were used as inputs for a neural network Weibull-Cox survival model to derive a prediction tool for future need of vasopressors. Domain adaptation updated parameters to optimize model performance in the validation site (two hospitals), a different healthcare system over 2,000 miles away. The cohorts at both sites were randomly split into training and testing sets (80% and 20%, respectively). When applied to the test set in the development site, the model predicted vasopressor use 4-24 hours in advance with an area under the receiver operator characteristic curve, specificity, and positive predictive value ranging from 0.80 to 0.81, 56.2% to 61.8%, and 5.6% to 12.1%, respectively. Domain adaptation improved performance of the model to predict vasopressor use within 4 hours at the validation site (area under the receiver operator characteristic curve 0.81 [CI, 0.80-0.81] from 0.77 [CI, 0.76-0.77], p < 0.01; specificity 59.7% [CI, 58.9-62.5%] from 49.9% [CI, 49.5-50.7%], p < 0.01; positive predictive value 8.9% [CI, 8.5-9.4%] from 7.3 [7.1-7.4%], p < 0.01). CONCLUSIONS: Domain adaptation improved performance of a model predicting sepsis-associated vasopressor use during external validation.

DeepAISE - An interpretable and recurrent neural survival model for early prediction of sepsis.

Sepsis, a dysregulated immune system response to infection, is among the leading causes of morbidity, mortality, and cost overruns in the Intensive Care Unit (ICU). Early prediction of sepsis can improve situational awareness among clinicians and facilitate timely, protective interventions. While the application of predictive analytics in ICU patients has shown early promising results, much of the work has been encumbered by high false-alarm rates and lack of trust by the end-users due to the 'black box' nature of these models. Here, we present DeepAISE (Deep Artificial Intelligence Sepsis Expert), a recurrent neural survival model for the early prediction of sepsis. DeepAISE automatically learns predictive features related to higher-order interactions and temporal patterns among clinical risk factors that maximize the data likelihood of observed time to septic events. A comparative study of four baseline models on data from hospitalized patients at three different healthcare systems indicates that DeepAISE produces the most accurate predictions (AUCs between 0.87 and 0.90) at the lowest false alarm rates (FARs between 0.20 and 0.25) while simultaneously producing interpretable representations of the clinical time series and risk factors.

Predicting Progression to Septic Shock in the Emergency Department Using an Externally Generalizable Machine-Learning Algorithm.

STUDY OBJECTIVE: Machine-learning algorithms allow improved prediction of sepsis syndromes in the emergency department (ED), using data from electronic medical records. Transfer learning, a new subfield of machine learning, allows generalizability of an algorithm across clinical sites. We aim to validate the Artificial Intelligence Sepsis Expert for the prediction of delayed septic shock in a cohort of patients treated in the ED and demonstrate the feasibility of transfer learning to improve external validity at a second site. METHODS: This was an observational cohort study using data from greater than 180,000 patients from 2 academic medical centers between 2014 and 2019, using multiple definitions of sepsis. The Artificial Intelligence Sepsis Expert algorithm was trained with 40 input variables at the development site to predict delayed septic shock (occurring greater than 4 hours after ED triage) at various prediction windows. We then validated the algorithm at a second site, using transfer learning to demonstrate generalizability of the algorithm. RESULTS: We identified 9,354 patients with severe sepsis, of whom 723 developed septic shock at least 4 hours after triage. The Artificial Intelligence Sepsis Expert algorithm demonstrated excellent area under the receiver operating characteristic curve (>0.8) at 8 and 12 hours for the prediction of delayed septic shock. Transfer learning significantly improved the test characteristics of the Artificial Intelligence Sepsis Expert algorithm and yielded comparable performance at the validation site. CONCLUSION: The Artificial Intelligence Sepsis Expert algorithm accurately predicted the development of delayed septic shock. The use of transfer learning allowed significantly improved external validity and generalizability at a second site. Future prospective studies are indicated to evaluate the clinical utility of this model.