Effectiveness of an Artificial Intelligence-Enabled Intervention for Detecting Clinical Deterioration.

Importance: Inpatient clinical deterioration is associated with substantial morbidity and mortality but may be easily missed by clinicians. Early warning scores have been developed to alert clinicians to patients at high risk of clinical deterioration, but there is limited evidence for their effectiveness. Objective: To evaluate the effectiveness of an artificial intelligence deterioration model-enabled intervention to reduce the risk of escalations in care among hospitalized patients using a study design that facilitates stronger causal inference. Design, Setting, and Participants: This cohort study used a regression discontinuity design that controlled for confounding and was based on Epic Deterioration Index (EDI; Epic Systems Corporation) prediction model scores. Compared with other observational research, the regression discontinuity design facilitates causal analysis. Hospitalized adults were included from 4 general internal medicine units in 1 academic hospital from January 17, 2021, through November 16, 2022. Exposure: An artificial intelligence deterioration model-enabled intervention, consisting of alerts based on an EDI score threshold with an associated collaborative workflow among nurses and physicians. Main Outcomes and Measures: The primary outcome was escalations in care, including rapid response team activation, transfer to the intensive care unit, or cardiopulmonary arrest during hospitalization. Results: During the study, 9938 patients were admitted to 1 of the 4 units, with 963 patients (median [IQR] age, 76.1 [64.2-86.2] years; 498 males [52.3%]) included within the primary regression discontinuity analysis. The median (IQR) Elixhauser Comorbidity Index score in the primary analysis cohort was 10 (0-24). The intervention was associated with a -10.4-percentage point (95% CI, -20.1 to -0.8 percentage points; P = .03) absolute risk reduction in the primary outcome for patients at the EDI score threshold. There was no evidence of a discontinuity in measured confounders at the EDI score threshold. Conclusions and Relevance: Using a regression discontinuity design, this cohort study found that the implementation of an artificial intelligence deterioration model-enabled intervention was associated with a significantly decreased risk of escalations in care among inpatients. These results provide evidence for the effectiveness of this intervention and support its further expansion and testing in other care settings.

Considerations in the reliability and fairness audits of predictive models for advance care planning.

Multiple reporting guidelines for artificial intelligence (AI) models in healthcare recommend that models be audited for reliability and fairness. However, there is a gap of operational guidance for performing reliability and fairness audits in practice. Following guideline recommendations, we conducted a reliability audit of two models based on model performance and calibration as well as a fairness audit based on summary statistics, subgroup performance and subgroup calibration. We assessed the Epic End-of-Life (EOL) Index model and an internally developed Stanford Hospital Medicine (HM) Advance Care Planning (ACP) model in 3 practice settings: Primary Care, Inpatient Oncology and Hospital Medicine, using clinicians' answers to the surprise question ("Would you be surprised if [patient X] passed away in [Y years]?") as a surrogate outcome. For performance, the models had positive predictive value (PPV) at or above 0.76 in all settings. In Hospital Medicine and Inpatient Oncology, the Stanford HM ACP model had higher sensitivity (0.69, 0.89 respectively) than the EOL model (0.20, 0.27), and better calibration (O/E 1.5, 1.7) than the EOL model (O/E 2.5, 3.0). The Epic EOL model flagged fewer patients (11%, 21% respectively) than the Stanford HM ACP model (38%, 75%). There were no differences in performance and calibration by sex. Both models had lower sensitivity in Hispanic/Latino male patients with Race listed as "Other." 10 clinicians were surveyed after a presentation summarizing the audit. 10/10 reported that summary statistics, overall performance, and subgroup performance would affect their decision to use the model to guide care; 9/10 said the same for overall and subgroup calibration. The most commonly identified barriers for routinely conducting such reliability and fairness audits were poor demographic data quality and lack of data access. This audit required 115 person-hours across 8-10 months. Our recommendations for performing reliability and fairness audits include verifying data validity, analyzing model performance on intersectional subgroups, and collecting clinician-patient linkages as necessary for label generation by clinicians. Those responsible for AI models should require such audits before model deployment and mediate between model auditors and impacted stakeholders.

A framework for making predictive models useful in practice.

OBJECTIVE: To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. MATERIALS AND METHODS: We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP. RESULTS: Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care. DISCUSSION: The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit. CONCLUSION: An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

Lean-Based Redesign of Multidisciplinary Rounds on General Medicine Service.

BACKGROUND: Multidisciplinary rounds (MDR) facilitate timely communication amongst the care team and with patients. We used Lean techniques to redesign MDR on the teaching general medicine service. OBJECTIVE: To examine if our Lean-based new model of MDR was associated with change in the primary outcome of length of stay (LOS) and secondary outcomes of discharges before noon, documentation of estimated discharge date (EDD), and patient satisfaction. DESIGN, SETTING, PATIENTS: This is a pre-post study. The preperiod (in which the old model of MDR was followed) comprised 4000 patients discharged between September 1, 2013, and October 22, 2014. The postperiod (in which the new model of MDR was followed) comprised 2085 patients between October 23, 2014, and April 30, 2015. INTERVENTION: Lean-based redesign of MDR. MEASUREMENTS: LOS, discharges before noon, EDD, and patient satisfaction. RESULTS: There was no change in the mean LOS. Discharges before noon increased from 6.9% to 10.7% (P < .001). Recording of EDD increased from 31.4% to 41.3% (P < .001). There was no change in patient satisfaction. CONCLUSIONS: Lean-based redesign of MDR was associated with an increase in discharges before noon and in recording of EDD.

A model of mild hypoglycemia.

OBJECTIVE: To investigate the impact associated with mild hypoglycemia among patients with type 2 diabetes (T2DM) in the United States and to identify risk factors among different subpopulations. METHODS: We performed a literature search to gather available data allowing estimation of rates of mild hypoglycemia. Because risk factors are interdependent, risk factors included in the model were based on those reported within multivariate analyses or judged to be biologically plausible by the medical community. Based on literature search results, we built a mathematical model predicting the rates of mild hypoglycemia in individual patients as a function of the patient's antidiabetic medications, hemoglobin A1c levels, duration of diabetes, kidney function, and body mass index. RESULTS: We estimated an overall average rate of mild hypoglycemia among US patients with T2DM of 2.2 ± 0.8 events per person per year. Patients taking oral antidiabetic medications only had an average rate of 1.9 ± 0.8 events per person per year. The average rate for all patients taking insulin, including those combining it with other antidiabetic medications, was 4.9 ± 2.0 events per person per year. Mild hypoglycemia rates increased with age, with 80-year-old patients experiencing 1.5 times the risk of 40-year-old patients. Based on published values for direct and indirect medical costs for mild hypoglycemia events, we determined that the economic impact in the US of mild hypoglycemic events is approximately $900 million per year, roughly equal to that of severe hypoglycemic events. One of the key limitations to our model is that it applies to the US population under standard medical care and not to clinical trials and does not include certain known risk factors such as rigorous exercise. CONCLUSIONS: Understanding the benefit versus risk of glycemic control and hypoglycemia is fundamental to the successful management of patients with T2DM. Our validated hypoglycemia model is an important step in addressing this issue and may be helpful to researchers, clinicians, and payers to determine the patients who are at the highest risk for hypoglycemia, whether a patient is experiencing events at 'higher-than-expected' rates, and the corresponding economic burden.

Simulating the impact of improved cardiovascular risk interventions on clinical and economic outcomes in Russia.

OBJECTIVES: Russia faces a high burden of cardiovascular disease. Prevalence of all cardiovascular risk factors, especially hypertension, is high. Elevated blood pressure is generally poorly controlled and medication usage is suboptimal. With a disease-model simulation, we forecast how various treatment programs aimed at increasing blood pressure control would affect cardiovascular outcomes. In addition, we investigated what additional benefit adding lipid control and smoking cessation to blood pressure control would generate in terms of reduced cardiovascular events. Finally, we estimated the direct health care costs saved by treating fewer cardiovascular events. METHODS: The Archimedes Model, a detailed computer model of human physiology, disease progression, and health care delivery was adapted to the Russian setting. Intervention scenarios of achieving systolic blood pressure control rates (defined as systolic blood pressure <140 mmHg) of 40% and 60% were simulated by modifying adherence rates of an antihypertensive medication combination and compared with current care (23.9% blood pressure control rate). Outcomes of major adverse cardiovascular events; cerebrovascular event (stroke), myocardial infarction, and cardiovascular death over a 10-year time horizon were reported. Direct health care costs of strokes and myocardial infarctions were derived from official Russian statistics and tariff lists. RESULTS: To achieve systolic blood pressure control rates of 40% and 60%, adherence rates to the antihypertensive treatment program were 29.4% and 65.9%. Cardiovascular death relative risk reductions were 13.2%, and 29.6%, respectively. For the current estimated 43,855,000-person Russian hypertensive population, each control-rate scenario resulted in an absolute reduction of 1.0 million and 2.4 million cardiovascular deaths, and a reduction of 1.2 million and 2.7 million stroke/myocardial infarction diagnoses, respectively. Averted direct costs from current care levels ($7.6 billion [in United States dollars]) were $1.1 billion and $2.6 billion, respectively.

Spatial clustering of ocular chlamydial infection over time following treatment, among households in a village in Tanzania.

PURPOSE: To observe the spatial distribution of households with high loads of ocular chlamydia infection in children, before and after mass treatment with azithromycin to determine whether there exists spatial clustering of households with high loads of infection and the spatial scale of the clustering. METHODS: All residents of a village in Tanzania were invited to participate in the study. A global positioning system unit recorded the location of each house. Mass treatment with azithromycin was offered, with participation above 80%. Active trachoma and swab samples of the conjunctiva were assessed at baseline and at 2, 6, 12, and 18 months after treatment. A k-function analysis was performed to detect clustering of households with high loads of ocular chlamydia in children younger than 8 years. RESULTS: A total of 1055 villagers were examined during the study; of these, 374 (35.4%) were children younger than 8 years. The total number of households was 215, with 182 (84.6%) households having at least one child. K-function analysis showed clustering of households with high loads of ocular chlamydia at distances up to 2 kilometers (km) at baseline; at 6 months, slight clustering existed within 0.5 km. At 12 and 18 months, high load households clustered at distances up to 1.3 km. CONCLUSIONS: This analysis suggests that infection spreads between households with children or that nearby households share the same risk factors for infection. Mass treatment has value in lowering infection prevalence within the community, and clustering of households with infection takes up to 1 year to reemerge at the same level as baseline. Re-treatment at yearly intervals may interrupt spread of infection.