Clinical Implementation of an AI Early Warning System Algorithm: Lessons Learned.

The Deterioration Index (DI) is an automatic early warning system that utilizes a machine learning algorithm integrated into the electronic health record and was implemented to improve risk stratification of inpatients. Our pilot implementation showed superior diagnostic accuracy than standard care. A score >60 had a specificity of 88.5% and a sensitivity of 59.8% (PPV 0.1758, NPP 0.9817). However, acceptance in the clinical workflow was divided; nurses preferred standard care, while providers found it helpful.

Prevalence, Severity, and Co-Occurrence of SPPADE Symptoms in 31,866 Patients With Cancer.

OBJECTIVES: To examine the prevalence, severity, and co-occurrence of SPPADE symptoms as well as their association with cancer type and patient characteristics. BACKGROUND: The SPPADE symptoms (sleep disturbance, pain, physical function impairment, anxiety, depression, and low energy /fatigue) are prevalent, co-occurring, and undertreated in oncology and other clinical populations. METHODS: Baseline SPPADE symptom data were analyzed from the E2C2 study, a stepped wedge pragmatic, population-level, cluster randomized clinical trial designed to evaluate a guideline-informed symptom management model targeting the six SPPADE symptoms. Symptom prevalence and severity were measured with a 0-10 numeric rating (NRS) scale for each of the six symptoms. Prevalence of severe (NRS ≥ 7) and potential clinically relevant (NRS ≥ 5) symptoms as well as co-occurrence of clinical symptoms were determined. Distribution-based methods were used to estimate the minimally important difference (MID). Associations of cancer type and patient characteristics with a SPPADE composite score were analyzed. RESULTS: A total of 31,886 patients were assessed for SPPADE symptoms prior to, during, or soon after an outpatient medical oncology encounter. The proportion of patients with a potential clinically relevant symptom ranged from 17.5% for depression to 33.4% for fatigue. Co-occurrence of symptoms was high, with the proportion of patients with three or more additional clinically relevant symptoms ranging from 45.2% for fatigue to 68.6% for depression. The summed SPPADE composite score demonstrated good internal reliability (Cronbach's alpha of 0.86), with preliminary MID estimates of 4.1-4.3. Symptom burden differed across several types of cancer but was generally similar across most sociodemographic characteristics. CONCLUSION: The high prevalence and co-occurrence of SPPADE symptoms in patients with all types of cancer warrants clinical approaches that optimize detection and management.

Novel machine learning model to improve performance of an early warning system in hospitalized patients: a retrospective multisite cross-validation study.

Background: Threshold-based early warning systems (EWS) are used to predict adverse events (Aes). Machine learning (ML) algorithms that incorporate all EWS scores prior to an event may perform better in hospitalized patients. Methods: The deterioration index (DI) is a proprietary EWS. A threshold of DI >60 is used to predict a composite AE: all-cause mortality, cardiac arrest, transfer to intensive care, and evaluation by the rapid response team in practice. The DI scores were collected for adult patients (≥18 y-o) hospitalized on medical or surgical services during 8-23-2021 to 3-31-2022 from four different Mayo Clinic sites in the United States. A novel ML model was developed and trained on a retrospective cohort of hospital encounters. DI scores were represented in a high-dimensional space using random convolution kernels to facilitate training of a classifier and the area under the receiver operator characteristics curve (AUC) was calculated. Multiple time intervals prior to an AE were analyzed. A leave-one-out cross-validation protocol was used to evaluate performance across separate clinic sites. Findings: Three different classifiers were trained on 59,617 encounter-derived DI scores in high-dimensional feature space and the AUCs were compared to two threshold models. All three tested classifiers improved the AUC over the threshold approaches from 0.56 and 0.57 to 0.76, 0.85 and 0.94. Time interval analysis of the top performing classifier showed best accuracy in the hour before an event occurred (AUC 0.91), but prediction held up even in the 12 h before an AE (AUC 0.80 at minus 12 h, 0.81 at minus 9 h, 0.85 at minus 6 h, and 0.88 at minus 3 h before an AE). Multisite cross-validation using leave-one-out approach on data from four different clinical sites showed broad generalization performance of the top performing ML model with AUC of 0.91, 0.91, 0.95, and 0.91. Interpretation: A novel ML model that incorporates all the longitudinal DI scores prior to an AE in a hospitalized patient performs better at outcome prediction than the currently used threshold model. The use of clinical data, a generalized ML technique, and successful multisite cross-validation demonstrate the feasibility of our model in clinical implementation. Funding: No funding to report.

Risk Prediction Model for 6-Month Mortality for Patients Discharged to Skilled Nursing Facilities.

OBJECTIVE: Hospitalized patients discharged to skilled nursing facilities (SNFs) for post-acute care are at high risk for adverse outcomes. Yet, absence of effective prognostic tools hinders optimal care planning and decision making. Our objective was to develop and validate a risk prediction model for 6-month all-cause death among hospitalized patients discharged to SNFs. DESIGN: Retrospective cohort study. SETTING AND PARTICIPANTS: Patients discharged from 1 of 2 hospitals to 1 of 10 SNFs for post-acute care in an integrated health care delivery system between January 1, 2009, and December 31, 2016. METHODS: Gradient-boosting machine modeling was used to predict all-cause death within 180 days of hospital discharge with use of patient demographic characteristics, comorbidities, pattern of prior health care use, and clinical parameters from the index hospitalization. Area under the receiver operating characteristic curve (AUC) was assessed for out-of-sample observations under 10-fold cross-validation. RESULTS: We identified 9803 unique patients with 11,647 hospital-to-SNF discharges [mean (SD) age, 80.72 (9.71) years; female sex, 61.4%]. These discharges involved 9803 patients alive at 180 days and 1844 patients who died between day 1 and day 180 of discharge. Age, comorbid burden, health care use in prior 6 months, abnormal laboratory parameters, and mobility status during hospital stay were the most important predictors of 6-month death (model AUC, 0.82). CONCLUSION AND IMPLICATIONS: We derived a robust prediction model with parameters available at discharge to SNFs to calculate risk of death within 6 months. This work may be useful to guide other clinicians wishing to develop mortality prediction instruments specific to their post-acute SNF populations.

Receptivity to a Nurse-Led Symptom Management Intervention Among Highly Symptomatic Patients With Cancer.

BACKGROUND: The symptom burden associated with cancer and its treatment can negatively affect patients' quality of life and survival. Symptom-focused collaborative care model (CCM) interventions can improve outcomes, but only if patients engage with them. We assessed the receptivity of severely symptomatic oncology patients to a remote nurse-led CCM intervention. METHODS: In a pragmatic, cluster-randomized, stepped-wedge trial conducted as part of the National Cancer Institute IMPACT Consortium (E2C2, NCT03892967), patients receiving cancer care were asked to rate their sleep disturbance, pain, anxiety, emotional distress, fatigue, and limitations in physical function. Patients reporting at least 1 severe symptom (≥7/10) were offered phone consultation with a nurse symptom care manager (RN SCM). Initially, patients had to "opt-in" to receive a call, but the protocol was later modified so they had to "opt-out" if they did not want a call. We assessed the impact of opt-in vs opt-out framing and patient characteristics on receptiveness to RN SCM calls. All statistical tests were 2-sided. RESULTS: Of the 1204 symptom assessments (from 864 patients) on which at least 1 severe symptom was documented, 469 (39.0%) indicated receptivity to an RN SCM phone call. The opt-out period (odds ratio [OR] = 1.61, 95% confidence interval [CI] = 1.12 to 2.32, P = .01), receiving care at a tertiary care center (OR = 3.59, 95% CI = 2.18 to 5.91, P < .001), and having severe pain (OR = 1.80, 95% CI = 1.24 to 2.62, P = .002) were associated with statistically significantly greater willingness to receive a call. CONCLUSIONS: Many severely symptomatic patients were not receptive to an RN SCM phone call. Better understanding of reasons for refusal and strategies for improving patient receptivity are needed.

Use of a Computerized Algorithm to Evaluate the Proportion and Causes of Potentially Preventable Readmissions Among Patients Discharged to Skilled Nursing Facilities.

OBJECTIVES: Older patients discharged to skilled nursing facilities (SNFs) for post-acute care are at high risk for hospital readmission. Yet, as in the community setting, some readmissions may be preventable with optimal transitional care. This study examined the proportion of 30-day hospital readmissions from SNFs that could be considered potentially preventable readmissions (PPRs) and evaluated the reasons for these readmissions. DESIGN: Retrospective cohort study. SETTING AND PARTICIPANTS: Post-acute practice of an integrated health care delivery system serving 11 SNFs in the US Midwest. Patients discharged from the hospital to an SNF and subsequently readmitted to the hospital within 30 days from January 1, 2009, through November 31, 2016. METHODS: A computerized algorithm evaluated the relationship between initial and repeat hospitalizations to determine whether the repeat hospitalization was a PPR. We assessed for changes in PPR rates across the system over the study period and evaluated the readmission categories to identify the most prevalent PPR categories. RESULTS: Of 11,976 discharges to SNFs for post-acute care among 8041 patients over the study period, 16.6% resulted in rehospitalization within 30 days, and 64.8% of these rehospitalizations were considered PPRs. Annual proportion of PPRs ranged from 58.2% to 66.4% [mean (standard deviation) 0.65 (0.03); 95% confidence interval CI 0.63-0.67; P = .36], with no discernable trend. Nearly one-half (46.2%) of all 30-day readmissions were classified as potentially preventable medical readmissions related to recurrence or continuation of the reason for initial admission or to complications from the initial hospitalization. CONCLUSIONS AND IMPLICATIONS: For this cohort of patients discharged to SNFs, a computerized algorithm categorized a large proportion of 30-day hospital readmissions as potentially preventable, with nearly one-half of those linked to the reason for the initial hospitalization. These findings indicate the importance of improvement in postdischarge transitional care for patients discharged to SNFs.

Pragmatic cluster randomized trial to evaluate effectiveness and implementation of enhanced EHR-facilitated cancer symptom control (E2C2).

BACKGROUND: The prevalence of inadequate symptom control among cancer patients is quite high despite the availability of definitive care guidelines and accurate and efficient assessment tools. METHODS: We will conduct a hybrid type 2 stepped wedge pragmatic cluster randomized clinical trial to evaluate a guideline-informed enhanced, electronic health record (EHR)-facilitated cancer symptom control (E2C2) care model. Teams of clinicians at five hospitals that care for patients with various cancers will be randomly assigned in steps to the E2C2 intervention. The E2C2 intervention will have two levels of care: level 1 will offer low-touch, automated self-management support for patients reporting moderate sleep disturbance, pain, anxiety, depression, and energy deficit symptoms or limitations in physical function (or both). Level 2 will offer nurse-managed collaborative care for patients reporting more intense (severe) symptoms or functional limitations (or both). By surveying and interviewing clinical staff, we will also evaluate whether the use of a multifaceted, evidence-based implementation strategy to support adoption and use of the E2C2 technologies improves patient and clinical outcomes. Finally, we will conduct a mixed methods evaluation to identify disparities in the adoption and implementation of the E2C2 intervention among elderly and rural-dwelling patients with cancer. DISCUSSION: The E2C2 intervention offers a pragmatic, scalable approach to delivering guideline-based symptom and function management for cancer patients. Since discrete EHR-imbedded algorithms drive defining aspects of the intervention, the approach can be efficiently disseminated and updated by specifying and modifying these centralized EHR algorithms. TRIAL REGISTRATION: ClinicalTrials.gov, NCT03892967. Registered on 25 March 2019.

Associations of Skilled Nursing Facility Quality Ratings With 30-Day Rehospitalizations and Emergency Department Visits.

Skilled nursing facilities (SNFs) increasingly provide care to patients after hospitalization. The Centers for Medicare & Medicaid Services reports ratings for SNFs for overall quality, staffing, health inspections, and clinical quality measures. However, the relationship between these ratings and patient outcomes remains unclear. In this retrospective cohort study, we reviewed the electronic health records of 3,923 adult patients discharged from the hospital and admitted to 9 SNFs served by a health care delivery system. We used Cox proportional hazards models to examine associations between the overall quality and individual ratings and our primary outcomes of 30-day rehospitalizations and 30-day emergency department visits. Patients in higher-rated facilities had a 13% lower risk of 30-day rehospitalization than patients in lower-rated facilities (hazard ratio, 0.87; 95% CI, 0.76-0.99). The risk of emergency department visits was also lower for patients in facilities with a higher overall quality rating and a higher quality measures rating. Staffing and health inspection ratings were not associated with our primary outcomes. These findings may help inform providers and nursing home policy makers.

Risk of 30-Day Hospital Readmission Among Patients Discharged to Skilled Nursing Facilities: Development and Validation of a Risk-Prediction Model.

OBJECTIVES: Patients discharged to a skilled nursing facility (SNF) for post-acute care have a high risk of hospital readmission. We aimed to develop and validate a risk-prediction model to prospectively quantify the risk of 30-day hospital readmission at the time of discharge to a SNF. DESIGN: Retrospective cohort study. SETTING: Ten independent SNFs affiliated with the post-acute care practice of an integrated health care delivery system. PARTICIPANTS: We evaluated 6032 patients who were discharged to SNFs for post-acute care after hospitalization. MEASUREMENTS: The primary outcome was all-cause 30-day hospital readmission. Patient demographics, medical comorbidity, prior use of health care, and clinical parameters during the index hospitalization were analyzed by using gradient boosting machine multivariable analysis to build a predictive model for 30-day hospital readmission. Area under the receiver operating characteristic curve (AUC) was assessed on out-of-sample observations under 10-fold cross-validation. RESULTS: Among 8616 discharges to SNFs from January 1, 2009, through June 30, 2014, a total of 1568 (18.2%) were readmitted to the hospital within 30 days. The 30-day hospital readmission prediction model had an AUC of 0.69, a 16% improvement over risk assessment using the Charlson Comorbidity Index alone. The final model included length of stay, abnormal laboratory parameters, and need for intensive care during the index hospitalization; comorbid status; and number of emergency department and hospital visits within the preceding 6 months. CONCLUSIONS AND IMPLICATIONS: We developed and validated a risk-prediction model for 30-day hospital readmission in patients discharged to a SNF for post-acute care. This prediction tool can be used to risk stratify the complex population of hospitalized patients who are discharged to SNFs to prioritize interventions and potentially improve the quality, safety, and cost-effectiveness of care.

Care Transitions Program for High-Risk Frail Older Adults is Most Beneficial for Patients with Cognitive Impairment.

BACKGROUND: Although posthospitalization care transitions programs (CTP) are highly diverse, their overall program thoroughness is most predictive of their success. OBJECTIVE: To identify components of a successful homebased CTP and patient characteristics that are most predictive of reduced 30-day readmissions. DESIGN: Retrospective cohort. PATIENTS: A total of 315 community-dwelling, hospitalized, older adults (≥60 years) at high risk for readmission (Elder Risk Assessment score ≥16), discharged home over the period of January 1, 2011 to June 30, 2013. SETTING: Midwest primary care practice in an integrated health system. INTERVENTION: Enrollment in a CTP during acute hospitalization. MEASUREMENTS: The primary outcome was all-cause readmission within 30 days of the first CTP evaluation. Logistic regression was used to examine independent variables, including patient demographics, comorbidities, number of medications, completion, and timing of program fidelity measures, and prior utilization of healthcare. RESULTS: The overall 30-day readmission rate was 17.1%. The intensity of follow-up varied among patients, with 17.1% and 50.8% of the patients requiring one and ≥3 home visits, respectively, within 30 days. More than half (54.6%) required visits beyond 30 days. Compared with patients who were not readmitted, readmitted patients were less likely to exhibit cognitive impairment (29.6% vs 46.0%; P = .03) and were more likely to have high medication use (59.3% vs 44.4%; P = .047), more emergency department (ED; 0.8 vs 0.4; P = .03) and primary care visits (4.0 vs 3.0; P = .018), and longer cumulative time in the hospital (4.6 vs 2.5 days; P = .03) within 180 days of the index hospitalization. Multivariable analysis indicated that only cognitive impairment and previous ED visits were important predictors of readmission. CONCLUSIONS: No single CTP component reliably predicted reduced readmission risk. Patients with cognitive impairment and polypharmacy derived the most benefit from the program.

Short-term and long-term effectiveness of a post-hospital care transitions program in an older, medically complex population.

BACKGROUND: Care transition programs can potentially reduce 30 day readmission; however, the effect on long-term hospital readmissions is still unclear. OBJECTIVE: We compared short-term (30 day) and long-term (180 day) utilization of participants enrolled in care transitions versus those matched referents eligible but not enrolled. DESIGN: This cohort study was conducted from January 1, 2011 until June 30, 2013 within a primary care academic practice. PARTICIPANTS: Patients at high risk for hospital readmission based on age and comorbid health conditions had participated in care transitions group (cases) or usual care (referent). MAIN MEASURES: The primary outcomes were 30, 90, and 180 day hospital readmissions.. Secondary outcomes included: mortality; emergency room visits and days; combined rehospitalizations and emergency room visits; and total intensive care unit days. Cox proportional hazard models using propensity score matching were used to assess rehospitalization, emergency room visits and mortality. Poisson regression models were used to compare the numbers of hospital days. KEY RESULTS: Compared to referent (n=365), Mayo Clinic Care Transitions patients exhibited a lower 30 day rehospitalization rate compared to referent; 12.4% (95% CI 8.9-15.7) versus 20.1% (95% CI 15.8-24.1%), respectively (P=0.002). At 180-days, there was no difference in rehospitalization between transitions and referent; 39.9% (95% CI 34.6-44.9%) versus 44.8% (95% CI 39.4-49.8%), (P=0.07). CONCLUSION: We observed a reduction in 30 day rehospitalization rates among those enrolled in care transitions compared to referent. However, this effect was not sustained at 180 days. More work is needed to identify how the intervention can be sustained beyond 30 days.

Incorporating the Last Four Digits of Social Security Numbers Substantially Improves Linking Patient Data from De-identified Hospital Claims Databases.

OBJECTIVE: Assess algorithms for linking patients across de-identified databases without compromising confidentiality. DATA SOURCES/STUDY SETTING: Hospital discharges from 11 Mayo Clinic hospitals during January 2008-September 2012 (assessment and validation data). Minnesota death certificates and hospital discharges from 2009 to 2012 for entire state (application data). STUDY DESIGN: Cross-sectional assessment of sensitivity and positive predictive value (PPV) for four linking algorithms tested by identifying readmissions and posthospital mortality on the assessment data with application to statewide data. DATA COLLECTION/EXTRACTION METHODS: De-identified claims included patient gender, birthdate, and zip code. Assessment records were matched with institutional sources containing unique identifiers and the last four digits of Social Security number (SSNL4). PRINCIPAL FINDINGS: Gender, birthdate, and five-digit zip code identified readmissions with a sensitivity of 98.0 percent and a PPV of 97.7 percent and identified postdischarge mortality with 84.4 percent sensitivity and 98.9 percent PPV. Inclusion of SSNL4 produced nearly perfect identification of readmissions and deaths. When applied statewide, regions bordering states with unavailable hospital discharge data had lower rates. CONCLUSION: Addition of SSNL4 to administrative data, accompanied by appropriate data use and data release policies, can enable trusted repositories to link data with nearly perfect accuracy without compromising patient confidentiality. States maintaining centralized de-identified databases should add SSNL4 to data specifications.

Health care outcomes and advance care planning in older adults who receive home-based palliative care: a pilot cohort study.

BACKGROUND: Approximately 20% of seniors live with five or more chronic medical illnesses. Terminal stages of their lives are often characterized by repeated burdensome hospitalizations and advance care directives are insufficiently addressed. This study reports on the preliminary results of a Palliative Care Homebound Program (PCHP) at the Mayo Clinic in Rochester, Minnesota to service these vulnerable populations. OBJECTIVE: The study objective was to evaluate inpatient hospital utilization and the adequacy of advance care planning in patients who receive home-based palliative care. METHODS: This is a retrospective pilot cohort study of patients enrolled in the PCHP between September 2012 and March 2013. Two control patients were matched to each intervention patient by propensity scoring methods that factor in risk and prognosis. Primary outcomes were six-month hospital utilization including ER visits. Secondary outcomes evaluated advance care directive completion and overall mortality. RESULTS: Patients enrolled in the PCHP group (n = 54) were matched to 108 controls with an average age of 87 years. Ninety-two percent of controls and 33% of PCHP patients were admitted to the hospital at least once. The average number of hospital admissions was 1.36 per patient for controls versus 0.35 in the PCHP (p < 0.001). Total hospital days were reduced by 5.13 days. There was no difference between rates of ER visits. Advanced care directive were completed more often in the intervention group (98%) as compared to controls (31%), with p < 0.001. Goals of care discussions were held at least once for all patients in the PCHP group, compared to 41% in the controls.

Future complexity of care tier affected by depression outcomes.

BACKGROUND: The inclusion of mental health issues in the evaluation of multimorbidity generally has been as the presence or absence of the condition rather than severity, complexity, or stage. The hypothesis for this study was that clinical outcome of the depression 6 months after enrollment into collaborative care management would have a role in predicting future complexity of care tier. METHODS: This study was a retrospective chart review of 1894 primary care patients who were diagnosed with major depressive disorder or dysthymia as of December 2012. Multiple logistic regression analysis was used to test the independent associations between each variable and the odds of being included in the higher tiers (HT) group. RESULTS: Age (odds ratio [OR] = 1.022, confidence interval [CI] = 1.013-1.030, P < .001), female gender (OR = 1.380, CI = 1.020-1.868, P = .037), being married (OR = 0.730, CI = 0.563-0.947, P = .018), and the presence of comorbidities (1, OR = 1.986, CI = 1.485-2.656, P < .001; ≥ 2, OR = 4.678, CI = 3.242-6.750, P < .001) were independently associated with future HT levels. The presence of persistent depressive symptoms (PHQ-9 ≥ 10) at 6 months conferred 2.280 (CI = 1.673-3.107, P < .001) times likely odds of HT level compared with clinical remission at 6 months. CONCLUSION: Patients with the diagnosis of major depression or dysthymia had greater odds of complex tier levels in the future, if depression was not treated to remission by 6 months. This study demonstrated the importance of the goal of significant improvement (ie, remission) of depression symptoms by 6 months (especially those older patients with more comorbidity) from entering into the higher complexity tiers.