Comparing Hospital Processes and Outcomes in California Medicare Beneficiaries: Simulation Prompts Reconsideration.

INTRODUCTION: This article is not a traditional research report. It describes how conducting a specific set of benchmarking analyses led us to broader reflections on hospital benchmarking. We reexamined an issue that has received far less attention from researchers than in the past: How variations in the hospital admission threshold might affect hospital rankings. Considering this threshold made us reconsider what benchmarking is and what future benchmarking studies might be like. Although we recognize that some of our assertions are speculative, they are based on our reading of the literature and previous and ongoing data analyses being conducted in our research unit. We describe the benchmarking analyses that led to these reflections. OBJECTIVES: The Centers for Medicare and Medicaid Services' Hospital Compare Web site includes data on fee-for-service Medicare beneficiaries but does not control for severity of illness, which requires physiologic data now available in most electronic medical records.To address this limitation, we compared hospital processes and outcomes among Kaiser Permanente Northern California's (KPNC) Medicare Advantage beneficiaries and non-KPNC California Medicare beneficiaries between 2009 and 2010. METHODS: We assigned a simulated severity of illness measure to each record and explored the effect of having the additional information on outcomes. RESULTS: We found that if the admission severity of illness in non-KPNC hospitals increased, KPNC hospitals' mortality performance would appear worse; conversely, if admission severity at non-KPNC hospitals' decreased, KPNC hospitals' performance would appear better. CONCLUSION: Future hospital benchmarking should consider the impact of variation in admission thresholds.

Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals.

Patients who deteriorate in the hospital outside the intensive care unit (ICU) have higher mortality and morbidity than those admitted directly to the ICU. As more hospitals deploy comprehensive inpatient electronic medical records (EMRs), attempts to support rapid response teams with automated early detection systems are becoming more frequent. We aimed to describe some of the technical and operational challenges involved in the deployment of an early detection system. This 2-hospital pilot, set within an integrated healthcare delivery system with 21 hospitals, had 2 objectives. First, it aimed to demonstrate that severity scores and probability estimates could be provided to hospitalists in real time. Second, it aimed to surface issues that would need to be addressed so that deployment of the early warning system could occur in all remaining hospitals. To achieve these objectives, we first established a rationale for the development of an early detection system through the analysis of risk-adjusted outcomes. We then demonstrated that EMR data could be employed to predict deteriorations. After addressing specific organizational mandates (eg, defining the clinical response to a probability estimate), we instantiated a set of equations into a Java application that transmits scores and probability estimates so that they are visible in a commercially available EMR every 6 hours. The pilot has been successful and deployment to the remaining hospitals has begun. Journal of Hospital Medicine 2016;11:S18-S24. © 2016 Society of Hospital Medicine.

Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU.

BACKGROUND: Patients in general medical-surgical wards who experience unplanned transfer to the intensive care unit (ICU) show evidence of physiologic derangement 6-24h prior to their deterioration. With increasing availability of electronic medical records (EMRs), automated early warning scores (EWSs) are becoming feasible. OBJECTIVE: To describe the development and performance of an automated EWS based on EMR data. MATERIALS AND METHODS: We used a discrete-time logistic regression model to obtain an hourly risk score to predict unplanned transfer to the ICU within the next 12h. The model was based on hospitalization episodes from all adult patients (18years) admitted to 21 Kaiser Permanente Northern California (KPNC) hospitals from 1/1/2010 to 12/31/2013. Eligible patients met these entry criteria: initial hospitalization occurred at a KPNC hospital; the hospitalization was not for childbirth; and the EMR had been operational at the hospital for at least 3months. We evaluated the performance of this risk score, called Advanced Alert Monitor (AAM) and compared it against two other EWSs (eCART and NEWS) in terms of their sensitivity, specificity, negative predictive value, positive predictive value, and area under the receiver operator characteristic curve (c statistic). RESULTS: A total of 649,418 hospitalization episodes involving 374,838 patients met inclusion criteria, with 19,153 of the episodes experiencing at least one outcome. The analysis data set had 48,723,248 hourly observations. Predictors included physiologic data (laboratory tests and vital signs); neurological status; severity of illness and longitudinal comorbidity indices; care directives; and health services indicators (e.g. elapsed length of stay). AAM showed better performance compared to NEWS and eCART in all the metrics and prediction intervals. The AAM AUC was 0.82 compared to 0.79 and 0.76 for eCART and NEWS, respectively. Using a threshold that generated 1 alert per day in a unit with a patient census of 35, the sensitivity of AAM was 49% (95% CI: 47.6-50.3%) compared to the sensitivities of eCART and NEWS scores of 44% (42.3-45.1) and 40% (38.2-40.9), respectively. For all three scores, about half of alerts occurred within 12h of the event, and almost two thirds within 24h of the event. CONCLUSION: The AAM score is an example of a score that takes advantage of multiple data streams now available in modern EMRs. It highlights the ability to harness complex algorithms to maximize signal extraction. The main challenge in the future is to develop detection approaches for patients in whom data are sparser because their baseline risk is lower.

Evaluation Following Staggered Implementation of the "Rethinking Critical Care" ICU Care Bundle in a Multicenter Community Setting.

OBJECTIVES: To evaluate process metrics and outcomes after implementation of the "Rethinking Critical Care" ICU care bundle in a community setting. DESIGN: Retrospective interrupted time-series analysis. SETTING: Three hospitals in the Kaiser Permanente Northern California integrated healthcare delivery system. PATIENTS: ICU patients admitted between January 1, 2009, and August 30, 2013. INTERVENTIONS: Implementation of the Rethinking Critical Care ICU care bundle which is designed to reduce potentially preventable complications by focusing on the management of delirium, sedation, mechanical ventilation, mobility, ambulation, and coordinated care. Rethinking Critical Care implementation occurred in a staggered fashion between October 2011 and November 2012. MEASUREMENTS AND MAIN RESULTS: We measured implementation metrics based on electronic medical record data and evaluated the impact of implementation on mortality with multivariable regression models for 24,886 first ICU episodes in 19,872 patients. After implementation, some process metrics (e.g., ventilation start and stop times) were achieved at high rates, whereas others (e.g., ambulation distance), available late in the study period, showed steep increases in compliance. Unadjusted mortality decreased from 12.3% to 10.9% (p < 0.01) before and after implementation, respectively. The adjusted odds ratio for hospital mortality after implementation was 0.85 (95% CI, 0.73-0.99) and for 30-day mortality was 0.88 (95% CI, 0.80-0.97) compared with before implementation. However, the mortality rate trends were not significantly different before and after Rethinking Critical Care implementation. The mean duration of mechanical ventilation and hospital stay also did not demonstrate incrementally greater declines after implementation. CONCLUSIONS: Rethinking Critical Care implementation was associated with changes in practice and a 12-15% reduction in the odds of short-term mortality. However, these findings may represent an evaluation of changes in practices and outcomes still in the midimplementation phase and cannot be directly attributed to the elements of bundle implementation.

An electronic order set for acute myocardial infarction is associated with improved patient outcomes through better adherence to clinical practice guidelines.

BACKGROUND: Adherence to evidence-based recommendations for acute myocardial infarction (AMI) remains unsatisfactory. OBJECTIVE: Quantifying association between using an electronic AMI order set (AMI-OS) and hospital processes and outcomes. DESIGN: Retrospective cohort study. SETTING: Twenty-one community hospitals. PATIENTS: A total of 5879 AMI patients were hospitalized between September 28, 2008 and December 31, 2010. MEASUREMENTS: We ascertained whether patients were treated using the AMI-OS or individual orders (a la carte). Dependent process variables were use of evidence-based care; outcome variables were mortality and rehospitalization. RESULTS: Use of individual and combined therapies improved outcomes (eg, 50% lower odds of 30-day mortality for patients with ≥3 therapies). The 3531 patients treated using the AMI-OS were more likely to receive evidence-based therapies (eg, 50% received 5 different therapies vs 36% a la carte). These patients had lower 30-day mortality (5.7% vs 8.5%) than the 2348 treated using a la carte orders. Although AMI-OS patients' predicted mortality risk was lower (3.2%) than that of a la carte patients (4.8%), the association of improved processes and outcomes with the use of the AMI-OS persisted after risk adjustment. For example, after inverse probability weighting, the relative risk for inpatient mortality in the AMI-OS group was 0.67 (95% confidence interval: 0.52-0.86). Inclusion of use of recommended therapies in risk adjustment eliminated the benefit of the AMI-OS, highlighting its mediating effect on adherence to evidence-based treatment. CONCLUSIONS: Use of an electronic order set is associated with increased adherence to evidence-based care and better AMI outcomes.

Stratification of risk of early-onset sepsis in newborns ≥ 34 weeks' gestation.

OBJECTIVE: To define a quantitative stratification algorithm for the risk of early-onset sepsis (EOS) in newborns ≥ 34 weeks' gestation. METHODS: We conducted a retrospective nested case-control study that used split validation. Data collected on each infant included sepsis risk at birth based on objective maternal factors, demographics, specific clinical milestones, and vital signs during the first 24 hours after birth. Using a combination of recursive partitioning and logistic regression, we developed a risk classification scheme for EOS on the derivation dataset. This scheme was then applied to the validation dataset. RESULTS: Using a base population of 608,014 live births ≥ 34 weeks' gestation at 14 hospitals between 1993 and 2007, we identified all 350 EOS cases <72 hours of age and frequency matched them by hospital and year of birth to 1063 controls. Using maternal and neonatal data, we defined a risk stratification scheme that divided the neonatal population into 3 groups: treat empirically (4.1% of all live births, 60.8% of all EOS cases, sepsis incidence of 8.4/1000 live births), observe and evaluate (11.1% of births, 23.4% of cases, 1.2/1000), and continued observation (84.8% of births, 15.7% of cases, incidence 0.11/1000). CONCLUSIONS: It is possible to combine objective maternal data with evolving objective neonatal clinical findings to define more efficient strategies for the evaluation and treatment of EOS in term and late preterm infants. Judicious application of our scheme could result in decreased antibiotic treatment in 80,000 to 240,000 US newborns each year.

Automated identification of pneumonia in chest radiograph reports in critically ill patients.

BACKGROUND: Prior studies demonstrate the suitability of natural language processing (NLP) for identifying pneumonia in chest radiograph (CXR) reports, however, few evaluate this approach in intensive care unit (ICU) patients. METHODS: From a total of 194,615 ICU reports, we empirically developed a lexicon to categorize pneumonia-relevant terms and uncertainty profiles. We encoded lexicon items into unique queries within an NLP software application and designed an algorithm to assign automated interpretations ('positive', 'possible', or 'negative') based on each report's query profile. We evaluated algorithm performance in a sample of 2,466 CXR reports interpreted by physician consensus and in two ICU patient subgroups including those admitted for pneumonia and for rheumatologic/endocrine diagnoses. RESULTS: Most reports were deemed 'negative' (51.8%) by physician consensus. Many were 'possible' (41.7%); only 6.5% were 'positive' for pneumonia. The lexicon included 105 terms and uncertainty profiles that were encoded into 31 NLP queries. Queries identified 534,322 'hits' in the full sample, with 2.7 ± 2.6 'hits' per report. An algorithm, comprised of twenty rules and probability steps, assigned interpretations to reports based on query profiles. In the validation set, the algorithm had 92.7% sensitivity, 91.1% specificity, 93.3% positive predictive value, and 90.3% negative predictive value for differentiating 'negative' from 'positive'/'possible' reports. In the ICU subgroups, the algorithm also demonstrated good performance, misclassifying few reports (5.8%). CONCLUSIONS: Many CXR reports in ICU patients demonstrate frank uncertainty regarding a pneumonia diagnosis. This electronic tool demonstrates promise for assigning automated interpretations to CXR reports by leveraging both terms and uncertainty profiles.

An electronic Simplified Acute Physiology Score-based risk adjustment score for critical illness in an integrated healthcare system.

OBJECTIVE: Risk adjustment is essential in evaluating the performance of an ICU; however, assigning scores is time-consuming. We sought to create an automated ICU risk adjustment score, based on the Simplified Acute Physiology Score 3, using only data available within the electronic medical record (Kaiser Permanente HealthConnect). DESIGN, SETTING, AND PATIENTS: The eSimplified Acute Physiology Score 3 was developed by adapting Kaiser Permanente HealthConnect structured data to Simplified Acute Physiology Score 3 criteria. The model was tested among 67,889 first-time ICU admissions at 21 hospitals between 2007 and 2011 to predict hospital mortality. Model performance was evaluated using published Simplified Acute Physiology Score 3 global and North American coefficients; a first-level customized version of the eSimplified Acute Physiology Score 3 was also developed in a 40% derivation cohort and tested in a 60% validation cohort. MEASUREMENTS: Electronic variables were considered "directly" available if they could be mapped exactly within Kaiser Permanente HealthConnect; they were considered "adapted" if no exact electronic corollary was identified. Model discrimination was evaluated with area under receiver operating characteristic curves; calibration was assessed using Hosmer-Lemeshow goodness-of-fit tests. MAIN RESULTS: Mean age at ICU admission was 65 ± 17 yrs. Mortality in the ICU was 6.2%; total in-hospital mortality was 11.2%. The majority of Simplified Acute Physiology Score 3 variables were considered "directly" available; others required adaptation based on diagnosis coding, medication records, or procedure tables. Mean eSimplified Acute Physiology Score 3 scores were 45 ± 13. Using published Simplified Acute Physiology Score 3 global and North American coefficients, the eSimplified Acute Physiology Score 3 demonstrated good discrimination (area under the receiver operating characteristic curve, 0.80-0.81); however, it overpredicted mortality. The customized eSimplified Acute Physiology Score 3 score demonstrated good discrimination (area under the receiver operating characteristic curve, 0.82) and calibration (Hosmer-Lemeshow goodness-of-fit chi-square p = 0.57) in the validation cohort. The eSimplified Acute Physiology Score 3 demonstrated stable performance when cohorts were limited to specific hospitals and years. CONCLUSIONS: The customized eSimplified Acute Physiology Score 3 shows good potential for providing automated risk adjustment in the intensive care unit.

Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record.

BACKGROUND: Ward patients who experience unplanned transfer to intensive care units have excess morbidity and mortality. OBJECTIVE: To develop a predictive model for prediction of unplanned transfer from the medical-surgical ward to intensive care (or death on the ward in a patient who was "full code") using data from a comprehensive inpatient electronic medical record (EMR). DESIGN: Retrospective case-control study; unit of analysis was a 12-hour patient shift. Shifts where a patient experienced an unplanned transfer were event shifts; shifts without a transfer were comparison shifts. Hospitalization records were transformed into 12-hour shift records, with 10 randomly selected comparison shifts identified for each event shift. Analysis employed logistic regression and split validation. SETTING: Integrated healthcare delivery system in Northern California. PATIENTS: Hospitalized adults at 14 hospitals with comprehensive inpatient EMRs. MEASUREMENTS: Predictors included vital signs, laboratory test results, severity of illness scores, longitudinal chronic illness burden scores, transpired hospital length of stay, and care directives. Patients were also given a retrospective, electronically (not manually assigned) Modified Early Warning Score, or MEWS(re). Outcomes were transfer to the intensive care unit (ICU) from the ward or transitional care unit, or death outside the ICU among patients who were "full code". RESULTS: We identified 4,036 events and 39,782 comparison shifts from a cohort of 102,422 patients' hospitalizations. The MEWS(re) had a c-statistic of 0.709 in the derivation and 0.698 in the validation dataset; corresponding values for the EMR-based model were 0.845 and 0.775. LIMITATIONS: Using these algorithms requires hospitals with comprehensive inpatient EMRs and longitudinal data. CONCLUSIONS: EMR-based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems.

The association between sepsis and potential medical injury among hospitalized patients.

BACKGROUND: Patient safety remains a national priority, but the role of disease-specific characteristics in safety is not well characterized. METHODS: We identified potentially preventable medical injuries using patient safety indicators (PSIs) and annual data from the Nationwide Inpatient Sample between 2003 and 2007. We compared the rate of selected PSIs among patients hospitalized with and without sepsis. Among patients with sepsis, we also compared PSI rates across severity strata. Using multivariable case-control matching and regression analyses, we estimated the excess adverse outcomes associated with PSI events in patients with sepsis. RESULTS: Patients hospitalized with sepsis accounted for 2% to 4% of hospital discharges; however, they accounted for 9% to 26% of all potential medical injuries. PSI rates varied considerably; among patients hospitalized for sepsis, they were lowest for accidental puncture or laceration and highest for postoperative respiratory failure. Nearly all PSI rates were higher among patients with sepsis compared with patients without sepsis. Among those with sepsis, most PSI rates increased as sepsis severity increased. Compared with matched sepsis control subjects, increased length of stay and hospital charges were associated with PSI events in sepsis cases. However, only decubitus ulcer, iatrogenic pneumothorax, and postoperative metabolic and physiologic derangement or respiratory failure were associated with excess mortality. CONCLUSION: Patients hospitalized for sepsis, compared with the general hospital population, were at a substantially increased risk of potential medical injury; their risk rose as disease severity increased. Future patient safety efforts may benefit from focusing on medically vulnerable populations.