Development and Performance of Electronic Pediatric Risk of Mortality and Pediatric Logistic Organ Dysfunction-2 Automated Acuity Scores.

OBJECTIVES: Develop and test the performance of electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV and electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 scores. DESIGN: Retrospective, single-center cohort derived from structured electronic health record data. SETTING: Large, quaternary PICU at a freestanding, university-affiliated children's hospital. PATIENTS: All encounters with a PICU admission between January 1, 2009, and December 31, 2017, identified using electronic definitions of inpatient encounter. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: The main outcome was predictive validity of each score for hospital mortality, assessed as model discrimination and calibration. Discrimination was examined with the area under the receiver operating characteristics curve and the area under the precision-recall curve. Calibration was assessed with the Hosmer-Lemeshow goodness of fit test and calculation of a standardized mortality ratio. Models were recalibrated with new regression coefficients in a training subset of 75% of encounters selected randomly from all years of the cohort and the calibrated models were tested in the remaining 25% of the cohort. Content validity was assessed by examining correlation between electronic versions of the scores and prospectively calculated data (electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV) and an alternative informatics approach (Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score). The cohort included 21,335 encounters. Correlation coefficients indicated strong agreement between different methods of score calculation. Uncalibrated area under the receiver operating characteristics curves were 0.96 (95% CI, 0.95-0.97) for electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score and 0.87 (95% CI, 0.85-0.89) for electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV for inpatient mortality. The uncalibrated electronic version of the Children's Hospital of Pittsburgh Pediatric Risk of Mortality-IV standardized mortality ratio was 0.63 (0.59-0.66), demonstrating strong agreement with previous, prospective evaluation at the study center. The uncalibrated electronic version of the Children's Hospital of Pittsburgh Pediatric Logistic Organ Dysfunction-2 score standardized mortality ratio was 0.20 (0.18-0.21). All models required recalibrating (all Hosmer-Lemeshow goodness-of-fit, p < 0.001) and subsequently demonstrated acceptable goodness-of-fit when examined in a test subset (n = 5,334) of the cohort. CONCLUSIONS: Electronically derived intensive care acuity scores demonstrate very good to excellent discrimination and can be calibrated to institutional outcomes. This approach can facilitate both performance improvement and research initiatives and may offer a scalable strategy for comparison of interinstitutional PICU outcomes.

Early prediction of critical events for infants with single-ventricle physiology in critical care using routinely collected data.

OBJECTIVE: Critical events are common and difficult to predict among infants with congenital heart disease and are associated with mortality and long-term sequelae. We aimed to achieve early prediction of critical events, that is, cardiopulmonary resuscitation, emergency endotracheal intubation, and extracorporeal membrane oxygenation in infants with single-ventricle physiology before second-stage surgery. We hypothesized that naïve Bayesian models learned from expert knowledge and clinical data can predict critical events early and accurately. METHODS: We collected 93 patients with single-ventricle physiology admitted to intensive care units in a single tertiary pediatric hospital between 2014 and 2017. Using knowledge elicited from experienced cardiac-intensive-care-unit providers and machine-learning techniques, we developed and evaluated the Cardiac-intensive-care Warning INdex (C-WIN) system, consisting of a set of naïve Bayesian models that leverage routinely collected data. We evaluated predictive performance using the area under the receiver operating characteristic curve, sensitivity, and specificity. We performed the evaluation at 5 different prediction horizons: 1, 2, 4, 6, and 8 hours before the onset of critical events. RESULTS: The area under the receiver operating characteristic curves of the C-WIN models ranged between 0.73 and 0.88 at different prediction horizons. At 1 hour before critical events, C-WIN was able to detect events with an area under the receiver operating characteristic curve of 0.88 (95% confidence interval, 0.84-0.92) and a sensitivity of 84% at the 81% specificity level. CONCLUSIONS: Predictive models may enhance clinicians' ability to identify infants with single-ventricle physiology at high risk of critical events. Early prediction of critical events may indicate the need to perform timely interventions, potentially reducing morbidity, mortality, and health care costs.