ABSTRACT
BACKGROUND: Mortality predictions calculated using scoring scales are often not accurate in populations other than those in which the scales were developed because of differences in case-mix. The present study investigates the effect of first-level customization, using a logistic regression technique, on discrimination and calibration of the Acute Physiology and Chronic Health Evaluation (APACHE) II and III scales. METHOD: Probabilities of hospital death for patients were estimated by applying APACHE II and III and comparing these with observed outcomes. Using the split sample technique, a customized model to predict outcome was developed by logistic regression. The overall goodness-of-fit of the original and the customized models was assessed. RESULTS: Of 3383 consecutive intensive care unit (ICU) admissions over 3 years, 2795 patients could be analyzed, and were split randomly into development and validation samples. The discriminative powers of APACHE II and III were unchanged by customization (areas under the receiver operating characteristic [ROC] curve 0.82 and 0.85, respectively). Hosmer-Lemeshow goodness-of-fit tests showed good calibration for APACHE II, but insufficient calibration for APACHE III. Customization improved calibration for both models, with a good fit for APACHE III as well. However, fit was different for various subgroups. CONCLUSIONS: The overall goodness-of-fit of APACHE III mortality prediction was improved significantly by customization, but uniformity of fit in different subgroups was not achieved. Therefore, application of the customized model provides no advantage, because differences in case-mix still limit comparisons of quality of care.
Subject(s)
APACHE , Critical Care/standards , Quality of Health Care , Critical Care/statistics & numerical data , Diagnosis-Related Groups , Humans , Intensive Care Units , Logistic Models , Middle Aged , Mortality , Prognosis , Reproducibility of ResultsABSTRACT
OBJECTIVE: To evaluate the ability of three scoring systems to predict hospital mortality in adult patients of an interdisciplinary intensive care unit in Germany. DESIGN: A prospective cohort study. SETTING: A mixed medical and surgical intensive care unit at a teaching hospital in Germany. PATIENTS: From a total of 3,108 patients, 2,795 patients (89.9%) for Acute Physiology and Chronic Health Evaluation (APACHE) II and 2,661 patients (85.6%) for APACHE III and Simplified Acute Physiology Score (SAPS) II could be enrolled to the study because of defined exclusion criteria. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Probabilities of hospital death for patients were estimated by applying APACHE II and III and SAPS II and compared with observed outcomes. The overall goodness-of-fit of the three models was assessed. Hospital death rates were equivalent to those predicted by APACHE II but higher than those predicted by APACHE III and SAPS II. Calibration was good for APACHE II. For the other systems, it was insufficient, but better for SAPS II than for APACHE III. The overall correct classification rate, applying a decision criterion of 50%, was 84% for APACHE II and 85% for APACHE III and SAPS II. The areas under the receiver operating characteristic curve were 0.832 for APACHE II and 0.846 for APACHE III and SAPS II. Risk estimates for surgical and medical admissions differed between the three systems. For all systems, risk predictions for diagnostic categories did not fit uniformly across the spectrum of disease categories. CONCLUSIONS: Our data more closely resemble those of the APACHE II database, demonstrating a higher degree of overall goodness-of-fit of APACHE II than APACHE III and SAPS II. Although discrimination was slightly better for the two new systems, calibration was good with a close fit for APACHE II only. Hospital mortality was higher than predicted for both new models but was underestimated to a greater degree by APACHE III. Both score systems demonstrated a considerable variation across the spectrum of diagnostic categories, which also differed between the two models.
Subject(s)
APACHE , Critical Illness/mortality , Hospital Mortality , Outcome Assessment, Health Care/standards , Adolescent , Adult , Aged , Cohort Studies , Female , Germany/epidemiology , Humans , Intensive Care Units , Length of Stay , Male , Middle Aged , Predictive Value of Tests , Prognosis , Prospective Studies , Severity of Illness IndexABSTRACT
OBJECTIVE: Scores like APACHE (Acute Physiology And Chronic Health Evaluation) were evaluated for unselected intensive care unit (ICU) admissions. Can they also be used for risk stratification and quality assurance in selected subgroups like elderly patients? METHODS: Over a 3-year period data of all admissions of a 12 bed interdisciplinary ICU were collected. APACHE II and III scores and probabilities of hospital deaths were compared with observed outcomes. The discriminatory power was evaluated by calculating the areas under the receiver operating characteristic (ROC) curves. Calibration was analyzed with standardized mortality ratios (SMR) and the Hosmer-Lemeshow goodness-of-fit statistic. RESULTS: Of 3382 admissions due to exclusion criteria, 2795 patients were analyzed, 1396 (49.9%) of these were > or = 65 years, mean age 75 (65-99) years. 62.5% were non-operative, 37.5% postoperative admissions, 35% after emergency operations. ICU mortality was 11.7%, hospital mortality 25.1%. The areas under the ROC curves were 0.77 for APACHE II and 0.79 for APACHE III (whole collective 0.83 and 0.85, respectively). The SMR was 1.17 for APACHE II and 1.23 for APACHE III compared with 1.06 and 1.22 for all patients, respectively. Calibration for elderly patients was insufficient for APACHE II (Hosmer-Lemeshow chi-square = 19, p < 0.025) as well as for APACHE III (chi-square = 41, p < 0.001), while it was good for all patients for APACHE II (chi-square = 12, p > 0.1) but not so for APACHE III (chi-square = 48, p < 0.001). CONCLUSIONS: APACHE II and III both show good discrimination for elderly patients although a little inferior than for all patients. Both scores can be used for risk stratification of elderly ICU patients. Mortality prognosis is not sufficient for geriatric patients although APACHE II calibrates well for all. Application of these scores for quality assurance in selected subgroups like elderly patients cannot be recommended based on these data.
