Predicting outcomes in patients with aortic stenosis using machine learning: the Aortic Stenosis Risk (ASteRisk) score.

OBJECTIVE: To use echocardiographic and clinical features to develop an explainable clinical risk prediction model in patients with aortic stenosis (AS), including those with low-gradient AS (LGAS), using machine learning (ML). METHODS: In 1130 patients with moderate or severe AS, we used bootstrap lasso regression (BLR), an ML method, to identify echocardiographic and clinical features important for predicting the combined outcome of all-cause mortality or aortic valve replacement (AVR) within 5 years after the initial echocardiogram. A separate hold out set, from a different centre (n=540), was used to test the generality of the model. We also evaluated model performance with respect to each outcome separately and in different subgroups, including patients with LGAS. RESULTS: Out of 69 available variables, 26 features were identified as predictive by BLR and expert knowledge was used to further reduce this set to 9 easily available and input features without loss of efficacy. A ridge logistic regression model constructed using these features had an area under the receiver operating characteristic curve (AUC) of 0.74 for the combined outcome of mortality/AVR. The model reliably identified patients at high risk of death in years 2-5 (HRs ≥2.0, upper vs other quartiles, for years 2-5, p<0.05, p=not significant in year 1) and was also predictive in the cohort with LGAS (n=383, HRs≥3.3, p<0.05). The model performed similarly well in the independent hold out set (AUC 0.78, HR ≥2.5 in years 1-5, p<0.05). CONCLUSION: In two separate longitudinal databases, ML identified prognostic features and produced an algorithm that predicts outcome for up to 5 years of follow-up in patients with AS, including patients with LGAS. Our algorithm, the Aortic Stenosis Risk (ASteRisk) score, is available online for public use.

Identifying unreliable predictions in clinical risk models.

The ability to identify patients who are likely to have an adverse outcome is an essential component of good clinical care. Therefore, predictive risk stratification models play an important role in clinical decision making. Determining whether a given predictive model is suitable for clinical use usually involves evaluating the model's performance on large patient datasets using standard statistical measures of success (e.g., accuracy, discriminatory ability). However, as these metrics correspond to averages over patients who have a range of different characteristics, it is difficult to discern whether an individual prediction on a given patient should be trusted using these measures alone. In this paper, we introduce a new method for identifying patient subgroups where a predictive model is expected to be poor, thereby highlighting when a given prediction is misleading and should not be trusted. The resulting "unreliability score" can be computed for any clinical risk model and is suitable in the setting of large class imbalance, a situation often encountered in healthcare settings. Using data from more than 40,000 patients in the Global Registry of Acute Coronary Events (GRACE), we demonstrate that patients with high unreliability scores form a subgroup in which the predictive model has both decreased accuracy and decreased discriminatory ability.

Choosing Clinical Variables for Risk Stratification Post-Acute Coronary Syndrome.

Most risk stratification methods use expert opinion to identify a fixed number of clinical variables that have prognostic significance. In this study our goal was to develop improved metrics that utilize a variable number of input parameters. We first used Bootstrap Lasso Regression (BLR) - a Machine Learning method for selecting important variables - to identify a prognostic set of features that identify patients at high risk of death 6-months after presenting with an Acute Coronary Syndrome. Using data derived from the Global Registry of Acute Coronary Events (GRACE) we trained a logistic regression model using these features and evaluated its performance on a development set (N = 43,063) containing patients who have values for all features, and a separate dataset (N = 6,363) that contains patients who have missing feature values. The final model, Ridge Logistic Regression with Variable Inputs (RLRVI), uses imputation to estimate values for missing features. BLR identified 19 features, 8 of which appear in the GRACE score. RLRVI had modest, yet statistically significant, improvement over the standard GRACE score on both datasets. Moreover, for patients who are relatively low-risk (GRACE≤87), RLRVI had an AUC and Hazard Ratio of 0.754 and 6.27, respectively, vs. 0.688 and 2.46 for GRACE, (p < 0.007). RLRVI has improved discriminatory performance on patients who have values for the 8 GRACE features plus any subset of the 11 non-GRACE features. Our results demonstrate that BLR and data imputation can be used to obtain improved risk stratification metrics, particularly for patients who are classified as low risk using traditional methods.

Machine Learning Improves Risk Stratification After Acute Coronary Syndrome.

The accurate assessment of a patient's risk of adverse events remains a mainstay of clinical care. Commonly used risk metrics have been based on logistic regression models that incorporate aspects of the medical history, presenting signs and symptoms, and lab values. More sophisticated methods, such as Artificial Neural Networks (ANN), form an attractive platform to build risk metrics because they can easily incorporate disparate pieces of data, yielding classifiers with improved performance. Using two cohorts consisting of patients admitted with a non-ST-segment elevation acute coronary syndrome, we constructed an ANN that identifies patients at high risk of cardiovascular death (CVD). The ANN was trained and tested using patient subsets derived from a cohort containing 4395 patients (Area Under the Curve (AUC) 0.743) and validated on an independent holdout set containing 861 patients (AUC 0.767). The ANN 1-year Hazard Ratio for CVD was 3.72 (95% confidence interval 1.04-14.3) after adjusting for the TIMI Risk Score, left ventricular ejection fraction, and B-type natriuretic peptide. A unique feature of our approach is that it captures small changes in the ST segment over time that cannot be detected by visual inspection. These findings highlight the important role that ANNs can play in risk stratification.

Sustainability of school-located influenza vaccination programs in Florida.

BACKGROUND: School-located influenza vaccination (SLIV) programs are a promising strategy for increasing vaccination coverage among schoolchildren. However, questions of economic sustainability have dampened enthusiasm for this approach in the United States. We evaluated SLIV sustainability of a health department led, county-wide SLIV program in Alachua County, Florida. Based on Alachua's outcome data, we modeled the sustainability of SLIV programs statewide using two different implementation costs and at different vaccination rates, reimbursement amount, and Vaccines for Children (VFC) coverage. METHODS: Mass vaccination clinics were conducted at 69 Alachua County schools in 2013 using VFC (for Medicaid and uninsured children) and non-VFC vaccines. Claims were processed after each clinic and submitted to insurance providers for reimbursement ($5 Medicaid and $47.04 from private insurers). We collected programmatic expenditures and volunteer hours to calculate fixed and variable costs for two different implementation costs (with or without in-kind costs included). We project program sustainability for Florida using publicly available county-specific student populations and health insurance enrollment data. RESULTS: Approximately 42% (n=12,853) of pre-kindergarten - 12th grade students participated in the SLIV program in Alachua. Of the 13,815 doses provided, 58% (8042) were non-VFC vaccine. Total implementation cost was $14.95/dose or $7.93/dose if "in-kind" costs were not included. The program generated a net surplus of $24,221, despite losing $4.68 on every VFC dose provided to Medicaid and uninsured children. With volunteers, 99% of Florida counties would be sustainable at a 50% vaccination rate and average reimbursement amount of $3.25 VFC and $37 non-VFC. Without volunteers, 69% of counties would be sustainable at 50% vaccination rate if all VFC recipients were on Medicaid and its reimbursement increased from $5 to $10 (amount private practices receive). CONCLUSIONS AND RELEVANCE: Key factors that contributed to the sustainability and success of an SLIV program are: targeting privately insured children and reducing administration cost through volunteers. Counties with a high proportion of VFC eligible children may not be sustainable without subsidies at $5 Medicaid reimbursement.

School-located influenza vaccination reduces community risk for influenza and influenza-like illness emergency care visits.

BACKGROUND: School-located influenza vaccination (SLIV) programs can substantially enhance the sub-optimal coverage achieved under existing delivery strategies. Randomized SLIV trials have shown these programs reduce laboratory-confirmed influenza among both vaccinated and unvaccinated children. This work explores the effectiveness of a SLIV program in reducing the community risk of influenza and influenza-like illness (ILI) associated emergency care visits. METHODS: For the 2011/12 and 2012/13 influenza seasons, we estimated age-group specific attack rates (AR) for ILI from routine surveillance and census data. Age-group specific SLIV program effectiveness was estimated as one minus the AR ratio for Alachua County versus two comparison regions: the 12 county region surrounding Alachua County, and all non-Alachua counties in Florida. RESULTS: Vaccination of ∼50% of 5-17 year-olds in Alachua reduced their risk of ILI-associated visits, compared to the rest of Florida, by 79% (95% confidence interval: 70, 85) in 2011/12 and 71% (63, 77) in 2012/13. The greatest indirect effectiveness was observed among 0-4 year-olds, reducing AR by 89% (84, 93) in 2011/12 and 84% (79, 88) in 2012/13. Among all non-school age residents, the estimated indirect effectiveness was 60% (54, 65) and 36% (31, 41) for 2011/12 and 2012/13. The overall effectiveness among all age-groups was 65% (61, 70) and 46% (42, 50) for 2011/12 and 2012/13. CONCLUSION: Wider implementation of SLIV programs can significantly reduce the influenza-associated public health burden in communities.

Implementing a community-supported school-based influenza immunization program.

School-based influenza immunization programs are increasingly recognized as a key component of community-based efforts to control annual influenza epidemics. Computer modeling suggests that immunizing 70% of schoolchildren could protect an entire community from the flu. Most of the school-based influenza immunization programs described in the literature have had support from industry or federal grants. This article describes a program that used only community resources to administer live, attenuated influenza vaccine supplied by the state health department. Beginning in 2006, the Alachua County Health Department and school system, working in collaboration with the University of Florida, began exploration of a non-mandatory community-wide school-based influenza immunization program, with the goal of achieving high levels of immunization of the ~22,000 public and private pre-K through grade 8 students in the county. In 2009-10 the program was repeated. This report describes the procedures developed to achieve the goal, the barriers that were encountered, and solutions to problems that occurred during the implementation of the program. Preliminary data suggest that the crude immunization rate in the schools was approximately 55% and that at least 10% more students were immunized by their health providers. At an operational level, it is possible to achieve high immunization rates if the stakeholders share a common vision and there is extensive community involvement.