Assessing racial bias in healthcare predictive models: Practical lessons from an empirical evaluation of 30-day hospital readmission models.

OBJECTIVE: Despite increased availability of methodologies to identify algorithmic bias, the operationalization of bias evaluation for healthcare predictive models is still limited. Therefore, this study proposes a process for bias evaluation through an empirical assessment of common hospital readmission models. The process includes selecting bias measures, interpretation, determining disparity impact and potential mitigations. METHODS: This retrospective analysis evaluated racial bias of four common models predicting 30-day unplanned readmission (i.e., LACE Index, HOSPITAL Score, and the CMS readmission measure applied as is and retrained). The models were assessed using 2.4 million adult inpatient discharges in Maryland from 2016 to 2019. Fairness metrics that are model-agnostic, easy to compute, and interpretable were implemented and apprised to select the most appropriate bias measures. The impact of changing model's risk thresholds on these measures was further assessed to guide the selection of optimal thresholds to control and mitigate bias. RESULTS: Four bias measures were selected for the predictive task: zero-one-loss difference, false negative rate (FNR) parity, false positive rate (FPR) parity, and generalized entropy index. Based on these measures, the HOSPITAL score and the retrained CMS measure demonstrated the lowest racial bias. White patients showed a higher FNR while Black patients resulted in a higher FPR and zero-one-loss. As the models' risk threshold changed, trade-offs between models' fairness and overall performance were observed, and the assessment showed all models' default thresholds were reasonable for balancing accuracy and bias. CONCLUSIONS: This study proposes an Applied Framework to Assess Fairness of Predictive Models (AFAFPM) and demonstrates the process using 30-day hospital readmission model as the example. It suggests the feasibility of applying algorithmic bias assessment to determine optimized risk thresholds so that predictive models can be used more equitably and accurately. It is evident that a combination of qualitative and quantitative methods and a multidisciplinary team are necessary to identify, understand and respond to algorithm bias in real-world healthcare settings. Users should also apply multiple bias measures to ensure a more comprehensive, tailored, and balanced view. The results of bias measures, however, must be interpreted with caution and consider the larger operational, clinical, and policy context.

Evaluating Algorithmic Bias in 30-Day Hospital Readmission Models: Retrospective Analysis.

BACKGROUND: The adoption of predictive algorithms in health care comes with the potential for algorithmic bias, which could exacerbate existing disparities. Fairness metrics have been proposed to measure algorithmic bias, but their application to real-world tasks is limited. OBJECTIVE: This study aims to evaluate the algorithmic bias associated with the application of common 30-day hospital readmission models and assess the usefulness and interpretability of selected fairness metrics. METHODS: We used 10.6 million adult inpatient discharges from Maryland and Florida from 2016 to 2019 in this retrospective study. Models predicting 30-day hospital readmissions were evaluated: LACE Index, modified HOSPITAL score, and modified Centers for Medicare & Medicaid Services (CMS) readmission measure, which were applied as-is (using existing coefficients) and retrained (recalibrated with 50% of the data). Predictive performances and bias measures were evaluated for all, between Black and White populations, and between low- and other-income groups. Bias measures included the parity of false negative rate (FNR), false positive rate (FPR), 0-1 loss, and generalized entropy index. Racial bias represented by FNR and FPR differences was stratified to explore shifts in algorithmic bias in different populations. RESULTS: The retrained CMS model demonstrated the best predictive performance (area under the curve: 0.74 in Maryland and 0.68-0.70 in Florida), and the modified HOSPITAL score demonstrated the best calibration (Brier score: 0.16-0.19 in Maryland and 0.19-0.21 in Florida). Calibration was better in White (compared to Black) populations and other-income (compared to low-income) groups, and the area under the curve was higher or similar in the Black (compared to White) populations. The retrained CMS and modified HOSPITAL score had the lowest racial and income bias in Maryland. In Florida, both of these models overall had the lowest income bias and the modified HOSPITAL score showed the lowest racial bias. In both states, the White and higher-income populations showed a higher FNR, while the Black and low-income populations resulted in a higher FPR and a higher 0-1 loss. When stratified by hospital and population composition, these models demonstrated heterogeneous algorithmic bias in different contexts and populations. CONCLUSIONS: Caution must be taken when interpreting fairness measures' face value. A higher FNR or FPR could potentially reflect missed opportunities or wasted resources, but these measures could also reflect health care use patterns and gaps in care. Simply relying on the statistical notions of bias could obscure or underplay the causes of health disparity. The imperfect health data, analytic frameworks, and the underlying health systems must be carefully considered. Fairness measures can serve as a useful routine assessment to detect disparate model performances but are insufficient to inform mechanisms or policy changes. However, such an assessment is an important first step toward data-driven improvement to address existing health disparities.

A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models.

OBJECTIVE: Health care providers increasingly rely upon predictive algorithms when making important treatment decisions, however, evidence indicates that these tools can lead to inequitable outcomes across racial and socio-economic groups. In this study, we introduce a bias evaluation checklist that allows model developers and health care providers a means to systematically appraise a model's potential to introduce bias. MATERIALS AND METHODS: Our methods include developing a bias evaluation checklist, a scoping literature review to identify 30-day hospital readmission prediction models, and assessing the selected models using the checklist. RESULTS: We selected 4 models for evaluation: LACE, HOSPITAL, Johns Hopkins ACG, and HATRIX. Our assessment identified critical ways in which these algorithms can perpetuate health care inequalities. We found that LACE and HOSPITAL have the greatest potential for introducing bias, Johns Hopkins ACG has the most areas of uncertainty, and HATRIX has the fewest causes for concern. DISCUSSION: Our approach gives model developers and health care providers a practical and systematic method for evaluating bias in predictive models. Traditional bias identification methods do not elucidate sources of bias and are thus insufficient for mitigation efforts. With our checklist, bias can be addressed and eliminated before a model is fully developed or deployed. CONCLUSION: The potential for algorithms to perpetuate biased outcomes is not isolated to readmission prediction models; rather, we believe our results have implications for predictive models across health care. We offer a systematic method for evaluating potential bias with sufficient flexibility to be utilized across models and applications.

Impact of the COVID-19 pandemic on pediatric and adolescent vaccinations and well child visits in the United States: A database analysis.

BACKGROUND: The COVID-19 pandemic has disrupted healthcare, including immunization practice and well child visit attendance. Maintaining vaccination coverage is important to prevent disease outbreaks and morbidity. We assessed the impact of the COVID-19 pandemic on pediatric and adolescent vaccination administration and well child visit attendance in the United States. METHODS: This cross-sectional study used IBM MarketScan Commercial Database (IMC) with Early View (healthcare claims database) and TriNetX Dataworks Global Network (electronic medical records database) from January 2018-March 2021. Individuals ≤ 18 years of age who were enrolled during the analysis month of interest (IMC with Early View) or had ≥ 1 health encounter at a participating institution (TriNetX Dataworks) were included. We calculated the monthly percent difference between well child visit attendance and vaccine administration rates for 10 recommended pediatric/adolescent vaccines in 2020 and 2021 compared with 2018-2019. Data were stratified by the age groups 0-2 years, 4-6 years, and 9-16 years. RESULTS: In IMC with Early View, the average monthly enrollment for children 0-18 years of age was 5.2 million. In TriNetX Dataworks, 12.2 million eligible individuals were included. Well child visits and vaccinations reached the lowest point in April 2020 compared with 2018-2019. Well child visit attendance and vaccine administration rates were inversely related to age, with initial reductions highest for adolescents and lowest for ages 0-2 years. Rates rebounded in June and September 2020 and stabilized to pre-pandemic levels in Fall 2020. Rates dropped below baseline in early 2021 for groups 0-2 years and 4-6 years. CONCLUSIONS: We found substantial disruptions in well child visit attendance and vaccination administration for children and adolescents during the COVID-19 pandemic in 2020 and early 2021. Continued efforts are needed to monitor recovery and catch up to avoid outbreaks and morbidity associated with vaccine-preventable diseases.

Process evaluation of Healthy Bodies, Healthy Souls: a church-based health intervention program in Baltimore City.

Soaring obesity rates in the United States demand comprehensive health intervention strategies that simultaneously address dietary patterns, physical activity, psychosocial factors and the food environment. Healthy Bodies, Healthy Souls (HBHS) is a church-based, community-participatory, cluster-randomized health intervention trial conducted in Baltimore City to reduce diabetes risk among urban African Americans by promoting healthy dietary intake, increased physical activity and improvement to the church food environment. HBHS was organized into five 3-8-week phases: Healthy Beverages, Healthy Desserts, Healthy Cooking, Healthy Snacking and Eating Out and Physical Activity. A three-part process evaluation was adopted to evaluate implementation success: an in-church instrument to assess the reach, dose delivered and fidelity of interactive sessions; a post-intervention exposure survey to assess individual-level dose received in a sample of congregants and an evaluation form to assess the church food environment. Print materials were implemented with moderate to high fidelity and high dose. Program reach was low, which may reflect inaccuracies in church attendance rather than study implementation issues. Intervention components with the greatest dose received were giveaways (42.0-61.7%), followed by taste tests (48.7-53.7%) and posters (34.3-65.0%). The dose received of general program information was moderate to high. The results indicate successful implementation of the HBHS program.