An Objective Search for Unrecognized Bias in Validated COVID-19 Prediction Models

The growing recognition of algorithmic bias has spurred discussions about fairness in artificial intelligence (AI) / machine learning (ML) algorithms. The increasing translation of predictive models into clinical practice brings an increased risk of direct harm from algorithmic bias; however, bias remains incompletely measured in many medical AI applications. Using data from over 56 thousand Mass General Brigham (MGB) patients with confirmed severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), we evaluate unrecognized bias in four AI models developed during the early months of the pandemic in Boston, Massachusetts that predict risks of hospital admission, ICU admission, mechanical ventilation, and death after a SARS-CoV-2 infection purely based on their pre-infection longitudinal medical records. We discuss that while a model can be biased against certain protected groups (i.e., perform worse) in certain tasks, it can be at the same time biased towards another protected group (i.e., perform better). As such, current bias evaluation studies may lack a full depiction of the variable effects of a model on its subpopulations. If the goal is to make a change in a positive way, the underlying roots of bias need to be fully explored in medical AI. Only a holistic evaluation, a diligent search for unrecognized bias, can provide enough information for an unbiased judgment of AI bias that can invigorate follow-up investigations on identifying the underlying roots of bias and ultimately make a change.

The Digital Analytic Patient Reviewer (DAPR) for COVID-19 Data Mart Validation

ObjectiveTo provide high-quality data for COVID-19 research, we validated COVID-19 clinical indicators and 22 associated computed phenotypes, which were derived by machine learning algorithms, in the Mass General Brigham (MGB) COVID-19 Data Mart. Materials and MethodsFifteen reviewers performed a manual chart review for 150 COVID-19 positive patients in the data mart. To support rapid chart review for a wide range of target data, we offered the Digital Analytic Patient Reviewer (DAPR). DAPR is a web-based chart review tool that integrates patient notes and provides note search functionalities and a patient-specific summary view linked with relevant notes. Within DAPR, we developed a COVID-19 validation task-oriented view and information extraction logic, enabled fast access to data, and considered privacy and security issues. ResultsThe concepts for COVID-19 positive cohort, COVID-19 index date, COVID-19 related admission, and the admission date were shown to have high values in all evaluation metrics. For phenotypes, the overall specificities, PPVs, and NPVs were high. However, sensitivities were relatively low. Based on these results, we removed 3 phenotypes from our data mart. In the survey about using the tool, participants expressed positive attitudes towards using DAPR for chart review. They assessed the validation was easy and DAPR helped find relevant information. Some validation difficulties were also discussed. Discussion and ConclusionDAPRs patient summary view accelerated the validation process. We are in the process of automating the workflow to use DAPR for chart reviews. Moreover, we will extend its use case to other domains.

Evolving Phenotypes of non-hospitalized Patients that Indicate Long Covid

For some SARS-CoV-2 survivors, recovery from the acute phase of the infection has been grueling with lingering effects. Many of the symptoms characterized as the post-acute sequelae of COVID-19 (PASC) could have multiple causes or are similarly seen in non-COVID patients. Accurate identification of phenotypes will be important to guide future research and help the healthcare system focus its efforts and resources on adequately controlled age- and gender-specific sequelae of a COVID-19 infection. In this retrospective electronic health records (EHR) cohort study, we applied a computational framework for knowledge discovery from clinical data, MLHO, to identify phenotypes that positively associate with a past positive reverse transcription-polymerase chain reaction (RT-PCR) test for COVID-19. We evaluated the post-test phenotypes in two temporal windows at 3-6 and 6-9 months after the test and by age and gender. Data from longitudinal diagnosis records stored in EHRs from Mass General Brigham in the Boston metropolitan area was used for the analyses. Statistical analyses were performed on data from March 2020 to June 2021. Study participants included over 96 thousand patients who had tested positive or negative for COVID-19 and were not hospitalized. We identified 33 phenotypes among different age/gender cohorts or time windows that were positively associated with past SARS-CoV-2 infection. All identified phenotypes were newly recorded in patients medical records two months or longer after a COVID-19 RT-PCR test in non-hospitalized patients regardless of the test result. Among these phenotypes, a new diagnosis record for anosmia and dysgeusia (OR: 2.60, 95% CI [1.94 - 3.46]), alopecia (OR: 3.09, 95% CI [2.53 - 3.76]), chest pain (OR: 1.27, 95% CI [1.09 - 1.48]), chronic fatigue syndrome (OR 2.60, 95% CI [1.22-2.10]), shortness of breath (OR 1.41, 95% CI [1.22 - 1.64]), pneumonia (OR 1.66, 95% CI [1.28 - 2.16]), and type 2 diabetes mellitus (OR 1.41, 95% CI [1.22 - 1.64]) are some of the most significant indicators of a past COVID-19 infection. Additionally, more new phenotypes were found with increased confidence among the cohorts who were younger than 65. Our approach avoids a flood of false positive discoveries while offering a more robust probabilistic approach compared to the standard linear phenome-wide association study (PheWAS). The findings of this study confirm many of the post-COVID symptoms and suggest that a variety of new diagnoses, including new diabetes mellitus and neurological disorder diagnoses, are more common among those with a history of COVID-19 than those without the infection. Additionally, more than 63 percent of PASC phenotypes were observed in patients under 65 years of age, pointing out the importance of vaccination to minimize the risk of debilitating post-acute sequelae of COVID-19 among younger adults.

A Web-based, Mobile Responsive Application to Screen Healthcare Workers for COVID Symptoms: Descriptive Study

BackgroundThe COVID-19 pandemic has impacted over 1 million people across the globe, with over 330,000 cases in the United States. To help limit the spread in Massachusetts, the Department of Public Health required that all healthcare workers must be screened for symptoms daily - individuals with symptoms may not work. We rapidly created a digital COVID-19 symptom screening tool for a large, academic, integrated healthcare delivery system, Partners HealthCare, in Boston, Massachusetts. ObjectiveWe describe the design and development of the COVID-19 symptom screening application and report on aggregate usage data from the first week of use across the organization. MethodsUsing agile principles, we designed, tested and implemented a solution over the span of a week using progressively custom development approaches as the requirements and use case become more solidified. We developed the minimum viable product (MVP) of a mobile responsive, web-based self-service application using REDCap (Research Electronic Data Capture). For employees without access to a computer or mobile device to use the self-service application, we established a manual process where in-person, socially distanced screeners asked employees entering the site if they have symptoms and then manually recorded the responses in an Office 365 Form. A custom .NET Framework application was developed solution as COVID Pass was scaled. We collected log data from the .NET application, REDCap and Office 365 from the first week of full enterprise deployment (March 30, 2020 - April 5, 2020). Aggregate descriptive statistics including overall employee attestations by day and site, employee attestations by application method (COVID Pass automatic screening vs. manual screening), employee attestations by time of day, and percentage of employees reporting COVID-19 symptoms ResultsWe rapidly created the MVP and gradually deployed it across the hospitals in our organization. By the end of the first week of enterprise deployment, the screening application was being used by over 25,000 employees each weekday. Over the first full week of deployment, 154,730 employee attestation logs were processed across the system. Over this 7-day period, 558 (0.36%) employees reported positive symptoms. In most clinical locations, the majority of employees ([~]80-90%) used the self-service application, with a smaller percentage ([~]10-20%) using manual attestation. Hospital staff continued to work around the clock, but as expected, staff attestations peaked during shift changes between 7-8am, 2-3pm, 4-6pm, and 11pm-midnight. ConclusionsUsing rapid, agile development, we quickly created and deployed a dedicated employee attestation application that gained widespread adoption and use within our health system. Further, we have identified over 500 symptomatic employees that otherwise would have possibly come to work, potentially putting others at risk. We share the story of our implementation, lessons learned, and source code (via GitHub) for other institutions who may want to implement similar solutions.