Deep learning models for COVID-19 chest x-ray classification: Preventing shortcut learning using feature disentanglement

In response to the COVID-19 global pandemic, recent research has proposed creating deep learning based models that use chest radiographs (CXRs) in a variety of clinical tasks to help manage the crisis. However, the size of existing datasets of CXRs from COVID-19+ patients are relatively small, and researchers often pool CXR data from multiple sources, for example, using different x-ray machines in various patient populations under different clinical scenarios. Deep learning models trained on such datasets have been shown to overfit to erroneous features instead of learning pulmonary characteristics - a phenomenon known as shortcut learning. We propose adding feature disentanglement to the training process, forcing the models to identify pulmonary features from the images while penalizing them for learning features that can discriminate between the original datasets that the images come from. We find that models trained in this way indeed have better generalization performance on unseen data; in the best case we found that it improved AUC by 0.13 on held out data. We further find that this outperforms masking out non-lung parts of the CXRs and performing histogram equalization, both of which are recently proposed methods for removing biases in CXR datasets.

Improvement and Multi-Population Generalizability of a Deep Learning-Based Chest Radiograph Severity Score for COVID-19

PurposeTo improve and test the generalizability of a deep learning-based model for assessment of COVID-19 lung disease severity on chest radiographs (CXRs) from different patient populations. Materials and MethodsA published convolutional Siamese neural network-based model previously trained on hospitalized patients with COVID-19 was tuned using 250 outpatient CXRs. This model produces a quantitative measure of COVID-19 lung disease severity (pulmonary x-ray severity (PXS) score). The model was evaluated on CXRs from four test sets, including 3 from the United States (patients hospitalized at an academic medical center (N=154), patients hospitalized at a community hospital (N=113), and outpatients (N=108)) and 1 from Brazil (patients at an academic medical center emergency department (N=303)). Radiologists from both countries independently assigned reference standard CXR severity scores, which were correlated with the PXS scores as a measure of model performance (Pearson r). The Uniform Manifold Approximation and Projection (UMAP) technique was used to visualize the neural network results. ResultsTuning the deep learning model with outpatient data improved model performance in two United States hospitalized patient datasets (r=0.88 and r=0.90, compared to baseline r=0.86). Model performance was similar, though slightly lower, when tested on the United States outpatient and Brazil emergency department datasets (r=0.86 and r=0.85, respectively). UMAP showed that the model learned disease severity information that generalized across test sets. ConclusionsPerformance of a deep learning-based model that extracts a COVID-19 severity score on CXRs improved using training data from a different patient cohort (outpatient versus hospitalized) and generalized across multiple populations.

COVID-19 Outpatient Screening: a Prediction Score for Adverse Events

BackgroundWe sought to develop an automatable score to predict hospitalization, critical illness, or death in patients at risk for COVID-19 presenting for urgent care during the Massachusetts outbreak. MethodsSingle-center study of adult outpatients seen in respiratory illness clinics (RICs) or the emergency department (ED), including development (n = 9381, March 7-May 2) and prospective (n = 2205, May 3-14) cohorts. Data was queried from Partners Enterprise Data Warehouse. Outcomes were hospitalization, critical illness or death within 7 days. We developed the COVID-19 Acuity Score (CoVA) using automatically extracted data from the electronic medical record and learning-to-rank ordinal logistic regression modeling. Calibration was assessed using predicted-to-observed ratio (E/O). Discrimination was assessed by C-statistics (AUC). ResultsIn the development cohort, 27.3%, 7.2%, and 1.1% of patients experienced hospitalization, critical illness, or death, respectively; and in the prospective cohort, 26.1%, 6.3%, and 0.5%. CoVA showed excellent performance in the development cohort (concurrent validation) for hospitalization (E/O: 1.00, AUC: 0.80); for critical illness (E/O: 1.00, AUC: 0.82); and for death (E/O: 1.00, AUC: 0.87). Performance in the prospective cohort (prospective validation) was similar for hospitalization (E/O: 1.01, AUC: 0.76); for critical illness (E/O 1.03, AUC: 0.79); and for death (E/O: 1.63, AUC=0.93). Among 30 predictors, the top five were age, diastolic blood pressure, blood oxygen saturation, COVID-19 testing status, and respiratory rate. ConclusionsCoVA is a prospectively validated automatable score to assessing risk for adverse outcomes related to COVID-19 infection in the outpatient setting.

Automated assessment of COVID-19 pulmonary disease severity on chest radiographs using convolutional Siamese neural networks

PurposeTo develop an automated measure of COVID-19 pulmonary disease severity on chest radiographs (CXRs), for longitudinal disease evaluation and clinical risk stratification. Materials and MethodsA convolutional Siamese neural network-based algorithm was trained to output a measure of pulmonary disease severity on anterior-posterior CXRs (pulmonary x-ray severity (PXS) score), using weakly-supervised pretraining on ~160,000 images from CheXpert and transfer learning on 314 CXRs from patients with COVID-19. The algorithm was evaluated on internal and external test sets from different hospitals, containing 154 and 113 CXRs respectively. The PXS score was correlated with a radiographic severity score independently assigned by two thoracic radiologists and one in-training radiologist. For 92 internal test set patients with follow-up CXRs, the change in PXS score was compared to radiologist assessments of change. The association between PXS score and subsequent intubation or death was assessed. ResultsThe PXS score correlated with the radiographic pulmonary disease severity score assigned to CXRs in the COVID-19 internal and external test sets ({rho}=0.84 and {rho}=0.78 respectively). The direction of change in PXS score in follow-up CXRs agreed with radiologist assessment ({rho}=0.74). In patients not intubated on the admission CXR, the PXS score predicted subsequent intubation or death within three days of hospital admission (area under the receiver operator characteristic curve=0.80 (95%CI 0.75-0.85)). ConclusionA Siamese neural network-based severity score automatically measures COVID-19 pulmonary disease severity in chest radiographs, which can be scaled and rapidly deployed for clinical triage and workflow optimization. SUMMARYA convolutional Siamese neural network-based algorithm can calculate a continuous radiographic pulmonary disease severity score in COVID-19 patients, which can be used for longitudinal disease evaluation and clinical risk stratification. KEY RESULTSO_LIA Siamese neural network-based severity score correlates with radiologist-annotated pulmonary disease severity on chest radiographs from patients with COVID-19 ({rho}=0.84 and {rho}=0.78 in internal and external test sets respectively). C_LIO_LIThe direction of change in the severity score in follow-up radiographs is concordant with radiologist assessment ({rho}=0.74). C_LIO_LIThe admission chest radiograph severity score can help predict subsequent intubation or death within three days of admission (receiver operator characteristic area under the curve=0.80). C_LI

Errors in Quantitative Image Analysis due to Platform-Dependent Image Scaling.

PURPOSE: To evaluate the ability of various software (SW) tools used for quantitative image analysis to properly account for source-specific image scaling employed by magnetic resonance imaging manufacturers. METHODS: A series of gadoteridol-doped distilled water solutions (0%, 0.5%, 1%, and 2% volume concentrations) was prepared for manual substitution into one (of three) phantom compartments to create "variable signal," whereas the other two compartments (containing mineral oil and 0.25% gadoteriol) were held unchanged. Pseudodynamic images were acquired over multiple series using four scanners such that the histogram of pixel intensities varied enough to provoke variable image scaling from series to series. Additional diffusion-weighted images were acquired of an ice-water phantom to generate scanner-specific apparent diffusion coefficient (ADC) maps. The resulting pseudodynamic images and ADC maps were analyzed by eight centers of the Quantitative Imaging Network using 16 different SW tools to measure compartment-specific region-of-interest intensity. RESULTS: Images generated by one of the scanners appeared to have additional intensity scaling that was not accounted for by the majority of tested quantitative image analysis SW tools. Incorrect image scaling leads to intensity measurement bias near 100%, compared to nonscaled images. CONCLUSION: Corrective actions for image scaling are suggested for manufacturers and quantitative imaging community.