Tuberculosis Chest X-Ray Image Retrieval System Using Deep Learning Based Biomarker Predictions.

The world health organization's global tuberculosis (TB) report for 2022 identifies TB, with an estimated 1.6 million, as a leading cause of death. The number of new cases has risen since 2020, particularly the number of new drug-resistant cases, estimated at 450,000 in 2021. This is concerning, as treatment of patients with drug resistant TB is complex and may not always be successful. The NIAID TB Portals program is an international consortium with a primary focus on patient centric data collection and analysis for drug resistant TB. The data includes images, their associated radiological findings, clinical records, and socioeconomic information. This work describes a TB Portals' Chest X-ray based image retrieval system which enables precision medicine. An input image is used to retrieve similar images and the associated patient specific information, thus facilitating inspection of outcomes and treatment regimens from comparable patients. Image similarity is defined using clinically relevant biomarkers: gender, age, body mass index (BMI), and the percentage of lung affected per sextant. The biomarkers are predicted using variations of the DenseNet169 convolutional neural network. A multi-task approach is used to predict gender, age and BMI incorporating transfer learning from an initial training on the NIH Clinical Center CXR dataset to the TB portals dataset. The resulting gender AUC, age and BMI mean absolute errors were 0.9854, 4.03years and 1.67kgm2. For the percentage of sextant affected by lesions the mean absolute errors ranged between 7% to 12% with higher error values in the middle and upper sextants which exhibit more variability than the lower sextants. The retrieval system is currently available from https://rap.tbportals.niaid.nih.gov/find_similar_cxr.

Automated Pulmonary Tuberculosis Severity Assessment on Chest X-rays.

According to the 2022 World Health Organization's Global Tuberculosis (TB) report, an estimated 10.6 million people fell ill with TB, and 1.6 million died from the disease in 2021. In addition, 2021 saw a reversal of a decades-long trend of declining TB infections and deaths, with an estimated increase of 4.5% in the number of people who fell ill with TB compared to 2020, and an estimated yearly increase of 450,000 cases of drug resistant TB. Estimating the severity of pulmonary TB using frontal chest X-rays (CXR) can enable better resource allocation in resource constrained settings and monitoring of treatment response, enabling prompt treatment modifications if disease severity does not decrease over time. The Timika score is a clinically used TB severity score based on a CXR reading. This work proposes and evaluates three deep learning-based approaches for predicting the Timika score with varying levels of explainability. The first approach uses two deep learning-based models, one to explicitly detect lesion regions using YOLOV5n and another to predict the presence of cavitation using DenseNet121, which are then utilized in score calculation. The second approach uses a DenseNet121-based regression model to directly predict the affected lung percentage and another to predict cavitation presence using a DenseNet121-based classification model. Finally, the third approach directly predicts the Timika score using a DenseNet121-based regression model. The best performance is achieved by the second approach with a mean absolute error of 13-14% and a Pearson correlation of 0.7-0.84 using three held-out datasets for evaluating generalization.

Multi-omic profiling of follicular lymphoma reveals changes in tissue architecture and enhanced stromal remodeling in high-risk patients.

Follicular lymphoma (FL) is a generally incurable malignancy that evolves from developmentally blocked germinal center (GC) B cells. To promote survival and immune escape, tumor B cells undergo significant genetic changes and extensively remodel the lymphoid microenvironment. Dynamic interactions between tumor B cells and the tumor microenvironment (TME) are hypothesized to contribute to the broad spectrum of clinical behaviors observed among FL patients. Despite the urgent need, existing clinical tools do not reliably predict disease behavior. Using a multi-modal strategy, we examined cell-intrinsic and -extrinsic factors governing progression and therapeutic outcomes in FL patients enrolled onto a prospective clinical trial. By leveraging the strengths of each platform, we identify several tumor-specific features and microenvironmental patterns enriched in individuals who experience early relapse, the most high-risk FL patients. These features include stromal desmoplasia and changes to the follicular growth pattern present 20 months before first progression and first relapse.

Harmonizing the Generation and Pre-publication Stewardship of FAIR Image data.

Together with the molecular knowledge of genes and proteins, biological images promise to significantly enhance the scientific understanding of complex cellular systems and to advance predictive and personalized therapeutic products for human health. For this potential to be realized, quality-assured image data must be shared among labs at a global scale to be compared, pooled, and reanalyzed, thus unleashing untold potential beyond the original purpose for which the data was generated. There are two broad sets of requirements to enable image data sharing in the life sciences. One set of requirements is articulated in the companion White Paper entitled "Enabling Global Image Data Sharing in the Life Sciences," which is published in parallel and addresses the need to build the cyberinfrastructure for sharing the digital array data (arXiv:2401.13023 [q-bio.OT], https://doi.org/10.48550/arXiv.2401.13023). In this White Paper, we detail a broad set of requirements, which involves collecting, managing, presenting, and propagating contextual information essential to assess the quality, understand the content, interpret the scientific implications, and reuse image data in the context of the experimental details. We start by providing an overview of the main lessons learned to date through international community activities, which have recently made considerable progress toward generating community standard practices for imaging Quality Control (QC) and metadata. We then provide a clear set of recommendations for amplifying this work. The driving goal is to address remaining challenges, and democratize access to common practices and tools for a spectrum of biomedical researchers, regardless of their expertise, access to resources, and geographical location.

Combining Radiological and Genomic TB Portals Data for Drug Resistance Analysis.

Tuberculosis (TB) drug resistance is a worldwide public health problem. It decreases the likelihood of a positive outcome for the individual patient and increases the likelihood of disease spread. Therefore, early detection of TB drug resistance is crucial for improving outcomes and controlling disease transmission. While drug-sensitive tuberculosis cases are declining worldwide because of effective treatment, the threat of drug-resistant tuberculosis is growing, and the success rate of drug-resistant tuberculosis treatment is only around 60%. The TB Portals program provides a publicly accessible repository of TB case data with an emphasis on collecting drug-resistant cases. The dataset includes multi-modal information such as socioeconomic/geographic data, clinical characteristics, pathogen genomics, and radiological features. The program is an international collaboration whose participants are typically under a substantial burden of drug-resistant tuberculosis, with data collected from standard clinical care provided to the patients. Consequentially, the TB Portals dataset is heterogenous in nature, with data representing multiple treatment centers in different countries and containing cross-domain information. This study presents the challenges and methods used to address them when working with this real-world dataset. Our goal was to evaluate whether combining radiological features derived from a chest X-ray of the host and genomic features from the pathogen can potentially improve the identification of the drug susceptibility type, drug-sensitive (DS-TB) or drug-resistant (DR-TB), and the length of the first successful drug regimen. To perform these studies, significantly imbalanced data needed to be processed, which included a much larger number of DR-TB cases than DS-TB, many more cases with radiological findings than genomic ones, and the sparse high dimensional nature of the genomic information. Three evaluation studies were carried out. First, the DR-TB/DS-TB classification model achieved an average accuracy of 92.4% when using genomic features alone or when combining radiological and genomic features. Second, the regression model for the length of the first successful treatment had a relative error of 53.5% using radiological features, 25.6% using genomic features, and 22.0% using both radiological and genomic features. Finally, the relative error of the third regression model predicting the length of the first treatment using the most common drug combination varied depending on the feature type used. When using radiological features alone, the relative error was 17.8%. For genomic features alone, the relative error increased to 19.9%. The model had a relative error of 19.0% when both radiological and genomic features were combined. Although combining radiological and genomic features did not improve upon the use of genomic features when classifying DR-TB/DS-TB, the combination of the two feature types improved the relative error of the predictive model for the length of the first successful treatment. Furthermore, the regression model trained on radiological features achieved the best performance when predicting the treatment length of the most common drug combination.

Generalization Challenges in Drug-Resistant Tuberculosis Detection from Chest X-rays.

Classification of drug-resistant tuberculosis (DR-TB) and drug-sensitive tuberculosis (DS-TB) from chest radiographs remains an open problem. Our previous cross validation performance on publicly available chest X-ray (CXR) data combined with image augmentation, the addition of synthetically generated and publicly available images achieved a performance of 85% AUC with a deep convolutional neural network (CNN). However, when we evaluated the CNN model trained to classify DR-TB and DS-TB on unseen data, significant performance degradation was observed (65% AUC). Hence, in this paper, we investigate the generalizability of our models on images from a held out country's dataset. We explore the extent of the problem and the possible reasons behind the lack of good generalization. A comparison of radiologist-annotated lesion locations in the lung and the trained model's localization of areas of interest, using GradCAM, did not show much overlap. Using the same network architecture, a multi-country classifier was able to identify the country of origin of the X-ray with high accuracy (86%), suggesting that image acquisition differences and the distribution of non-pathological and non-anatomical aspects of the images are affecting the generalization and localization of the drug resistance classification model as well. When CXR images were severely corrupted, the performance on the validation set was still better than 60% AUC. The model overfitted to the data from countries in the cross validation set but did not generalize to the held out country. Finally, we applied a multi-task based approach that uses prior TB lesions location information to guide the classifier network to focus its attention on improving the generalization performance on the held out set from another country to 68% AUC.

IBEX: an iterative immunolabeling and chemical bleaching method for high-content imaging of diverse tissues.

High-content imaging is needed to catalog the variety of cellular phenotypes and multicellular ecosystems present in metazoan tissues. We recently developed iterative bleaching extends multiplexity (IBEX), an iterative immunolabeling and chemical bleaching method that enables multiplexed imaging (>65 parameters) in diverse tissues, including human organs relevant for international consortia efforts. IBEX is compatible with >250 commercially available antibodies and 16 unique fluorophores, and can be easily adopted to different imaging platforms using slides and nonproprietary imaging chambers. The overall protocol consists of iterative cycles of antibody labeling, imaging and chemical bleaching that can be completed at relatively low cost in 2-5 d by biologists with basic laboratory skills. To support widespread adoption, we provide extensive details on tissue processing, curated lists of validated antibodies and tissue-specific panels for multiplex imaging. Furthermore, instructions are included on how to automate the method using competitively priced instruments and reagents. Finally, we present a software solution for image alignment that can be executed by individuals without programming experience using open-source software and freeware. In summary, IBEX is a noncommercial method that can be readily implemented by academic laboratories and scaled to achieve high-content mapping of diverse tissues in support of a Human Reference Atlas or other such applications.

Differentiating between drug-sensitive and drug-resistant tuberculosis with machine learning for clinical and radiological features.

BACKGROUND: Tuberculosis (TB) drug resistance is a worldwide public health problem that threatens progress made in TB care and control. Early detection of drug resistance is important for disease control, with discrimination between drug-resistant TB (DR-TB) and drug-sensitive TB (DS-TB) still being an open problem. The objective of this work is to investigate the relevance of readily available clinical data and data derived from chest X-rays (CXRs) in DR-TB prediction and to investigate the possibility of applying machine learning techniques to selected clinical and radiological features for discrimination between DR-TB and DS-TB. We hypothesize that the number of sextants affected by abnormalities such as nodule, cavity, collapse and infiltrate may serve as a radiological feature for DR-TB identification, and that both clinical and radiological features are important factors for machine classification of DR-TB and DS-TB. METHODS: We use data from the NIAID TB Portals program (https://tbportals.niaid.nih.gov), 1,455 DR-TB cases and 782 DS-TB cases from 11 countries. We first select three clinical features and 26 radiological features from the dataset. Then, we perform Pearson's chi-squared test to analyze the significance of the selected clinical and radiological features. Finally, we train machine classifiers based on different features and evaluate their ability to differentiate between DR-TB and DS-TB. RESULTS: Pearson's chi-squared test shows that two clinical features and 23 radiological features are statistically significant regarding DR-TB vs. DS-TB. A ten-fold cross-validation using a support vector machine shows that automatic discrimination between DR-TB and DS-TB achieves an average accuracy of 72.34% and an average AUC value of 78.42%, when combing all 25 statistically significant features. CONCLUSIONS: Our study suggests that the number of affected lung sextants can be used for predicting DR-TB, and that automatic discrimination between DR-TB and DS-TB is possible, with a combination of clinical features and radiological features providing the best performance.

Identifying Drug-Resistant Tuberculosis in Chest Radiographs: Evaluation of CNN Architectures and Training Strategies.

Tuberculosis (TB) is a serious infectious disease that mainly affects the lungs. Drug resistance to the disease makes it more challenging to control. Early diagnosis of drug resistance can help with decision making resulting in appropriate and successful treatment. Chest X-rays (CXRs) have been pivotal to identifying tuberculosis and are widely available. In this work, we utilize CXRs to distinguish between drug-resistant and drug-sensitive tuberculosis. We incorporate Convolutional Neural Network (CNN) based models to discriminate the two types of TB, and employ standard and deep learning based data augmentation methods to improve the classification. Using labeled data from NIAID TB Portals and additional non-labeled sources, we were able to achieve an Area Under the ROC Curve (AUC) of up to 85% using a pretrained InceptionV3 network.

Dual-view light-sheet imaging through a tilted glass interface using a deformable mirror.

Light-sheet microscopy has become indispensable for imaging developing organisms, and imaging from multiple directions (views) is essential to improve its spatial resolution. We combine multi-view light-sheet microscopy with microfluidics using adaptive optics (deformable mirror) which corrects aberrations introduced by the 45o-tilted glass coverslip. The optimal shape of the deformable mirror is computed by an iterative algorithm that optimizes the point-spread function in two orthogonal views. Simultaneous correction in two optical arms is achieved via a knife-edge mirror that splits the excitation path and combines the detection paths. Our design allows multi-view light-sheet microscopy with microfluidic devices for precisely controlled experiments and high-content screening.

PD-1 blockade exacerbates Mycobacterium tuberculosis infection in rhesus macaques.

Boosting immune cell function by targeting the coinhibitory receptor PD-1 may have applications in the treatment of chronic infections. Here, we examine the role of PD-1 during Mycobacterium tuberculosis (Mtb) infection of rhesus macaques. Animals treated with anti-PD-1 monoclonal antibody developed worse disease and higher granuloma bacterial loads compared with isotype control-treated monkeys. PD-1 blockade increased the number and functionality of granuloma Mtb-specific CD8 T cells. In contrast, Mtb-specific CD4 T cells in anti-PD-1-treated macaques were not increased in number or function in granulomas, expressed increased levels of CTLA-4, and exhibited reduced intralesional trafficking in live imaging studies. In granulomas of anti-PD-1-treated animals, multiple proinflammatory cytokines were elevated, and more cytokines correlated with bacterial loads, leading to the identification of a role for caspase 1 in the exacerbation of tuberculosis after PD-1 blockade. Last, increased Mtb bacterial loads after PD-1 blockade were found to associate with the composition of the intestinal microbiota before infection in individual macaques. Therefore, PD-1-mediated coinhibition is required for control of Mtb infection in macaques, perhaps because of its role in dampening detrimental inflammation and allowing for normal CD4 T cell responses.

IBEX: A versatile multiplex optical imaging approach for deep phenotyping and spatial analysis of cells in complex tissues.

The diverse composition of mammalian tissues poses challenges for understanding the cell-cell interactions required for organ homeostasis and how spatial relationships are perturbed during disease. Existing methods such as single-cell genomics, lacking a spatial context, and traditional immunofluorescence, capturing only two to six molecular features, cannot resolve these issues. Imaging technologies have been developed to address these problems, but each possesses limitations that constrain widespread use. Here we report a method that overcomes major impediments to highly multiplex tissue imaging. "Iterative bleaching extends multiplexity" (IBEX) uses an iterative staining and chemical bleaching method to enable high-resolution imaging of >65 parameters in the same tissue section without physical degradation. IBEX can be employed with various types of conventional microscopes and permits use of both commercially available and user-generated antibodies in an "open" system to allow easy adjustment of staining panels based on ongoing marker discovery efforts. We show how IBEX can also be used with amplified staining methods for imaging strongly fixed tissues with limited epitope retention and with oligonucleotide-based staining, allowing potential cross-referencing between flow cytometry, cellular indexing of transcriptomes and epitopes by sequencing, and IBEX analysis of the same tissue. To facilitate data processing, we provide an open-source platform for automated registration of iterative images. IBEX thus represents a technology that can be rapidly integrated into most current laboratory workflows to achieve high-content imaging to reveal the complex cellular landscape of diverse organs and tissues.

DicomAnnotator: a Configurable Open-Source Software Program for Efficient DICOM Image Annotation.

Modern, supervised machine learning approaches to medical image classification, image segmentation, and object detection usually require many annotated images. As manual annotation is usually labor-intensive and time-consuming, a well-designed software program can aid and expedite the annotation process. Ideally, this program should be configurable for various annotation tasks, enable efficient placement of several types of annotations on an image or a region of an image, attribute annotations to individual annotators, and be able to display Digital Imaging and Communications in Medicine (DICOM)-formatted images. No current open-source software program fulfills these requirements. To fill this gap, we developed DicomAnnotator, a configurable open-source software program for DICOM image annotation. This program fulfills the above requirements and provides user-friendly features to aid the annotation process. In this paper, we present the design and implementation of DicomAnnotator. Using spine image annotation as a test case, our evaluation showed that annotators with various backgrounds can use DicomAnnotator to annotate DICOM images efficiently. DicomAnnotator is freely available at https://github.com/UW-CLEAR-Center/DICOM-Annotator under the GPLv3 license.

A distance map regularized CNN for cardiac cine MR image segmentation.

PURPOSE: Cardiac image segmentation is a critical process for generating personalized models of the heart and for quantifying cardiac performance parameters. Fully automatic segmentation of the left ventricle (LV), the right ventricle (RV), and the myocardium from cardiac cine MR images is challenging due to variability of the normal and abnormal anatomy, as well as the imaging protocols. This study proposes a multi-task learning (MTL)-based regularization of a convolutional neural network (CNN) to obtain accurate segmenation of the cardiac structures from cine MR images. METHODS: We train a CNN network to perform the main task of semantic segmentation, along with the simultaneous, auxiliary task of pixel-wise distance map regression. The network also predicts uncertainties associated with both tasks, such that their losses are weighted by the inverse of their corresponding uncertainties. As a result, during training, the task featuring a higher uncertainty is weighted less and vice versa. The proposed distance map regularizer is a decoder network added to the bottleneck layer of an existing CNN architecture, facilitating the network to learn robust global features. The regularizer block is removed after training, so that the original number of network parameters does not change. The trained network outputs per-pixel segmentation when a new patient cine MR image is provided as an input. RESULTS: We show that the proposed regularization method improves both binary and multi-class segmentation performance over the corresponding state-of-the-art CNN architectures. The evaluation was conducted on two publicly available cardiac cine MRI datasets, yielding average Dice coefficients of 0.84 ± 0.03 and 0.91 ± 0.04. We also demonstrate improved generalization performance of the distance map regularized network on cross-dataset segmentation, showing as much as 42% improvement in myocardium Dice coefficient from 0.56 ± 0.28 to 0.80 ± 0.14. CONCLUSIONS: We have presented a method for accurate segmentation of cardiac structures from cine MR images. Our experiments verify that the proposed method exceeds the segmentation performance of three existing state-of-the-art methods. Furthermore, several cardiac indices that often serve as diagnostic biomarkers, specifically blood pool volume, myocardial mass, and ejection fraction, computed using our method are better correlated with the indices computed from the reference, ground truth segmentation. Hence, the proposed method has the potential to become a non-invasive screening and diagnostic tool for the clinical assessment of various cardiac conditions, as well as a reliable aid for generating patient specific models of the cardiac anatomy for therapy planning, simulation, and guidance.

Correction to: SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible Research.

This paper had published originally without open access, but has since been republished with open access.

Left Ventricle Segmentation and Quantification from Cardiac Cine MR Images via Multi-task Learning.

Segmentation of the left ventricle and quantification of various cardiac contractile functions is crucial for the timely diagnosis and treatment of cardiovascular diseases. Traditionally, the two tasks have been tackled independently. Here we propose a convolutional neural network based multi-task learning approach to perform both tasks simultaneously, such that, the network learns better representation of the data with improved generalization performance. Probabilistic formulation of the problem enables learning the task uncertainties during the training, which are used to automatically compute the weights for the tasks. We performed a five fold cross-validation of the myocardium segmentation obtained from the proposed multi-task network on 97 patient 4-dimensional cardiac cine-MRI datasets available through the STA-COM LV segmentation challenge against the provided gold-standard myocardium segmentation, obtaining a Dice overlap of 0.849 ± 0.036 and mean surface distance of 0.274 ± 0.083 mm, while simultaneously estimating the myocardial area with mean absolute difference error of 205 ± 198 mm2.

A marker-free registration method for standing X-ray panorama reconstruction for hip-knee-ankle axis deformity assessment.

Accurate measurement of knee alignment, quantified by the hip-knee-ankle (HKA) angle (varus-valgus), serves as an essential biomarker in the diagnosis of various orthopaedic conditions and selection of appropriate therapies. Such angular deformities are assessed from standing X-ray panoramas. However, the limited field-of-view of traditional X-ray imaging systems necessitates the acquisition of several sector images to capture an individual's standing posture, and their subsequent 'stitching' to reconstruct a panoramic image. Such panoramas are typically constructed manually by an X-ray imaging technician, often using various external markers attached to the individual's clothing and visible in two adjacent sector images. To eliminate human error, user-induced variability, improve consistency and reproducibility, and reduce the time associated with the traditional manual 'stitching' protocol, here we propose an automatic panorama construction method that only relies on anatomical features reliably detected in the images, eliminating the need for any external markers or manual input from the technician. The method first performs a rough segmentation of the femur and the tibia, then the sector images are registered by evaluating a distance metric between the corresponding bones along their medial edge. The identified translations are then used to generate the standing panorama image. The method was evaluated on 95 patient image datasets from a database of X-ray images acquired across 10 clinical sites as part of the screening process for a multi-site clinical trial. The panorama reconstruction parameters yielded by the proposed method were compared to those used for the manual panorama construction, which served as gold-standard. The horizontal translation differences were 0:43 ± 1:95 mm 0:26 ± 1:43 mm for the femur and tibia respectively, while the vertical translation differences were 3:76 ± 22:35 mm and 1:85 ± 6:79 mm for the femur and tibia, respectively. Our results showed no statistically significant differences between the HKA angles measured using the automated vs. the manually generated panoramas, and also led to similar decisions with regards to the patient inclusion/exclusion in the clinical trial. Thus, the proposed method was shown to provide comparable performance to manual panorama construction, with increased efficiency, consistency and robustness.

Cine Cardiac MRI Slice Misalignment Correction Towards Full 3D Left Ventricle Segmentation.

Accurate segmentation of the left ventricle (LV) blood-pool and myocardium is required to compute cardiac function assessment parameters or generate personalized cardiac models for pre-operative planning of minimally invasive therapy. Cardiac Cine Magnetic Resonance Imaging (MRI) is the preferred modality for high resolution cardiac imaging thanks to its capability of imaging the heart throughout the cardiac cycle, while providing tissue contrast superior to other imaging modalities without ionizing radiation. However, there exists an inevitable misalignment between the slices in cine MRI due to the 2D + time acquisition, rendering 3D segmentation methods ineffective. A large part of published work on cardiac MR image segmentation focuses on 2D segmentation methods that yield good results in mid-slices, however with less accurate results for the apical and basal slices. Here, we propose an algorithm to correct for the slice misalignment using a Convolutional Neural Network (CNN)-based regression method, and then perform a 3D graph-cut based segmentation of the LV using atlas shape prior. Our algorithm is able to reduce the median slice misalignment error from 3.13 to 2.07 pixels, and obtain the blood-pool segmentation with an accuracy characterized by a 0.904 mean dice overlap and 0.56 mm mean surface distance with respect to the gold-standard blood-pool segmentation for 9 test cine MR datasets.