Radiomics-Guided Global-Local Transformer for Weakly Supervised Pathology Localization in Chest X-Rays.

Before the recent success of deep learning methods for automated medical image analysis, practitioners used handcrafted radiomic features to quantitatively describe local patches of medical images. However, extracting discriminative radiomic features relies on accurate pathology localization, which is difficult to acquire in real-world settings. Despite advances in disease classification and localization from chest X-rays, many approaches fail to incorporate clinically-informed domainspecific radiomic features. For these reasons, we propose a Radiomics-Guided Transformer (RGT) that fuses global image information with local radiomics-guided auxiliary information to provide accurate cardiopulmonary pathology localization and classification without any bounding box annotations. RGT consists of an image Transformer branch, a radiomics Transformer branch, and fusion layers that aggregate image and radiomics information. Using the learned self-attention of its image branch, RGT extracts a bounding box for which to compute radiomic features, which are further processed by the radiomics branch; learned image and radiomic features are then fused and mutually interact via cross-attention layers. Thus, RGT utilizes a novel end-to-end feedback loop that can bootstrap accurate pathology localization only using image-level disease labels. Experiments on the NIH ChestXRay dataset demonstrate that RGT outperforms prior works in weakly supervised disease localization (by an average margin of 3.6% over various intersection-over-union thresholds) and classification (by 1.1% in average area under the receiver operating characteristic curve). We publicly release our codes and pre-trained models at https://github.com/VITAGroup/chext.

Knowledge-Augmented Contrastive Learning for Abnormality Classification and Localization in Chest X-rays with Radiomics using a Feedback Loop.

Accurate classification and localization of abnormalities in chest X-rays play an important role in clinical diagnosis and treatment planning. Building a highly accurate predictive model for these tasks usually requires a large number of manually annotated labels and pixel regions (bounding boxes) of abnormalities. However, it is expensive to acquire such annotations, especially the bounding boxes. Recently, contrastive learning has shown strong promise in leveraging unlabeled natural images to produce highly generalizable and discriminative features. However, extending its power to the medical image domain is under-explored and highly non-trivial, since medical images are much less amendable to data augmentations. In contrast, their prior knowledge, as well as radiomic features, is often crucial. To bridge this gap, we propose an end-to-end semi-supervised knowledge-augmented contrastive learning framework, that simultaneously performs disease classification and localization tasks. The key knob of our framework is a unique positive sampling approach tailored for the medical images, by seamlessly integrating radiomic features as a knowledge augmentation. Specifically, we first apply an image encoder to classify the chest X-rays and to generate the image features. We next leverage Grad-CAM to highlight the crucial (abnormal) regions for chest X-rays (even when unannotated), from which we extract radiomic features. The radiomic features are then passed through another dedicated encoder to act as the positive sample for the image features generated from the same chest X-ray. In this way, our framework constitutes a feedback loop for image and radiomic features to mutually reinforce each other. Their contrasting yields knowledge-augmented representations that are both robust and interpretable. Extensive experiments on the NIH Chest X-ray dataset demonstrate that our approach outperforms existing baselines in both classification and localization tasks.

Brain Signals to Rescue Aphasia, Apraxia and Dysarthria Speech Recognition.

In this paper, we propose a deep learning-based algorithm to improve the performance of automatic speech recognition (ASR) systems for aphasia, apraxia, and dysarthria speech by utilizing electroencephalography (EEG) features recorded synchronously with aphasia, apraxia, and dysarthria speech. We demonstrate a significant decoding performance improvement by more than 50% during test time for isolated speech recognition task and we also provide preliminary results indicating performance improvement for the more challenging continuous speech recognition task by utilizing EEG features. The results presented in this paper show the first step towards demonstrating the possibility of utilizing non-invasive neural signals to design a real-time robust speech prosthetic for stroke survivors recovering from aphasia, apraxia, and dysarthria. Our aphasia, apraxia, and dysarthria speech-EEG data set will be released to the public to help further advance this interesting and crucial research.

Deep J-Sense: Accelerated MRI Reconstruction via Unrolled Alternating Optimization.

Accelerated multi-coil magnetic resonance imaging reconstruction has seen a substantial recent improvement combining compressed sensing with deep learning. However, most of these methods rely on estimates of the coil sensitivity profiles, or on calibration data for estimating model parameters. Prior work has shown that these methods degrade in performance when the quality of these estimators are poor or when the scan parameters differ from the training conditions. Here we introduce Deep J-Sense as a deep learning approach that builds on unrolled alternating minimization and increases robustness: our algorithm refines both the magnetization (image) kernel and the coil sensitivity maps. Experimental results on a subset of the knee fastMRI dataset show that this increases reconstruction performance and provides a significant degree of robustness to varying acceleration factors and calibration region sizes.

PNEUMONIA DETECTION ON CHEST X-RAY USING RADIOMIC FEATURES AND CONTRASTIVE LEARNING.

Chest X-ray becomes one of the most common medical diagnoses due to its noninvasiveness. The number of chest X-ray images has skyrocketed, but reading chest X-rays still has been manually performed by radiologists, which creates huge burnouts and delays. Traditionally, radiomics, as a subfield of radiology that can extract a large number of quantitative features from medical images, demonstrates its potential to facilitate medical imaging diagnosis before the deep learning era. With the rise of deep learning, the explainability of deep neural networks on chest X-ray diagnosis remains opaque. In this study, we proposed a novel framework that leverages radiomics features and contrastive learning to detect pneumonia in chest X-ray. Experiments on the RSNA Pneumonia Detection Challenge dataset show that our model achieves superior results to several state-of-the-art models (> 10% in F1-score) and increases the model's interpretability.

Using Radiomics as Prior Knowledge for Thorax Disease Classification and Localization in Chest X-rays.

Chest X-ray becomes one of the most common medical diagnoses due to its noninvasiveness. The number of chest X-ray images has skyrocketed, but reading chest X-rays still have been manually performed by radiologists, which creates huge burnouts and delays. Traditionally, radiomics, as a subfield of radiology that can extract a large number of quantitative features from medical images, demonstrates its potential to facilitate medical imaging diagnosis before the deep learning era. In this paper, we develop an end-to-end framework, ChexRadiNet, that can utilize the radiomics features to improve the abnormality classification performance. Specifically, ChexRadiNet first applies a light-weight but efficient triplet-attention mechanism to classify the chest X-rays and highlight the abnormal regions. Then it uses the generated class activation map to extract radiomic features, which further guides our model to learn more robust image features. After a number of iterations and with the help of radiomic features, our framework can converge to more accurate image regions. We evaluate the ChexRadiNet framework using three public datasets: NIH ChestX-ray, CheXpert, and MIMIC-CXR. We find that ChexRadiNet outperforms the state-of-the-art on both disease detection (0.843 in AUC) and localization (0.679 in T(IoU) = 0.1). We make the code publicly available at https://github. com/bionlplab/lung_disease_detection_amia2021, with the hope that this method can facilitate the development of automatic systems with a higher-level understanding of the radiological world.

Overcoming Long-Term Variability in Local Field Potentials Using an Adaptive Decoder.

Long-term variability remains one of the major hurdles in using intracortical recordings like local field potentials for brain computer interfaces (BCI). Practical neural decoders need to overcome time instability of neural signals to estimate subject behavior accurately and faithfully over the long term. This paper presents a novel decoder that 1) characterizes each behavioral task (i.e., different movement directions under different force conditions) with multiple neural patterns and 2) adapts to the long-term variations in neural features by identifying the stable neural patterns. This adaptation can be performed in both an unsupervised and a semisupervised learning framework requiring minimal feedback from the user. To achieve generalization over time, the proposed decoder uses redundant sparse regression models that adapt to day-to-day variations in neural patterns. While this update requires no explicit feedback from the BCI user, any feedback (explicit or derived) to the BCI improves its performance. With this adaptive decoder, we investigated the effects of long-term neural modulation especially when subjects encountered new external forces against movement. The proposed decoder predicted eight hand-movement directions with an accuracy of 95% over two weeks (when there was no external forces); and 85% in later acquisition sessions spanning up to 42 days (when the monkeys countered external field forces). Since the decoder can operate with or without manual intervention, it could alleviate user frustration associated with BCI.

Block design enhances classification of 3D reach targets from electroencephalographic signals.

To date, decoding accuracy of actual or imagined pointing movements to targets in 3D space from electroencephalographic (EEG) signals has remained modest. The reason may pertain to the fact that these movements activate essentially the same neural networks. In this study, we aimed at testing whether repetitive pointing movements to each of the targets promotes the development of segregated neural patterns, resulting in enhanced decoding accuracy. Six human subjects generated slow or fast repetitive pointing movements with their right dominant arm to one of five targets distributed in 3D space, followed by repetitive imagery of movements to the same target or to a different target. Nine naive subjects generated both repetitive and non-repetitive slow actual movements to each of the five targets to test the effect of block design on decoding accuracy. In order to assure that base line drift and low frequency motion artifacts do not contaminate the data, the data were high-pass filtered in 4-30Hz, leaving out the delta and gamma band. For the repetitive trials, the model decoded target location with 81% accuracy, which is significantly higher than chance level. The average decoding rate of target location was only 30% for the non-repetitive trials, which is not significantly different than chance level. A subset of electrodes, mainly over the contralateral sensorimotor areas, was found to provide most of the discriminative features for all tested conditions. Time proximity between trained and tested blocks was found to enhance decoding accuracy of target location both by target non-specific and specific mechanisms. Our findings suggest that movement repetition promotes the development of distinct neural patterns, presumably by the formation of target-specific kinesthetic memory.

Long-term decoding of arm movement using Spatial Distribution of Neural Patterns.

Day to day variability and non-stationarity caused by changes in subject motivation, learning and behavior pose a challenge in using local field potentials (LFP) for practical Brain Computer Interfaces. Pattern recognition algorithms require that the features possess little to no variation from the training to test data. As such models developed on one day fail to represent the characteristics on the other day. This paper provides a solution in the form of adaptive spatial features. We propose an algorithm to capture the local spatial variability of LFP patterns and provide accurate long-term decoding. This algorithm achieved more than 95% decoding of eight movement directions two weeks after its initial training.

Source localization techniques for direction decoding from local field potentials.

Local Field Potential (LFP) recordings are one type of intracortical recordings, (besides Single Unit Activity) that can help decode movement direction successfully. In the longterm however, using LFPs for decoding presents some major challenges like inherent instability and non-stationarity. Our approach to overcome this challenge bases around the hypothesis that each task has a signature source-location pattern. The methodology involves introduction of source localization, and tracking of sources over a period of time that enables us to decode movement direction in an eight-direction center-out-reach-task. We establish that such tracking can be used for long term decoding, with preliminary results indicating consistent patterns. In fact, tracking task related source locations render up to 66% accuracy in decoding movement direction one week after the decoding model was learnt.

Robust movement direction decoders from local field potentials using spatio-temporal qualitative patterns.

A major drawback of using Local Field Potentials (LFP) for Brain Computer Interface (BCI) is their inherent instability and non-stationarity. Specifically, even when a well-trained subject performs the same task over a period of time, the neural data observed are unstable. To overcome this problem in decoding movement direction, this paper proposes the use of qualitative information in the form of spatial patterns of inter-channel ranking of multi-channel LFP recordings. The quality of the decoding was further refined by concentrating on the statistical distributions of the top powered channels. Decoding of movement direction was performed using Support Vector Machines (SVM) to construct decoders, instead of the traditional spatial patterns. Our algorithm provides a decoding power of up to 74% on average over a period of two weeks, compared with the state-of-the-art methods in the literature that yield only 33%. Furthermore, it provides 62.5% direction decoding in novel motor environments, compared with 29.5% with conventional methods. Finally, a comparison with the traditional methods and other surveyed literature is presented.

Real time 3D visualization of intraoperative organ deformations using structured dictionary.

Restricted visualization of the surgical field is one of the most critical challenges for minimally invasive surgery (MIS). Current intraoperative visualization systems are promising. However, they can hardly meet the requirements of high resolution and real time 3D visualization of the surgical scene to support the recognition of anatomic structures for safe MIS procedures. In this paper, we present a new approach for real time 3D visualization of organ deformations based on optical imaging patches with limited field-of-view and a single preoperative scan of magnetic resonance imaging (MRI) or computed tomography (CT). The idea for reconstruction is motivated by our empirical observation that the spherical harmonic coefficients corresponding to distorted surfaces of a given organ lie in lower dimensional subspaces in a structured dictionary that can be learned from a set of representative training surfaces. We provide both theoretical and practical designs for achieving these goals. Specifically, we discuss details about the selection of limited optical views and the registration of partial optical images with a single preoperative MRI/CT scan. The design proposed in this paper is evaluated with both finite element modeling data and ex vivo experiments. The ex vivo test is conducted on fresh porcine kidneys using 3D MRI scans with 1.2 mm resolution and a portable laser scanner with an accuracy of 0.13 mm. Results show that the proposed method achieves a sub-3 mm spatial resolution in terms of Hausdorff distance when using only one preoperative MRI scan and the optical patch from the single-sided view of the kidney. The reconstruction frame rate is between 10 frames/s and 39 frames/s depending on the complexity of the test model.

Sparse Representation of Deformable 3D Organs with Spherical Harmonics and Structured Dictionary.

This paper proposed a novel algorithm to sparsely represent a deformable surface (SRDS) with low dimensionality based on spherical harmonic decomposition (SHD) and orthogonal subspace pursuit (OSP). The key idea in SRDS method is to identify the subspaces from a training data set in the transformed spherical harmonic domain and then cluster each deformation into the best-fit subspace for fast and accurate representation. This algorithm is also generalized into applications of organs with both interior and exterior surfaces. To test the feasibility, we first use the computer models to demonstrate that the proposed approach matches the accuracy of complex mathematical modeling techniques and then both ex vivo and in vivo experiments are conducted using 3D magnetic resonance imaging (MRI) scans for verification in practical settings. All results demonstrated that the proposed algorithm features sparse representation of deformable surfaces with low dimensionality and high accuracy. Specifically, the precision evaluated as maximum error distance between the reconstructed surface and the MRI ground truth is better than 3 mm in real MRI experiments.

Prediction of pharmacologically induced baroreflex sensitivity from local time and frequency domain indices of R-R interval and systolic blood pressure signals obtained during deep breathing.

Pharmacological measurement of baroreflex sensitivity (BRS) is widely accepted and used in clinical practice. Following the introduction of pharmacologically induced BRS (p-BRS), alternative assessment methods eliminating the use of drugs were in the center of interest of the cardiovascular research community. In this study we investigated whether p-BRS using phenylephrine injection can be predicted from non-pharmacological time and frequency domain indices computed from electrocardiogram (ECG) and blood pressure (BP) data acquired during deep breathing. In this scheme, ECG and BP data were recorded from 16 subjects in a two-phase experiment. In the first phase the subjects performed irregular deep breaths and in the second phase the subjects received phenylephrine injection. From the first phase of the experiment, a large pool of predictors describing the local characteristic of beat-to-beat interval tachogram (RR) and systolic blood pressure (SBP) were extracted in time and frequency domains. A subset of these indices was selected using twelve subjects with an exhaustive search fused with a leave one subject out cross validation procedure. The selected indices were used to predict the p-BRS on the remaining four test subjects. A multivariate regression was used in all prediction steps. The algorithm achieved best prediction accuracy with only two features extracted from the deep breathing data, one from the frequency and the other from the time domain. The normalized L2-norm error was computed as 22.9% and the correlation coefficient was 0.97 (p=0.03). These results suggest that the p-BRS can be estimated from non-pharmacological indices computed from ECG and invasive BP data related to deep breathing.

Intragenic rearrangement and altered RNA splicing of the androgen receptor in a cell-based model of prostate cancer progression.

Androgen depletion for advanced prostate cancer (PCa) targets activity of the androgen receptor (AR), a steroid receptor transcription factor required for PCa growth. The emergence of lethal castration-resistant PCa (CRPCa) is marked by aberrant reactivation of the AR despite ongoing androgen depletion. Recently, alternative splicing has been described as a mechanism giving rise to COOH-terminally truncated, constitutively active AR isoforms that can support the CRPCa phenotype. However, the pathologic origin of these truncated AR isoforms is unknown. The goal of this study was to investigate alterations in AR expression arising in a cell-based model of PCa progression driven by truncated AR isoform activity. We show that stable, high-level expression of truncated AR isoforms in 22Rv1 CRPCa cells is associated with intragenic rearrangement of an approximately 35-kb AR genomic segment harboring a cluster of previously described alternative AR exons. Analysis of genomic data from clinical specimens indicated that related AR intragenic copy number alterations occurred in CRPCa in the context of AR amplification. Cloning of the break fusion junction in 22Rv1 cells revealed long interspersed nuclear elements (LINE-1) flanking the rearranged segment and a DNA repair signature consistent with microhomology-mediated, break-induced replication. This rearrangement served as a marker for the emergence of a rare subpopulation of CRPCa cells expressing high levels of truncated AR isoforms during PCa progression in vitro. Together, these data provide the first report of AR intragenic rearrangements in CRPCa and an association with pathologic expression of truncated AR isoforms in a cell-based model of PCa progression.

A subspace approach to learning recurrent features from brain activity.

This paper introduces a novel technique to address the instability and time variability challenges associated with brain activity recorded on different days. A critical challenge when working with brain signal activity is the variability in their characteristics when the signals are collected in different sessions separated by a day or more. Such variability is due to the acute and chronic responses of the brain tissue after implantation, variations as the subject learns to optimize performance, physiological changes in a subject due to prior activity or rest periods and environmental conditions. We propose a novel approach to tackle signal variability by focusing on learning subspaces which are recurrent over time. Furthermore, we illustrate how we can use projections on those subspaces to improve classification for an application such as brain-machine interface (BMI). In this paper, we illustrate the merits of finding recurrent subspaces in the context of movement direction decoding using local field potential (LFP). We introduce two methods for using the learned subspaces in movement direction decoding and show a decoding power improvement from 76% to 88% for a particularly unstable subject and consistent decoding across subjects.

Semi-supervised event detection using higher order statistics for multidimensional time series accelerometer data.

In this study, we target to automatically detect stereotypical behavioral patterns (stereotypy) and self-injurious behaviors (SIB) of Autistic children which can lead to critical damages or wounds as they tend to repeatedly harm oneself. Our custom designed accelerometer based wearable sensors are placed at wrists, ankles and upper body to detect stereotypy and SIB. The analysis was done on four children diagnosed with ASD who showed repeated behaviors that involve part of the body such as flapping arms, body rocking and self-injurious behaviors such as punching their face, or hitting their legs. Our goal of detecting novel events relies on the fact that the limitation of training data and variability in the possible combination of signals and events also make it impossible to design a single algorithm to understand all events in natural setting. Therefore, a semi-supervised method to discover and track unknown events in a multidimensional sensor data rises as a very important topic in classification and detection problems. In this paper, we show how the Higher Order Statistics (HOS) features can be used to design dictionaries and to detect novel events in a multichannel time series data. We explain our methods to detect novel events in a multidimensional time series data and combine the proposed semi-supervised learning method to improve the adaptability of the system while maintaining comparable detection accuracy as the supervised method. We, compare our results to the supervised methods that we have previously developed and show that although semi-supervised method do not achieve better performance compared to supervised methods, it can efficiently find new events and anomalies in multidimensional time series data with similar performance of the supervised method. We show that our proposed method achieves recall rate of 93.3% compared to 94.1% for the supervised method studied earlier.

Automatic characterization and detection of behavioral patterns using linear predictive coding of accelerometer sensor data.

In this study, we target to automatically detect behavioral patterns of patients with autism. Many stereotypical behavioral patterns may hinder their learning ability as a child and patterns such as self-injurious behaviors (SIB) can lead to critical damages or wounds as they tend to repeatedly harm one single location. Our custom designed accelerometer based wearable sensor can be placed at various locations of the body to detect stereotypical self-stimulatory behaviors (stereotypy) and self-injurious behaviors of patients with Autism Spectrum Disorder (ASD). A microphone was used to record sounds so that we may understand the surrounding environment and video provided ground truth for analysis. The analysis was done on four children diagnosed with ASD who showed repeated self-stimulatory behaviors that involve part of the body such as flapping arms, body rocking and self-injurious behaviors such as punching their face, or hitting their legs. The goal of this study is to devise novel algorithms to detect these events and open possibility for design of intervention methods. In this paper, we have shown time domain pattern matching with linear predictive coding (LPC) of data to design detection and classification of these ASD behavioral events. We observe clusters of pole locations from LPC roots to select candidates and apply pattern matching for classification. We also show novel event detection using online dictionary update method. We show that our proposed method achieves recall rate of 95.5% for SIB, 93.5% for flapping, and 95.5% for rocking which is an increase of approximately 5% compared to flapping events detected by using wrist worn sensors in our previous study.

Time robust movement direction decoding in Local Field Potentials using channel ranking.

Movement direction for Brain Machine Interface (BMI) can be decoded successfully using Local Field Potentials (LFP) and Single Unit Activity (SUA). A major challenge when dealing with the intra-cortical recordings is to develop decoders that are robust in time. In this paper we present for the first time a technique that uses the qualitative information derived from multiple LFP channels rather than the absolute power of the recorded signals. In this novel method, we use a power based inter-channel ranking system to define the quality of a channel in multi-channel LFP. This representation enables us to bypass the problems associated with the dynamic ranges of absolute power. We also introduce a parameter based ranking system that provides the same rank to channels that have comparable powers. We show that using our algorithms, we can develop models that provide stable decoding of eight movement directions with an average efficiency of above 56% over a period of two weeks. Moreover, the decoding power using this method is 46% at the end of two weeks versus the 13% using the traditional approaches. We also applied these models to decoding movements performed in a force field and again achieved significantly higher decoding power than the existing methods.

A real-time cardiac surface tracking system using Subspace Clustering.

Catheter based radio frequency ablation of atrial fibrillation requires real-time 3D tracking of cardiac surfaces with sub-millimeter accuracy. To best of our knowledge, there are no commercial or non-commercial systems capable to do so. In this paper, a system for high-accuracy 3D tracking of cardiac surfaces in real-time is proposed and results applied to a real patient dataset are presented. Proposed system uses Subspace Clustering algorithm to identify the potential deformation subspaces for cardiac surfaces during the training phase from pre-operative MRI scan based training set. In Tracking phase, using low-density outer cardiac surface samples, active deformation subspace is identified and complete inner & outer cardiac surfaces are reconstructed in real-time under a least squares formulation.