Self-supervised Natural Image Reconstruction and Large-scale Semantic Classification from Brain Activity.

Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs of images and their corresponding fMRI responses, which span the huge space of natural images, is prohibitive. We present a novel self-supervised approach that goes well beyond the scarce paired data, for achieving both: (i) state-of-the art fMRI-to-image reconstruction, and (ii) first-ever large-scale semantic classification from fMRI responses. By imposing cycle consistency between a pair of deep neural networks (from image-to-fMRI & from fMRI-to-image), we train our image reconstruction network on a large number of "unpaired" natural images (images without fMRI recordings) from many novel semantic categories. This enables to adapt our reconstruction network to a very rich semantic coverage without requiring any explicit semantic supervision. Specifically, we find that combining our self-supervised training with high-level perceptual losses, gives rise to new reconstruction & classification capabilities. In particular, this perceptual training enables to classify well fMRIs of never-before-seen semantic classes, without requiring any class labels during training. This gives rise to: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing), and (ii) Large-scale semantic classification of categories that were never-before-seen during network training. Such large-scale (1000-way) semantic classification from fMRI recordings has never been demonstrated before. Finally, we provide evidence for the biological consistency of our learned model.

Natural Language Generation Model for Mammography Reports Simulation.

Extending the size of labeled corpora of medical reports is a major step towards a successful training of machine learning algorithms. Simulating new text reports is a key solution for reports augmentation, which extends the cohort size. However, text generation in the medical domain is challenging because it needs to preserve both content and style that are typical for real reports, without risking the patients' privacy. In this paper, we present a conditioned LSTM-RNN architecture for simulating realistic mammography reports. We evaluated the performance by analyzing the characteristics of the simulated reports and classifying them into benign and malignant classes. An average classification AUC was calculated over two distinct test sets. A qualitative analysis was also performed in which a masked radiologist classified 0.75 of the simulated reports as real reports, showing that both the style and content of the simulated reports were similar to real reports. Finally, we compared our RNN-LSTM generative model with Markov Random Fields. The RNN-LSTM provided significantly better and more stable performance than MRFs ( , Wilcoxon).

Spatial Characterization of Tumor Perfusion Properties from 3D DCE-US Perfusion Maps are Early Predictors of Cancer Treatment Response.

There is a need for noninvasive repeatable biomarkers to detect early cancer treatment response and spare non-responders unnecessary morbidities and costs. Here, we introduce three-dimensional (3D) dynamic contrast enhanced ultrasound (DCE-US) perfusion map characterization as inexpensive, bedside and longitudinal indicator of tumor perfusion for prediction of vascular changes and therapy response. More specifically, we developed computational tools to generate perfusion maps in 3D of tumor blood flow, and identified repeatable quantitative features to use in machine-learning models to capture subtle multi-parametric perfusion properties, including heterogeneity. Models were developed and trained in mice data and tested in a separate mouse cohort, as well as early validation clinical data consisting of patients receiving therapy for liver metastases. Models had excellent (ROC-AUC > 0.9) prediction of response in pre-clinical data, as well as proof-of-concept clinical data. Significant correlations with histological assessments of tumor vasculature were noted (Spearman R > 0.70) in pre-clinical data. Our approach can identify responders based on early perfusion changes, using perfusion properties correlated to gold-standard vascular properties.

Automated Quantitative Imaging Measurements of Disease Severity in Patients with Nonthrombotic Iliac Vein Compression.

PURPOSE: An automated segmentation technique (AST) for computed tomography (CT) venography was developed to quantify measures of disease severity before and after stent placement in patients with left-sided nonthrombotic iliac vein compression. MATERIALS AND METHODS: Twenty-one patients with left-sided nonthrombotic iliac vein compression who underwent venous stent placement were retrospectively identified. Pre- and poststent CT venography studies were quantitatively analyzed using an AST to determine leg volume, skin thickness, and water content of fat. These measures were compared between diseased and nondiseased limbs and between pre- and poststent images, using patients as their own controls. Additionally, patients with and without postthrombotic lesions were compared. RESULTS: The AST detected significantly increased leg volume (12,437 cm3 vs 10,748 cm3, P < .0001), skin thickness (0.531 cm vs 0.508 cm, P < .0001), and water content of fat (8.2% vs 5.0%, P < .0001) in diseased left limbs compared with the contralateral nondiseased limbs, on prestent imaging. After stent placement in the left leg, there was a significant decrease in the water content of fat in the right (4.9% vs 2.7%, P < .0001) and left (8.2% vs 3.2%, P < .0001) legs. There were no significant changes in leg volume or skin thickness in either leg after stent placement. There were no significant differences between patients with or without postthrombotic lesions in their poststent improvement across the 3 measures of disease severity. CONCLUSIONS: ASTs can be used to quantify measures of disease severity and postintervention changes on CT venography for patients with lower extremity venous disease. Further investigation may clarify the clinical benefit of such technologies.

Automated dendritic spine detection using convolutional neural networks on maximum intensity projected microscopic volumes.

BACKGROUND: Dendritic spines are structural correlates of excitatory synapses in the brain. Their density and structure are shaped by experience, pointing to their role in memory encoding. Dendritic spine imaging, followed by manual analysis, is a primary way to study spines. However, an approach that analyses dendritic spines images in an automated and unbiased manner is needed to fully capture how spines change with normal experience, as well as in disease. NEW METHOD: We propose an approach based on fully convolutional neural networks (FCNs) to detect dendritic spines in two-dimensional maximum-intensity projected images from confocal fluorescent micrographs. We experiment on both fractionally strided convolution and efficient sub-pixel convolutions. Dendritic spines far from the dendritic shaft are pruned by extraction of the shaft to reduce false positives. Performance of the proposed method is evaluated by comparing predicted spine positions to those manually marked by experts. RESULTS: The averaged distance between predicted and manually annotated spines is 2.81 ± 2.63 pixels (0.082 ± 0.076 microns) and 2.87 ± 2.33 pixels (0.084 ± 0.068 microns) based on two different experts. FCN-based detection achieves F scores > 0.80 for both sets of expert annotations. COMPARISON WITH EXISTING METHODS: Our method significantly outperforms two well-known software, NeuronStudio and Neurolucida (p-value < 0.02). CONCLUSIONS: FCN architectures used in this work allow for automated dendritic spine detection. Superior outcomes are possible even with small training data-sets. The proposed method may generalize to other datasets on larger scales.

A curated mammography data set for use in computer-aided detection and diagnosis research.

Published research results are difficult to replicate due to the lack of a standard evaluation data set in the area of decision support systems in mammography; most computer-aided diagnosis (CADx) and detection (CADe) algorithms for breast cancer in mammography are evaluated on private data sets or on unspecified subsets of public databases. This causes an inability to directly compare the performance of methods or to replicate prior results. We seek to resolve this substantial challenge by releasing an updated and standardized version of the Digital Database for Screening Mammography (DDSM) for evaluation of future CADx and CADe systems (sometimes referred to generally as CAD) research in mammography. Our data set, the CBIS-DDSM (Curated Breast Imaging Subset of DDSM), includes decompressed images, data selection and curation by trained mammographers, updated mass segmentation and bounding boxes, and pathologic diagnosis for training data, formatted similarly to modern computer vision data sets. The data set contains 753 calcification cases and 891 mass cases, providing a data-set size capable of analyzing decision support systems in mammography.

Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions.

Quantitative analysis of brain MRI is routine for many neurological diseases and conditions and relies on accurate segmentation of structures of interest. Deep learning-based segmentation approaches for brain MRI are gaining interest due to their self-learning and generalization ability over large amounts of data. As the deep learning architectures are becoming more mature, they gradually outperform previous state-of-the-art classical machine learning algorithms. This review aims to provide an overview of current deep learning-based segmentation approaches for quantitative brain MRI. First we review the current deep learning architectures used for segmentation of anatomical brain structures and brain lesions. Next, the performance, speed, and properties of deep learning approaches are summarized and discussed. Finally, we provide a critical assessment of the current state and identify likely future developments and trends.

Adaptive local window for level set segmentation of CT and MRI liver lesions.

We propose a novel method, the adaptive local window, for improving level set segmentation technique. The window is estimated separately for each contour point, over iterations of the segmentation process, and for each individual object. Our method considers the object scale, the spatial texture, and the changes of the energy functional over iterations. Global and local statistics are considered by calculating several gray level co-occurrence matrices. We demonstrate the capabilities of the method in the domain of medical imaging for segmenting 233 images with liver lesions. To illustrate the strength of our method, those lesions were screened by either Computed Tomography or Magnetic Resonance Imaging. Moreover, we analyzed images using three different energy models. We compared our method to a global level set segmentation, to a local framework that uses predefined fixed-size square windows and to a local region-scalable fitting model. The results indicate that our proposed method outperforms the other methods in terms of agreement with the manual marking and dependence on contour initialization or the energy model used. In case of complex lesions, such as low contrast lesions, heterogeneous lesions, or lesions with a noisy background, our method shows significantly better segmentation with an improvement of 0.25 ± 0.13 in Dice similarity coefficient, compared with state of the art fixed-size local windows (Wilcoxon, p < 0.001).

Adaptive Estimation of Active Contour Parameters Using Convolutional Neural Networks and Texture Analysis.

In this paper, we propose a generalization of the level set segmentation approach by supplying a novel method for adaptive estimation of active contour parameters. The presented segmentation method is fully automatic once the lesion has been detected. First, the location of the level set contour relative to the lesion is estimated using a convolutional neural network (CNN). The CNN has two convolutional layers for feature extraction, which lead into dense layers for classification. Second, the output CNN probabilities are then used to adaptively calculate the parameters of the active contour functional during the segmentation process. Finally, the adaptive window size surrounding each contour point is re-estimated by an iterative process that considers lesion size and spatial texture. We demonstrate the capabilities of our method on a dataset of 164 MRI and 112 CT images of liver lesions that includes low contrast and heterogeneous lesions as well as noisy images. To illustrate the strength of our method, we evaluated it against state of the art CNN-based and active contour techniques. For all cases, our method, as assessed by Dice similarity coefficients, performed significantly better than currently available methods. An average Dice improvement of 0.27 was found across the entire dataset over all comparisons. We also analyzed two challenging subsets of lesions and obtained a significant Dice improvement of 0.24 with our method (p <;0.001, Wilcoxon).

Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles.

Computerized analysis of digital pathology images offers the potential of improving clinical care (e.g. automated diagnosis) and catalyzing research (e.g. discovering disease subtypes). There are two key challenges thwarting computerized analysis of digital pathology images: first, whole slide pathology images are massive, making computerized analysis inefficient, and second, diverse tissue regions in whole slide images that are not directly relevant to the disease may mislead computerized diagnosis algorithms. We propose a method to overcome both of these challenges that utilizes a coarse-to-fine analysis of the localized characteristics in pathology images. An initial surveying stage analyzes the diversity of coarse regions in the whole slide image. This includes extraction of spatially localized features of shape, color and texture from tiled regions covering the slide. Dimensionality reduction of the features assesses the image diversity in the tiled regions and clustering creates representative groups. A second stage provides a detailed analysis of a single representative tile from each group. An Elastic Net classifier produces a diagnostic decision value for each representative tile. A weighted voting scheme aggregates the decision values from these tiles to obtain a diagnosis at the whole slide level. We evaluated our method by automatically classifying 302 brain cancer cases into two possible diagnoses (glioblastoma multiforme (N = 182) versus lower grade glioma (N = 120)) with an accuracy of 93.1% (p << 0.001). We also evaluated our method in the dataset provided for the 2014 MICCAI Pathology Classification Challenge, in which our method, trained and tested using 5-fold cross validation, produced a classification accuracy of 100% (p << 0.001). Our method showed high stability and robustness to parameter variation, with accuracy varying between 95.5% and 100% when evaluated for a wide range of parameters. Our approach may be useful to automatically differentiate between the two cancer subtypes.

Improved Patch-Based Automated Liver Lesion Classification by Separate Analysis of the Interior and Boundary Regions.

The bag-of-visual-words (BoVW) method with construction of a single dictionary of visual words has been used previously for a variety of classification tasks in medical imaging, including the diagnosis of liver lesions. In this paper, we describe a novel method for automated diagnosis of liver lesions in portal-phase computed tomography (CT) images that improves over single-dictionary BoVW methods by using an image patch representation of the interior and boundary regions of the lesions. Our approach captures characteristics of the lesion margin and of the lesion interior by creating two separate dictionaries for the margin and the interior regions of lesions ("dual dictionaries" of visual words). Based on these dictionaries, visual word histograms are generated for each region of interest within the lesion and its margin. For validation of our approach, we used two datasets from two different institutions, containing CT images of 194 liver lesions (61 cysts, 80 metastasis, and 53 hemangiomas). The final diagnosis of each lesion was established by radiologists. The classification accuracy for the images from the two institutions was 99% and 88%, respectively, and 93% for a combined dataset. Our new BoVW approach that uses dual dictionaries shows promising results. We believe the benefits of our approach may generalize to other application domains within radiology.

Weighted locality-constrained linear coding for lesion classification in CT images.

Computed tomography is a popular imaging modality for detecting abnormalities associated with abdominal organs such as the liver, kidney and uterus. In this paper, we propose a novel weighted locality-constrained linear coding (LLC) method followed by a weighted max-pooling method to classify liver lesions into three classes: cysts, metastases, hemangiomas. We first divide the lesions into same-size patches. Then, we extract the raw features in all patches followed by Principal Components Analysis (PCA) and apply K means to obtain a single LLC dictionary. Since the interior lesion patches and the boundary patches contribute different information in the image, we assign different weights on these two types of patches to obtain the LLC codes. Moreover, a weighted max pooling approach is also proposed to further evaluate the importance of these two types of patches in feature pooling. Experiments on 109 images of liver lesions were carried out to validate the proposed method. The proposed method achieves a best lesion classification accuracy of 96.33%, which appears to be superior compared with traditional image coding methods: LLC method and Bag-of-words method (BoW) and traditional features: Local Binary Pattern (LBP) features, uniform LBP and complete LBP, demonstrating that the proposed method provides better classification.

New quantification methods for carotid intra-plaque neovascularization using contrast-enhanced ultrasound.

As carotid intra-plaque neovascularization (IPN) is linked to progressive atherosclerotic disease and plaque vulnerability, its accurate quantification might allow early detection of plaque vulnerability. We therefore developed several new quantitative methods for analyzing IPN perfusion and structure. From our analyses, we derived six quantitative parameters-IPN surface area (IPNSA), IPN surface ratio (IPNSR), plaque mean intensity, plaque-to-lumen enhancement ratio, mean plaque contrast percentage and number of micro-vessels (MVN)-and compared these with visual grading of IPN by two independent physicians. A total of 45 carotid arteries with symptomatic stenosis in 23 patients were analyzed. IPNSA (correlation r = 0.719), IPNSR (r = 0.538) and MVN (r = 0.484) were found to be significantly correlated with visual scoring (p < 0.01). IPNSA was the best match to visual scoring. These results indicate that IPNSA, IPNSR and MVN may have the potential to replace qualitative visual scoring and to measure the degree of carotid IPN.

Evaluation of a 3D technique for quantifying neovascularization within plaques imaged by contrast enhanced ultrasound.

Intra-plaque neovascularization and inflammation are considered as important indicators of plaque vulnerability, which when ruptured, may cause stroke or acute myocardial infarction. The purpose of this research was to validate and evaluate a semi-automatic method, which allows quantification of carotid plaque neovascularization using contrast-enhanced ultrasound cines, thus enabling assessment of plaque vulnerability. The method detects contrast clusters in the images, and tracks them, to generate over time a path that portrays the neovasculature. It classifies the paths as either artifacts or `blood vessels' and reconstructs the 3D arterial tree. Software-based phantom was developed to represent volumetric structures of the carotid lumen, the plaque, and `objects' passing through the intra-plaque neovasculature. These 3D objects, which mimic microbubbles or clusters of microbubbles, were based on original 2D formations, imaged during clinical examinations using contrast-enhanced ultrasound. Within a plaque, several paths were constructed, representing flow inside blood vessels, and several isolated objects were added, representing artifacts. Different paths were generated, classified into 4 groups: separate paths, paths that merge at some point, paths that branch and intersecting paths. The phantom was used to generate sets of cines, which were then processed by the method. The method identified artifacts and different paths, which were then compared to the `true' ones. Sixty-four 'objects' in 16 movies were examined. All of them were detected. 79% of those objects were well tracked and classified to either artifacts or real blood vessels. The results of this study show that the method accurately identifies artifacts and paths, which allows reconstruction of intra-plaque vascular tree and quantification of the plaque neovasculature, which is associated with plaque vulnerability.

Quantitative analysis of ultrasound contrast flow behavior in carotid plaque neovasculature.

Intraplaque neovascularization is considered as an important indication for plaque vulnerability. We propose a semiautomatic algorithm for quantification of neovasculature, thus, enabling assessment of plaque vulnerability. The algorithm detects and tracks contrast spots using multidimensional dynamic programming. Classification of contrast tracks into blood vessels and artifacts was performed. The results were compared with manual tracking, visual classification and maximal intensity projection. In 28 plaques, 97% of the contrast spots were detected. In 89% of the objects, the automatic tracking determined the contrast motion with an average distance of less than 0.5 mm from the manual marking. Furthermore, 75% were correctly classified into artifacts and vessels. The automated neovascularization grading agreed within 1 grade with visual analysis in 91% of the cases, which was comparable to the interobserver variability of visual grading. These results show that the method can successfully quantify features that are linked to vulnerability of the carotid plaque.

Carotid plaque vulnerability: quantification of neovascularization on contrast-enhanced ultrasound with histopathologic correlation.

OBJECTIVE: The purpose of this research is to develop a computerized method to quantify carotid plaque neovascularization on contrast-enhanced ultrasound images and to compare the results with the histopathologic analysis of the plaque. SUBJECTS AND METHODS: Twenty-seven patients (age range, 48-84 years; mean [± SD] age, 68.4 ± 9.72 years) were recruited before endarterectomy. Contrast-enhanced ultrasound examination of the carotid artery was performed by applying low mechanical index and harmonics with pulse inversion. An algorithm was developed that implemented several image processing methods to automatically quantify neovascularization and reconstruct the vascular tree in the atheromatous plaque. Neovascularization and the number of inflammatory cells seen on histopathologic analysis of the plaque after endarterectomy were compared with neovascularization determined by the computerized method. The mean (± SD) ratios of the ultrasound and histopathologic measurements were calculated. RESULTS: In five patients, heavy calcification of the plaque prevented visualization of plaque texture. Intraplaque neovascularization on contrast-enhanced ultrasound images was significant in 19 patients and low in three patients. The ratio of the neovascularization area to the total plaque area on contrast-enhanced ultrasound images was well correlated with the same histopathologic ratio (R(2) = 0.7905) and with the number of inflammatory cells present in the plaque (R(2) = 0.6109). The histopathologic ratio and the number of intraplaque inflammatory cells also were well correlated (R(2) = 0.7034). CONCLUSION: The newly developed method allowed quantification of the intraplaque neovascularization as a feature of vulnerability in the carotid plaque and proved to be highly correlated with histopathologic results.

Subharmonic response of encapsulated microbubbles: conditions for existence and amplification.

The response of encapsulated microbubbles at half the ultrasound insonation frequency, termed subharmonic response, may have potential applications in diagnosis and therapy. The subharmonic signal, emitted by Definity microbubble cloud sonicated by ultrasound was studied theoretically and experimentally. The size distribution of the microbubbles was optically analyzed and resonance frequency of 2.7 MHz was determined. An asymptotic model has been developed that generates subharmonic response of a single and of a cloud of bubbles of various sizes. Threshold conditions for existence and the intensity of the subharmonic signal are predicted to depend on microbubbles size distribution and shell properties, as well as on the driving field frequency and pressure. Thin tubes filled with Definity solution were insonated at acoustic pressures from 100 to 630 kPa. The intensities of the emitted fundamental harmonics and subharmonics were measured. At frequency 5.5MHz, twice the resonance frequency, the subharmonic signals were observed only at pressures greater than 190 kPa. The subharmonic to fundamental harmonics intensity ratio was within -12 to -1 dB. The experimental results showed good correlation with the theoretical results in the range of validity of the asymptotic solution, thus supporting the model assumptions.