Unified Multi-Modal Image Synthesis for Missing Modality Imputation.

Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.

PELE scores: pelvic X-ray landmark detection with pelvis extraction and enhancement.

PURPOSE: Pelvic X-ray (PXR) is widely utilized in clinical decision-making associated with the pelvis, the lower part of the trunk that supports and balances the trunk. In particular, PXR-based landmark detection facilitates downstream analysis and computer-assisted diagnosis and treatment of pelvic diseases. Although PXR has the advantages of low radiation and reduced cost compared to computed tomography (CT), it characterizes the 2D pelvis-tissue superposition of 3D structures, which may affect the accuracy of landmark detection in some cases. However, the superposition nature of PXR is implicitly handled by existing deep learning-based landmark detection methods, which mainly design the deep network structures for better detection performances. Explicit handling of the superposition nature of PXR is rarely done. METHODS: In this paper, we explicitly focus on the superposition of X-ray images. Specifically, we propose a pelvis extraction (PELE) module that consists of a decomposition network, a domain adaptation network, and an enhancement module, which utilizes 3D prior anatomical knowledge in CT to guide and well isolate the pelvis from PXR, thereby eliminating the influence of soft tissue for landmark detection. The extracted pelvis image, after enhancement, is then used for landmark detection. RESULTS: We conduct an extensive evaluation based on two public and one private dataset, totaling 850 PXRs. The experimental results show that the proposed PELE module significantly improves the accuracy of PXRs landmark detection and achieves state-of-the-art performances in several benchmark metrics. CONCLUSION: The design of PELE module can improve the accuracy of different pelvic landmark detection baselines, which we believe is obviously conducive to the positioning and inspection of clinical landmarks and critical structures, thus better serving downstream tasks. Our project has been open-sourced at https://github.com/ECNUACRush/PELEscores .

Adversarial Medical Image with Hierarchical Feature Hiding.

Deep learning based methods for medical images can be easily compromised by adversarial examples (AEs), posing a great security flaw in clinical decision-making. It has been discovered that conventional adversarial attacks like PGD which optimize the classification logits, are easy to distinguish in the feature space, resulting in accurate reactive defenses. To better understand this phenomenon and reassess the reliability of the reactive defenses for medical AEs, we thoroughly investigate the characteristic of conventional medical AEs. Specifically, we first theoretically prove that conventional adversarial attacks change the outputs by continuously optimizing vulnerable features in a fixed direction, thereby leading to outlier representations in the feature space. Then, a stress test is conducted to reveal the vulnerability of medical images, by comparing with natural images. Interestingly, this vulnerability is a double-edged sword, which can be exploited to hide AEs. We then propose a simple-yet-effective hierarchical feature constraint (HFC), a novel add-on to conventional white-box attacks, which assists to hide the adversarial feature in the target feature distribution. The proposed method is evaluated on three medical datasets, both 2D and 3D, with different modalities. The experimental results demonstrate the superiority of HFC, i.e., it bypasses an array of state-of-the-art adversarial medical AE detectors more efficiently than competing adaptive attacks1, which reveals the deficiencies of medical reactive defense and allows to develop more robust defenses in future.

Combo loss: Handling input and output imbalance in multi-organ segmentation.

Simultaneous segmentation of multiple organs from different medical imaging modalities is a crucial task as it can be utilized for computer-aided diagnosis, computer-assisted surgery, and therapy planning. Thanks to the recent advances in deep learning, several deep neural networks for medical image segmentation have been introduced successfully for this purpose. In this paper, we focus on learning a deep multi-organ segmentation network that labels voxels. In particular, we examine the critical choice of a loss function in order to handle the notorious imbalance problem that plagues both the input and output of a learning model. The input imbalance refers to the class-imbalance in the input training samples (i.e., small foreground objects embedded in an abundance of background voxels, as well as organs of varying sizes). The output imbalance refers to the imbalance between the false positives and false negatives of the inference model. In order to tackle both types of imbalance during training and inference, we introduce a new curriculum learning based loss function. Specifically, we leverage Dice similarity coefficient to deter model parameters from being held at bad local minima and at the same time gradually learn better model parameters by penalizing for false positives/negatives using a cross entropy term. We evaluated the proposed loss function on three datasets: whole body positron emission tomography (PET) scans with 5 target organs, magnetic resonance imaging (MRI) prostate scans, and ultrasound echocardigraphy images with a single target organ i.e., left ventricular. We show that a simple network architecture with the proposed integrative loss function can outperform state-of-the-art methods and results of the competing methods can be improved when our proposed loss is used.

Material damage diagnosis and characterization for turbine rotors using three-dimensional adaptive ultrasonic NDE data reconstruction techniques.

Damage diagnosis for turbine rotors plays an essential role in power plant management. Ultrasonic non-destructive examinations (NDEs) have increasingly been utilized as an effective tool to provide comprehensive information for damage diagnosis. This study presents a general methodology of damage diagnosis for turbine rotors using three-dimensional adaptive ultrasonic NDE data reconstruction techniques. Volume reconstruction algorithms and data fusion schemes are proposed to map raw ultrasonic NDE data back to the structural model of the object being examined. The reconstructed volume is used for automatic damage identification and quantification using region-growing algorithms and the method of distance-gain-size. Key reconstruction parameters are discussed and suggested based on industrial experiences. A software tool called AutoNDE Rotor is developed to automate the overall analysis workflow. Effectiveness of the proposed methods and AutoNDE Rotor are explored using realistic ultrasonic NDE data.

Spine detection in CT and MR using iterated marginal space learning.

Examinations of the spinal column with both, Magnetic Resonance (MR) imaging and Computed Tomography (CT), often require a precise three-dimensional positioning, angulation and labeling of the spinal disks and the vertebrae. A fully automatic and robust approach is a prerequisite for an automated scan alignment as well as for the segmentation and analysis of spinal disks and vertebral bodies in Computer Aided Diagnosis (CAD) applications. In this article, we present a novel method that combines Marginal Space Learning (MSL), a recently introduced concept for efficient discriminative object detection, with a generative anatomical network that incorporates relative pose information for the detection of multiple objects. It is used to simultaneously detect and label the spinal disks. While a novel iterative version of MSL is used to quickly generate candidate detections comprising position, orientation, and scale of the disks with high sensitivity, the anatomical network selects the most likely candidates using a learned prior on the individual nine dimensional transformation spaces. Finally, we propose an optional case-adaptive segmentation approach that allows to segment the spinal disks and vertebrae in MR and CT respectively. Since the proposed approaches are learning-based, they can be trained for MR or CT alike. Experimental results based on 42 MR and 30 CT volumes show that our system not only achieves superior accuracy but also is among the fastest systems of its kind in the literature. On the MR data set the spinal disks of a whole spine are detected in 11.5s on average with 98.6% sensitivity and 0.073 false positive detections per volume. On the CT data a comparable sensitivity of 98.0% with 0.267 false positives is achieved. Detected disks are localized with an average position error of 2.4 mm/3.2 mm and angular error of 3.9°/4.5° in MR/CT, which is close to the employed hypothesis resolution of 2.1 mm and 3.3°.