A 3D hierarchical cross-modality interaction network using transformers and convolutions for brain glioma segmentation in MR images.

BACKGROUND: Precise glioma segmentation from multi-parametric magnetic resonance (MR) images is essential for brain glioma diagnosis. However, due to the indistinct boundaries between tumor sub-regions and the heterogeneous appearances of gliomas in volumetric MR scans, designing a reliable and automated glioma segmentation method is still challenging. Although existing 3D Transformer-based or convolution-based segmentation networks have obtained promising results via multi-modal feature fusion strategies or contextual learning methods, they widely lack the capability of hierarchical interactions between different modalities and cannot effectively learn comprehensive feature representations related to all glioma sub-regions. PURPOSE: To overcome these problems, in this paper, we propose a 3D hierarchical cross-modality interaction network (HCMINet) using Transformers and convolutions for accurate multi-modal glioma segmentation, which leverages an effective hierarchical cross-modality interaction strategy to sufficiently learn modality-specific and modality-shared knowledge correlated to glioma sub-region segmentation from multi-parametric MR images. METHODS: In the HCMINet, we first design a hierarchical cross-modality interaction Transformer (HCMITrans) encoder to hierarchically encode and fuse heterogeneous multi-modal features by Transformer-based intra-modal embeddings and inter-modal interactions in multiple encoding stages, which effectively captures complex cross-modality correlations while modeling global contexts. Then, we collaborate an HCMITrans encoder with a modality-shared convolutional encoder to construct the dual-encoder architecture in the encoding stage, which can learn the abundant contextual information from global and local perspectives. Finally, in the decoding stage, we present a progressive hybrid context fusion (PHCF) decoder to progressively fuse local and global features extracted by the dual-encoder architecture, which utilizes the local-global context fusion (LGCF) module to efficiently alleviate the contextual discrepancy among the decoding features. RESULTS: Extensive experiments are conducted on two public and competitive glioma benchmark datasets, including the BraTS2020 dataset with 494 patients and the BraTS2021 dataset with 1251 patients. Results show that our proposed method outperforms existing Transformer-based and CNN-based methods using other multi-modal fusion strategies in our experiments. Specifically, the proposed HCMINet achieves state-of-the-art mean DSC values of 85.33% and 91.09% on the BraTS2020 online validation dataset and the BraTS2021 local testing dataset, respectively. CONCLUSIONS: Our proposed method can accurately and automatically segment glioma regions from multi-parametric MR images, which is beneficial for the quantitative analysis of brain gliomas and helpful for reducing the annotation burden of neuroradiologists.

A 3D boundary-guided hybrid network with convolutions and Transformers for lung tumor segmentation in CT images.

-Accurate lung tumor segmentation from Computed Tomography (CT) scans is crucial for lung cancer diagnosis. Since the 2D methods lack the volumetric information of lung CT images, 3D convolution-based and Transformer-based methods have recently been applied in lung tumor segmentation tasks using CT imaging. However, most existing 3D methods cannot effectively collaborate the local patterns learned by convolutions with the global dependencies captured by Transformers, and widely ignore the important boundary information of lung tumors. To tackle these problems, we propose a 3D boundary-guided hybrid network using convolutions and Transformers for lung tumor segmentation, named BGHNet. In BGHNet, we first propose the Hybrid Local-Global Context Aggregation (HLGCA) module with parallel convolution and Transformer branches in the encoding phase. To aggregate local and global contexts in each branch of the HLGCA module, we not only design the Volumetric Cross-Stripe Window Transformer (VCSwin-Transformer) to build the Transformer branch with local inductive biases and large receptive fields, but also design the Volumetric Pyramid Convolution with transformer-based extensions (VPConvNeXt) to build the convolution branch with multi-scale global information. Then, we present a Boundary-Guided Feature Refinement (BGFR) module in the decoding phase, which explicitly leverages the boundary information to refine multi-stage decoding features for better performance. Extensive experiments were conducted on two lung tumor segmentation datasets, including a private dataset (HUST-Lung) and a public benchmark dataset (MSD-Lung). Results show that BGHNet outperforms other state-of-the-art 2D or 3D methods in our experiments, and it exhibits superior generalization performance in both non-contrast and contrast-enhanced CT scans.

CT-based quantification of intratumoral heterogeneity for predicting pathologic complete response to neoadjuvant immunochemotherapy in non-small cell lung cancer.

Objectives: To investigate the prediction of pathologic complete response (pCR) in patients with non-small cell lung cancer (NSCLC) undergoing neoadjuvant immunochemotherapy (NAIC) using quantification of intratumoral heterogeneity from pre-treatment CT image. Methods: This retrospective study included 178 patients with NSCLC who underwent NAIC at 4 different centers. The training set comprised 108 patients from center A, while the external validation set consisted of 70 patients from center B, center C, and center D. The traditional radiomics model was contrasted using radiomics features. The radiomics features of each pixel within the tumor region of interest (ROI) were extracted. The optimal division of tumor subregions was determined using the K-means unsupervised clustering method. The internal tumor heterogeneity habitat model was developed using the habitats features from each tumor sub-region. The LR algorithm was employed in this study to construct a machine learning prediction model. The diagnostic performance of the model was evaluated using criteria such as area under the receiver operating characteristic curve (AUC), accuracy, specificity, sensitivity, positive predictive value (PPV), and negative predictive value (NPV). Results: In the training cohort, the traditional radiomics model achieved an AUC of 0.778 [95% confidence interval (CI): 0.688-0.868], while the tumor internal heterogeneity habitat model achieved an AUC of 0.861 (95% CI: 0.789-0.932). The tumor internal heterogeneity habitat model exhibits a higher AUC value. It demonstrates an accuracy of 0.815, surpassing the accuracy of 0.685 achieved by traditional radiomics models. In the external validation cohort, the AUC values of the two models were 0.723 (CI: 0.591-0.855) and 0.781 (95% CI: 0.673-0.889), respectively. The habitat model continues to exhibit higher AUC values. In terms of accuracy evaluation, the tumor heterogeneity habitat model outperforms the traditional radiomics model, achieving a score of 0.743 compared to 0.686. Conclusion: The quantitative analysis of intratumoral heterogeneity using CT to predict pCR in NSCLC patients undergoing NAIC holds the potential to inform clinical decision-making for resectable NSCLC patients, prevent overtreatment, and enable personalized and precise cancer management.

An advanced nomogram model using deep learning radiomics and clinical data for predicting occult lymph node metastasis in lung adenocarcinoma.

PURPOSE: To evaluate the effectiveness of deep learning radiomics nomogram in distinguishing the occult lymph node metastasis (OLNM) status in clinical stage IA lung adenocarcinoma. METHODS: A cohort of 473 cases of lung adenocarcinomas from two hospitals was included, with 404 cases allocated to the training cohort and 69 cases to the testing cohort. Clinical characteristics and semantic features were collected, and radiomics features were extracted from the computed tomography (CT) images. Additionally, deep transfer learning (DTL) features were generated using RseNet50. Predictive models were developed using the logistic regression (LR) machine learning algorithm. Moreover, gene analysis was conducted on RNA sequencing data from 14 patients to explore the underlying biological basis of deep learning radiomics scores. RESULT: The training and testing cohorts achieved AUC values of 0.826 and 0.775 for the clinical model, 0.865 and 0.801 for the radiomics model, 0.927 and 0.885 for the DTL-radiomics model, and 0.928 and 0.898 for the nomogram model. The nomogram model demonstrated superiority over the clinical model. The decision curve analysis (DCA) revealed a net benefit in predicting OLNM for all models. The investigation into the biological basis of deep learning radiomics scores identified an association between high scores and pathways related to tumor proliferation and immune cell infiltration in the microenvironment. CONCLUSIONS: The nomogram model, incorporating clinical-semantic features, radiomics, and DTL features, exhibited promising performance in predicting OLNM. It has the potential to provide valuable information for non-invasive lymph node staging and individualized therapeutic approaches.

Magnetically driven capsules with multimodal response and multifunctionality for biomedical applications.

Untethered capsules hold clinical potential for the diagnosis and treatment of gastrointestinal diseases. Although considerable progress has been achieved recently in this field, the constraints imposed by the narrow spatial structure of the capsule and complex gastrointestinal tract environment cause many open-ended problems, such as poor active motion and limited medical functions. In this work, we describe the development of small-scale magnetically driven capsules with a distinct magnetic soft valve made of dual-layer ferromagnetic soft composite films. A core technological advancement achieved is the flexible opening and closing of the magnetic soft valve by using the competitive interactions between magnetic gradient force and magnetic torque, laying the foundation for the functional integration of both drug release and sampling. Meanwhile, we propose a magnetic actuation strategy based on multi-frequency response control and demonstrate that it can achieve effective decoupled regulation of the capsule's global motion and local responses. Finally, through a comprehensive approach encompassing ideal models, animal ex vivo models, and in vivo assessment, we demonstrate the versatility of the developed magnetic capsules and their multiple potential applications in the biomedical field, such as targeted drug delivery and sampling, selective dual-drug release, and light/thermal-assisted therapy.

Development and Validation of a Deep Learning Radiomics Model to Predict High-Risk Pathologic Pulmonary Nodules Using Preoperative Computed Tomography.

RATIONALE AND OBJECTIVES: To accurately identify the high-risk pathological factors of pulmonary nodules, our study constructed a model combined with clinical features, radiomics features, and deep transfer learning features to predict high-risk pathological pulmonary nodules. MATERIALS AND METHODS: The study cohort consisted of 469 cases of lung adenocarcinoma patients, divided into a training cohort (n = 400) and an external validation cohort (n = 69). We obtained computed tomography (CT) semantic features and clinical characteristics, as well as extracted radiomics and deep transfer learning (DTL) features from the CT images. Selected features were used for constructing prediction models using the logistic regression (LR) algorithm. The performance of the models was evaluated through metrics including the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, calibration curve, and decision curve analysis. RESULTS: The clinical model achieved an AUC of 0.774 (95% CI: 0.728-0.821) in the training cohort and 0.762 (95% confidence interval [CI]: 0.650-0.873) in the external validation cohort. The radiomics model demonstrated an AUC of 0.847 (95% CI: 0.810-0.884) in the training cohort and 0.800 (95% CI: 0.693-0.907) in the external validation cohort. The radiomics-DTL (Rad-DTL) model showed an AUC of 0.871 (95% CI: 0.838-0.905) in the training cohort and 0.806 (95% CI: 0.698-0.914) in the external validation cohort. The proposed combined model yielded AUC values of 0.872 and 0.814 in the training and external validation cohorts, respectively. The combined model demonstrated superiority over both the clinical model and the Rad-DTL model. There were no statistically significant differences observed in the comparison between the combined model incorporating clinical features and the Rad-DTL model. Decision curve analysis (DCA) indicated that the models provided a net benefit in predicting high-risk pathologic pulmonary nodules. CONCLUSION: Rad-DTL signature is a potential biomarker for predicting high-risk pathologic pulmonary nodules using preoperative CT, determining the appropriate surgical strategy, and guiding the extent of resection.

Combining external-latent attention for medical image segmentation.

The attention mechanism comes as a new entry point for improving the performance of medical image segmentation. How to reasonably assign weights is a key element of the attention mechanism, and the current popular schemes include the global squeezing and the non-local information interactions using self-attention (SA) operation. However, these approaches over-focus on external features and lack the exploitation of latent features. The global squeezing approach crudely represents the richness of contextual information by the global mean or maximum value, while non-local information interactions focus on the similarity of external features between different regions. Both ignore the fact that the contextual information is presented more in terms of the latent features like the frequency change within the data. To tackle above problems and make proper use of attention mechanisms in medical image segmentation, we propose an external-latent attention collaborative guided image segmentation network, named TransGuider. This network consists of three key components: 1) a latent attention module that uses an improved entropy quantification method to accurately explore and locate the distribution of latent contextual information. 2) an external self-attention module using sparse representation, which can preserve external global contextual information while reducing computational overhead by selecting representative feature description map for SA operation. 3) a multi-attention collaborative module to guide the network to continuously focus on the region of interest, refining the segmentation mask. Our experimental results on several benchmark medical image segmentation datasets show that TransGuider outperforms the state-of-the-art methods, and extensive ablation experiments demonstrate the effectiveness of the proposed components. Our code will be available at https://github.com/chasingone/TransGuider.

FSA-Net: Rethinking the attention mechanisms in medical image segmentation from releasing global suppressed information.

Attention mechanism-based medical image segmentation methods have developed rapidly recently. For the attention mechanisms, it is crucial to accurately capture the distribution weights of the effective features contained in the data. To accomplish this task, most attention mechanisms prefer using the global squeezing approach. However, it will lead to a problem of over-focusing on the global most salient effective features of the region of interest, while suppressing the secondary salient ones. Making partial fine-grained features are abandoned directly. To address this issue, we propose to use a multiple-local perception method to aggregate global effective features, and design a fine-grained medical image segmentation network, named FSA-Net. This network consists of two key components: 1) the novel Separable Attention Mechanisms which replace global squeezing with local squeezing to release the suppressed secondary salient effective features. 2) a Multi-Attention Aggregator (MAA) which can fuse multi-level attention to efficiently aggregate task-relevant semantic information. We conduct extensive experimental evaluations on five publicly available medical image segmentation datasets: MoNuSeg, COVID-19-CT100, GlaS, CVC-ClinicDB, ISIC2018, and DRIVE datasets. Experimental results show that FSA-Net outperforms state-of-the-art methods in medical image segmentation.

A bidirectional registration neural network for cardiac motion tracking using cine MRI images.

Using cine magnetic resonance imaging (cine MRI) images to track cardiac motion helps users to analyze the myocardial strain, and is of great importance in clinical applications. At present, most of the automatic deep learning-based motion tracking methods compare two images without considering temporal information between MRI frames, which easily leads to the lack of consistency of the generated motion fields. Even though a small number of works take into account the temporal factor, they are usually computationally intensive or have limitations on image length. To solve this problem, we propose a bidirectional convolution neural network for motion tracking of cardiac cine MRI images. This network leverages convolutional blocks to extract spatial features from three-dimensional (3D) image registration pairs, and models the temporal relations through a bidirectional recurrent neural network to obtain the Lagrange motion field between the reference image and other images. Compared with previous pairwise registration methods, the proposed method can automatically learn spatiotemporal information from multiple images with fewer parameters. We evaluated our model on three public cardiac cine MRI datasets. The experimental results demonstrated that the proposed method can significantly improve the motion tracking accuracy. The average Dice coefficient between estimated segmentation and manual segmentation has reached almost 0.85 on the widely used Automatic Cardiac Diagnostic Challenge (ACDC) dataset.

A discontinuity-preserving regularization for deep learning-based cardiac image registration.

Objective. Sliding motion may occur between organs in anatomical regions due to respiratory motion and heart beating. This issue is often neglected in previous studies, resulting in poor image registration performance. A new approach is proposed to handle discontinuity at the boundary and improve registration accuracy.Approach. The proposed discontinuity-preserving regularization (DPR) term can maintain local discontinuities. It leverages the segmentation mask to find organ boundaries and then relaxes the displacement field constraints in these boundary regions. A weakly supervised method using mask dissimilarity loss (MDL) is also proposed. It employs a simple formula to calculate the similarity between the fixed image mask and the deformed moving image mask. These two strategies are added to the loss function during network training to guide the model better to update parameters. Furthermore, during inference time, no segmentation mask information is needed.Main results. Adding the proposed DPR term increases the Dice coefficients by 0.005, 0.009, and 0.081 for three existing registration neural networks CRNet, VoxelMorph, and ViT-V-Net, respectively. It also shows significant improvements in other metrics, including Hausdorff Distance and Average Surface Distance. All quantitative indicator results with MDL have been slightly improved within 1%. After applying these two regularization terms, the generated displacement field is more reasonable at the boundary, and the deformed moving image is closer to the fixed image.Significance. This study demonstrates that the proposed regularization terms can effectively handle discontinuities at the boundaries of organs and improve the accuracy of deep learning-based cardiac image registration methods. Besides, they are generic to be extended to other networks.

A modality-collaborative convolution and transformer hybrid network for unpaired multi-modal medical image segmentation with limited annotations.

BACKGROUND: Multi-modal learning is widely adopted to learn the latent complementary information between different modalities in multi-modal medical image segmentation tasks. Nevertheless, the traditional multi-modal learning methods require spatially well-aligned and paired multi-modal images for supervised training, which cannot leverage unpaired multi-modal images with spatial misalignment and modality discrepancy. For training accurate multi-modal segmentation networks using easily accessible and low-cost unpaired multi-modal images in clinical practice, unpaired multi-modal learning has received comprehensive attention recently. PURPOSE: Existing unpaired multi-modal learning methods usually focus on the intensity distribution gap but ignore the scale variation problem between different modalities. Besides, within existing methods, shared convolutional kernels are frequently employed to capture common patterns in all modalities, but they are typically inefficient at learning global contextual information. On the other hand, existing methods highly rely on a large number of labeled unpaired multi-modal scans for training, which ignores the practical scenario when labeled data is limited. To solve the above problems, we propose a modality-collaborative convolution and transformer hybrid network (MCTHNet) using semi-supervised learning for unpaired multi-modal segmentation with limited annotations, which not only collaboratively learns modality-specific and modality-invariant representations, but also could automatically leverage extensive unlabeled scans for improving performance. METHODS: We make three main contributions to the proposed method. First, to alleviate the intensity distribution gap and scale variation problems across modalities, we develop a modality-specific scale-aware convolution (MSSC) module that can adaptively adjust the receptive field sizes and feature normalization parameters according to the input. Secondly, we propose a modality-invariant vision transformer (MIViT) module as the shared bottleneck layer for all modalities, which implicitly incorporates convolution-like local operations with the global processing of transformers for learning generalizable modality-invariant representations. Third, we design a multi-modal cross pseudo supervision (MCPS) method for semi-supervised learning, which enforces the consistency between the pseudo segmentation maps generated by two perturbed networks to acquire abundant annotation information from unlabeled unpaired multi-modal scans. RESULTS: Extensive experiments are performed on two unpaired CT and MR segmentation datasets, including a cardiac substructure dataset derived from the MMWHS-2017 dataset and an abdominal multi-organ dataset consisting of the BTCV and CHAOS datasets. Experiment results show that our proposed method significantly outperforms other existing state-of-the-art methods under various labeling ratios, and achieves a comparable segmentation performance close to single-modal methods with fully labeled data by only leveraging a small portion of labeled data. Specifically, when the labeling ratio is 25%, our proposed method achieves overall mean DSC values of 78.56% and 76.18% in cardiac and abdominal segmentation, respectively, which significantly improves the average DSC value of two tasks by 12.84% compared to single-modal U-Net models. CONCLUSIONS: Our proposed method is beneficial for reducing the annotation burden of unpaired multi-modal medical images in clinical applications.

Prostate lesion segmentation based on a 3D end-to-end convolution neural network with deep multi-scale attention.

Prostate cancer is one of the deadest cancers among human beings. To better diagnose the prostate cancer, prostate lesion segmentation becomes a very important work, but its progress is very slow due to the prostate lesions small in size, irregular in shape, and blurred in contour. Therefore, automatic prostate lesion segmentation from mp-MRI is a great significant work and a challenging task. However, the most existing multi-step segmentation methods based on voxel-level classification are time-consuming, may introduce errors in different steps and lead to error accumulation. To decrease the computation time, harness richer 3D spatial features, and fuse the multi-level contextual information of mp-MRI, we present an automatic segmentation method in which all steps are optimized conjointly as one step to form our end-to-end convolutional neural network. The proposed end-to-end network DMSA-V-Net consists of two parts: (1) a 3D V-Net is used as the backbone network, it is the first attempt in employing 3D convolutional neural network for CS prostate lesion segmentation, (2) a deep multi-scale attention mechanism is introduced into the 3D V-Net which can highly focus on the ROI while suppressing the redundant background. As a merit, the attention can adaptively re-align the context information between the feature maps at different scales and the saliency maps in high-levels. We performed experiments based on five cross-fold validation with data including 97 patients. The results show that the Dice and sensitivity are 0.7014 and 0.8652 respectively, which demonstrates that our segmentation approach is more significant and accurate compared to other methods.

Automatic 3D CT liver segmentation based on fast global minimization of probabilistic active contour.

PURPOSE: Automatic liver segmentation from computed tomography (CT) images is an essential preprocessing step for computer-aided diagnosis of liver diseases. However, due to the large differences in liver shapes, low-contrast to adjacent tissues, and existence of tumors or other abnormalities, liver segmentation has been very challenging. This study presents an accurate and fast liver segmentation method based on a novel probabilistic active contour (PAC) model and its fast global minimization scheme (3D-FGMPAC), which is explainable as compared with deep learning methods. METHODS: The proposed method first constructs a slice-indexed-histogram to localize the volume of interest (VOI) and estimate the probability that a voxel belongs to the liver according its intensity. The probabilistic image would be used to initialize the 3D PAC model. Secondly, a new contour indicator function, which is a component of the model, is produced by combining the gradient-based edge detection and Hessian-matrix-based surface detection. Then, a fast numerical scheme derived for the 3D PAC model is performed to evolve the initial probabilistic image into the global minimizer of the model, which is a smoothed probabilistic image showing a distinctly highlighted liver. Next, a simple region-growing strategy is applied to extract the whole liver mask from the smoothed probabilistic image. Finally, a B-spline surface is constructed to fit the patch of the rib cage to prevent possible leakage into adjacent intercostal tissues. RESULTS: The proposed method is evaluated on two public datasets. The average Dice score, volume overlap error, volume difference, symmetric surface distance and volume processing time are 0.96, 7.35%, 0.02%, 1.17 mm and 19.8 s for the Sliver07 dataset, and 0.95, 8.89%, - 0.02 % $-0.02\%$ , 1.45 mm and 23.08 s for the 3Dircadb dataset, respectively. CONCLUSIONS: The proposed fully-automatic approach can effectively segment the liver from low-contrast and complex backgrounds. The quantitative and qualitative results demonstrate that the proposed segmentation method outperforms state-of-the-art traditional automatic liver segmentation algorithms and achieves very competitive performance compared with recent deep leaning-based methods.

A 3D Cross-Modality Feature Interaction Network With Volumetric Feature Alignment for Brain Tumor and Tissue Segmentation.

Accurate volumetric segmentation of brain tumors and tissues is beneficial for quantitative brain analysis and brain disease identification in multi-modal Magnetic Resonance (MR) images. Nevertheless, due to the complex relationship between modalities, 3D Fully Convolutional Networks (3D FCNs) using simple multi-modal fusion strategies hardly learn the complex and nonlinear complementary information between modalities. Meanwhile, the indiscriminative feature aggregation between low-level and high-level features easily causes volumetric feature misalignment in 3D FCNs. On the other hand, the 3D convolution operations of 3D FCNs are excellent at modeling local relations but typically inefficient at capturing global relations between distant regions in volumetric images. To tackle these issues, we propose an Aligned Cross-Modality Interaction Network (ACMINet) for segmenting the regions of brain tumors and tissues from MR images. In this network, the cross-modality feature interaction module is first designed to adaptively and efficiently fuse and refine multi-modal features. Secondly, the volumetric feature alignment module is developed for dynamically aligning low-level and high-level features by the learnable volumetric feature deformation field. Thirdly, we propose the volumetric dual interaction graph reasoning module for graph-based global context modeling in spatial and channel dimensions. Our proposed method is applied to brain glioma, vestibular schwannoma, and brain tissue segmentation tasks, and we performed extensive experiments on BraTS2018, BraTS2020, Vestibular Schwannoma, and iSeg-2017 datasets. Experimental results show that ACMINet achieves state-of-the-art segmentation performance on all four benchmark datasets and obtains the highest DSC score of hard-segmented enhanced tumor region on the validation leaderboard of the BraTS2020 challenge.

A bidirectional multilayer contrastive adaptation network with anatomical structure preservation for unpaired cross-modality medical image segmentation.

Multi-modal medical image segmentation has achieved great success through supervised deep learning networks. However, because of domain shift and limited annotation information, unpaired cross-modality segmentation tasks are still challenging. The unsupervised domain adaptation (UDA) methods can alleviate the segmentation degradation of cross-modality segmentation by knowledge transfer between different domains, but current methods still suffer from the problems of model collapse, adversarial training instability, and mismatch of anatomical structures. To tackle these issues, we propose a bidirectional multilayer contrastive adaptation network (BMCAN) for unpaired cross-modality segmentation. The shared encoder is first adopted for learning modality-invariant encoding representations in image synthesis and segmentation simultaneously. Secondly, to retain the anatomical structure consistency in cross-modality image synthesis, we present a structure-constrained cross-modality image translation approach for image alignment. Thirdly, we construct a bidirectional multilayer contrastive learning approach to preserve the anatomical structures and enhance encoding representations, which utilizes two groups of domain-specific multilayer perceptron (MLP) networks to learn modality-specific features. Finally, a semantic information adversarial learning approach is designed to learn structural similarities of semantic outputs for output space alignment. Our proposed method was tested on three different cross-modality segmentation tasks: brain tissue, brain tumor, and cardiac substructure segmentation. Compared with other UDA methods, experimental results show that our proposed BMCAN achieves state-of-the-art segmentation performance on the above three tasks, and it has fewer training components and better feature representations for overcoming overfitting and domain shift problems. Our proposed method can efficiently reduce the annotation burden of radiologists in cross-modality image analysis.

Automatic CT liver Couinaud segmentation based on key bifurcation detection with attentive residual hourglass-based cascaded network.

This paper presents an automatic Couinaud segmentation method based on deep learning of key point detection. Assuming that the liver mask has been extracted, the proposed method can automatically divide the liver into eight anatomical segments according to Couinaud's definition. Firstly, an attentive residual hourglass-based cascaded network (ARH-CNet) is proposed to identify six key bifurcation points of the hepatic vascular system. Subsequently, the detected points are used to derive the planes that divide the liver into different functional units, and the caudate lobe is segmented slice-by-slice based on the circles defined by the detected points. We comprehensively evaluate our method on a public dataset from MICCAI 2018. Experiments firstly demonstrate the effectiveness of our landmark detection network ARH-CNet, which is superior to that of two baseline methods, also robust to noisy data. The average error distance of all predicted key points is 4.68 ± 3.17 mm, and the average accuracy of all points is 90% with the detection error distance of 7 mm. We also verify that summation of the corresponding heat-maps can improve the accuracy of point localization. Furthermore, the overlap-based accuracy and the Dice score of our landmark-derived Couinaud segmentation are respectively 91% and 84%, which are better than the performance of the direct segmentation approach and the traditional plane-based method, thus our method can be regarded as a good alternative for automatic Couinaud segmentation.

APRNet: A 3D Anisotropic Pyramidal Reversible Network With Multi-Modal Cross-Dimension Attention for Brain Tissue Segmentation in MR Images.

Brain tissue segmentation in multi-modal magnetic resonance (MR) images is significant for the clinical diagnosis of brain diseases. Due to blurred boundaries, low contrast, and intricate anatomical relationships between brain tissue regions, automatic brain tissue segmentation without prior knowledge is still challenging. This paper presents a novel 3D fully convolutional network (FCN) for brain tissue segmentation, called APRNet. In this network, we first propose a 3D anisotropic pyramidal convolutional reversible residual sequence (3DAPC-RRS) module to integrate the intra-slice information with the inter-slice information without significant memory consumption; secondly, we design a multi-modal cross-dimension attention (MCDA) module to automatically capture the effective information in each dimension of multi-modal images; then, we apply 3DAPC-RRS modules and MCDA modules to a 3D FCN with multiple encoded streams and one decoded stream for constituting the overall architecture of APRNet. We evaluated APRNet on two benchmark challenges, namely MRBrainS13 and iSeg-2017. The experimental results show that APRNet yields state-of-the-art segmentation results on both benchmark challenge datasets and achieves the best segmentation performance on the cerebrospinal fluid region. Compared with other methods, our proposed approach exploits the complementary information of different modalities to segment brain tissue regions in both adult and infant MR images, and it achieves the average Dice coefficient of 87.22% and 93.03% on the MRBrainS13 and iSeg-2017 testing data, respectively. The proposed method is beneficial for quantitative brain analysis in the clinical study, and our code is made publicly available.

Lung-CRNet: A convolutional recurrent neural network for lung 4DCT image registration.

PURPOSE: Deformable image registration (DIR) of lung four-dimensional computed tomography (4DCT) plays a vital role in a wide range of clinical applications. Most of the existing deep learning-based lung 4DCT DIR methods focus on pairwise registration which aims to register two images with large deformation. However, the temporal continuities of deformation fields between phases are ignored. This paper proposes a fast and accurate deep learning-based lung 4DCT DIR approach that leverages the temporal component of 4DCT images. METHODS: We present Lung-CRNet, an end-to-end convolutional recurrent registration neural network for lung 4DCT images and reformulate 4DCT DIR as a spatiotemporal sequence predicting problem in which the input is a sequence of three-dimensional computed tomography images from the inspiratory phase to the expiratory phase in a respiratory cycle. The first phase in the sequence is selected as the only reference image and the rest as moving images. Multiple convolutional gated recurrent units (ConvGRUs) are stacked to capture the temporal clues between images. The proposed network is trained in an unsupervised way using a spatial transformer layer. During inference, Lung-CRNet is able to yield the respective displacement field for each reference-moving image pair in the input sequence. RESULTS: We have trained the proposed network using a publicly available lung 4DCT dataset and evaluated performance on the widely used the DIR-Lab dataset. The mean and standard deviation of target registration error are 1.56 ± 1.05 mm on the DIR-Lab dataset. The computation time for each forward prediction is less than 1 s on average. CONCLUSIONS: The proposed Lung-CRNet is comparable to the existing state-of-the-art deep learning-based 4DCT DIR methods in both accuracy and speed. Additionally, the architecture of Lung-CRNet can be generalized to suit other groupwise registration tasks which align multiple images simultaneously.

Learning U-Net Based Multi-Scale Features in Encoding-Decoding for MR Image Brain Tissue Segmentation.

Accurate brain tissue segmentation of MRI is vital to diagnosis aiding, treatment planning, and neurologic condition monitoring. As an excellent convolutional neural network (CNN), U-Net is widely used in MR image segmentation as it usually generates high-precision features. However, the performance of U-Net is considerably restricted due to the variable shapes of the segmented targets in MRI and the information loss of down-sampling and up-sampling operations. Therefore, we propose a novel network by introducing spatial and channel dimensions-based multi-scale feature information extractors into its encoding-decoding framework, which is helpful in extracting rich multi-scale features while highlighting the details of higher-level features in the encoding part, and recovering the corresponding localization to a higher resolution layer in the decoding part. Concretely, we propose two information extractors, multi-branch pooling, called MP, in the encoding part, and multi-branch dense prediction, called MDP, in the decoding part, to extract multi-scale features. Additionally, we designed a new multi-branch output structure with MDP in the decoding part to form more accurate edge-preserving predicting maps by integrating the dense adjacent prediction features at different scales. Finally, the proposed method is tested on datasets MRbrainS13, IBSR18, and ISeg2017. We find that the proposed network performs higher accuracy in segmenting MRI brain tissues and it is better than the leading method of 2018 at the segmentation of GM and CSF. Therefore, it can be a useful tool for diagnostic applications, such as brain MRI segmentation and diagnosing.

An Explainable System for Diagnosis and Prognosis of COVID-19.

The outbreak of Coronavirus Disease-2019 (COVID-19) has posed a threat to world health. With the increasing number of people infected, healthcare systems, especially those in developing countries, are bearing tremendous pressure. There is an urgent need for the diagnosis of COVID-19 and the prognosis of inpatients. To alleviate these problems, a data-driven medical assistance system is put forward in this article. Based on two real-world data sets in Wuhan, China, the proposed system integrates data from different sources with tools of machine learning (ML) to predict COVID-19 infected probability of suspected patients in their first visit, and then predict mortality of confirmed cases. Rather than choosing an interpretable algorithm, this system separates the explanations from ML models. It can do help to patient triaging and provide some useful advice for doctors.