Comprehensive learning and adaptive teaching: Distilling multi-modal knowledge for pathological glioma grading.

The fusion of multi-modal data, e.g., pathology slides and genomic profiles, can provide complementary information and benefit glioma grading. However, genomic profiles are difficult to obtain due to the high costs and technical challenges, thus limiting the clinical applications of multi-modal diagnosis. In this work, we investigate the realistic problem where paired pathology-genomic data are available during training, while only pathology slides are accessible for inference. To solve this problem, a comprehensive learning and adaptive teaching framework is proposed to improve the performance of pathological grading models by transferring the privileged knowledge from the multi-modal teacher to the pathology student. For comprehensive learning of the multi-modal teacher, we propose a novel Saliency-Aware Masking (SA-Mask) strategy to explore richer disease-related features from both modalities by masking the most salient features. For adaptive teaching of the pathology student, we first devise a Local Topology Preserving and Discrepancy Eliminating Contrastive Distillation (TDC-Distill) module to align the feature distributions of the teacher and student models. Furthermore, considering the multi-modal teacher may include incorrect information, we propose a Gradient-guided Knowledge Refinement (GK-Refine) module that builds a knowledge bank and adaptively absorbs the reliable knowledge according to their agreement in the gradient space. Experiments on the TCGA GBM-LGG dataset show that our proposed distillation framework improves the pathological glioma grading and outperforms other KD methods. Notably, with the sole pathology slides, our method achieves comparable performance with existing multi-modal methods. The code is available at https://github.com/CUHK-AIM-Group/MultiModal-learning.

Medical federated learning with joint graph purification for noisy label learning.

In terms of increasing privacy issues, Federated Learning (FL) has received extensive attention in medical imaging. Through collaborative training, FL can produce superior diagnostic models with global knowledge, while preserving private data locally. In practice, medical diagnosis suffers from intra-/inter-observer variability, thus label noise is inevitable in dataset preparation. Different from existing studies on centralized datasets, the label noise problem in FL scenarios confronts more challenges, due to data inaccessibility and even noise heterogeneity. In this work, we propose a federated framework with joint Graph Purification (FedGP) to address the label noise in FL through server and clients collaboration. Specifically, to overcome the impact of label noise on local training, we first devise a noisy graph purification on the client side to generate reliable pseudo labels by progressively expanding the purified graph with topological knowledge. Then, we further propose a graph-guided negative ensemble loss to exploit the topology of the client-side purified graph with robust complementary supervision against label noise. Moreover, to address the FL label noise with data silos, we propose a global centroid aggregation on the server side to produce a robust classifier with global knowledge, which can be optimized collaboratively in the FL framework. Extensive experiments are conducted on endoscopic and pathological images with the comparison under the homogeneous, heterogeneous, and real-world label noise for medical FL. Among these diverse noisy FL settings, our FedGP framework significantly outperforms denoising and noisy FL state-of-the-arts by a large margin. The source code is available at https://github.com/CUHK-AIM-Group/FedGP.

Gradient modulated contrastive distillation of low-rank multi-modal knowledge for disease diagnosis.

The fusion of multi-modal data, e.g., medical images and genomic profiles, can provide complementary information and further benefit disease diagnosis. However, multi-modal disease diagnosis confronts two challenges: (1) how to produce discriminative multi-modal representations by exploiting complementary information while avoiding noisy features from different modalities. (2) how to obtain an accurate diagnosis when only a single modality is available in real clinical scenarios. To tackle these two issues, we present a two-stage disease diagnostic framework. In the first multi-modal learning stage, we propose a novel Momentum-enriched Multi-Modal Low-Rank (M3LR) constraint to explore the high-order correlations and complementary information among different modalities, thus yielding more accurate multi-modal diagnosis. In the second stage, the privileged knowledge of the multi-modal teacher is transferred to the unimodal student via our proposed Discrepancy Supervised Contrastive Distillation (DSCD) and Gradient-guided Knowledge Modulation (GKM) modules, which benefit the unimodal-based diagnosis. We have validated our approach on two tasks: (i) glioma grading based on pathology slides and genomic data, and (ii) skin lesion classification based on dermoscopy and clinical images. Experimental results on both tasks demonstrate that our proposed method consistently outperforms existing approaches in both multi-modal and unimodal diagnoses.

Boundary-enhanced semi-supervised retinal layer segmentation in optical coherence tomography images using fewer labels.

Automatic segmentation of multiple layers in retinal optical coherence tomography (OCT) images is crucial for eye disease diagnosis and treatment. Despite the success of deep learning algorithms, it still remains a challenge due to the blurry layer boundaries and lack of adequate pixel-wise annotations. To tackle these issues, we propose a Boundary-Enhanced Semi-supervised Network (BE-SemiNet) that exploits an auxiliary distance map regression task to improve retinal layer segmentation with scarce labeled data and abundant unlabeled data. Specifically, a novel Unilaterally Truncated Distance Map (UTDM) is firstly introduced to alleviate the class imbalance problem and enhance the layer boundary learning in the regression task. Then for the pixel-wise segmentation and UTDM regression branches, we impose task-level and data-level consistency regularization on unlabeled data to enrich the diversity of unsupervised information and improve the regularization effects. Pseudo supervision is incorporated in consistency regularization to bridge the task prediction spaces for consistency and expand training labeled data. Experiments on two public retinal OCT datasets show that our method can greatly improve the supervised baseline performance with only 5 annotations and outperform the state-of-the-art methods. Since it is difficult and labor-expensive to obtain adequate pixel-wise annotations in practice, our method has a promising application future in clinical retinal OCT image analysis.

Multi-level attention graph neural network based on co-expression gene modules for disease diagnosis and prognosis.

MOTIVATION: Advanced deep learning techniques have been widely applied in disease diagnosis and prognosis with clinical omics, especially gene expression data. In the regulation of biological processes and disease progression, genes often work interactively rather than individually. Therefore, investigating gene association information and co-functional gene modules can facilitate disease state prediction. RESULTS: To explore the gene modules and inter-gene relational information contained in the omics data, we propose a novel multi-level attention graph neural network (MLA-GNN) for disease diagnosis and prognosis. Specifically, we format omics data into co-expression graphs via weighted correlation network analysis, and then construct multi-level graph features, finally fuse them through a well-designed multi-level graph feature fully fusion module to conduct predictions. For model interpretation, a novel full-gradient graph saliency mechanism is developed to identify the disease-relevant genes. MLA-GNN achieves state-of-the-art performance on transcriptomic data from TCGA-LGG/TCGA-GBM and proteomic data from coronavirus disease 2019 (COVID-19)/non-COVID-19 patient sera. More importantly, the relevant genes selected by our model are interpretable and are consistent with the clinical understanding. AVAILABILITYAND IMPLEMENTATION: The codes are available at https://github.com/TencentAILabHealthcare/MLA-GNN.

Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer with Deep Learning.

Objective. To develop an artificial intelligence method predicting lymph node metastasis (LNM) for patients with colorectal cancer (CRC). Impact Statement. A novel interpretable multimodal AI-based method to predict LNM for CRC patients by integrating information of pathological images and serum tumor-specific biomarkers. Introduction. Preoperative diagnosis of LNM is essential in treatment planning for CRC patients. Existing radiology imaging and genomic tests approaches are either unreliable or too costly. Methods. A total of 1338 patients were recruited, where 1128 patients from one centre were included as the discovery cohort and 210 patients from other two centres were involved as the external validation cohort. We developed a Multimodal Multiple Instance Learning (MMIL) model to learn latent features from pathological images and then jointly integrated the clinical biomarker features for predicting LNM status. The heatmaps of the obtained MMIL model were generated for model interpretation. Results. The MMIL model outperformed preoperative radiology-imaging diagnosis and yielded high area under the curve (AUCs) of 0.926, 0.878, 0.809, and 0.857 for patients with stage T1, T2, T3, and T4 CRC, on the discovery cohort. On the external cohort, it obtained AUCs of 0.855, 0.832, 0.691, and 0.792, respectively (T1-T4), which indicates its prediction accuracy and potential adaptability among multiple centres. Conclusion. The MMIL model showed the potential in the early diagnosis of LNM by referring to pathological images and tumor-specific biomarkers, which is easily accessed in different institutes. We revealed the histomorphologic features determining the LNM prediction indicating the model ability to learn informative latent features.

Zoom in Lesions for Better Diagnosis: Attention Guided Deformation Network for WCE Image Classification.

Wireless capsule endoscopy (WCE) is a novel imaging tool that allows noninvasive visualization of the entire gastrointestinal (GI) tract without causing discomfort to patients. Convolutional neural networks (CNNs), though perform favorably against traditional machine learning methods, show limited capacity in WCE image classification due to the small lesions and background interference. To overcome these limits, we propose a two-branch Attention Guided Deformation Network (AGDN) for WCE image classification. Specifically, the attention maps of branch1 are utilized to guide the amplification of lesion regions on the input images of branch2, thus leading to better representation and inspection of the small lesions. What's more, we devise and insert Third-order Long-range Feature Aggregation (TLFA) modules into the network. By capturing long-range dependencies and aggregating contextual features, TLFAs endow the network with a global contextual view and stronger feature representation and discrimination capability. Furthermore, we propose a novel Deformation based Attention Consistency (DAC) loss to refine the attention maps and achieve the mutual promotion of the two branches. Finally, the global feature embeddings from the two branches are fused to make image label predictions. Extensive experiments show that the proposed AGDN outperforms state-of-the-art methods with an overall classification accuracy of 91.29% on two public WCE datasets. The source code is available at https://github.com/hathawayxxh/WCE-AGDN.

Bleeding Detection in Wireless Capsule Endoscopy Image Video Using Superpixel-Color Histogram and a Subspace KNN Classifier.

Wireless Capsule Endoscopy (WCE) has become increasingly popular in clinical gastrointestinal (GI) disease diagnosis, benefiting from its painless and noninvasive examination. However, reviewing a large number of images is time-consuming for doctors, thus a computer-aided diagnosis (CAD) system is in high demand. In this paper, we present an automatic bleeding detection algorithm that consists of three stages. The first stage is the preprocessing, including key frame extraction and edge removal. In the second stage, we discriminate the bleeding frames using a novel superpixelcolor histogram (SPCH) feature based on the principle color spectrum, and then the decision is made by a subspace KNN classifier. Thirdly, we further segment the bleeding regions by extracting a 9-D color feature vector from the multiple color spaces at the superpixel level. Experimental results with an accuracy of 0.9922 illustrate that our proposed method outperforms the state-of-the-art methods in GI bleeding detection with low computational costs.

The photocurrent response in the perovskite device based on coordination polymers: structure, topology, band gap and matched energy levels.

Using a rigid ditopic ligand, 4,5-di(4'-carboxylphenyl)benzene (H2L), three coordination polymers (CPs) formulated as MnL(H2O)2 (1), CdL(H2O) (2) and Mn2L2(DMF)3 (3) have been synthesized and structurally characterized by single-crystal X-ray diffraction. These three CPs display 2D architectures but with different topologies. The experimental data and DFT calculation indicate that CP 2 is a semiconductor, and its CB/VB energy levels match with those of the perovskite CH3NH3PbI3. A FTO/TiO2/CH3NH3PbI3/CP 2 device is fabricated and the CP-based device shows much larger photoresponse under visible light illumination (650 nm > λ > 350 nm, 100 mW cm-2) than the individual CP 2. At 0 V vs. AgCl/Ag, the largest photocurrent density yielded by the CP-based perovskite device is ca. 200 times that of CP 2, which is due to the matched energy levels of all the materials in the device, leading the photogenerated electron-hole pairs to be separated effectively. Meanwhile, the coverage of the insoluble CP on the surface of the perovskite CH3NH3PbI3 can improve the stability of the perovskite against water.