A review of deep learning-based information fusion techniques for multimodal medical image classification.

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

Hybrid Fusion of High-Resolution and Ultra-Widefield OCTA Acquisitions for the Automatic Diagnosis of Diabetic Retinopathy.

Optical coherence tomography angiography (OCTA) can deliver enhanced diagnosis for diabetic retinopathy (DR). This study evaluated a deep learning (DL) algorithm for automatic DR severity assessment using high-resolution and ultra-widefield (UWF) OCTA. Diabetic patients were examined with 6×6 mm2 high-resolution OCTA and 15×15 mm2 UWF-OCTA using PLEX®Elite 9000. A novel DL algorithm was trained for automatic DR severity inference using both OCTA acquisitions. The algorithm employed a unique hybrid fusion framework, integrating structural and flow information from both acquisitions. It was trained on data from 875 eyes of 444 patients. Tested on 53 patients (97 eyes), the algorithm achieved a good area under the receiver operating characteristic curve (AUC) for detecting DR (0.8868), moderate non-proliferative DR (0.8276), severe non-proliferative DR (0.8376), and proliferative/treated DR (0.9070). These results significantly outperformed detection with the 6×6 mm2 (AUC = 0.8462, 0.7793, 0.7889, and 0.8104, respectively) or 15×15 mm2 (AUC = 0.8251, 0.7745, 0.7967, and 0.8786, respectively) acquisitions alone. Thus, combining high-resolution and UWF-OCTA acquisitions holds the potential for improved early and late-stage DR detection, offering a foundation for enhancing DR management and a clear path for future works involving expanded datasets and integrating additional imaging modalities.

Is There a Nonperfusion Threshold on OCT Angiography Associated With New Vessels Detected on Ultra-Wide-Field Imaging in Diabetic Retinopathy?

Purpose: To determine whether the nonperfusion index (NPI) measured on widefield (WF) optical coherence tomography angiography (OCTA) could be used as an alternative method for the diagnosis of proliferative diabetic retinopathy (PDR) and to study the relationship between the NPI and the location of new vessels (NV) in eyes with PDR. Methods: Fifty-one treatment-naïve eyes with either severe nonproliferative DR (NPDR) or PDR were imaged using ultra-wide-field imaging and wide-field OCTA. Results: The NPI was significantly higher in eyes with PDR (18.94% vs. 7.51%; P < 0.01). Using the NPI on the whole image to assess PDR status, the area under the curve was 0.770, but the area under the curve increased when the NPI of the most peripheral circle was used (area under the curve of 0.792). Four eyes with PDR (17%) had NV outside the OCTA image field, and their mean NPI (6.15 %) did not differ from that measured in severe NPDR eyes (7.51%; P = 0.67) and was lower than in other eyes with PDR (21.49%; P = 0.023). The presence of NV in a sector was associated with a higher NPI in the same sector (29.2% vs. 6.0%; P < 10-15). Conclusions: Although the NPI was significantly higher in eyes with PDR compared with severe NPDR eyes, its measurement on the whole wide-field OCTA image was not sensitive enough to replace the detection of NV for the diagnosis of PDR. Translational Relevance: Because the presence of new vessels was related to the local nonperfusion index in the same sector, the assessment of nonperfusion outside the optical coherence tomography angiography field is important in diabetic retinopathy.

Correlation between clinical examination and perineal ultrasound in women treated for pelvic organ prolapse.

INTRODUCTION: Lifetime risk of surgery for female pelvic organ prolapse (FPOP) is estimated at 10 to 20%. Prolapse assessment is mostly done by clinical examination. Perineal ultrasound is easily available and performed to evaluate and stage FPOP. This study's aim is to evaluate the agreement between clinical examination by POP-Q and perineal sonography in women presenting pelvic organ prolapse. MATERIALS AND METHODS: We carried out a prospective study from December 2015 to March 2018 in the gynecologic department of a teaching hospital. Consecutive woman requiring a surgery for pelvic organ prolapse were included. All women underwent clinical examination by POP-Q, perineal ultrasound with measurements of each compartment descent, levator hiatus area and posterior perineal angle. They also answered several functional questionnaires (PFDI 20, PFIQ7, EQ-5D and PISQ12) before and after surgery. Data for clinical and sonographic assessments were compared with Spearman's test and correlation with functional questionnaires was tested. RESULTS: 82 women were included. We found no significant agreement between POP-Q and sonographic measures of bladder prolapse, surface of the perineal hiatus or perineal posterior angle. There was a significant improvement of most of the functional scores after surgery. DISCUSSION: Our study does not suggest correlation between clinical POP-Q and sonographic assessment of bladder prolapse, hiatus surface or perineal posterior angle. Ultrasound datasets were limited by an important number of missing data resulting in a lack of power.

Impact of image averaging on vessel detection using optical coherence tomography angiography in eyes with macular oedema and in healthy eyes.

PURPOSE: To assess the repeatability of multiple automatic vessel density (VD) measurements and the effect of image averaging on vessel detection by optical coherence tomography angiography (OCTA). METHODS: An observational study was conducted in a series of healthy volunteers and patients with macular oedema. Five sequential OCTA images were acquired for each eye using the OptoVue HD device. The effect of the averaging of the 5 acquisitions on vessel detection was analysed quantitatively using a pixel-by-pixel automated analysis. In addition, two independent retina experts qualitatively assessed the change in vessel detection in averaged images segmented in 9 boxes and compared to the first non-averaged image. RESULTS: The automatic VD measurement in OCTA images showed a good repeatability with an overall mean intra-class correlation coefficient (ICC) of 0.924. The mean ICC was higher in healthy eyes compared to eyes with macular oedema (0.877 versus 0.960; p < 0.001) and in the superficial vascular plexus versus the deep vascular complex (0.967 versus 0.888; p = 0.001). The quantitative analysis of the effect of the averaging showed that averaged images had a mean gain of 790.4 pixels/box, located around or completing interruptions in the vessel walls, and a mean loss of 727.2 pixels/box. The qualitative analysis of the averaged images showed that 99.6% of boxes in the averaged images had a gain in vessel detection (i.e., vessels detected in the averaged image but not in the non-averaged image). The loss of pixels was due to a reduction in background noise and motion artifacts in all cases and no case of loss of vessel detection was observed. CONCLUSION: The automatic VD measurement using the OptoVue HD device showed a good repeatability in 5 acquisitions in a row setting. Averaging images increased vessel detection, and in about a third of boxes, decreased the background noise, both in healthy eyes and, in a greater proportion, in eyes with macular oedema.