Robustness evaluation of deep neural networks for endoscopic image analysis: Insights and strategies.

Computer-aided detection and diagnosis systems (CADe/CADx) in endoscopy are commonly trained using high-quality imagery, which is not representative for the heterogeneous input typically encountered in clinical practice. In endoscopy, the image quality heavily relies on both the skills and experience of the endoscopist and the specifications of the system used for screening. Factors such as poor illumination, motion blur, and specific post-processing settings can significantly alter the quality and general appearance of these images. This so-called domain gap between the data used for developing the system and the data it encounters after deployment, and the impact it has on the performance of deep neural networks (DNNs) supportive endoscopic CAD systems remains largely unexplored. As many of such systems, for e.g. polyp detection, are already being rolled out in clinical practice, this poses severe patient risks in particularly community hospitals, where both the imaging equipment and experience are subject to considerable variation. Therefore, this study aims to evaluate the impact of this domain gap on the clinical performance of CADe/CADx for various endoscopic applications. For this, we leverage two publicly available data sets (KVASIR-SEG and GIANA) and two in-house data sets. We investigate the performance of commonly-used DNN architectures under synthetic, clinically calibrated image degradations and on a prospectively collected dataset including 342 endoscopic images of lower subjective quality. Additionally, we assess the influence of DNN architecture and complexity, data augmentation, and pretraining techniques for improved robustness. The results reveal a considerable decline in performance of 11.6% (±1.5) as compared to the reference, within the clinically calibrated boundaries of image degradations. Nevertheless, employing more advanced DNN architectures and self-supervised in-domain pre-training effectively mitigate this drop to 7.7% (±2.03). Additionally, these enhancements yield the highest performance on the manually collected test set including images with lower subjective quality. By comprehensively assessing the robustness of popular DNN architectures and training strategies across multiple datasets, this study provides valuable insights into their performance and limitations for endoscopic applications. The findings highlight the importance of including robustness evaluation when developing DNNs for endoscopy applications and propose strategies to mitigate performance loss.

Toward improved endoscopic surveillance with multidiameter single fiber reflectance spectroscopy in patients with Barrett's esophagus.

Patients with Barrett's esophagus are at an increased risk to develop esophageal cancer and, therefore, undergo regular endoscopic surveillance. Early detection of neoplasia enables endoscopic treatment, which improves outcomes. However, early Barrett's neoplasia is easily missed during endoscopic surveillance. This study investigates multidiameter single fiber reflectance spectroscopy (MDSFR) to improve Barrett's surveillance. Based on the concept of field cancerization, it may be possible to identify the presence of a neoplastic lesion from measurements elsewhere in the esophagus or even the oral cavity. In this study, MDSFR measurements are performed on non-dysplastic Barrett's mucosa, squamous mucosa, oral mucosa, and the neoplastic lesion (if present). Based on logistic regression analysis on the scattering parameters measured by MDSFR, a classifier is developed that can predict the presence of neoplasia elsewhere in the Barrett's segment from measurements on the non-dysplastic Barrett's mucosa (sensitivity 91%, specificity 71%, AUC = 0.77). Classifiers obtained from logistic regression analysis for the squamous and oral mucosa do not result in an AUC significantly different from 0.5.

A computer-assisted algorithm for narrow-band imaging-based tissue characterization in Barrett's esophagus.

BACKGROUND AND AIMS: The endoscopic evaluation of narrow-band imaging (NBI) zoom imagery in Barrett's esophagus (BE) is associated with suboptimal diagnostic accuracy and poor interobserver agreement. Computer-aided diagnosis (CAD) systems may assist endoscopists in the characterization of Barrett's mucosa. Our aim was to demonstrate the feasibility of a deep-learning CAD system for tissue characterization of NBI zoom imagery in BE. METHODS: The CAD system was first trained using 494,364 endoscopic images of general endoscopic imagery. Next, 690 neoplastic BE and 557 nondysplastic BE (NDBE) white-light endoscopy overview images were used for refinement training. Subsequently, a third dataset of 112 neoplastic and 71 NDBE NBI zoom images with histologic correlation was used for training and internal validation. Finally, the CAD system was further trained and validated with a fourth, histologically confirmed dataset of 59 neoplastic and 98 NDBE NBI zoom videos. Performance was evaluated using fourfold cross-validation. The primary outcome was the diagnostic performance of the CAD system for classification of neoplasia in NBI zoom videos. RESULTS: The CAD system demonstrated accuracy, sensitivity, and specificity for detection of BE neoplasia using NBI zoom images of 84%, 88%, and 78%, respectively. In total, 30,021 individual video frames were analyzed by the CAD system. Accuracy, sensitivity, and specificity of the video-based CAD system were 83% (95% confidence interval [CI], 78%-89%), 85% (95% CI, 76%-94%), and 83% (95% CI, 76%-90%), respectively. The mean assessment speed was 38 frames per second. CONCLUSION: We have demonstrated promising diagnostic accuracy of predicting the presence/absence of Barrett's neoplasia on histologically confirmed unaltered NBI zoom videos with fast corresponding assessment time.

Advanced Imaging and Sampling in Barrett's Esophagus: Artificial Intelligence to the Rescue?

Because the current Barrett's esophagus (BE) surveillance protocol suffers from sampling error of random biopsies and a high miss-rate of early neoplastic lesions, many new endoscopic imaging and sampling techniques have been developed. None of these techniques, however, have significantly increased the diagnostic yield of BE neoplasia. In fact, these techniques have led to an increase in the amount of visible information, yet endoscopists and pathologists inevitably suffer from variations in intra- and interobserver agreement. Artificial intelligence systems have the potential to overcome these endoscopist-dependent limitations.

Prospective development and validation of a volumetric laser endomicroscopy computer algorithm for detection of Barrett's neoplasia.

BACKGROUND AND AIMS: Volumetric laser endomicroscopy (VLE) is an advanced imaging modality used to detect Barrett's esophagus (BE) dysplasia. However, real-time interpretation of VLE scans is complex and time-consuming. Computer-aided detection (CAD) may help in the process of VLE image interpretation. Our aim was to train and validate a CAD algorithm for VLE-based detection of BE neoplasia. METHODS: The multicenter, VLE PREDICT study, prospectively enrolled 47 patients with BE. In total, 229 nondysplastic BE and 89 neoplastic (high-grade dysplasia/esophageal adenocarcinoma) targets were laser marked under VLE guidance and subsequently underwent a biopsy for histologic diagnosis. Deep convolutional neural networks were used to construct a CAD algorithm for differentiation between nondysplastic and neoplastic BE tissue. The CAD algorithm was trained on a set consisting of the first 22 patients (134 nondysplastic BE and 38 neoplastic targets) and validated on a separate test set from patients 23 to 47 (95 nondysplastic BE and 51 neoplastic targets). The performance of the algorithm was benchmarked against the performance of 10 VLE experts. RESULTS: Using the training set to construct the algorithm resulted in an accuracy of 92%, sensitivity of 95%, and specificity of 92%. When performance was assessed on the test set, accuracy, sensitivity, and specificity were 85%, 91%, and 82%, respectively. The algorithm outperformed all 10 VLE experts, who demonstrated an overall accuracy of 77%, sensitivity of 70%, and specificity of 81%. CONCLUSIONS: We developed, validated, and benchmarked a VLE CAD algorithm for detection of BE neoplasia using prospectively collected and biopsy-correlated VLE targets. The algorithm detected neoplasia with high accuracy and outperformed 10 VLE experts. (The Netherlands National Trials Registry (NTR) number: NTR 6728.).

Deep learning algorithm detection of Barrett's neoplasia with high accuracy during live endoscopic procedures: a pilot study (with video).

BACKGROUND AND AIMS: We assessed the preliminary diagnostic accuracy of a recently developed computer-aided detection (CAD) system for detection of Barrett's neoplasia during live endoscopic procedures. METHODS: The CAD system was tested during endoscopic procedures in 10 patients with nondysplastic Barrett's esophagus (NDBE) and 10 patients with confirmed Barrett's neoplasia. White-light endoscopy images were obtained at every 2-cm level of the Barrett's segment and immediately analyzed by the CAD system, providing instant feedback to the endoscopist. At every level, 3 images were evaluated by the CAD system. Outcome measures were diagnostic performance of the CAD system per level and per patient, defined as accuracy, sensitivity, and specificity (ground truth was established by expert assessment and corresponding histopathology), and concordance of 3 sequential CAD predictions per level. RESULTS: Accuracy, sensitivity, and specificity of the CAD system in a per-level analyses were 90%, 91%, and 89%, respectively. Nine of 10 neoplastic patients were correctly diagnosed. The single lesion not detected by CAD showed NDBE in the endoscopic resection specimen. In only 1 NDBE patient, the CAD system produced false-positive predictions. In 75% of all levels, the CAD system produced 3 concordant predictions. CONCLUSIONS: This is one of the first studies to evaluate a CAD system for Barrett's neoplasia during live endoscopic procedures. The system detected neoplasia with high accuracy, with only a small number of false-positive predictions and with a high concordance rate between separate predictions. The CAD system is thereby ready for testing in larger, multicenter trials. (Clinical trial registration number: NL7544.).

Blue-light imaging and linked-color imaging improve visualization of Barrett's neoplasia by nonexpert endoscopists.

BACKGROUND AND AIMS: Endoscopic recognition of early Barrett's neoplasia is challenging. Blue-light imaging (BLI) and linked-color imaging (LCI) may assist endoscopists in appreciation of neoplasia. Our aim was to evaluate BLI and LCI for visualization of Barrett's neoplasia in comparison with white-light endoscopy (WLE) alone, when assessed by nonexpert endoscopists. METHODS: In this web-based assessment, corresponding WLE, BLI, and LCI images of 30 neoplastic Barrett's lesions were delineated by 3 expert endoscopists to establish ground truth. These images were then scored and delineated by 76 nonexpert endoscopists from 3 countries and with different levels of expertise, in 4 separate assessment phases with a washout period of 2 weeks. Assessments were as follows: assessment 1, WLE only; assessment 2, WLE + BLI; assessment 3, WLE + LCI; assessment 4, WLE + BLI + LCI. The outcomes were (1) appreciation of macroscopic appearance and ability to delineate lesions (visual analog scale [VAS] scores); (2) preferred technique (ordinal scores); and (3) assessors' delineation performance in terms of overlap with expert ground truth. RESULTS: Median VAS scores for phases 2 to 4 were significantly higher than in phase 1 (P < .001). Assessors preferred BLI and LCI over WLE for appreciation of macroscopic appearance (P < .001) and delineation (P < .001). Linear mixed-effect models showed that delineation performance increased significantly in phase 4. CONCLUSIONS: The use of BLI and LCI has significant additional value for the visualization of Barrett's neoplasia when used by nonexpert endoscopists. Assessors appreciated the addition of BLI and LCI better than the use of WLE alone. Furthermore, this addition led to improved delineation performance, thereby allowing for better acquisition of targeted biopsy samples. (The Netherlands Trial Registry number: NL7541.).

Deep-Learning System Detects Neoplasia in Patients With Barrett's Esophagus With Higher Accuracy Than Endoscopists in a Multistep Training and Validation Study With Benchmarking.

BACKGROUND & AIMS: We aimed to develop and validate a deep-learning computer-aided detection (CAD) system, suitable for use in real time in clinical practice, to improve endoscopic detection of early neoplasia in patients with Barrett's esophagus (BE). METHODS: We developed a hybrid ResNet-UNet model CAD system using 5 independent endoscopy data sets. We performed pretraining using 494,364 labeled endoscopic images collected from all intestinal segments. Then, we used 1704 unique esophageal high-resolution images of rigorously confirmed early-stage neoplasia in BE and nondysplastic BE, derived from 669 patients. System performance was assessed by using data sets 4 and 5. Data set 5 was also scored by 53 general endoscopists with a wide range of experience from 4 countries to benchmark CAD system performance. Coupled with histopathology findings, scoring of images that contained early-stage neoplasia in data sets 2-5 were delineated in detail for neoplasm position and extent by multiple experts whose evaluations served as the ground truth for segmentation. RESULTS: The CAD system classified images as containing neoplasms or nondysplastic BE with 89% accuracy, 90% sensitivity, and 88% specificity (data set 4, 80 patients and images). In data set 5 (80 patients and images) values for the CAD system vs those of the general endoscopists were 88% vs 73% accuracy, 93% vs 72% sensitivity, and 83% vs 74% specificity. The CAD system achieved higher accuracy than any of the individual 53 nonexpert endoscopists, with comparable delineation performance. CAD delineations of the area of neoplasm overlapped with those from the BE experts in all detected neoplasia in data sets 4 and 5. The CAD system identified the optimal site for biopsy of detected neoplasia in 97% and 92% of cases (data sets 4 and 5, respectively). CONCLUSIONS: We developed, validated, and benchmarked a deep-learning computer-aided system for primary detection of neoplasia in patients with BE. The system detected neoplasia with high accuracy and near-perfect delineation performance. The Netherlands National Trials Registry, Number: NTR7072.

Blue-light imaging has an additional value to white-light endoscopy in visualization of early Barrett's neoplasia: an international multicenter cohort study.

BACKGROUND AND AIMS: Endoscopic features of early neoplasia in Barrett's esophagus (BE) are subtle. Blue-light imaging (BLI) may improve visualization of neoplastic lesions. The aim of this study was to evaluate BLI in visualization of Barrett's neoplasia. METHODS: Corresponding white-light endoscopy (WLE) and BLI images of 40 BE lesions were obtained prospectively and assessed by 6 international experts in 3 assessments. Each assessment consisted of overview and magnification images. Assessments were as follows: assessment 1, WLE only; assessment 2, BLI only; and assessment 3, corresponding WLE and BLI images. Outcome parameters were as follows: (1) appreciation of macroscopic appearance and surface relief (visual analog scale scores); (2) ability to delineate lesions (visual analog scale scores); (3) preferred technique for delineation (ordinal scores); and (4) quantitative agreement on delineations (AND/OR scores). RESULTS: Experts appreciated BLI significantly better than WLE for visualization of macroscopic appearance (median 8.0 vs 7.0, P < .001) and surface relief (8.0 vs 6.0, P < .001). For both overview and magnification images, experts appreciated BLI significantly better than WLE for ability to delineate lesions (8.0 vs 6.0, P < .001 and 8.0 vs 5.0, P < .001). There was no overall significant difference in AND/OR scores of WLE + BLI when compared with WLE, yet agreement increased significantly with WLE + BLI for cases with a low baseline AND/OR score on WLE, both in overview (mean difference, 0.15; P = .015) and magnification (mean difference, 0.10; P = .01). CONCLUSIONS: BLI has additional value for visualization of BE neoplasia. Experts appreciated BLI better than WLE for visualization and delineation of BE neoplasia. Quantitative agreement increased significantly when BLI was offered next to WLE for lesions that were hard to delineate with WLE alone. (ISRCTN registry study ID: ISRCTN15916689.).

Feasibility of laser marking in Barrett's esophagus with volumetric laser endomicroscopy: first-in-man pilot study.

BACKGROUND AND AIM: Volumetric laser endomicroscopy (VLE) provides a circumferential scan of the esophageal wall layers and has potential to improve detection of neoplasia in Barrett's esophagus (BE). The novel VLE laser marking system enables direct in vivo marking of suspicious areas as identified on VLE. These laser marked areas can subsequently be targeted for biopsies. The aim was to evaluate the visibility and positional accuracy of laser marks (LMs) in different esophageal tissue types on white light endoscopy (WLE) and VLE. METHODS: Patients with BE with or without neoplasia underwent imaging with VLE. Protocol refinements were practiced in a learning phase. In the second phase, visibility of LMs was assessed by random marking in squamous, BE, and gastric tissue. In phase 3, positional accuracy of the LMs was tested by identifying and laser marking surrogate targets (endoscopically placed cautery marks). In the final phase, the most suspicious areas for neoplasia were identified in each patient using VLE, targeted by LMs, and biopsy samples subsequently obtained. RESULTS: Sixteen patients with BE were included (14 men; median age, 68 years), 1 of whom was included twice in different study phases. Worst histologic diagnoses were 9 non-dysplastic Barrett's esophagus (NDBE), 3 low-grade dysplasia (LGD), 4 high-grade dysplasia (HGD), and 1 early adenocarcinoma (EAC). In total, 222 LMs were placed, of which 97% was visible on WLE. All LMs were visible on VLE directly after marking, and 86% could be confirmed during post hoc analysis. LM targeting was successful with positional accuracy in 85% of cautery marks. Inaccurate targeting was caused by system errors or difficult cautery mark visualization on VLE. In the final phase (5 patients), 18 areas suspicious on VLE were identified, which were all successfully targeted by LMs (3 EAC, 3 HGD, 1 LGD, and 11 NDBE). Mean VLE procedure time was 22 minutes (±6 minutes standard deviation); mean endoscopy time was 56 minutes (±17 minutes). No adverse events were reported. CONCLUSIONS: This first-in-human study of VLE-guided laser marking was found to be feasible and safe in 17 procedures. Most LMs were visible on WLE and VLE. Targeting VLE areas of interest proved to be highly successful. VLE-guided laser marking may improve the detection and delineation of Barrett's neoplasia in the future.