Artificial Intelligence for Colonoscopy: Past, Present, and Future.

During the past decades, many automated image analysis methods have been developed for colonoscopy. Real-time implementation of the most promising methods during colonoscopy has been tested in clinical trials, including several recent multi-center studies. All trials have shown results that may contribute to prevention of colorectal cancer. We summarize the past and present development of colonoscopy video analysis methods, focusing on two categories of artificial intelligence (AI) technologies used in clinical trials. These are (1) analysis and feedback for improving colonoscopy quality and (2) detection of abnormalities. Our survey includes methods that use traditional machine learning algorithms on carefully designed hand-crafted features as well as recent deep-learning methods. Lastly, we present the gap between current state-of-the-art technology and desirable clinical features and conclude with future directions of endoscopic AI technology development that will bridge the current gap.

Polyp-Alert: near real-time feedback during colonoscopy.

We present a software system called "Polyp-Alert" to assist the endoscopist find polyps by providing visual feedback during colonoscopy. Polyp-Alert employs our previous edge-cross-section visual features and a rule-based classifier to detect a polyp edge-an edge along the contour of a polyp. The technique employs tracking of detected polyp edge(s) to group a sequence of images covering the same polyp(s) as one polyp shot. In our experiments, the software correctly detected 97.7% (42 of 43) of polyp shots on 53 randomly selected video files of entire colonoscopy procedures. However, Polyp-Alert incorrectly marked only 4.3% of a full-length colonoscopy procedure as showing a polyp when they do not. The test data set consists of about 18 h worth of video data from Olympus and Fujinon endoscopes. The technique is extensible to other brands of colonoscopes. Furthermore, Polyp-Alert can provide as high as ten feedbacks per second for a smooth display of feedback. The performance of our system is by far the most promising to potentially assist the endoscopist find more polyps in clinical practice during a routine screening colonoscopy.

Abnormal Image Detection in Endoscopy Videos Using a Filter Bank and Local Binary Patterns.

Finding mucosal abnormalities (e.g., erythema, blood, ulcer, erosion, and polyp) is one of the most essential tasks during endoscopy video review. Since these abnormalities typically appear in a small number of frames (around 5% of the total frame number), automated detection of frames with an abnormality can save physician's time significantly. In this paper, we propose a new multi-texture analysis method that effectively discerns images showing mucosal abnormalities from the ones without any abnormality since most abnormalities in endoscopy images have textures that are clearly distinguishable from normal textures using an advanced image texture analysis method. The method uses a "texton histogram" of an image block as features. The histogram captures the distribution of different "textons" representing various textures in an endoscopy image. The textons are representative response vectors of an application of a combination of Leung and Malik (LM) filter bank (i.e., a set of image filters) and a set of Local Binary Patterns on the image. Our experimental results indicate that the proposed method achieves 92% recall and 91.8% specificity on wireless capsule endoscopy (WCE) images and 91% recall and 90.8% specificity on colonoscopy images.

Part-based multiderivative edge cross-sectional profiles for polyp detection in colonoscopy.

This paper presents a novel technique for automated detection of protruding polyps in colonoscopy images using edge cross-section profiles (ECSP). We propose a part-based multiderivative ECSP that computes derivative functions of an edge cross-section profile and segments each of these profiles into parts. Therefore, we can model or extract features suitable for each part. Our features obtained from the parts can effectively describe complex properties of protruding polyps including the shape of the parts, texture, and protrusion and smoothness of the polyp surface. We evaluated our method against two existing polyp image detection techniques on 42 different polyps, including those with little protrusion. Each polyp has a large variation of appearance in viewing angles, light conditions, and scales in different images. The evaluation showed that our technique outperformed the existing techniques in both accuracy and analysis time. Our method has a higher area under the free-response receiver operating characteristic curve. For instance, when both techniques have a true positive rate for polyp image detection of 81.4%, the average number of false regions per image of our technique is 0.32 compared to 1.8 of the best existing technique under study. Additionally, our technique can precisely mark edges of candidate polyp regions as visual feedback. These results altogether indicate that our technique is promising to provide visual feedback of polyp regions in clinical practice.

3D Reconstruction of virtual colon structures from colonoscopy images.

This paper presents the first fully automated reconstruction technique of 3D virtual colon segments from individual colonoscopy images. It is the basis of new software applications that may offer great benefits for improving quality of care for colonoscopy patients. For example, a 3D map of the areas inspected and uninspected during colonoscopy can be shown on request of the endoscopist during the procedure. The endoscopist may revisit the suggested uninspected areas to reduce the chance of missing polyps that reside in these areas. The percentage of the colon surface seen by the endoscopist can be used as a coarse objective indicator of the quality of the procedure. The derived virtual colon models can be stored for post-procedure training of new endoscopists to teach navigation techniques that result in a higher level of procedure quality. Our technique does not require a prior CT scan of the colon or any global positioning device. Our experiments on endoscopy images of an Olympus synthetic colon model reveal encouraging results with small average reconstruction errors (4.1 mm for the fold depths and 12.1 mm for the fold circumferences).

SAPPHIRE: a toolkit for building efficient stream programs for medical video analysis.

This paper describes the design and implementation of SAPPHIRE--a novel middleware and software development kit for stream programing on a heterogeneous system of multi-core multi-CPUs with optional hardware accelerators such as graphics processing unit (GPU). A stream program consists of a set of tasks where the same tasks are repeated over multiple iterations of data (e.g., video frames). Examples of such programs are video analysis applications for computer-aided diagnosis and computer-assisted surgeries. Our design goal is to reduce the implementation efforts and ease collaborative software development of stream programs while supporting efficient execution of the programs on the target hardware. To validate the toolkit, we implemented EM-Automated-RT software with the toolkit and reported our experience. EM-Automated-RT performs real-time video analysis for quality of a colonoscopy procedure and provides visual feedback to assist the endoscopist to achieve optimal inspection of the colon during the procedure. The software has been deployed in a hospital setting to conduct a clinical trial.

Automatic real-time detection of endoscopic procedures using temporal features.

Endoscopy is used for inspection of the inner surface of organs such as the colon. During endoscopic inspection of the colon or colonoscopy, a tiny video camera generates a video signal, which is displayed on a monitor for interpretation in real-time by physicians. In practice, these images are not typically captured, which may be attributed by lack of fully automated tools for capturing, analysis of important contents, and quick and easy retrieval of these contents. This paper presents the description and evaluation results of our novel software that uses new metrics based on image color and motion over time to automatically record all images of an individual endoscopic procedure into a single digitized video file. The software automatically discards out-patient video frames between different endoscopic procedures. We validated our software system on 2464 h of live video (over 265 million frames) from endoscopy units where colonoscopy and upper endoscopy were performed. Our previous classification method achieved a frame-based sensitivity of 100.00%, but only a specificity of 89.22%. Our new method achieved a frame-based sensitivity and specificity of 99.90% and 99.97%, a significant improvement. Our system is robust for day-to-day use in medical practice.

Detection of quality visualization of appendiceal orifices using local edge cross-section profile features and near pause detection.

Colonoscopy is an endoscopic technique that allows a physician to inspect the inside of the human colon. The appearance of the appendiceal orifice during colonoscopy indicates a complete traversal of the colon, which is an important quality indicator of the colon examination. In this paper, we present two new algorithms. The first algorithm determines whether an image shows the clearly seen appendiceal orifice. This algorithm uses our new local features based on geometric shape, illumination difference, and intensity changes along the norm direction (cross section) of an edge. The second algorithm determines whether the video is an appendix video (the video showing at least 3 s of the appendiceal orifice inspection). Such a video indicates good visualization of the appendiceal orifice. This algorithm utilizes frame intensity histograms to detect a near camera pause during the apendiceal orifice inspection. We tested our algorithms on 23 videos captured from two types of endoscopy procedures. The average sensitivity and specificity for the detection of appendiceal orifice images with the often seen crescent appendiceal orifice shape are 96.86% and 90.47%, respectively. The average accuracy for the detection of appendix videos is 91.30%.

Measuring objective quality of colonoscopy.

Advances in video technology are being incorporated into today's healthcare practices. Colonoscopy is regarded as one of the most important diagnostic tools for colorectal cancer. Indeed, colonoscopy has contributed to a decline in the number of colorectal-cancer-related deaths. Although colonoscopy has become the preferred screening modality for prevention of colorectal cancer, recent data suggest that there is a significant miss rate for the detection of large polyps and cancers, and methods to investigate why this occurs are needed. To address this problem, we present a new computer-based method that analyzes a digitized video file of a colonoscopic procedure and produces a number of metrics that likely reflect the quality of the procedure. The method consists of a set of novel image-processing algorithms designed to address new technical challenges due to uncommon characteristics of videos captured during colonoscopy. As these measurements can be obtained automatically, our method enables future quality control in large-scale day-to-day medical practice, which is currently not feasible. In addition, our method can be adapted to other endoscopic procedures such as upper gastrointestinal endoscopy, enteroscopy, and bronchoscopy. Last but not least, our method may be useful to assess progress during colonoscopy training.

Edge cross-section features for detection of appendiceal orifice appearance in colonoscopy videos.

Colonoscopy is an endoscopic technique that allows a physician to inspect the inside of the human colon. The appearance of the appendiceal orifice during colonoscopy indicates a complete traversal of the colon, which is one of important quality indicators of examination of the colon. In this paper, we propose a new algorithm that detects appendix images-images showing the appendiceal orifice. We introduce new features based on geometric shape, saturation and intensity changes along the norm direction (cross-section) of an edge to discriminate appendix images. Our experimental results on real colonoscopic images show the average sensitivity and specificity of 88.12% and 94.25%, respectively.

Stool detection in colonoscopy videos.

Colonoscopy is the accepted screening method for detection of colorectal cancer or its precursor lesions, colorectal polyps. Indeed, colonoscopy has contributed to a decline in the number of colorectal cancer related deaths. However, not all cancers or large polyps are detected at the time of colonoscopy, and methods to investigate why this occurs are needed. One of the main factors affecting the diagnostic accuracy of colonoscopy is the quality of bowel preparation. The quality of bowel cleansing is generally assessed by the quantity of solid or liquid stool in the lumen. Despite a large body of published data on methods that could optimize cleansing, a substantial level of inadequate cleansing occurs in 10% to 75% of patients in randomized controlled trials. In this paper, a machine learning approach to the detection of stool in images of digitized colonoscopy video files is presented. The method involves the classification based on color features using a support vector machine (SVM) classifier. Our experiments show that the proposed stool image classification method is very accurate.

Quadrant coverage histogram: a new method for measuring quality of colonoscopic procedures.

Colonoscopy is an endoscopic technique that allows a physician to inspect the inside of the human colon. Although millions of colonoscopic procedures are performed annually, an objective method that estimates how much effort was undertaken to insure maximal inspection of the visible parts of the inside of the colon, does not exist. Experts agree that it is desirable to inspect all quadrants of the colon wall while the endoscope is gradually withdrawn. In this paper, we present a new computer-based method that constructs a Quadrant Coverage Histogram to determine the number of quadrants of the colon wall inspected during the withdrawal phase of colonoscopy. The proposed method is part of our novel computer-aided quality control system for colonoscopy intended for use in routine clinical practice.

Arthemis: annotation software in an integrated capturing and analysis system for colonoscopy.

Colonoscopy is an endoscopic technique that allows physicians to inspect the inside of the human colon. During a colonoscopic procedure, a tiny video camera at the tip of the endoscope generates a video signal of the internal mucosa of the colon. In current practice, the entire colonoscopic procedure is not routinely captured. Software tools providing easy access to important contents of videos that are digitally captured during colonoscopy are not available. Hence, it is very time consuming to review an entire video, locate important contents, annotate them, and extract the annotated contents for research, teaching, and training purposes. Arthemis, a software application, was developed to facilitate this process. For convenient data sharing, Arthemis allows annotation according to the European Gastrointestinal Society for Endoscopy (ESGE) Minimal Standard Terminology (MST), an internationally accepted standard for digestive endoscopy. Arthemis is part of our integrated capturing and content analysis system for colonoscopy called Endoscopic Multimedia Information System (EMIS). This paper presents Arthemis as a component of EMIS, the design and implementation of Arthemis, and key lessons learned from the development process.

Computer-aided detection of diagnostic and therapeutic operations in colonoscopy videos.

Colonoscopy is an endoscopic technique that allows a physician to inspect the inside of the human colon and to perform--if deemed necessary-at the same time a number of diagnostic and therapeutic operations. In order to see the inside of the colon, a video signal of the internal mucosa of the colon is generated by a tiny video camera at the tip of the endoscope and displayed on a monitor for real-time analysis by the physician. We have captured and stored these videos in digital format and call these colonoscopy videos. Based on new algorithms for instrument detection and shot segmentation, we introduce new spatio-temporal analysis techniques to automatically identify an operation shot--a segment of visual data in a colonoscopy video that corresponds to a diagnostic or therapeutic operation. Our experiments on real colonoscopy videos demonstrate the effectiveness of the proposed approach. The proposed techniques and software are useful for 1) postprocedure review for causes of complications due to diagnostic or therapeutic operations; 2) establishment of an effective content-based retrieval system to facilitate endoscopic research and education; 3) development of a systematic approach to assess and improve the procedural skills of endoscopists.

Informative frame classification for endoscopy video.

Advances in video technology allow inspection, diagnosis and treatment of the inside of the human body without or with very small scars. Flexible endoscopes are used to inspect the esophagus, stomach, small bowel, colon, and airways, whereas rigid endoscopes are used for a variety of minimal invasive surgeries (i.e., laparoscopy, arthroscopy, endoscopic neurosurgery). These endoscopes come in various sizes, but all have a tiny video camera at the tip. During an endoscopic procedure, the tiny video camera generates a video signal of the interior of the human organ, which is displayed on a monitor for real-time analysis by the physician. However, many out-of-focus frames are present in endoscopy videos because current endoscopes are equipped with a single, wide-angle lens that cannot be focused. We need to distinguish the out-of-focus frames from the in-focus frames to utilize the information of the out-of-focus and/or the in-focus frames for further automatic or semi-automatic computer-aided diagnosis (CAD). This classification can reduce the number of images to be viewed by a physician and to be analyzed by a CAD system. We call an out-of-focus frame a non-informative frame and an in-focus frame an informative frame. The out-of-focus frames have characteristics that are different from those of in-focus frames. In this paper, we propose two new techniques (edge-based and clustering-based) to classify video frames into two classes, informative and non-informative frames. However, because intensive specular reflections reduce the accuracy of the classification we also propose a specular reflection detection technique, and use the detected specular reflection information to increase the accuracy of informative frame classification. Our experimental studies indicate that precision, sensitivity, specificity, and accuracy for the specular reflection detection technique and the two informative frame classification techniques are greater than 90% and 95%, respectively.

Automatic classification of images with appendiceal orifice in colonoscopy videos.

Colonoscopy is an endoscopic technique that allows a physician to inspect the inside of the human colon. In current practice, videos captured from colonoscopic procedures are not routinely stored for either manual or automated post-procedure analysis. In this paper, we introduce new algorithms for automated detection of the presence of the shape of the opening of the appendix in a colonoscopy video frame. The appearance of the appendix in colonoscopy videos indicates traversal of the colon, which is an important measurement for evaluating the quality of colonoscopic procedures. The proposed techniques are valuable for (1) establishment of an effective content-based retrieval system to facilitate endoscopic research and education; and (2) assessment and improvement of the procedural skills of endoscopists, both in training and practice.