Deep learning with cinematic rendering: fine-tuning deep neural networks using photorealistic medical images.

Deep learning has emerged as a powerful artificial intelligence tool to interpret medical images for a growing variety of applications. However, the paucity of medical imaging data with high-quality annotations that is necessary for training such methods ultimately limits their performance. Medical data is challenging to acquire due to privacy issues, shortage of experts available for annotation, limited representation of rare conditions and cost. This problem has previously been addressed by using synthetically generated data. However, networks trained on synthetic data often fail to generalize to real data. Cinematic rendering simulates the propagation and interaction of light passing through tissue models reconstructed from CT data, enabling the generation of photorealistic images. In this paper, we present one of the first applications of cinematic rendering in deep learning, in which we propose to fine-tune synthetic data-driven networks using cinematically rendered CT data for the task of monocular depth estimation in endoscopy. Our experiments demonstrate that: (a) convolutional neural networks (CNNs) trained on synthetic data and fine-tuned on photorealistic cinematically rendered data adapt better to real medical images and demonstrate more robust performance when compared to networks with no fine-tuning, (b) these fine-tuned networks require less training data to converge to an optimal solution, and (c) fine-tuning with data from a variety of photorealistic rendering conditions of the same scene prevents the network from learning patient-specific information and aids in generalizability of the model. Our empirical evaluation demonstrates that networks fine-tuned with cinematically rendered data predict depth with 56.87% less error for rendered endoscopy images and 27.49% less error for real porcine colon endoscopy images.

Detailed quantitative assessment of colonic morphology at CT colonography using novel software: a feasibility and reproducibility study.

The aim of this study was to evaluate feasibility and reproducibility of quantitative assessment of colonic morphology on CT colonography (CTC). CTC datasets from 60 patients with optimal colonic distension were assessed using prototype software. Metrics potentially associated with poor endoscopic performance were calculated for the total colon and each segment including: length, volume, tortuosity (number of high curvature points <90°), and compactness (volume of box containing centerline divided by centerline length). Sigmoid apex height relative to the lumbosacral junction was also measured. Datasets were quantified twice each, and intra-reader reliability was evaluated using concordance correlation coefficient and Bland-Altman plot. Complete quantitative datasets including the five proposed metrics were generated from 58 of 60 (97 %) CTC examinations. The sigmoid and transverse segments were the longest (55.9 and 51.4 cm), had the largest volumes (0.410 and 0.609 L), and were the most tortuous (3.39 and 2.75 high curvature points) and least compact (3347 and 3595 mm2), noting high inter-patient variability for all metrics. Mean height of the sigmoid apex was 6.7 cm, also with high inter-patient variability (SD 6.8 cm). Intra-reader reliability was high for total and segmental lengths and sigmoid apex height (CCC = 0.9991) with excellent repeatability coefficient (CR = 3.0-3.3). There was low percent variance of metrics dependent upon length (median 5 %). Detailed automated quantitative assessment of colonic morphology on routine CTC datasets is feasible and reproducible, requiring minimal reader interaction.

Colon unfolding via skeletal subspace deformation.

We present an efficient method to digitally straighten a colon volume using mesh skinning, a technique well known in computer graphics to deform a polygonal mesh attached to a skeleton hierarchy. In our case, the colon centerline is used as the skeleton structure and the polyhedral model of the lumen as the skin that is to be deformed as the centerline is straightened. Once the colon has been straightened, we use standard rendering techniques to compute the virtual dissection. Our approach is significantly more efficient than previously proposed techniques.

Panoramic views for virtual endoscopy.

This paper describes a panoramic projection designed to increase the surface visibility during virtual endoscopies. The proposed projection renders five faces of a cubic viewing space into the plane in a continuous fashion. Using this real-time and interactive visualization technique as a screening method for colon cancer could lead to significantly shorter evaluation time. It avoids having to fly through the colon in both directions and prevents the occlusion of potential polyps behind haustral folds.