NerveTracker: a Python-based software toolkit for visualizing and tracking groups of nerve fibers in serial block-face microscopy with ultraviolet surface excitation images.

Significance: Information about the spatial organization of fibers within a nerve is crucial to our understanding of nerve anatomy and its response to neuromodulation therapies. A serial block-face microscopy method [three-dimensional microscopy with ultraviolet surface excitation (3D-MUSE)] has been developed to image nerves over extended depths ex vivo. To routinely visualize and track nerve fibers in these datasets, a dedicated and customizable software tool is required. Aim: Our objective was to develop custom software that includes image processing and visualization methods to perform microscopic tractography along the length of a peripheral nerve sample. Approach: We modified common computer vision algorithms (optic flow and structure tensor) to track groups of peripheral nerve fibers along the length of the nerve. Interactive streamline visualization and manual editing tools are provided. Optionally, deep learning segmentation of fascicles (fiber bundles) can be applied to constrain the tracts from inadvertently crossing into the epineurium. As an example, we performed tractography on vagus and tibial nerve datasets and assessed accuracy by comparing the resulting nerve tracts with segmentations of fascicles as they split and merge with each other in the nerve sample stack. Results: We found that a normalized Dice overlap ( Dice norm ) metric had a mean value above 0.75 across several millimeters along the nerve. We also found that the tractograms were robust to changes in certain image properties (e.g., downsampling in-plane and out-of-plane), which resulted in only a 2% to 9% change to the mean Dice norm values. In a vagus nerve sample, tractography allowed us to readily identify that subsets of fibers from four distinct fascicles merge into a single fascicle as we move ∼ 5 mm along the nerve's length. Conclusions: Overall, we demonstrated the feasibility of performing automated microscopic tractography on 3D-MUSE datasets of peripheral nerves. The software should be applicable to other imaging approaches. The code is available at https://github.com/ckolluru/NerveTracker.

Prediction of stent under-expansion in calcified coronary arteries using machine learning on intravascular optical coherence tomography images.

It can be difficult/impossible to fully expand a coronary artery stent in a heavily calcified coronary artery lesion. Under-expanded stents are linked to later complications. Here we used machine/deep learning to analyze calcifications in pre-stent intravascular optical coherence tomography (IVOCT) images and predicted the success of vessel expansion. Pre- and post-stent IVOCT image data were obtained from 110 coronary lesions. Lumen and calcifications in pre-stent images were segmented using deep learning, and lesion features were extracted. We analyzed stent expansion along the lesion, enabling frame, segmental, and whole-lesion analyses. We trained regression models to predict the post-stent lumen area and then computed the stent expansion index (SEI). Best performance (root-mean-square-error = 0.04 ± 0.02 mm2, r = 0.94 ± 0.04, p < 0.0001) was achieved when we used features from both lumen and calcification to train a Gaussian regression model for segmental analysis of 31 frames in length. Stents with minimum SEI > 80% were classified as "well-expanded;" others were "under-expanded." Under-expansion classification results (e.g., AUC = 0.85 ± 0.02) were significantly improved over a previous, simple calculation, as well as other machine learning solutions. Promising results suggest that such methods can identify lesions at risk of under-expansion that would be candidates for intervention lesion preparation (e.g., atherectomy).

Transport pathways and kinetics of cerebrospinal fluid tracers in mouse brain observed by dynamic contrast-enhanced MRI.

Recent studies have suggested the glymphatic system as a key mechanism of waste removal in the brain. Dynamic contrast-enhanced MRI (DCE-MRI) using intracisternally administered contrast agents is a promising tool for assessing glymphatic function in the whole brain. In this study, we evaluated the transport kinetics and distribution of three MRI contrast agents with vastly different molecular sizes in mice. Our results demonstrate that oxygen-17 enriched water (H217O), which has direct access to parenchymal tissues via aquaporin-4 water channels, exhibited significantly faster and more extensive transport compared to the two gadolinium-based contrast agents (Gd-DTPA and GadoSpin). Time-lagged correlation and clustering analyses also revealed different transport pathways for Gd-DTPA and H217O. Furthermore, there were significant differences in transport kinetics of the three contrast agents to the lateral ventricles, reflecting the differences in forces that drive solute transport in the brain. These findings suggest the size-dependent transport pathways and kinetics of intracisternally administered contrast agents and the potential of DCE-MRI for assessing multiple aspects of solute transport in the glymphatic system.

Deep-learning segmentation of fascicles from microCT of the human vagus nerve.

Introduction: MicroCT of the three-dimensional fascicular organization of the human vagus nerve provides essential data to inform basic anatomy as well as the development and optimization of neuromodulation therapies. To process the images into usable formats for subsequent analysis and computational modeling, the fascicles must be segmented. Prior segmentations were completed manually due to the complex nature of the images, including variable contrast between tissue types and staining artifacts. Methods: Here, we developed a U-Net convolutional neural network (CNN) to automate segmentation of fascicles in microCT of human vagus nerve. Results: The U-Net segmentation of ~500 images spanning one cervical vagus nerve was completed in 24 s, versus ~40 h for manual segmentation, i.e., nearly four orders of magnitude faster. The automated segmentations had a Dice coefficient of 0.87, a measure of pixel-wise accuracy, thus suggesting a rapid and accurate segmentation. While Dice coefficients are a commonly used metric to assess segmentation performance, we also adapted a metric to assess fascicle-wise detection accuracy, which showed that our network accurately detects the majority of fascicles, but may under-detect smaller fascicles. Discussion: This network and the associated performance metrics set a benchmark, using a standard U-Net CNN, for the application of deep-learning algorithms to segment fascicles from microCT images. The process may be further optimized by refining tissue staining methods, modifying network architecture, and expanding the ground-truth training data. The resulting three-dimensional segmentations of the human vagus nerve will provide unprecedented accuracy to define nerve morphology in computational models for the analysis and design of neuromodulation therapies.

Transport Pathways and Kinetics of Cerebrospinal Fluid Tracers in Mouse Brain Observed by Dynamic Contrast-Enhanced MRI.

Background: Recent studies have suggested the glymphatic system as a solute transport pathway and waste removal mechanism in the brain. Imaging intracisternally administered tracers provides the opportunity of assessing various aspects of the glymphatic function. Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) allows the evaluation of both the kinetics and spatial distribution of tracer transport in the whole brain. However, assessing mouse glymphatic function by DCE-MRI has been challenged by the small size of a mouse brain and the limited volume of fluids that can be delivered intracisternally without significantly altering the intracranial pressure. Further, previous studies in rats suggest that assessment of glymphatic function by DCE-MRI is dependent on the molecular size of the contrast agents. Methods: We established and validated an intracisternal infusion protocol in mice that allowed the measurements of the entire time course of contrast agent transport for 2 hours. The transport kinetics and distribution of three MRI contrast agents with drastically different molecular weights (MWs): Gd-DTPA (MW=661.8 Da, n=7), GadoSpin-P (MW=200 kDa, n=6), and oxygen-17 enriched water (H 2 17 O, MW=19 Da, n=7), were investigated. Results: The transport of H 2 17 O was significantly faster and more extensive than the two gadolinium-based contrast agents. Time-lagged correlation analysis and clustering analysis comparing the kinetics of Gd-DTPA and H 2 17 O transport also showed different cluster patterns and lag time between different regions of the brain, suggesting different transport pathways for H 2 17 O because of its direct access to parenchymal tissues via the aquaporin-4 water channels. Further, there were also significant differences in the transport kinetics of the three tracers to the lateral ventricles, which reflects the differences in forces that drive tracer transport in the brain. Conclusions: Comparison of the transport kinetics and distribution of three MRI contrast agents with different molecular sizes showed drastically different transport profiles and clustering patterns, suggesting that the transport pathways and kinetics in the glymphatic system are size-dependent.

OCTOPUS - Optical coherence tomography plaque and stent analysis software.

Background and objective: Compared with other imaging modalities, intravascular optical coherence tomography (IVOCT) has significant advantages for guiding percutaneous coronary interventions, assessing their outcomes, and characterizing plaque components. To aid IVOCT research studies, we developed the Optical Coherence TOmography PlaqUe and Stent (OCTOPUS) analysis software, which provides highly automated, comprehensive analysis of coronary plaques and stents in IVOCT images. Methods: User specifications for OCTOPUS were obtained from detailed, iterative discussions with IVOCT analysts in the Cardiovascular Imaging Core Laboratory at University Hospitals Cleveland Medical Center, a leading laboratory for IVOCT image analysis. To automate image analysis results, the software includes several important algorithmic steps: pre-processing, deep learning plaque segmentation, machine learning identification of stent struts, and registration of pullbacks for sequential comparisons. Intuitive, interactive visualization and manual editing of segmentations were included in the software. Quantifications include stent deployment characteristics (e.g., stent area and stent strut malapposition), strut level analysis, calcium angle, and calcium thickness measurements. Interactive visualizations include (x,y) anatomical, en face, and longitudinal views with optional overlays (e.g., segmented calcifications). To compare images over time, linked visualizations were enabled to display up to four registered vessel segments at a time. Results: OCTOPUS has been deployed for nearly 1 year and is currently being used in multiple IVOCT studies. Underlying plaque segmentation algorithm yielded excellent pixel-wise results (86.2% sensitivity and 0.781 F1 score). Using OCTOPUS on 34 new pullbacks, we determined that following automated segmentation, only 13% and 23% of frames needed any manual touch up for detailed lumen and calcification labeling, respectively. Only up to 3.8% of plaque pixels were modified, leading to an average editing time of only 7.5 s/frame, an approximately 80% reduction compared to manual analysis. Regarding stent analysis, sensitivity and precision were both greater than 90%, and each strut was successfully classified as either covered or uncovered with high sensitivity (94%) and specificity (90%). We demonstrated use cases for sequential analysis. To analyze plaque progression, we loaded multiple pullbacks acquired at different points (e.g., pre-stent, 3-month follow-up, and 18-month follow-up) and evaluated frame-level development of in-stent neo-atherosclerosis. In ex vivo cadaver experiments, the OCTOPUS software enabled visualization and quantitative evaluation of irregular stent deployment in the presence of calcifications identified in pre-stent images. Conclusions: We introduced and evaluated the clinical application of a highly automated software package, OCTOPUS, for quantitative plaque and stent analysis in IVOCT images. The software is currently used as an offline tool for research purposes; however, the software's embedded algorithms may also be useful for real-time treatment planning.

Automatic microchannel detection using deep learning in intravascular optical coherence tomography images.

Microchannel formation is known to be a significant marker of plaque vulnerability, plaque rupture, and intraplaque hemorrhage, which are responsible for plaque progression. We developed a fully-automated method for detecting microchannels in intravascular optical coherence tomography (IVOCT) images using deep learning. A total of 3,075 IVOCT image frames across 41 patients having 62 microchannel segments were analyzed. Microchannel was manually annotated by expert cardiologists, according to previously established criteria. In order to improve segmentation performance, pre-processing including guidewire detection/removal, lumen segmentation, pixel-shifting, and noise filtering was applied to the raw (r,θ) IVOCT image. We used the DeepLab-v3 plus deep learning model with the Xception backbone network for identifying microchannel candidates. After microchannel candidate detection, each candidate was classified as either microchannel or no-microchannel using a convolutional neural network (CNN) classification model. Our method provided excellent segmentation of microchannel with a Dice coefficient of 0.811, sensitivity of 92.4%, and specificity of 99.9%. We found that pre-processing and data augmentation were very important to improve results. In addition, a CNN classification step was also helpful to rule out false positives. Furthermore, automated analysis missed only 3% of frames having microchannels and showed no false positives. Our method has great potential to enable highly automated, objective, repeatable, and comprehensive evaluations of vulnerable plaques and treatments. We believe that this method is promising for both research and clinical applications.

Automated analysis of fibrous cap in intravascular optical coherence tomography images of coronary arteries.

Thin-cap fibroatheroma (TCFA) and plaque rupture have been recognized as the most frequent risk factor for thrombosis and acute coronary syndrome. Intravascular optical coherence tomography (IVOCT) can identify TCFA and assess cap thickness, which provides an opportunity to assess plaque vulnerability. We developed an automated method that can detect lipidous plaque and assess fibrous cap thickness in IVOCT images. This study analyzed a total of 4360 IVOCT image frames of 77 lesions among 41 patients. Expert cardiologists manually labeled lipidous plaque based on established criteria. To improve segmentation performance, preprocessing included lumen segmentation, pixel-shifting, and noise filtering on the raw polar (r, θ) IVOCT images. We used the DeepLab-v3 plus deep learning model to classify lipidous plaque pixels. After lipid detection, we automatically detected the outer border of the fibrous cap using a special dynamic programming algorithm and assessed the cap thickness. Our method provided excellent discriminability of lipid plaque with a sensitivity of 85.8% and A-line Dice coefficient of 0.837. By comparing lipid angle measurements between two analysts following editing of our automated software, we found good agreement by Bland-Altman analysis (difference 6.7° ± 17°; mean ~ 196°). Our method accurately detected the fibrous cap from the detected lipid plaque. Automated analysis required a significant modification for only 5.5% frames. Furthermore, our method showed a good agreement of fibrous cap thickness between two analysts with Bland-Altman analysis (4.2 ± 14.6 µm; mean ~ 175 µm), indicating little bias between users and good reproducibility of the measurement. We developed a fully automated method for fibrous cap quantification in IVOCT images, resulting in good agreement with determinations by analysts. The method has great potential to enable highly automated, repeatable, and comprehensive evaluations of TCFAs.

Fascicles split or merge every ∼560 microns within the human cervical vagus nerve.

Objective.Vagus nerve stimulation (VNS) is Food and Drug Administration-approved for epilepsy, depression, and obesity, and stroke rehabilitation; however, the morphological anatomy of the vagus nerve targeted by stimulatation is poorly understood. Here, we used microCT to quantify the fascicular structure and neuroanatomy of human cervical vagus nerves (cVNs).Approach.We collected eight mid-cVN specimens from five fixed cadavers (three left nerves, five right nerves). Analysis focused on the 'surgical window': 5 cm of length, centered around the VNS implant location. Tissue was stained with osmium tetroxide, embedded in paraffin, and imaged on a microCT scanner. We visualized and quantified the merging and splitting of fascicles, and report a morphometric analysis of fascicles: count, diameter, and area.Main results.In our sample of human cVNs, a fascicle split or merge event was observed every ∼560µm (17.8 ± 6.1 events cm-1). Mean morphological outcomes included: fascicle count (6.6 ± 2.8 fascicles; range 1-15), fascicle diameter (514 ± 142µm; range 147-1360µm), and total cross-sectional fascicular area (1.32 ± 0.41 mm2; range 0.58-2.27 mm).Significance.The high degree of fascicular splitting and merging, along with wide range in key fascicular morphological parameters across humans may help to explain the clinical heterogeneity in patient responses to VNS. These data will enable modeling and experimental efforts to determine the clinical effect size of such variation. These data will also enable efforts to design improved VNS electrodes.

Imaging peripheral nerve micro-anatomy with MUSE, 2D and 3D approaches.

Understanding peripheral nerve micro-anatomy can assist in the development of safe and effective neuromodulation devices. However, current approaches for imaging nerve morphology at the fiber level are either cumbersome, require substantial instrumentation, have a limited volume of view, or are limited in resolution/contrast. We present alternative methods based on MUSE (Microscopy with Ultraviolet Surface Excitation) imaging to investigate peripheral nerve morphology, both in 2D and 3D. For 2D imaging, fixed samples are imaged on a conventional MUSE system either label free (via auto-fluorescence) or after staining with fluorescent dyes. This method provides a simple and rapid technique to visualize myelinated nerve fibers at specific locations along the length of the nerve and perform measurements of fiber morphology (e.g., axon diameter and g-ratio). For 3D imaging, a whole-mount staining and MUSE block-face imaging method is developed that can be used to characterize peripheral nerve micro-anatomy and improve the accuracy of computational models in neuromodulation. Images of rat sciatic and human cadaver tibial nerves are presented, illustrating the applicability of the method in different preclinical models.

In vivo Visualization of Pig Vagus Nerve "Vagotopy" Using Ultrasound.

Background: Placement of the clinical vagus nerve stimulating cuff is a standard surgical procedure based on anatomical landmarks, with limited patient specificity in terms of fascicular organization or vagal anatomy. As such, the therapeutic effects are generally limited by unwanted side effects of neck muscle contractions, demonstrated by previous studies to result from stimulation of (1) motor fibers near the cuff in the superior laryngeal and (2) motor fibers within the cuff projecting to the recurrent laryngeal. Objective: Conventional non-invasive ultrasound, where the transducer is placed on the surface of the skin, has been previously used to visualize the vagus with respect to other landmarks such as the carotid and internal jugular vein. However, it lacks sufficient resolution to provide details about the vagus fascicular organization, or detail about smaller neural structures such as the recurrent and superior laryngeal branch responsible for therapy limiting side effects. Here, we characterize the use of ultrasound with the transducer placed in the surgical pocket to improve resolution without adding significant additional risk to the surgical procedure in the pig model. Methods: Ultrasound images were obtained from a point of known functional organization at the nodose ganglia to the point of placement of stimulating electrodes within the surgical window. Naïve volunteers with minimal training were then asked to use these ultrasound videos to trace afferent groupings of fascicles from the nodose to their location within the surgical window where a stimulating cuff would normally be placed. Volunteers were asked to select a location for epineural electrode placement away from the fascicles containing efferent motor nerves responsible for therapy limiting side effects. 2-D and 3-D reconstructions of the ultrasound were directly compared to post-mortem histology in the same animals. Results: High-resolution ultrasound from the surgical pocket enabled 2-D and 3-D reconstruction of the cervical vagus and surrounding structures that accurately depicted the functional vagotopy of the pig vagus nerve as confirmed via histology. Although resolution was not sufficient to match specific fascicles between ultrasound and histology 1 to 1, it was sufficient to trace fascicle groupings from a point of known functional organization at the nodose ganglia to their locations within the surgical window at stimulating electrode placement. Naïve volunteers were able place an electrode proximal to the sensory afferent grouping of fascicles and away from the motor nerve efferent grouping of fascicles in each subject (n = 3). Conclusion: The surgical pocket itself provides a unique opportunity to obtain higher resolution ultrasound images of neural targets responsible for intended therapeutic effect and limiting off-target effects. We demonstrate the increase in resolution is sufficient to aid patient-specific electrode placement to optimize outcomes. This simple technique could be easily adopted for multiple neuromodulation targets to better understand how patient specific anatomy impacts functional outcomes.

Learning With Fewer Images via Image Clustering: Application to Intravascular OCT Image Segmentation.

Deep learning based methods are routinely used to segment various structures of interest in varied medical imaging modalities. Acquiring annotations for a large number of images requires a skilled analyst, and the process is both time consuming and challenging. Our approach to reduce effort is to reduce the number of images needing detailed annotation. For intravascular optical coherence tomography (IVOCT) image pullbacks, we tested 10% to 100% of training images derived from two schemes: equally-spaced image subsampling and deep-learning- based image clustering. The first strategy involves selecting images at equally spaced intervals from the volume, accounting for the high spatial correlation between neighboring images. In clustering, we used an autoencoder to create a deep feature space representation, performed k-medoids clustering, and then used the cluster medians for training. For coronary calcifications, a baseline U-net model was trained on all images from volumes of interest (VOIs) and compared with fewer images from the sub-sampling strategies. For a given sampling ratio, the clustering based method performed better or similar as compared to the equally spaced sampling approach (e.g., with 10% of images, mean F1 score for calcific class increased from 0.52 to 0.63, with equal spacing and with clustering, respectively). Additionally, for a fixed number of training images, sampling images from more VOIs performed better than otherwise. In conclusion, we recommend the clustering based approach to annotate a small fraction of images, creating a baseline model, which potentially can be improved further by annotating images selected using methods described in active learning research.

3D imaging of the vagus nerve fascicular anatomy with cryo-imaging and UV excitation.

Vagus nerve stimulation (VNS) is a method to treat drug-resistant epilepsy and depression, but therapeutic outcomes are often not ideal. Newer electrode designs such as intra-fascicular electrodes offer potential improvements in reducing off-target effects but require a detailed understanding of the fascicular anatomy of the vagus nerve. We have adapted a section-and-image technique, cryo-imaging, with UV excitation to visualize fascicles along the length of the vagus nerve. In addition to offering optical sectioning at the surface via reduced penetration depth, UV illumination also produces sufficient contrast between fascicular structures and connective tissue. Here we demonstrate the utility of this approach in pilot experiments. We imaged fixed, cadaver vagus nerve samples, segmented fascicles, and demonstrated 3D tracking of fascicles. Such data can serve as input for computer models of vagus nerve stimulation.

Quantitative and qualitative evaluation of deep learning automatic segmentations of corneal endothelial cell images of reduced image quality obtained following cornea transplant.

We are developing automated analysis of corneal-endothelial-cell-layer, specular microscopic images so as to determine quantitative biomarkers indicative of corneal health following corneal transplantation. Especially on these images of varying quality, commercial automated image analysis systems can give inaccurate results, and manual methods are very labor intensive. We have developed a method to automatically segment endothelial cells with a process that included image flattening, U-Net deep learning, and postprocessing to create individual cell segmentations. We used 130 corneal endothelial cell images following one type of corneal transplantation (Descemet stripping automated endothelial keratoplasty) with expert-reader annotated cell borders. We obtained very good pixelwise segmentation performance (e.g., Dice coefficient = 0.87 ± 0.17 , Jaccard index = 0.80 ± 0.18 , across 10 folds). The automated method segmented cells left unmarked by analysts and sometimes segmented cells differently than analysts (e.g., one cell was split or two cells were merged). A clinically informative visual analysis of the held-out test set showed that 92% of cells within manually labeled regions were acceptably segmented and that, as compared to manual segmentation, automation added 21% more correctly segmented cells. We speculate that automation could reduce 15 to 30 min of manual segmentation to 3 to 5 min of manual review and editing.

Fully automated plaque characterization in intravascular OCT images using hybrid convolutional and lumen morphology features.

For intravascular OCT (IVOCT) images, we developed an automated atherosclerotic plaque characterization method that used a hybrid learning approach, which combined deep-learning convolutional and hand-crafted, lumen morphological features. Processing was done on innate A-line units with labels fibrolipidic (fibrous tissue followed by lipidous tissue), fibrocalcific (fibrous tissue followed by calcification), or other. We trained/tested on an expansive data set (6,556 images), and performed an active learning, relabeling step to improve noisy ground truth labels. Conditional random field was an important post-processing step to reduce classification errors. Sensitivities/specificities were 84.8%/97.8% and 91.4%/95.7% for fibrolipidic and fibrocalcific plaques, respectively. Over lesions, en face classification maps showed automated results that agreed favorably to manually labeled counterparts. Adding lumen morphological features gave statistically significant improvement (p < 0.05), as compared to classification with convolutional features alone. Automated assessments of clinically relevant plaque attributes (arc angle and length), compared favorably to those from manual labels. Our hybrid approach gave statistically improved results as compared to previous A-line classification methods using deep learning or hand-crafted features alone. This plaque characterization approach is fully automated, robust, and promising for live-time treatment planning and research applications.

Automatic A-line coronary plaque classification using combined deep learning and textural features in intravascular OCT images.

We developed a fully automated method for classifying A-line coronary plaques in intravascular optical coherence tomography images using combined deep learning and textural features. The proposed method was trained on 4,292 images from 48 pullbacks giving 80 manually labeled, volumes of interest. Preprocessing steps including guidewire/shadow removal, lumen boundary detection, pixel shifting, and noise reduction were employed. We built a convolutional neural network to extract the deep learning features from the preprocessed image. Traditional textural features were also extracted and combined with deep learning features. Feature selection was performed using the minimum redundancy maximum relevance method. Combined features were utilized as inputs for a random forest classifier. After classification, conditional random field (CRF) method was used for classification noise cleaning. We determined a sub-feature set with the most predictive power. With CRF noise cleaning, sensitivities/specificities were 82.2%/90.8% and 82.4%/89.2% for fibrolipidic and fibrocalcific classes, respectively, with good Dice coefficients. The classification noise cleaning step improved performance metrics by nearly 10-15%. The predicted en face classification maps of entire pullbacks agreed favorably to the manually labeled counterparts. Both assessments suggested that our automated measurements gave clinically relevant results. The proposed method is very promising with regards to both clinical treatment planning and research applications.

Segmentation of Coronary Calcified Plaque in Intravascular OCT Images Using a Two-Step Deep Learning Approach.

We developed a fully automated, two-step deep learning approach for characterizing coronary calcified plaque in intravascular optical coherence tomography (IVOCT) images. First, major calcification lesions were detected from an entire pullback using a 3D convolutional neural network (CNN). Second, a SegNet deep learning model with the Tversky loss function was used to segment calcified plaques in the major calcification lesions. The fully connected conditional random field and the frame interpolation of the missing calcification frames were used to reduce classification errors. We trained/tested the networks on a large dataset comprising 8,231 clinical images from 68 patients with 68 vessels and 4,320 ex vivo cadaveric images from 4 hearts with 4 vessels. The 3D CNN model detected major calcifications with high sensitivity (97.7%), specificity (87.7%), and F1 score (0.922). Compared to the standard one-step approach, our two-step deep learning approach significantly improved sensitivity (from 77.5% to 86.2%), precision (from 73.5% to 75.8%), and F1 score (from 0.749 to 0.781). We investigated segmentation performance for varying numbers of training samples; at least 3,900 images were required to obtain stable segmentation results. We also found very small differences in calcification attributes (e.g., angle, thickness, and depth) and identical calcium scores on repetitive pullbacks, indicating excellent reproducibility. Applied to new clinical pullbacks, our method has implications for real-time treatment planning and imaging research.

Automated plaque characterization using deep learning on coronary intravascular optical coherence tomographic images.

Accurate identification of coronary plaque is very important for cardiologists when treating patients with advanced atherosclerosis. We developed fully-automated semantic segmentation of plaque in intravascular OCT images. We trained/tested a deep learning model on a folded, large, manually annotated clinical dataset. The sensitivities/specificities were 87.4%/89.5% and 85.1%/94.2% for pixel-wise classification of lipidous and calcified plaque, respectively. Automated clinical lesion metrics, potentially useful for treatment planning and research, compared favorably (<4%) with those derived from ground-truth labels. When we converted the results to A-line classification, they were significantly better (p < 0.05) than those obtained previously by using deep learning classifications of A-lines.

Machine learning for segmenting cells in corneal endothelium images.

Images of the endothelial cell layer of the cornea can be used to evaluate corneal health. Quantitative biomarkers extracted from these images such as cell density, coefficient of variation of cell area, and cell hexagonality are commonly used to evaluate the status of the endothelium. Currently, fully-automated endothelial image analysis systems in use often give inaccurate results, while semi-automated methods, requiring trained image analysis readers to identify cells manually, are both challenging and time-consuming. We are investigating two deep learning methods to automatically segment cells in such images. We compare the performance of two deep neural networks, namely U-Net and SegNet. To train and test the classifiers, a dataset of 130 images was collected, with expert reader annotated cell borders in each image. We applied standard training and testing techniques to evaluate pixel-wise segmentation performance, and report corresponding metrics such as the Dice and Jaccard coefficients. Visual evaluation of results showed that most pixel-wise errors in the U-Net were rather non-consequential. Results from the U-Net approach are being applied to create endothelial cell segmentations and quantify important morphological measurements for evaluating cornea health.

Automated A-line coronary plaque classification of intravascular optical coherence tomography images using handcrafted features and large datasets.

We developed machine learning methods to identify fibrolipidic and fibrocalcific A-lines in intravascular optical coherence tomography (IVOCT) images using a comprehensive set of handcrafted features. We incorporated features developed in previous studies (e.g., optical attenuation and A-line peaks). In addition, we included vascular lumen morphology and three-dimensional (3-D) digital edge and texture features. Classification methods were developed using expansive datasets (∼7000 images), consisting of both clinical in-vivo images and an ex-vivo dataset, which was validated using 3-D cryo-imaging/histology. Conditional random field was used to perform 3-D classification noise cleaning of classification results. We tested various multiclass approaches, classifiers, and feature selection schemes and found that a three-class support vector machine with minimal-redundancy-maximal-relevance feature selection gave the best performance. We found that inclusion of our morphological and 3-D features improved overall classification accuracy. On a held-out test set consisting of >1700 images, we obtained an overall accuracy of 81.58%, with the following (sensitivity/specificity) for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). The en-face views of classification results showed that automated classification easily captured the preponderance of a disease segment (e.g., a calcified segment had large regions of fibrocalcific classifications). Finally, we demonstrated proof-of-concept for streamlining A-line classification output with existing fibrolipidic and fibrocalcific boundary segmentation methods, to enable fully automated plaque quantification. The results suggest that our classification approach is a viable step toward fully automated IVOCT plaque classification and segmentation for live-time treatment planning and for offline assessment of drug and biologic therapeutics.