Deep Learning Based Parameterization of Diffeomorphic Image Registration for Cardiac Image Segmentation.

Cardiac segmentation from magnetic resonance imaging (MRI) is one of the essential tasks in analyzing the anatomy and function of the heart for the assessment and diagnosis of cardiac diseases. However, cardiac MRI generates hundreds of images per scan, and manual annotation of them is challenging and time-consuming, and therefore processing these images automatically is of interest. This study proposes a novel end-to-end supervised cardiac MRI segmentation framework based on a diffeomorphic deformable registration that can segment cardiac chambers from 2D and 3D images or volumes. To represent actual cardiac deformation, the method parameterizes the transformation using radial and rotational components computed via deep learning, with a set of paired images and segmentation masks used for training. The formulation guarantees transformations that are invertible and prevents mesh folding, which is essential for preserving the topology of the segmentation results. A physically plausible transformation is achieved by employing diffeomorphism in computing the transformations and activation functions that constrain the range of the radial and rotational components. The method was evaluated over three different data sets and showed significant improvements compared to exacting learning and non-learning based methods in terms of the Dice score and Hausdorff distance metrics.

Training Convolutional Neural Networks and Compressed Sensing End-to-End for Microscopy Cell Detection.

Automated cell detection and localization from microscopy images are significant tasks in biomedical research and clinical practice. In this paper, we design a new cell detection and localization algorithm that combines deep convolutional neural network (CNN) and compressed sensing (CS) or sparse coding (SC) for end-to-end training. We also derive, for the first time, a backpropagation rule, which is applicable to train any algorithm that implements a sparse code recovery layer. The key innovation behind our algorithm is that the cell detection task is structured as a point object detection task in computer vision, where the cell centers (i.e., point objects) occupy only a tiny fraction of the total number of pixels in an image. Thus, we can apply compressed sensing (or equivalently SC) to compactly represent a variable number of cells in a projected space. Subsequently, CNN regresses this compressed vector from the input microscopy image. The SC/CS recovery algorithm ( L 1 optimization) can then recover sparse cell locations from the output of CNN. We train this entire processing pipeline end-to-end and demonstrate that end-to-end training improves accuracy over a training paradigm that treats CNN and CS-recovery layers separately. We have validated our algorithm on five benchmark datasets with excellent results.

Fast Large-Scale Spectral Clustering via Explicit Feature Mapping.

We propose an efficient spectral clustering method for large-scale data. The main idea in our method consists of employing random Fourier features to explicitly represent data in kernel space. The complexity of spectral clustering thus is shown lower than existing Nyström approximations on large-scale data. With m training points from a total of n data points, Nyström method requires O(nmd+m3+nm2) operations, where d is the input dimension. In contrast, our proposed method requires O(nDd+D3+n'D2) , where n' is the number of data points needed until convergence and D is the kernel mapped dimension. In large-scale datasets where n' << n hold true, our explicitly mapping method can significantly speed up eigenvector approximation and benefit prediction speed in spectral clustering. For instance, on MNIST (60 000 data points), the proposed method is similar in clustering accuracy to Nyström methods while its speed is twice as fast as Nyström.

Object Classification With Joint Projection and Low-Rank Dictionary Learning.

For an object classification system, the most critical obstacles toward real-world applications are often caused by large intra-class variability, arising from different lightings, occlusion, and corruption, in limited sample sets. Most methods in the literature would fail when the training samples are heavily occluded, corrupted or have significant illumination or viewpoint variations. Besides, most of the existing methods and especially deep learning-based methods, need large training sets to achieve a satisfactory recognition performance. Although using the pre-trained network on a generic large-scale data set and fine-tune it to the small-sized target data set is a widely used technique, this would not help when the content of base and target data sets are very different. To address these issues simultaneously, we propose a joint projection and low-rank dictionary learning method using dual graph constraints. Specifically, a structured class-specific dictionary is learned in the low-dimensional space, and the discrimination is further improved by imposing a graph constraint on the coding coefficients, that maximizes the intra-class compactness and inter-class separability. We enforce structural incoherence and low-rank constraints on sub-dictionaries to reduce the redundancy among them, and also make them robust to variations and outliers. To preserve the intrinsic structure of data, we introduce a supervised neighborhood graph into the framework to make the proposed method robust to small-sized and high-dimensional data sets. Experimental results on several benchmark data sets verify the superior performance of our method for object classification of small-sized data sets, which include a considerable amount of different kinds of variation, and may have high-dimensional feature vectors.

New algorithm for semiautomatic segmentation of nasal cavity and pharyngeal airway in comparison with manual segmentation using cone-beam computed tomography.

INTRODUCTION: Our objectives were to assess reliability, validity, and time efficiency of semiautomatic segmentation using Segura software of the nasal and pharyngeal airways, against manual segmentation with point-based analysis with color mapping. METHODS: Pharyngeal and nasal airways from 10 cone-beam computed tomography image sets were segmented manually and semiautomatically using Segura (University of Alberta, Edmonton, Alberta, Canada). To test intraexaminer and interexaminer reliabilities, semiautomatic segmentation was repeated 3 times by 1 examiner and then by 3 examiners. In addition to volume and surface area, point-based analysis was completed to assess the reconstructed 3-dimensional models from Segura against manual segmentation. The times of both methods of segmentation were also recorded to assess time efficiency. RESULTS: The reliability and validity of Segura were excellent (intraclass correlation coefficient, >0.9 for volume and surface area). Part analysis showed small differences between the Segura and manually segmented 3-dimensional models (greatest difference did not exceed 4.3 mm). Time of segmentation using Segura was significantly shorter than that for manual segmentation, 49 ± 11.0 vs 109 ± 9.4 minutes (P <0.001). CONCLUSIONS: Semiautomatic segmentation of the pharyngeal and nasal airways using Segura was found to be reliable, valid, and time efficient. Part analysis with color mapping was the key to explaining differences in upper airway volume and provides meaningful and clinically relevant analysis of 3-dimensional changes.

MISTICA: Minimum Spanning Tree-Based Coarse Image Alignment for Microscopy Image Sequences.

Registration of an in vivo microscopy image sequence is necessary in many significant studies, including studies of atherosclerosis in large arteries and the heart. Significant cardiac and respiratory motion of the living subject, occasional spells of focal plane changes, drift in the field of view, and long image sequences are the principal roadblocks. The first step in such a registration process is the removal of translational and rotational motion. Next, a deformable registration can be performed. The focus of our study here is to remove the translation and/or rigid body motion that we refer to here as coarse alignment. The existing techniques for coarse alignment are unable to accommodate long sequences often consisting of periods of poor quality images (as quantified by a suitable perceptual measure). Many existing methods require the user to select an anchor image to which other images are registered. We propose a novel method for coarse image sequence alignment based on minimum weighted spanning trees (MISTICA) that overcomes these difficulties. The principal idea behind MISTICA is to reorder the images in shorter sequences, to demote nonconforming or poor quality images in the registration process, and to mitigate the error propagation. The anchor image is selected automatically making MISTICA completely automated. MISTICA is computationally efficient. It has a single tuning parameter that determines graph width, which can also be eliminated by the way of additional computation. MISTICA outperforms existing alignment methods when applied to microscopy image sequences of mouse arteries.

Intravital live cell triggered imaging system reveals monocyte patrolling and macrophage migration in atherosclerotic arteries.

Intravital multiphoton imaging of arteries is technically challenging because the artery expands with every heartbeat, causing severe motion artifacts. To study leukocyte activity in atherosclerosis, we developed the intravital live cell triggered imaging system (ILTIS). This system implements cardiac triggered acquisition as well as frame selection and image registration algorithms to produce stable movies of myeloid cell movement in atherosclerotic arteries in live mice. To minimize tissue damage, no mechanical stabilization is used and the artery is allowed to expand freely. ILTIS performs multicolor high frame-rate two-dimensional imaging and full-thickness three-dimensional imaging of beating arteries in live mice. The external carotid artery and its branches (superior thyroid and ascending pharyngeal arteries) were developed as a surgically accessible and reliable model of atherosclerosis. We use ILTIS to demonstrate Cx3cr1GFP monocytes patrolling the lumen of atherosclerotic arteries. Additionally, we developed a new reporter mouse (Apoe−/−Cx3cr1GFP/+Cd11cYFP) to image GFP+ and GFP+YFP + macrophages "dancing on the spot" and YFP+ macrophages migrating within intimal plaque. ILTIS will be helpful to answer pertinent open questions in the field, including monocyte recruitment and transmigration, macrophage and dendritic cell activity, and motion of other immune cells.

Cell tracking in microscopic video using matching and linking of bipartite graphs.

Automated visual tracking of cells from video microscopy has many important biomedical applications. In this paper, we track human monocyte cells in a fluorescent microscopic video using matching and linking of bipartite graphs. Tracking of cells over a pair of frames is modeled as a maximum cardinality minimum weight matching problem for a bipartite graph with a novel cost function. The tracking results are further refined using a rank-based filtering mechanism. Linking of cell trajectories over different frames are achieved through composition of bipartite matches. The proposed solution does not require any explicit motion model, is highly scalable, and, can effectively handle the entry and exit of cells. Our tracking accuracy of (97.97±0.94)% is superior than several existing methods [(95.66±2.39)%, (94.42±2.08)%, (81.22±5.75)%, (78.31±4.70)%] and is highly comparable (98.20±1.22)% to a recently published algorithm.

Object detection with DoG scale-space: a multiple kernel learning approach.

Difference of Gaussians (DoG) scale-space for an image is a significant way to generate features for object detection and classification. While applying DoG scale-space features for object detection/classification, we face two inevitable issues: dealing with high dimensional data and selecting/weighting of proper scales. The scale selection process is mostly ad-hoc to date. In this paper, we propose a multiple kernel learning (MKL) method for both DoG scale selection/weighting and dealing with high dimensional scale-space data. We design a novel shift invariant kernel function for DoG scale-space. To select only the useful scales in the DoG scale-space, a novel framework of MKL is also proposed. We utilize a 1-norm support vector machine (SVM) in the MKL optimization problem for sparse weighting of scales from DoG scale-space. The optimized data-dependent kernel accommodates only a few scales that are most discriminatory according to the large margin principle. With a 2-norm SVM this learned kernel is applied to a challenging detection problem in oil sand mining: to detect large lumps in oil sand videos. We tested our method on several challenging oil sand data sets. Our method yields encouraging results on these difficult-to-process images and compares favorably against other popular multiple kernel methods.

Quick detection of brain tumors and edemas: a bounding box method using symmetry.

A significant medical informatics task is indexing patient databases according to size, location, and other characteristics of brain tumors and edemas, possibly based on magnetic resonance (MR) imagery. This requires segmenting tumors and edemas within images from different MR modalities. To date, automated brain tumor or edema segmentation from MR modalities remains a challenging, computationally intensive task. In this paper, we propose a novel automated, fast, and approximate segmentation technique. The input is a patient study consisting of a set of MR slices, and its output is a subset of the slices that include axis-parallel boxes that circumscribe the tumors. Our approach is based on an unsupervised change detection method that searches for the most dissimilar region (axis-parallel bounding boxes) between the left and the right halves of a brain in an axial view MR slice. This change detection process uses a novel score function based on Bhattacharya coefficient computed with gray level intensity histograms. We prove that this score function admits a very fast (linear in image height and width) search to locate the bounding box. The average dice coefficients for localizing brain tumors and edemas, over ten patient studies, are 0.57 and 0.52, respectively, which significantly exceeds the scores for two other competitive region-based bounding box techniques.

Computation of fluid and particle motion from a time-sequenced image pair: a global outlier identification approach.

Fluid motion estimation from time-sequenced images is a significant image analysis task. Its application is widespread in experimental fluidics research and many related areas like biomedical engineering and atmospheric sciences. In this paper, we present a novel flow computation framework to estimate the flow velocity vectors from two consecutive image frames. In an energy minimization-based flow computation, we propose a novel data fidelity term, which: 1) can accommodate various measures, such as cross-correlation or sum of absolute or squared differences of pixel intensities between image patches; 2) has a global mechanism to control the adverse effect of outliers arising out of motion discontinuities, proximity of image borders; and 3) can go hand-in-hand with various spatial smoothness terms. Further, the proposed data term and related regularization schemes are both applicable to dense and sparse flow vector estimations. We validate these claims by numerical experiments on benchmark flow data sets.

Automatic segmentation of spinal cord MRI using symmetric boundary tracing.

We develop an adaptive active contour tracing algorithm for extraction of spinal cord from MRI that is fully automatic, unlike existing approaches that need manually chosen seeds. We can accurately extract the target spinal cord and construct the volume of interest to provide visual guidance for strategic rehabilitation surgery planning.

An affine transformation invariance approach to cell tracking.

Accurate and robust methods for automatically tracking rolling leukocytes facilitate inflammation research as leukocyte motion is a primary indicator of inflammatory response in the microvasculature. This paper reports on an affine transformation invariance approach we proposed to track rolling leukocyte in intravital microscopy image sequences. The method is based on the affine transformation invariance property, which enables the accommodation of linear affine transformations (translation, rotation, and/or scaling) of the target, and a particle filter that overcomes the effect of image clutter. In our data set of 50 sequences, we compared the new approach with an active contour tracking method and a Monte Carlo tracker. With the manual tracking result provided by an operator as the reference, the root mean square errors for the active contour tracking method, the Monte Carlo tracker and the affine transformation invariance approach were 0.95 microm, 0.79 microm and 0.74 microm, respectively, and the percentage of frames tracked were 72%, 75% and 89%, respectively. The affine transformation invariance approach demonstrated more robust (being able to successfully locate target leukocyte in more frames) and more accurate (lower root mean square error) tracking performance. We also separately studied the ability of the affine transformation invariance approach to track a dark target leukocyte and a bright target leukocyte by using the number of frames tracked as an evaluation measure. Dark target leukocyte possesses similar image intensity to the background, making it difficult to be located. In 20 sequences where the target leukocyte was dark, the affine transformation invariance approach tracked more frames in 18 sequences and fewer frames in 2 sequences when compared with the active contour tracking method. In comparison with the Monte Carlo tracker, the affine invariance method tracked more frames in 9 sequences, the same number of frames in 7 sequences and fewer frames in 4 sequences. In tracking a bright target leukocyte in 30 sequences, the affine transformation invariance approach demonstrated superior performance in 7 sequences and inferior performance in 1 sequence when compared with the active contour tracking method. It outperformed the Monte Carlo tracker in 15 sequences and underperformed in 1 sequence.

Inclusion filters: a class of self-dual connected operators.

In this paper, we define a connected operator that either fills or retains the holes of the connected sets depending on application-specific criteria that are increasing in the set theoretic sense. We refer to this class of connected operators as inclusion filters, which are shown to be increasing, idempotent, and self dual (gray-level inversion invariance). We demonstrate self duality for 8-adjacency on a discrete Cartesian grid. Inclusion filters are defined first for binary-valued images, and then the definition is extended to grayscale imagery. It is also shown that inclusion filters are levelings, a larger class of connected operators. Several important applications of inclusion filters are demonstrated-automatic segmentation of the lung cavities from magnetic resonance imagery, user interactive shape delineation in content-based image retrieval, registration of intravital microscopic video sequences, and detection and tracking of cells from these sequences. The numerical performance measures on 100-cell tracking experiments show that the use of inclusion filter improves the total number of frames successfully tracked by five times and provides a threefold reduction in the overall position error.

Data acceptance for automated leukocyte tracking through segmentation of spatiotemporal images.

A crucial task in inflammation research and inflammatory drug validation is leukocyte velocity data collection from microscopic video imagery. Since manual methods are bias-prone and extremely time consuming, automated tracking methods are required to compute cell velocities. However, an automated tracking method is of little practical use unless it is accompanied by a mechanism to validate the tracker output. In this paper, we propose a validation technique that accepts or rejects the output of automated tracking methods. The proposed method first generates a spatiotemporal image from the cell locations given by a tracking method; then, it segments the spatiotemporal image to detect the presence or absence of a leukocyte. For segmenting the spatiotemporal images, we employ an edge-direction sensitive nonlinear filter followed by an active contour based technique. The proposed nonlinear filter, the maximum absolute average directional derivative (MAADD), first computes the magnitude of the mean directional derivative over an oriented line segment and then chooses the maximum of all such values within a range of orientations of the line segment. The proposed active contour segmentation is obtained via growing contours controlled by a two-dimensional force field, which is constructed by imposing a Dirichlet boundary condition on the gradient vector flow (GVF) field equations. The performance of the proposed validation method is reported here for the outputs of three different tracking techniques: the method was successful in 97% of the trials using manual tracking, in 94% using correlation tracking and in 93% using active contour tracking.

Intravital leukocyte detection using the gradient inverse coefficient of variation.

The problem of identifying and counting rolling leukocytes within intravital microscopy is of both theoretical and practical interest. Currently, methods exist for tracking rolling leukocytes in vivo, but these methods rely on manual detection of the cells. In this paper we propose a technique for accurately detecting rolling leukocytes based on Bayesian classification. The classification depends on a feature score, the gradient inverse coefficient of variation (GICOV), which serves to discriminate rolling leukocytes from a cluttered environment. The leukocyte detection process consists of three sequential steps: the first step utilizes an ellipse matching algorithm to coarsely identify the leukocytes by finding the ellipses with a locally maximal GICOV. In the second step, starting from each of the ellipses found in the first step, a B-spline snake is evolved to refine the leukocytes boundaries by maximizing the associated GICOV score. The third and final step retains only the extracted contours that have a GICOV score above the analytically determined threshold. Experimental results using 327 rolling leukocytes were compared to those of human experts and currently used methods. The proposed GICOV method achieves 78.6% leukocyte detection accuracy with 13.1% false alarm rate.

Motion gradient vector flow: an external force for tracking rolling leukocytes with shape and size constrained active contours.

Recording rolling leukocyte velocities from intravital microscopic video imagery is a critical task in inflammation research and drug validation. Since manual tracking is excessively time consuming, an automated method is desired. This paper illustrates an active contour based automated tracking method, where we propose a novel external force to guide the active contour that takes the hemodynamic flow direction into account. The construction of the proposed force field, referred to as motion gradient vector flow (MGVF), is accomplished by minimizing an energy functional involving the motion direction, and the image gradient magnitude. The tracking experiments demonstrate that MGVF can be used to track both slow- and fast-rolling leukocytes, thus extending the capture range of previously designed cell tracking techniques.

Level set analysis for leukocyte detection and tracking.

We propose a cell detection and tracking solution using image-level sets computed via threshold decomposition. In contrast to existing methods where manual initialization is required to track individual cells, the proposed approach can automatically identify and track multiple cells by exploiting the shape and intensity characteristics of the cells. The capture of the cell boundary is considered as an evolution of a closed curve that maximizes image gradient along the curve enclosing a homogeneous region. An energy functional dependent upon the gradient magnitude along the cell boundary, the region homogeneity within the cell boundary and the spatial overlap of the detected cells is minimized using a variational approach. For tracking between frames, this energy functional is modified considering the spatial and shape consistency of a cell as it moves in the video sequence. The integrated energy functional complements shape-based segmentation with a spatial consistency based tracking technique. We demonstrate that an acceptable, expedient solution of the energy functional is possible through a search of the image-level lines: boundaries of connected components within the level sets obtained by threshold decomposition. The level set analysis can also capture multiple cells in a single frame rather than iteratively computing a single active contour for each individual cell. Results of cell detection using the energy functional approach and the level set approach are presented along with the associated processing time. Results of successful tracking of rolling leukocytes from a number of digital video sequences are reported and compared with the results from a correlation tracking scheme.

Merging parametric active contours within homogeneous image regions for MRI-based lung segmentation.

Inhaled hyperpolarized helium-3 (3He) gas is a new magnetic resonance (MR) contrast agent that is being used to study lung functionality. To evaluate the total lung ventilation from the hyperpolarized 3He MR images, it is necessary to segment the lung cavities. This is difficult to accomplish using only the hyperpolarized 3He MR images, so traditional proton (1H) MR images are frequently obtained concurrent with the hyperpolarized 3He MR examination. Segmentation of the lung cavities from traditional proton (1H) MRI is a necessary first step in the analysis of hyperpolarized 3He MR images. In this paper, we develop an active contour model that provides a smooth boundary and accurately captures the high curvature features of the lung cavities from the 1H MR images. This segmentation method is the first parametric active contour model that facilitates straightforward merging of multiple contours. The proposed method of merging computes an external force field that is based on the solution of partial differential equations with boundary condition defined by the initial positions of the evolving contours. A theoretical connection with fluid flow in porous media and the proposed force field is established. Then by using the properties of fluid flow we prove that the proposed method indeed achieves merging and the contours stop at the object boundary as well. Experimental results involving merging in synthetic images are provided. The segmentation technique has been employed in lung 1H MR imaging for segmenting the total lung air space. This technology plays a key role in computing the functional air space from MR images that use hyperpolarized 3He gas as a contrast agent.

Tracking leukocytes in vivo with shape and size constrained active contours.

Inflammatory disease is initiated by leukocytes (white blood cells) rolling along the inner surface lining of small blood vessels called postcapillary venules. Studying the number and velocity of rolling leukocytes is essential to understanding and successfully treating inflammatory diseases. Potential inhibitors of leukocyte recruitment can be screened by leukocyte rolling assays and successful inhibitors validated by intravital microscopy. In this paper, we present an active contour or snake-based technique to automatically track the movement of the leukocytes. The novelty of the proposed method lies in the energy functional that constrains the shape and size of the active contour. This paper introduces a significant enhancement over existing gradient-based snakes in the form of a modified gradient vector flow. Using the gradient vector flow, we can track leukocytes rolling at high speeds that are not amenable to tracking with the existing edge-based techniques. We also propose a new energy-based implicit sampling method of the points on the active contour that replaces the computationally expensive explicit method. To enhance the performance of this shape and size constrained snake model, we have coupled it with Kalman filter so that during coasting (when the leukocytes are completely occluded or obscured), the tracker may infer the location of the center of the leukocyte. Finally, we have compared the performance of the proposed snake tracker with that of the correlation and centroid-based trackers. The proposed snake tracker results in superior performance measures, such as reduced error in locating the leukocyte under tracking and improvements in the percentage of frames successfully tracked. For screening and drug validation, the tracker shows promise as an automated data collection tool.