A novel deep learning-based 3D cell segmentation framework for future image-based disease detection.

Cell segmentation plays a crucial role in understanding, diagnosing, and treating diseases. Despite the recent success of deep learning-based cell segmentation methods, it remains challenging to accurately segment densely packed cells in 3D cell membrane images. Existing approaches also require fine-tuning multiple manually selected hyperparameters on the new datasets. We develop a deep learning-based 3D cell segmentation pipeline, 3DCellSeg, to address these challenges. Compared to the existing methods, our approach carries the following novelties: (1) a robust two-stage pipeline, requiring only one hyperparameter; (2) a light-weight deep convolutional neural network (3DCellSegNet) to efficiently output voxel-wise masks; (3) a custom loss function (3DCellSeg Loss) to tackle the clumped cell problem; and (4) an efficient touching area-based clustering algorithm (TASCAN) to separate 3D cells from the foreground masks. Cell segmentation experiments conducted on four different cell datasets show that 3DCellSeg outperforms the baseline models on the ATAS (plant), HMS (animal), and LRP (plant) datasets with an overall accuracy of 95.6%, 76.4%, and 74.7%, respectively, while achieving an accuracy comparable to the baselines on the Ovules (plant) dataset with an overall accuracy of 82.2%. Ablation studies show that the individual improvements in accuracy is attributable to 3DCellSegNet, 3DCellSeg Loss, and TASCAN, with the 3DCellSeg demonstrating robustness across different datasets and cell shapes. Our results suggest that 3DCellSeg can serve a powerful biomedical and clinical tool, such as histo-pathological image analysis, for cancer diagnosis and grading.

Size control of the inner ear via hydraulic feedback.

Animals make organs of precise size, shape, and symmetry but how developing embryos do this is largely unknown. Here, we combine quantitative imaging, physical theory, and physiological measurement of hydrostatic pressure and fluid transport in zebrafish to study size control of the developing inner ear. We find that fluid accumulation creates hydrostatic pressure in the lumen leading to stress in the epithelium and expansion of the otic vesicle. Pressure, in turn, inhibits fluid transport into the lumen. This negative feedback loop between pressure and transport allows the otic vesicle to change growth rate to control natural or experimentally-induced size variation. Spatiotemporal patterning of contractility modulates pressure-driven strain for regional tissue thinning. Our work connects molecular-driven mechanisms, such as osmotic pressure driven strain and actomyosin tension, to the regulation of tissue morphogenesis via hydraulic feedback to ensure robust control of organ size. Editorial note: This article has been through an editorial process in which the authors decide how to respond to the issues raised during peer review. The Reviewing Editor's assessment is that all the issues have been addressed (see decision letter).

Cortical column and whole-brain imaging with molecular contrast and nanoscale resolution.

Optical and electron microscopy have made tremendous inroads toward understanding the complexity of the brain. However, optical microscopy offers insufficient resolution to reveal subcellular details, and electron microscopy lacks the throughput and molecular contrast to visualize specific molecular constituents over millimeter-scale or larger dimensions. We combined expansion microscopy and lattice light-sheet microscopy to image the nanoscale spatial relationships between proteins across the thickness of the mouse cortex or the entire Drosophila brain. These included synaptic proteins at dendritic spines, myelination along axons, and presynaptic densities at dopaminergic neurons in every fly brain region. The technology should enable statistically rich, large-scale studies of neural development, sexual dimorphism, degree of stereotypy, and structural correlations to behavior or neural activity, all with molecular contrast.

Lamellar projections in the endolymphatic sac act as a relief valve to regulate inner ear pressure.

The inner ear is a fluid-filled closed-epithelial structure whose function requires maintenance of an internal hydrostatic pressure and fluid composition. The endolymphatic sac (ES) is a dead-end epithelial tube connected to the inner ear whose function is unclear. ES defects can cause distended ear tissue, a pathology often seen in hearing and balance disorders. Using live imaging of zebrafish larvae, we reveal that the ES undergoes cycles of slow pressure-driven inflation followed by rapid deflation. Absence of these cycles in lmx1bb mutants leads to distended ear tissue. Using serial-section electron microscopy and adaptive optics lattice light-sheet microscopy, we find a pressure relief valve in the ES comprised of partially separated apical junctions and dynamic overlapping basal lamellae that separate under pressure to release fluid. We propose that this lmx1-dependent pressure relief valve is required to maintain fluid homeostasis in the inner ear and other fluid-filled cavities.

Feedback between tissue packing and neurogenesis in the zebrafish neural tube.

Balancing the rate of differentiation and proliferation in developing tissues is essential to produce organs of robust size and composition. Although many molecular regulators have been established, how these connect to physical and geometrical aspects of tissue architecture is poorly understood. Here, using high-resolution timelapse imaging, we find that changes to cell geometry associated with dense tissue packing play a significant role in regulating differentiation rate in the zebrafish neural tube. Specifically, progenitors that are displaced away from the apical surface due to crowding, tend to differentiate in a Notch-dependent manner. Using simulations we show that interplay between progenitor density, cell shape and changes in differentiation rate could naturally result in negative-feedback control on progenitor cell number. Given these results, we suggest a model whereby differentiation rate is regulated by density dependent effects on cell geometry to: (1) correct variability in cell number; and (2) balance the rates of proliferation and differentiation over development to 'fill' the available space.

Observing the cell in its native state: Imaging subcellular dynamics in multicellular organisms.

True physiological imaging of subcellular dynamics requires studying cells within their parent organisms, where all the environmental cues that drive gene expression, and hence the phenotypes that we actually observe, are present. A complete understanding also requires volumetric imaging of the cell and its surroundings at high spatiotemporal resolution, without inducing undue stress on either. We combined lattice light-sheet microscopy with adaptive optics to achieve, across large multicellular volumes, noninvasive aberration-free imaging of subcellular processes, including endocytosis, organelle remodeling during mitosis, and the migration of axons, immune cells, and metastatic cancer cells in vivo. The technology reveals the phenotypic diversity within cells across different organisms and developmental stages and may offer insights into how cells harness their intrinsic variability to adapt to different physiological environments.

Recovery of shape and size in a developing organ pair.

BACKGROUND: Paired organs in animals are largely bilaterally symmetric despite inherent noise in most biological processes. How is precise organ shape and size achieved during development despite this noise? Examining paired organ development is a challenge because it requires repeated quantification of two structures in parallel within living embryos. Here we combine bilateral quantification of morphology through time with asymmetric perturbations to study regulation of organ shape, size, and symmetry in developing organ pairs. RESULTS: We present quantitative live imaging tools to measure the shape and size of the developing inner ears on both the left and right side simultaneously over time. By quantifying variation between the left and right inner ear (intrinsic noise) and between different individuals (extrinsic noise), we find that initial variability decreases over time in normal development to achieve symmetry. Early asymmetry is increased by environmental stress, but symmetry is still recovered over subsequent developmental time. Using multiple unilateral perturbations including Fgf signaling and ultraviolet light, we find that growth can be adjusted to compensate for a range of initial size and shape differences. CONCLUSIONS: We propose that symmetry in developmental systems does not emerge through precise deterministic bilateral development, but rather through feedback mechanisms that adjust morphogenesis rates to account for variation. Developmental Dynamics 246:451-465, 2016. © 2017 Wiley Periodicals, Inc.

Membrane dynamics of dividing cells imaged by lattice light-sheet microscopy.

Membrane remodeling is an essential part of transferring components to and from the cell surface and membrane-bound organelles and for changes in cell shape, which are particularly critical during cell division. Earlier analyses, based on classical optical live-cell imaging and mostly restricted by technical necessity to the attached bottom surface, showed persistent formation of endocytic clathrin pits and vesicles during mitosis. Taking advantage of the resolution, speed, and noninvasive illumination of the newly developed lattice light-sheet fluorescence microscope, we reexamined their assembly dynamics over the entire cell surface and found that clathrin pits form at a lower rate during late mitosis. Full-cell imaging measurements of cell surface area and volume throughout the cell cycle of single cells in culture and in zebrafish embryos showed that the total surface increased rapidly during the transition from telophase to cytokinesis, whereas cell volume increased slightly in metaphase and was relatively constant during cytokinesis. These applications demonstrate the advantage of lattice light-sheet microscopy and enable a new standard for imaging membrane dynamics in single cells and multicellular assemblies.

Improved Long-Term Imaging of Embryos with Genetically Encoded α-Bungarotoxin.

Rapid advances in microscopy and genetic labeling strategies have created new opportunities for time-lapse imaging of embryonic development. However, methods for immobilizing embryos for long periods while maintaining normal development have changed little. In zebrafish, current immobilization techniques rely on the anesthetic tricaine. Unfortunately, prolonged tricaine treatment at concentrations high enough to immobilize the embryo produces undesirable side effects on development. We evaluate three alternative immobilization strategies: combinatorial soaking in tricaine and isoeugenol, injection of α-bungarotoxin protein, and injection of α-bungarotoxin mRNA. We find evidence for co-operation between tricaine and isoeugenol to give immobility with improved health. However, even in combination these anesthetics negatively affect long-term development. α-bungarotoxin is a small protein from snake venom that irreversibly binds and inactivates acetylcholine receptors. We find that α-bungarotoxin either as purified protein from snakes or endogenously expressed in zebrafish from a codon-optimized synthetic gene can immobilize embryos for extended periods of time with few health effects or developmental delays. Using α-bungarotoxin mRNA injection we obtain complete movies of zebrafish embryogenesis from the 1-cell stage to 3 days post fertilization, with normal health and no twitching. These results demonstrate that endogenously expressed α-bungarotoxin provides unprecedented immobility and health for time-lapse microscopy.

Interplay of cell shape and division orientation promotes robust morphogenesis of developing epithelia.

Epithelial cells acquire functionally important shapes (e.g., squamous, cuboidal, columnar) during development. Here, we combine theory, quantitative imaging, and perturbations to analyze how tissue geometry, cell divisions, and mechanics interact to shape the presumptive enveloping layer (pre-EVL) on the zebrafish embryonic surface. We find that, under geometrical constraints, pre-EVL flattening is regulated by surface cell number changes following differentially oriented cell divisions. The division pattern is, in turn, determined by the cell shape distribution, which forms under geometrical constraints by cell-cell mechanical coupling. An integrated mathematical model of this shape-division feedback loop recapitulates empirical observations. Surprisingly, the model predicts that cell shape is robust to changes of tissue surface area, cell volume, and cell number, which we confirm in vivo. Further simulations and perturbations suggest the parameter linking cell shape and division orientation contributes to epithelial diversity. Together, our work identifies an evolvable design logic that enables robust cell-level regulation of tissue-level development.

Specified neural progenitors sort to form sharp domains after noisy Shh signaling.

Sharply delineated domains of cell types arise in developing tissues under instruction of inductive signal (morphogen) gradients, which specify distinct cell fates at different signal levels. The translation of a morphogen gradient into discrete spatial domains relies on precise signal responses at stable cell positions. However, cells in developing tissues undergoing morphogenesis and proliferation often experience complex movements, which may affect their morphogen exposure, specification, and positioning. How is a clear pattern achieved with cells moving around? Using in toto imaging of the zebrafish neural tube, we analyzed specification patterns and movement trajectories of neural progenitors. We found that specified progenitors of different fates are spatially mixed following heterogeneous Sonic Hedgehog signaling responses. Cell sorting then rearranges them into sharply bordered domains. Ectopically induced motor neuron progenitors also robustly sort to correct locations. Our results reveal that cell sorting acts to correct imprecision of spatial patterning by noisy inductive signals.

An efficient, scalable, and adaptable framework for solving generic systems of level-set PDEs.

In the last decade, level-set methods have been actively developed for applications in image registration, segmentation, tracking, and reconstruction. However, the development of a wide variety of level-set PDEs and their numerical discretization schemes, coupled with hybrid combinations of PDE terms, stopping criteria, and reinitialization strategies, has created a software logistics problem. In the absence of an integrative design, current toolkits support only specific types of level-set implementations which restrict future algorithm development since extensions require significant code duplication and effort. In the new NIH/NLM Insight Toolkit (ITK) v4 architecture, we implemented a level-set software design that is flexible to different numerical (continuous, discrete, and sparse) and grid representations (point, mesh, and image-based). Given that a generic PDE is a summation of different terms, we used a set of linked containers to which level-set terms can be added or deleted at any point in the evolution process. This container-based approach allows the user to explore and customize terms in the level-set equation at compile-time in a flexible manner. The framework is optimized so that repeated computations of common intensity functions (e.g., gradient and Hessians) across multiple terms is eliminated. The framework further enables the evolution of multiple level-sets for multi-object segmentation and processing of large datasets. For doing so, we restrict level-set domains to subsets of the image domain and use multithreading strategies to process groups of subdomains or level-set functions. Users can also select from a variety of reinitialization policies and stopping criteria. Finally, we developed a visualization framework that shows the evolution of a level-set in real-time to help guide algorithm development and parameter optimization. We demonstrate the power of our new framework using confocal microscopy images of cells in a developing zebrafish embryo.

ACME: automated cell morphology extractor for comprehensive reconstruction of cell membranes.

The quantification of cell shape, cell migration, and cell rearrangements is important for addressing classical questions in developmental biology such as patterning and tissue morphogenesis. Time-lapse microscopic imaging of transgenic embryos expressing fluorescent reporters is the method of choice for tracking morphogenetic changes and establishing cell lineages and fate maps in vivo. However, the manual steps involved in curating thousands of putative cell segmentations have been a major bottleneck in the application of these technologies especially for cell membranes. Segmentation of cell membranes while more difficult than nuclear segmentation is necessary for quantifying the relations between changes in cell morphology and morphogenesis. We present a novel and fully automated method to first reconstruct membrane signals and then segment out cells from 3D membrane images even in dense tissues. The approach has three stages: 1) detection of local membrane planes, 2) voting to fill structural gaps, and 3) region segmentation. We demonstrate the superior performance of the algorithms quantitatively on time-lapse confocal and two-photon images of zebrafish neuroectoderm and paraxial mesoderm by comparing its results with those derived from human inspection. We also compared with synthetic microscopic images generated by simulating the process of imaging with fluorescent reporters under varying conditions of noise. Both the over-segmentation and under-segmentation percentages of our method are around 5%. The volume overlap of individual cells, compared to expert manual segmentation, is consistently over 84%. By using our software (ACME) to study somite formation, we were able to segment touching cells with high accuracy and reliably quantify changes in morphogenetic parameters such as cell shape and size, and the arrangement of epithelial and mesenchymal cells. Our software has been developed and tested on Windows, Mac, and Linux platforms and is available publicly under an open source BSD license (https://github.com/krm15/ACME).

Streaming level set algorithm for 3D segmentation of confocal microscopy images.

We present a high performance variant of the popular geodesic active contours which are used for splitting cell clusters in microscopy images. Previously, we implemented a linear pipelined version that incorporates as many cues as possible into developing a suitable level-set speed function so that an evolving contour exactly segments a cell/nuclei blob. We use image gradients, distance maps, multiple channel information and a shape model to drive the evolution. We also developed a dedicated seeding strategy that uses the spatial coherency of the data to generate an over complete set of seeds along with a quality metric which is further used to sort out which seed should be used for a given cell. However, the computational performance of any level-set methodology is quite poor when applied to thousands of 3D data-sets each containing thousands of cells. Those data-sets are common in confocal microscopy. In this work, we explore methods to stream the algorithm in shared memory, multi-core environments. By partitioning the input and output using spatial data structures we insure the spatial coherency needed by our seeding algorithm as well as improve drastically the speed without memory overhead. Our results show speed-ups up to a factor of six.

Robust 3D reconstruction and identification of dendritic spines from optical microscopy imaging.

In neurobiology, the 3D reconstruction of neurons followed by the identification of dendritic spines is essential for studying neuronal morphology, function and biophysical properties. Most existing methods suffer from problems of low reliability, poor accuracy and require much user interaction. In this paper, we present a method to reconstruct dendrites using a surface representation of the neuron. The skeleton of the dendrite is extracted by a procedure based on the medial geodesic function that is robust and topology preserving, and it is used to accurately identify spines. The sensitivity of the algorithm on the various parameters is explored in detail and the method is shown to be robust.

Tensor classification of N-point correlation function features for histology tissue segmentation.

In this paper, we utilize the N-point correlation functions (N-pcfs) to construct an appropriate feature space for achieving tissue segmentation in histology-stained microscopic images. The N-pcfs estimate microstructural constituent packing densities and their spatial distribution in a tissue sample. We represent the multi-phase properties estimated by the N-pcfs in a tensor structure. Using a variant of higher-order singular value decomposition (HOSVD) algorithm, we realize a robust classifier that provides a multi-linear description of the tensor feature space. Validated results of the segmentation are presented in a case-study that focuses on understanding the genetic phenotyping differences in mouse placentae.

Detection of spatially correlated objects in 3D images using appearance models and coupled active contours.

We consider the problem of segmenting 3D images that contain a dense collection of spatially correlated objects, such as fluorescent labeled cells in tissue. Our approach involves an initial modeling phase followed by a data-fitting segmentation phase. In the first phase, cell shape (membrane bound) is modeled implicitly using a parametric distribution of correlation function estimates. The nucleus is modeled for its shape as well as image intensity distribution inspired from the physics of its image formation. In the second phase, we solve the segmentation problem using a variational level-set strategy with coupled active contours to minimize a novel energy functional. We demonstrate the utility of our approach on multispectral fluorescence microscopy images.

An imaging workflow for characterizing phenotypical change in large histological mouse model datasets.

MOTIVATION: This paper presents a workflow designed to quantitatively characterize the 3D structural attributes of macroscopic tissue specimens acquired at a micron level resolution using light microscopy. The specific application is a study of the morphological change in a mouse placenta induced by knocking out the retinoblastoma gene. RESULT: This workflow includes four major components: (i) serial section image acquisition, (ii) image preprocessing, (iii) image analysis involving 2D pair-wise registration, 2D segmentation and 3D reconstruction, and (iv) visualization and quantification of phenotyping parameters. Several new algorithms have been developed within each workflow component. The results confirm the hypotheses that (i) the volume of labyrinth tissue decreases in mutant mice with the retinoblastoma (Rb) gene knockout and (ii) there is more interdigitation at the surface between the labyrinth and spongiotrophoblast tissues in mutant placenta. Additional confidence stem from agreement in the 3D visualization and the quantitative results generated. AVAILABILITY: The source code is available upon request.

Reconstruction of cellular biological structures from optical microscopy data.

Developments in optical microscopy imaging have generated large high-resolution data sets that have spurred medical researchers to conduct investigations into mechanisms of disease, including cancer at cellular and subcellular levels. The work reported here demonstrates that a suitable methodology can be conceived that isolates modality-dependent effects from the larger segmentation task and that 3D reconstructions can be cognizant of shapes as evident in the available 2D planar images. In the current realization, a method based on active geodesic contours is first deployed to counter the ambiguity that exists in separating overlapping cells on the image plane. Later, another segmentation effort based on a variant of Voronoi tessellations improves the delineation of the cell boundaries using a Bayesian formulation. In the next stage, the cells are interpolated across the third dimension thereby mitigating the poor structural correlation that exists in that dimension. We deploy our methods on three separate data sets obtained from light, confocal, and phase-contrast microscopy and validate the results appropriately.

Detection and visualization of surface-pockets to enable phenotyping studies.

In this paper, we propose a technique for detecting pockets on a surface-of-interest. A sequence of propagating fronts converging to the target surface is used as the basis for inspection. We compute a correspondence function between the initial and the target surface. This leads to a natural definition of the local feature size measured as the evolution distance between mapped points. Surface pockets are then extracted as salient clusters embedded in the feature space. The level-set initialization also determines the scale-space of the extracted pockets. Results are presented on a case-study in which the focus is to chronicle the phenotyping differences in genetically modified mouse placenta. Our results are validated based on manually verified ground-truth.