Improving CNNs classification with pathologist-based expertise: the renal cell carcinoma case study.

The prognosis of renal cell carcinoma (RCC) malignant neoplasms deeply relies on an accurate determination of the histological subtype, which currently involves the light microscopy visual analysis of histological slides, considering notably tumor architecture and cytology. RCC subtyping is therefore a time-consuming and tedious process, sometimes requiring expert review, with great impact on diagnosis, prognosis and treatment of RCC neoplasms. In this study, we investigate the automatic RCC subtyping classification of 91 patients, diagnosed with clear cell RCC, papillary RCC, chromophobe RCC, or renal oncocytoma, through deep learning based methodologies. We show how the classification performance of several state-of-the-art Convolutional Neural Networks (CNNs) are perfectible among the different RCC subtypes. Thus, we introduce a new classification model leveraging a combination of supervised deep learning models (specifically CNNs) and pathologist's expertise, giving birth to a hybrid approach that we termed ExpertDeepTree (ExpertDT). Our findings prove ExpertDT's superior capability in the RCC subtyping task, with respect to traditional CNNs, and suggest that introducing some expert-based knowledge into deep learning models may be a valuable solution for complex classification cases.

A machine learning framework for the quantification of experimental uveitis in murine OCT.

This paper presents methods for the detection and assessment of non-infectious uveitis, a leading cause of vision loss in working age adults. In the first part, we propose a classification model that can accurately predict the presence of uveitis and differentiate between different stages of the disease using optical coherence tomography (OCT) images. We utilize the Grad-CAM visualization technique to elucidate the decision-making process of the classifier and gain deeper insights into the results obtained. In the second part, we apply and compare three methods for the detection of detached particles in the retina that are indicative of uveitis. The first is a fully supervised detection method, the second is a marked point process (MPP) technique, and the third is a weakly supervised segmentation that produces per-pixel masks as output. The segmentation model is used as a backbone for a fully automated pipeline that can segment small particles of uveitis in two-dimensional (2-D) slices of the retina, reconstruct the volume, and produce centroids as points distribution in space. The number of particles in retinas is used to grade the disease, and point process analysis on centroids in three-dimensional (3-D) shows clustering patterns in the distribution of the particles on the retina.

Age-dependent changes in epidermal architecture explored using an automated image analysis algorithm on in vivo reflectance confocal microscopy images.

BACKGROUND: Reflectance confocal microscopy (RCM) allows for real-time in vivo visualization of the epidermis at the cellular level noninvasively. Parameters relating to tissue architecture can be extracted from RCM images, however, analysis of such images requires manual identification of cells to derive these parameters, which can be time-consuming and subject to human error, highlighting the need for an automated cell identification method. METHODS: First, the region-of-interest (ROI) containing cells needs to be identified, followed by the identification of individual cells within the ROI. To perform this task, we use successive applications of Sato and Gabor filters. The final step is post-processing improvement of cell detection and removal of size outliers. The proposed algorithm is evaluated on manually annotated real data. It is then applied to 5345 images to study the evolution of epidermal architecture in children and adults. The images were acquired on the volar forearm of healthy children (3 months to 10 years) and women (25-80 years), and on the volar forearm and cheek of women (40-80 years). Following the identification of cell locations, parameters such as cell area, cell perimeter, and cell density are calculated, as well as the probability distribution of the number of nearest neighbors per cell. The thicknesses of the Stratum Corneum and supra-papillary epidermis are also calculated using a hybrid deep-learning method. RESULTS: Epidermal keratinocytes are significantly larger (area and perimeter) in the granular layer than in the spinous layer and they get progressively larger with a child's age. Skin continues to mature dynamically during adulthood, as keratinocyte size continues to increase with age on both the cheeks and volar forearm, but the topology and cell aspect ratio remain unchanged across different epidermal layers, body sites, and age. Stratum Corneum and supra-papillary epidermis thicknesses increase with age, at a faster rate in children than in adults. CONCLUSIONS: The proposed methodology can be applied to large datasets to automate image analysis and the calculation of parameters relevant to skin physiology. These data validate the dynamic nature of skin maturation during childhood and skin aging in adulthood.

Automatic granular and spinous epidermal cell identification and analysis on in vivo reflectance confocal microscopy images using cell morphological features.

Significance: Reflectance confocal microscopy (RCM) allows for real-time in vivo visualization of the skin at the cellular level. The study of RCM images provides information on the structural properties of the epidermis. These may change in each layer of the epidermis, depending on the subject's age and the presence of certain dermatological conditions. Studying RCM images requires manual identification of cells to derive these properties, which is time consuming and subject to human error, highlighting the need for an automated cell identification method. Aim: We aim to design an automated pipeline for the analysis of the structure of the epidermis from RCM images of the Stratum granulosum and Stratum spinosum. Approach: We identified the region of interest containing the epidermal cells and the individual cells in the segmented tissue area using tubeness filters to highlight membranes. We used prior biological knowledge on cell size to process the resulting detected cells, removing cells that were too small and reapplying the used filters locally on detected regions that were too big to be considered a single cell. The proposed full image analysis pipeline (FIAP) was compared with machine learning-based approaches (cell cutter, different U-Net configurations, and loss functions). Results: All methods were evaluated both on simulated data (four images) and on manually annotated RCM data (seven images). Accuracy was measured using recall and precision metrics. Both accuracy metrics were higher in the proposed FIAP for both real ( precision = 0.720 ± 0.068 , recall = 0.850 ± 0.11 ) and synthetic images ( precision = 0.835 ± 0.067 , recall = 0.925 ± 0.012 ). The tested machine learning methods failed to identify and segment keratinocytes on RCM images with a satisfactory accuracy. Conclusions: We showed that automatic cell segmentation can be achieved using a pipeline based on membrane detection, with an accuracy that matches expert manual cell identification. To our knowledge, this is the first method based on membrane detection to study healthy skin using RCM images evaluated against manually identified cell positions.

A spatial statistical framework for the parametric study of fiber networks: Application to fibronectin deposition by normal and activated fibroblasts.

Due to the complex architectural diversity of biological networks, there is an increasing need to complement statistical analyses with a qualitative and local description of their spatial properties. One such network is the extracellular matrix (ECM), a biological scaffold for which changes in its spatial organization significantly impact tissue functions in health and disease. Quantifying variations in the fibrillar architecture of major ECM proteins should considerably advance our understanding of the link between tissue structure and function. Inspired by the analysis of functional magnetic resonance imaging (fMRI) images, we propose a novel statistical analysis approach embedded into a machine learning paradigm, to measure and detect local variations of meaningful ECM parameters. We show that parametric maps representing fiber length and pore directionality can be analyzed within the proposed framework to differentiate among various tissue states. The parametric maps are derived from graph-based representations that reflect the network architecture of fibronectin (FN) fibers in a normal, or disease-mimicking in vitro setting. Such tools can potentially lead to a better characterization of dynamic matrix networks within fibrotic tumor microenvironments and contribute to the development of better imaging modalities for monitoring their remodeling and normalization following therapeutic intervention.

Automating reflectance confocal microscopy image analysis for dermatological research: a review.

SIGNIFICANCE: Reflectance confocal microscopy (RCM) is a noninvasive, in vivo technology that offers near histopathological resolution at the cellular level. It is useful in the study of phenomena for which obtaining a biopsy is impractical or would cause unnecessary tissue damage and trauma to the patient. AIM: This review covers the use of RCM in the study of skin and the use of machine learning to automate information extraction. It has two goals: (1) an overview of information provided by RCM on skin structure and how it changes over time in response to stimuli and in disease and (2) an overview of machine learning approaches developed to automate the extraction of key morphological features from RCM images. APPROACH: A PubMed search was conducted with additional literature obtained from references lists. RESULTS: The application of RCM as an in vivo tool in dermatological research and the biologically relevant information derived from it are presented. Algorithms for image classification to epidermal layers, delineation of the dermal-epidermal junction, classification of skin lesions, and demarcation of individual cells within an image, all important factors in the makeup of the skin barrier, were reviewed. Application of image analysis methods in RCM is hindered by low image quality due to noise and/or poor contrast. Use of supervised machine learning is limited by time-consuming manual labeling of RCM images. CONCLUSIONS: RCM has great potential in the study of skin structures. The use of artificial intelligence could enable an easier, more reproducible, precise, and rigorous study of RCM images for the understanding of skin structures, skin barrier, and skin inflammation and lesions. Although several attempts have been made, further work is still needed to provide a definite gold standard and overcome issues related to image quality, limited labeled datasets, and lack of phenotype variability in available databases.

Detecting Stress Granules in Drosophila Neurons.

Stress granules (SGs) are cytoplasmic ribonucleoprotein condensates that dynamically and reversibly assemble in response to stress. They are thought to contribute to the adaptive stress response by storing translationally inactive mRNAs as well as signaling molecules. Recent work has shown that SG composition and properties depend on both stress and cell types, and that neurons exhibit a complex SG proteome and a strong vulnerability to mutations in SG proteins. Drosophila has emerged as a powerful genetically tractable organism where to study the physiological regulation and functions of SGs in normal and pathological contexts. In this chapter, we describe a protocol enabling quantitative analysis of SG properties in both larval and adult Drosophila CNS samples. In this protocol, fluorescently tagged SGs are induced upon acute ex vivo stress or chronic in vivo stress, imaged at high-resolution via confocal microscopy and detected automatically, using a dedicated software.

Fibronectin Extra Domains tune cellular responses and confer topographically distinct features to fibril networks.

Cellular fibronectin (FN; also known as FN1) variants harboring one or two alternatively spliced so-called extra domains (EDB and EDA) play a central bioregulatory role during development, repair processes and fibrosis. Yet, how the extra domains impact fibrillar assembly and function of the molecule remains unclear. Leveraging a unique biological toolset and image analysis pipeline for direct comparison of the variants, we demonstrate that the presence of one or both extra domains impacts FN assembly, function and physical properties of the matrix. When presented to FN-null fibroblasts, extra domain-containing variants differentially regulate pH homeostasis, survival and TGF-ß signaling by tuning the magnitude of cellular responses, rather than triggering independent molecular switches. Numerical analyses of fiber topologies highlight significant differences in variant-specific structural features and provide a first step for the development of a generative model of FN networks to unravel assembly mechanisms and investigate the physical and functional versatility of extracellular matrix landscapes.This article has an associated First Person interview with the first author of the paper.

Detecting and quantifying stress granules in tissues of multicellular organisms with the Obj.MPP analysis tool.

Stress granules (SGs) are macromolecular assemblies induced by stress and composed of proteins and mRNAs stalled in translation initiation. SGs play an important role in the response to stress and in the modulation of signaling pathways. Furthermore, these structures are related to the pathological ribonucleoprotein (RNP) aggregates found in neurodegenerative disease contexts, highlighting the need to understand how they are formed and recycled in normal and pathological contexts. Although genetically tractable multicellular organisms have been key in identifying modifiers of RNP aggregate toxicity, in vivo analysis of SG properties and regulation has lagged behind, largely due to the difficulty of detecting SG from images of intact tissues. Here, we describe the object detector software Obj.MPP and show how it overcomes the limits of classical object analyzers to extract the properties of SGs from wide-field and confocal images of Caenorhabditis elegans and Drosophila tissues, respectively. We demonstrate that Obj.MPP enables the identification of genes modulating the assembly of endogenous and pathological SGs, and thus that it will be useful in the context of future genetic screens and in vivo studies.

3D analysis of the whole subcutaneous adipose tissue reveals a complex spatial network of interconnected lobules with heterogeneous browning ability.

Adipose tissue, as the main energy storage organ and through its endocrine activity, is interconnected with all physiological functions. It plays a fundamental role in energy homeostasis and in the development of metabolic disorders. Up to now, this tissue has been analysed as a pool of different cell types with very little attention paid to the organization and putative partitioning of cells. Considering the absence of a complete picture of the intimate architecture of this large soft tissue, we developed a method that combines tissue clearing, acquisition of autofluorescence or lectin signals by confocal microscopy, segmentation procedures based on contrast enhancement, and a new semi-automatic image analysis process, allowing accurate and quantitative characterization of the whole 3D fat pad organization. This approach revealed the unexpected anatomic complexity of the murine subcutaneous fat pad. Although the classical picture of adipose tissue corresponds to a superposition of simple and small ellipsoidal lobules of adipose cells separated by mesenchymal spans, our results show that segmented lobules display complex 3D poly-lobular shapes. Despite differences in shape and size, the number of these poly-lobular subunits is similar from one fat pad to another. Finally, investigation of the relationships of these subunits between each other revealed a never-described organization in two clusters with distinct molecular signatures and specific vascular and sympathetic nerve densities correlating with different browning abilities. This innovative procedure reveals that subcutaneous adipose tissue exhibits a subtle functional heterogeneity with partitioned areas, and opens new perspectives towards understanding its functioning and plasticity.

A stochastic framework to model axon interactions within growing neuronal populations.

The confined and crowded environment of developing brains imposes spatial constraints on neuronal cells that have evolved individual and collective strategies to optimize their growth. These include organizing neurons into populations extending their axons to common target territories. How individual axons interact with each other within such populations to optimize innervation is currently unclear and difficult to analyze experimentally in vivo. Here, we developed a stochastic model of 3D axon growth that takes into account spatial environmental constraints, physical interactions between neighboring axons, and branch formation. This general, predictive and robust model, when fed with parameters estimated on real neurons from the Drosophila brain, enabled the study of the mechanistic principles underlying the growth of axonal populations. First, it provided a novel explanation for the diversity of growth and branching patterns observed in vivo within populations of genetically identical neurons. Second, it uncovered that axon branching could be a strategy optimizing the overall growth of axons competing with others in contexts of high axonal density. The flexibility of this framework will make it possible to investigate the rules underlying axon growth and regeneration in the context of various neuronal populations.

Multiple objects detection in biological images using a marked point process framework.

The marked point process framework has been successfully developed in the field of image analysis to detect a configuration of predefined objects. The goal of this paper is to show how it can be particularly applied to biological imagery. We present a simple model that shows how some of the challenges specific to biological data are well addressed by the methodology. We further describe an extension to this first model to address other challenges due, for example, to the shape variability in biological material. We finally show results that illustrate the MPP framework using the "simcep" algorithm for simulating populations of cells.

Morphological analysis and feature extraction of neurons from mouse cortices multiscale 3D microscopic images.

In this paper, we propose a framework to analyze the morphology of mouse neurons in the layer V of the cortex from 3D microscopic images. We are given 8 sets of images, each of which is composed of a 10x image showing the whole neurons, and a few (2 to 5) 40x images focusing on the somas. The framework consists in segmenting the neurons on both types of images to compute a set of specific morphological features, and in matching the neurons in the 40x images to their counterparts in the 10x images to combine the features we obtained, in a fully automatic fashion.

Discrete stochastic model for the generation of axonal trees.

In this work we propose a 2D discrete stochastic model for the simulation of axonal biogenesis. The model is defined by a third order Markov Chain. The model considers two main processes: the growth process that models the elongation and shape of the neurites and the bifurcation process that models the generation of branches. The growth process depends, among other variables, on the external attraction field generated by a chemoattractant molecule secreted by the target area. We propose an estimation scheme of the involved parameters from real fluorescent confocal microscopy images of single neurons within intact adult Drosophila fly brains. Both normal neurons and neurons in which certain genes were inactivated have been considered (two mutations). In total, 53 images (18 normal, 21 type 1 mutant and 14 type 2 mutant) were used. The model parameters allow us to describe pathological characteristics of the mutated populations.

Geometric feature extraction by a multimarked point process.

This paper presents a new stochastic marked point process for describing images in terms of a finite library of geometric objects. Image analysis based on conventional marked point processes has already produced convincing results but at the expense of parameter tuning, computing time, and model specificity. Our more general multimarked point process has simpler parametric setting, yields notably shorter computing times, and can be applied to a variety of applications. Both linear and areal primitives extracted from a library of geometric objects are matched to a given image using a probabilistic Gibbs model, and a Jump-Diffusion process is performed to search for the optimal object configuration. Experiments with remotely sensed images and natural textures show that the proposed approach has good potential. We conclude with a discussion about the insertion of more complex object interactions in the model by studying the compromise between model complexity and efficiency.

Structural approach for building reconstruction from a single DSM.

We present a new approach for building reconstruction from a single Digital Surface Model (DSM). It treats buildings as an assemblage of simple urban structures extracted from a library of 3D parametric blocks (like a LEGO set). First, the 2D-supports of the urban structures are extracted either interactively or automatically. Then, 3D-blocks are placed on the 2D-supports using a Gibbs model which controls both the block assemblage and the fitting to data. A Bayesian decision finds the optimal configuration of 3D-blocks using a Markov Chain Monte Carlo sampler associated with original proposition kernels. This method has been validated on multiple data set in a wide-resolution interval such as 0.7 m satellite and 0.1 m aerial DSMs, and provides 3D representations on complex buildings and dense urban areas with various levels of detail.

Ant colony optimization for image regularization based on a nonstationary Markov modeling.

Ant colony optimization (ACO) has been proposed as a promising tool for regularization in image classification. The algorithm is applied here in a different way than the classical transposition of the graph color affectation problem. The ants collect information through the image, from one pixel to the others. The choice of the path is a function of the pixel label, favoring paths within the same image segment. We show that this corresponds to an automatic adaptation of the neighborhood to the segment form, and that it outperforms the fixed-form neighborhood used in classical Markov random field regularization techniques. The performance of this new approach is illustrated on a simulated image and on actual remote sensing images.

Point processes for unsupervised line network extraction in remote sensing.

This paper addresses the problem of unsupervised extraction of line networks (for example, road or hydrographic networks) from remotely sensed images. We model the target line network by an object process, where the objects correspond to interacting line segments. The prior model, called "Quality Candy," is designed to exploit as fully as possible the topological properties of the network under consideration, while the radiometric properties of the network are modeled using a data term based on statistical tests. Two techniques are used to compute this term: one is more accurate, the other more efficient. A calibration technique is used to choose the model parameters. Optimization is done via simulated annealing using a Reversible Jump Markov Chain Monte Carlo (RJMCMC) algorithm. We accelerate convergence of the algorithm by using appropriate proposal kernels. The results obtained on satellite and aerial images are quantitatively evaluated with respect to manual extractions. A comparison with the results obtained using a previous model, called the "Candy" model, shows the interest of adding quality coefficients with respect to interactions in the prior density. The relevance of using an offline computation of the data potential is shown, in particular, when a proposal kernel based on this computation is added in the RJMCMC algorithm.

An object-based approach for detecting small brain lesions: application to Virchow-Robin spaces.

This paper is concerned with the detection of multiple small brain lesions from magnetic resonance imaging (MRI) data. A model based on the marked point process framework is designed to detect Virchow-Robin spaces (VRSs). These tubular shaped spaces are due to retraction of the brain parenchyma from its supplying arteries. VRS are described by simple geometrical objects that are introduced as small tubular structures. Their radiometric properties are embedded in a data term. A prior model includes interactions describing the clustering property of VRS. A Reversible Jump Markov Chain Monte Carlo algorithm (RJMCMC) optimizes the proposed model, obtained by multiplying the prior and the data model. Example results are shown on T1-weighted MRI datasets of elderly subjects.