Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Possibilities of Using Multi-b-value Diffusion Magnetic Resonance Imaging for Classification of Brain Lesions.

In contrast to conventional diffusion magnetic resonance imaging (MRI), multi-b-value diffusion MRI methods are able to separate the signal from free water, pseudo-diffusion, and non-Gaussian components of water molecule diffusion. These approaches can then be utilised in so-called intravoxel incoherent motion imaging and diffusion kurtosis imaging. Various parameters provided by these methods can describe additional characteristics of the tissue microstructure and potentially help in the diagnosis and classification of various pathological processes. In this review, we present the basic principles and methods of analysing multi-b-value diffusion imaging data and specifically focus on the known possibilities for its use in the diagnosis of brain lesions. We also suggest possible directions for further research.

The Cell Tracking Challenge: 10 years of objective benchmarking.

The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here, we present a significant number of improvements introduced in the challenge since our 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver standard reference corpus based on the most competitive results, which will be of particular interest for data-hungry deep learning-based strategies. Furthermore, we present the up-to-date cell segmentation and tracking leaderboards, an in-depth analysis of the relationship between the performance of the state-of-the-art methods and the properties of the datasets and annotations, and two novel, insightful studies about the generalizability and the reusability of top-performing methods. These studies provide critical practical conclusions for both developers and users of traditional and machine learning-based cell segmentation and tracking algorithms.

Lactic Acidosis Interferes With Toxicity of Perifosine to Colorectal Cancer Spheroids: Multimodal Imaging Analysis.

Colorectal cancer (CRC) is a disease with constantly increasing incidence and high mortality. The treatment efficacy could be curtailed by drug resistance resulting from poor drug penetration into tumor tissue and the tumor-specific microenvironment, such as hypoxia and acidosis. Furthermore, CRC tumors can be exposed to different pH depending on the position in the intestinal tract. CRC tumors often share upregulation of the Akt signaling pathway. In this study, we investigated the role of external pH in control of cytotoxicity of perifosine, the Akt signaling pathway inhibitor, to CRC cells using 2D and 3D tumor models. In 3D settings, we employed an innovative strategy for simultaneous detection of spatial drug distribution and biological markers of proliferation/apoptosis using a combination of mass spectrometry imaging and immunohistochemistry. In 3D conditions, low and heterogeneous penetration of perifosine into the inner parts of the spheroids was observed. The depth of penetration depended on the treatment duration but not on the external pH. However, pH alteration in the tumor microenvironment affected the distribution of proliferation- and apoptosis-specific markers in the perifosine-treated spheroid. Accurate co-registration of perifosine distribution and biological response in the same spheroid section revealed dynamic changes in apoptotic and proliferative markers occurring not only in the perifosine-exposed cells, but also in the perifosine-free regions. Cytotoxicity of perifosine to both 2D and 3D cultures decreased in an acidic environment below pH 6.7. External pH affects cytotoxicity of the other Akt inhibitor, MK-2206, in a similar way. Our innovative approach for accurate determination of drug efficiency in 3D tumor tissue revealed that cytotoxicity of Akt inhibitors to CRC cells is strongly dependent on pH of the tumor microenvironment. Therefore, the effect of pH should be considered during the design and pre-clinical/clinical testing of the Akt-targeted cancer therapy.

BIAS: Transparent reporting of biomedical image analysis challenges.

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

Quantitative Assessment of Anti-Cancer Drug Efficacy From Coregistered Mass Spectrometry and Fluorescence Microscopy Images of Multicellular Tumor Spheroids.

Spheroids-three-dimensional aggregates of cells grown from a cancer cell line-represent a model of living tissue for chemotherapy investigation. Distribution of chemotherapeutics in spheroid sections was determined using the matrix-assisted laser desorption/ionization mass spectrometry imaging (MALDI MSI). Proliferating or apoptotic cells were immunohistochemically labeled and visualized by laser scanning confocal fluorescence microscopy (LSCM). Drug efficacy was evaluated by comparing coregistered MALDI MSI and LSCM data of drug-treated spheroids with LSCM only data of untreated control spheroids. We developed a fiducial-based workflow for coregistration of low-resolution MALDI MS with high-resolution LSCM images. To allow comparison of drug and cell distribution between the drug-treated and untreated spheroids of different shapes or diameters, we introduced a common diffusion-related coordinate, the distance from the spheroid boundary. In a procedure referred to as "peeling", we correlated average drug distribution at a certain distance with the average reduction in the affected cells between the untreated and the treated spheroids. This novel approach makes it possible to differentiate between peripheral cells that died due to therapy and the innermost cells which died naturally. Two novel algorithms-for MALDI MS image denoising and for weighting of MALDI MSI and LSCM data by the presence of cell nuclei-are also presented.

Drug Penetration Analysis in 3D Cell Cultures Using Fiducial-Based Semiautomatic Coregistration of MALDI MSI and Immunofluorescence Images.

In this paper, we present an easy-to-follow procedure for the analysis of tissue sections from 3D cell cultures (spheroids) by matrix-assisted laser desorption/ionization mass spectrometry imaging (MALDI MSI) and laser scanning confocal microscopy (LSCM). MALDI MSI was chosen to detect the distribution of the drug of interest, while fluorescence immunohistochemistry (IHC) followed by LSCM was used to localize the cells featuring specific markers of viability, proliferation, apoptosis and metastasis. The overlay of the mass spectrometry (MS) and IHC spheroid images, typically without any morphological features, required fiducial-based coregistration. The MALDI MSI protocol was optimized in terms of fiducial composition and antigen epitope preservation to allow MALDI MSI to be performed and directly followed by IHC analysis on exactly the same spheroid section. Once MS and IHC images were coregistered, the quantification of the MS and IHC signals was performed by an algorithm evaluating signal intensities along equidistant layers from the spheroid boundary to its center. This accurate colocalization of MS and IHC signals showed limited penetration of the clinically tested drug perifosine into spheroids during a 24 h period, revealing the fraction of proliferating and promigratory/proinvasive cells present in the perifosine-free areas, decrease of their abundance in the perifosine-positive regions, and distinguishing between apoptosis resulting from hypoxia/nutrient deprivation and drug exposure.

CytoPacq: a web-interface for simulating multi-dimensional cell imaging.

MOTIVATION: Objective assessment of bioimage analysis methods is an essential step towards understanding their robustness and parameter sensitivity, calling for the availability of heterogeneous bioimage datasets accompanied by their reference annotations. Because manual annotations are known to be arduous, highly subjective and barely reproducible, numerous simulators have emerged over past decades, generating synthetic bioimage datasets complemented with inherent reference annotations. However, the installation and configuration of these tools generally constitutes a barrier to their widespread use. RESULTS: We present a modern, modular web-interface, CytoPacq, to facilitate the generation of synthetic benchmark datasets relevant for multi-dimensional cell imaging. CytoPacq poses a user-friendly graphical interface with contextual tooltips and currently allows a comfortable access to various cell simulation systems of fluorescence microscopy, which have already been recognized and used by the scientific community, in a straightforward and self-contained form. AVAILABILITY AND IMPLEMENTATION: CytoPacq is a publicly available online service running at https://cbia.fi.muni.cz/simulator. More information about it as well as examples of generated bioimage datasets are available directly through the web-interface. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Author Correction: Why rankings of biomedical image analysis competitions should be interpreted with care.

In the original version of this Article the values in the rightmost column of Table 1 were inadvertently shifted relative to the other columns. This has now been corrected in the PDF and HTML versions of the Article.

Why rankings of biomedical image analysis competitions should be interpreted with care.

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

An objective comparison of cell-tracking algorithms.

We present a combined report on the results of three editions of the Cell Tracking Challenge, an ongoing initiative aimed at promoting the development and objective evaluation of cell segmentation and tracking algorithms. With 21 participating algorithms and a data repository consisting of 13 data sets from various microscopy modalities, the challenge displays today's state-of-the-art methodology in the field. We analyzed the challenge results using performance measures for segmentation and tracking that rank all participating methods. We also analyzed the performance of all of the algorithms in terms of biological measures and practical usability. Although some methods scored high in all technical aspects, none obtained fully correct solutions. We found that methods that either take prior information into account using learning strategies or analyze cells in a global spatiotemporal video context performed better than other methods under the segmentation and tracking scenarios included in the challenge.

Image-based surrogate biomarkers for molecular subtypes of colorectal cancer.

MOTIVATION: Whole genome expression profiling of large cohorts of different types of cancer led to the identification of distinct molecular subcategories (subtypes) that may partially explain the observed inter-tumoral heterogeneity. This is also the case of colorectal cancer (CRC) where several such categorizations have been proposed. Despite recent developments, the problem of subtype definition and recognition remains open, one of the causes being the intrinsic heterogeneity of each tumor, which is difficult to estimate from gene expression profiles. However, one of the observations of these studies indicates that there may be links between the dominant tumor morphology characteristics and the molecular subtypes. Benefiting from a large collection of CRC samples, comprising both gene expression and histopathology images, we investigated the possibility of building image-based classifiers able to predict the molecular subtypes. We employed deep convolutional neural networks for extracting local descriptors which were then used for constructing a dictionary-based representation of each tumor sample. A set of support vector machine classifiers were trained to solve different binary decision problems, their combined outputs being used to predict one of the five molecular subtypes. RESULTS: A hierarchical decomposition of the multi-class problem was obtained with an overall accuracy of 0.84 (95%CI=0.79-0.88). The predictions from the image-based classifier showed significant prognostic value similar to their molecular counterparts. CONTACT: popovici@iba.muni.cz. AVAILABILITY AND IMPLEMENTATION: Source code used for the image analysis is freely available from https://github.com/higex/qpath . SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Characterization of three-dimensional cancer cell migration in mixed collagen-Matrigel scaffolds using microfluidics and image analysis.

Microfluidic devices are becoming mainstream tools to recapitulate in vitro the behavior of cells and tissues. In this study, we use microfluidic devices filled with hydrogels of mixed collagen-Matrigel composition to study the migration of lung cancer cells under different cancer invasion microenvironments. We present the design of the microfluidic device, characterize the hydrogels morphologically and mechanically and use quantitative image analysis to measure the migration of H1299 lung adenocarcinoma cancer cells in different experimental conditions. Our results show the plasticity of lung cancer cell migration, which turns from mesenchymal in collagen only matrices, to lobopodial in collagen-Matrigel matrices that approximate the interface between a disrupted basement membrane and the underlying connective tissue. Our quantification of migration speed confirms a biphasic role of Matrigel. At low concentration, Matrigel facilitates migration, most probably by providing a supportive and growth factor retaining environment. At high concentration, Matrigel slows down migration, possibly due excessive attachment. Finally, we show that antibody-based integrin blockade promotes a change in migration phenotype from mesenchymal or lobopodial to amoeboid and analyze the effect of this change in migration dynamics, in regards to the structure of the matrix. In summary, we describe and characterize a robust microfluidic platform and a set of software tools that can be used to study lung cancer cell migration under different microenvironments and experimental conditions. This platform could be used in future studies, thus benefitting from the advantages introduced by microfluidic devices: precise control of the environment, excellent optical properties, parallelization for high throughput studies and efficient use of therapeutic drugs.

Virtual cell imaging: A review on simulation methods employed in image cytometry.

The simulations of cells and microscope images thereof have been used to facilitate the development, selection, and validation of image analysis algorithms employed in cytometry as well as for modeling and understanding cell structure and dynamics beyond what is visible in the eyepiece. The simulation approaches vary from simple parametric models of specific cell components-especially shapes of cells and cell nuclei-to learning-based synthesis and multi-stage simulation models for complex scenes that simultaneously visualize multiple object types and incorporate various properties of the imaged objects and laws of image formation. This review covers advances in artificial digital cell generation at scales ranging from particles up to tissue synthesis and microscope image simulation methods, provides examples of the use of simulated images for various purposes ranging from subcellular object detection to cell tracking, and discusses how such simulators have been validated. Finally, the future possibilities and limitations of simulation-based validation are considered. © 2016 International Society for Advancement of Cytometry.

Challenges and Benchmarks in Bioimage Analysis.

Similar to the medical imaging community, the bioimaging community has recently realized the need to benchmark various image analysis methods to compare their performance and assess their suitability for specific applications. Challenges sponsored by prestigious conferences have proven to be an effective means of encouraging benchmarking and new algorithm development for a particular type of image data. Bioimage analysis challenges have recently complemented medical image analysis challenges, especially in the case of the International Symposium on Biomedical Imaging (ISBI). This review summarizes recent progress in this respect and describes the general process of designing a bioimage analysis benchmark or challenge, including the proper selection of datasets and evaluation metrics. It also presents examples of specific target applications and biological research tasks that have benefited from these challenges with respect to the performance of automatic image analysis methods that are crucial for the given task. Finally, available benchmarks and challenges in terms of common features, possible classification and implications drawn from the results are analysed.

Cell Tracking Accuracy Measurement Based on Comparison of Acyclic Oriented Graphs.

Tracking motile cells in time-lapse series is challenging and is required in many biomedical applications. Cell tracks can be mathematically represented as acyclic oriented graphs. Their vertices describe the spatio-temporal locations of individual cells, whereas the edges represent temporal relationships between them. Such a representation maintains the knowledge of all important cellular events within a captured field of view, such as migration, division, death, and transit through the field of view. The increasing number of cell tracking algorithms calls for comparison of their performance. However, the lack of a standardized cell tracking accuracy measure makes the comparison impracticable. This paper defines and evaluates an accuracy measure for objective and systematic benchmarking of cell tracking algorithms. The measure assumes the existence of a ground-truth reference, and assesses how difficult it is to transform a computed graph into the reference one. The difficulty is measured as a weighted sum of the lowest number of graph operations, such as split, delete, and add a vertex and delete, add, and alter the semantics of an edge, needed to make the graphs identical. The measure behavior is extensively analyzed based on the tracking results provided by the participants of the first Cell Tracking Challenge hosted by the 2013 IEEE International Symposium on Biomedical Imaging. We demonstrate the robustness and stability of the measure against small changes in the choice of weights for diverse cell tracking algorithms and fluorescence microscopy datasets. As the measure penalizes all possible errors in the tracking results and is easy to compute, it may especially help developers and analysts to tune their algorithms according to their needs.

Performance and sensitivity evaluation of 3D spot detection methods in confocal microscopy.

Reliable 3D detection of diffraction-limited spots in fluorescence microscopy images is an important task in subcellular observation. Generally, fluorescence microscopy images are heavily degraded by noise and non-specifically stained background, making reliable detection a challenging task. In this work, we have studied the performance and parameter sensitivity of eight recent methods for 3D spot detection. The study is based on both 3D synthetic image data and 3D real confocal microscopy images. The synthetic images were generated using a simulator modeling the complete imaging setup, including the optical path as well as the image acquisition process. We studied the detection performance and parameter sensitivity under different noise levels and under the influence of uneven background signal. To evaluate the parameter sensitivity, we propose a novel measure based on the gradient magnitude of the F1 score. We measured the success rate of the individual methods for different types of the image data and found that the type of image degradation is an important factor. Using the F1 score and the newly proposed sensitivity measure, we found that the parameter sensitivity is not necessarily proportional to the success rate of a method. This also provided an explanation why the best performing method for synthetic data was outperformed by other methods when applied to the real microscopy images. On the basis of the results obtained, we conclude with the recommendation of the HDome method for data with relatively low variations in quality, or the Sorokin method for image sets in which the quality varies more. We also provide alternative recommendations for high-quality images, and for situations in which detailed parameter tuning might be deemed expensive.