An Empirical Study Into Annotator Agreement, Ground Truth Estimation, and Algorithm Evaluation.

Although agreement between the annotators who mark feature locations within images has been studied in the past from a statistical viewpoint, little work has attempted to quantify the extent to which this phenomenon affects the evaluation of foreground-background segmentation algorithms. Many researchers utilize ground truth (GT) in experimentation and more often than not this GT is derived from one annotator's opinion. How does the difference in opinion affects an algorithm's evaluation? A methodology is applied to four image-processing problems to quantify the interannotator variance and to offer insight into the mechanisms behind agreement and the use of GT. It is found that when detecting linear structures, annotator agreement is very low. The agreement in a structure's position can be partially explained through basic image properties. Automatic segmentation algorithms are compared with annotator agreement and it is found that there is a clear relation between the two. Several GT estimation methods are used to infer a number of algorithm performances. It is found that the rank of a detector is highly dependent upon the method used to form the GT, and that although STAPLE and LSML appear to represent the mean of the performance measured using individual annotations, when there are few annotations, or there is a large variance in them, these estimates tend to degrade. Furthermore, one of the most commonly adopted combination methods-consensus voting-accentuates more obvious features, resulting in an overestimation of performance. It is concluded that in some data sets, it is not possible to confidently infer an algorithm ranking when evaluating upon one GT.

Combat or surveillance? Evaluation of the heterogeneous inflammatory breast cancer microenvironment.

Evaluation of specific lymphocyte subsets is important in understanding the microenvironment in cancer and holds promise as a prognostic parameter in invasive breast cancer. To address this, we used digital image analysis to integrate cell abundance, distance metrics, neighbourhood relationships and sample heterogeneity into comprehensive assessment of immune infiltrates. Lymphocyte and macrophage subpopulations were detected by chromogenic duplex immunohistochemistry for CD3/perforin and CD68/CD163 in samples of invasive breast cancer. The analysis workflow combined commercial and open-source software modules. We confirmed the accuracy of automated detection of cells with lymphoid morphology [concordance correlation coefficient (CCC), 0.92 for CD3(+) -T lymphocytes], whereas variable morphology limited automated classification of macrophages as distinct cellular objects (CCC, 0.43 for object-based detection; 0.79 for pixel-based area analysis). Using a supervised learning algorithm that clustered image areas according to lymphocyte abundance, grouping behaviour and distance to tumour cells, we identified recurrent infiltration patterns reflecting different grades of direct interaction between tumour and immune effector cells. The approach provided comprehensive visual and statistical assessment of the inflammatory tumour microenvironment and allowed quantitative estimation of heterogeneous immune cell distribution. Cases with dense lymphocytic infiltrates (8/33) contained up to 65% of areas in which observed distances between tumour and immune cells suggested a low chance of direct contact, indicating the presence of regions where tumour cells might be protected from immune attack. In contrast, cases with moderate (11/33) or low (14/33) lymphocyte density occasionally comprised areas of focally intense interaction, likely not to be captured by conventional scores. Our approach improves the conventional evaluation of immune cell density scores by translating objective distance metrics into reproducible, largely observer-independent interaction patterns.

Discovering significant evolution patterns from satellite image time series.

Satellite Image Time Series (SITS) provide us with precious information on land cover evolution. By studying these series of images we can both understand the changes of specific areas and discover global phenomena that spread over larger areas. Changes that can occur throughout the sensing time can spread over very long periods and may have different start time and end time depending on the location, which complicates the mining and the analysis of series of images. This work focuses on frequent sequential pattern mining (FSPM) methods, since this family of methods fits the above-mentioned issues. This family of methods consists of finding the most frequent evolution behaviors, and is actually able to extract long-term changes as well as short term ones, whenever the change may start and end. However, applying FSPM methods to SITS implies confronting two main challenges, related to the characteristics of SITS and the domain's constraints. First, satellite images associate multiple measures with a single pixel (the radiometric levels of different wavelengths corresponding to infra-red, red, etc.), which makes the search space multi-dimensional and thus requires specific mining algorithms. Furthermore, the non evolving regions, which are the vast majority and overwhelm the evolving ones, challenge the discovery of these patterns. We propose a SITS mining framework that enables discovery of these patterns despite these constraints and characteristics. Our proposal is inspired from FSPM and provides a relevant visualization principle. Experiments carried out on 35 images sensed over 20 years show the proposed approach makes it possible to extract relevant evolution behaviors.