Reference standard for the evaluation of automatic segmentation algorithms: Quantification of inter observer variability of manual delineation of prostate contour on MRI.

PURPOSE: The purpose of this study was to investigate the relationship between inter-reader variability in manual prostate contour segmentation on magnetic resonance imaging (MRI) examinations and determine the optimal number of readers required to establish a reliable reference standard. MATERIALS AND METHODS: Seven radiologists with various experiences independently performed manual segmentation of the prostate contour (whole-gland [WG] and transition zone [TZ]) on 40 prostate MRI examinations obtained in 40 patients. Inter-reader variability in prostate contour delineations was estimated using standard metrics (Dice similarity coefficient [DSC], Hausdorff distance and volume-based metrics). The impact of the number of readers (from two to seven) on segmentation variability was assessed using pairwise metrics (consistency) and metrics with respect to a reference segmentation (conformity), obtained either with majority voting or simultaneous truth and performance level estimation (STAPLE) algorithm. RESULTS: The average segmentation DSC for two readers in pairwise comparison was 0.919 for WG and 0.876 for TZ. Variability decreased with the number of readers: the interquartile ranges of the DSC were 0.076 (WG) / 0.021 (TZ) for configurations with two readers, 0.005 (WG) / 0.012 (TZ) for configurations with three readers, and 0.002 (WG) / 0.0037 (TZ) for configurations with six readers. The interquartile range decreased slightly faster between two and three readers than between three and six readers. When using consensus methods, variability often reached its minimum with three readers (with STAPLE, DSC = 0.96 [range: 0.945-0.971] for WG and DSC = 0.94 [range: 0.912-0.957] for TZ, and interquartile range was minimal for configurations with three readers. CONCLUSION: The number of readers affects the inter-reader variability, in terms of inter-reader consistency and conformity to a reference. Variability is minimal for three readers, or three readers represent a tipping point in the variability evolution, with both pairwise-based metrics or metrics with respect to a reference. Accordingly, three readers may represent an optimal number to determine references for artificial intelligence applications.

Prostate volume prediction on MRI: tools, accuracy and variability.

OBJECTIVE: A reliable estimation of prostate volume (PV) is essential to prostate cancer management. The objective of our multi-rater study was to compare intra- and inter-rater variability of PV from manual planimetry and ellipsoid formulas. METHODS: Forty treatment-naive patients who underwent prostate MRI were selected from a local database. PV and corresponding PSA density (PSAd) were estimated on 3D T2-weighted MRI (3 T) by 7 independent radiologists using the traditional ellipsoid formula (TEF), the newer biproximate ellipsoid formula (BPEF), and the manual planimetry method (MPM) used as ground truth. Intra- and inter-rater variability was calculated using the mixed model-based intraclass correlation coefficient (ICC). RESULTS: Mean volumes were 67.00 (± 36.61), 66.07 (± 35.03), and 64.77 (± 38.27) cm3 with the TEF, BPEF, and MPM methods, respectively. Both TEF and BPEF overestimated PV relative to MPM, with the former presenting significant differences (+ 1.91 cm3, IQ = [- 0.33 cm3, 5.07 cm3], p val = 0.03). Both intra- (ICC > 0.90) and inter-rater (ICC > 0.90) reproducibility were excellent. MPM had the highest inter-rater reproducibility (ICC = 0.999). Inter-rater PV variation led to discrepancies in classification according to the clinical criterion of PSAd > 0.15 ng/mL for 2 patients (5%), 7 patients (17.5%), and 9 patients (22.5%) when using MPM, TEF, and BPEF, respectively. CONCLUSION: PV measurements using ellipsoid formulas and MPM are highly reproducible. MPM is a robust method for PV assessment and PSAd calculation, with the lowest variability. TEF showed a high degree of concordance with MPM but a slight overestimation of PV. Precise anatomic landmarks as defined with the BPEF led to a more accurate PV estimation, but also to a higher variability. KEY POINTS: • Manual planimetry used for prostate volume estimation is robust and reproducible, with the lowest variability between readers. • Ellipsoid formulas are accurate and reproducible but with higher variability between readers. • The traditional ellipsoid formula tends to overestimate prostate volume.

Challenge of prostate MRI segmentation on T2-weighted images: inter-observer variability and impact of prostate morphology.

BACKGROUND: Accurate prostate zonal segmentation on magnetic resonance images (MRI) is a critical prerequisite for automated prostate cancer detection. We aimed to assess the variability of manual prostate zonal segmentation by radiologists on T2-weighted (T2W) images, and to study factors that may influence it. METHODS: Seven radiologists of varying levels of experience segmented the whole prostate gland (WG) and the transition zone (TZ) on 40 axial T2W prostate MRI images (3D T2W images for all patients, and both 3D and 2D images for a subgroup of 12 patients). Segmentation variabilities were evaluated based on: anatomical and morphological variation of the prostate (volume, retro-urethral lobe, intensity contrast between zones, presence of a PI-RADS ≥ 3 lesion), variation in image acquisition (3D vs 2D T2W images), and reader's experience. Several metrics including Dice Score (DSC) and Hausdorff Distance were used to evaluate differences, with both a pairwise and a consensus (STAPLE reference) comparison. RESULTS: DSC was 0.92 (± 0.02) and 0.94 (± 0.03) for WG, 0.88 (± 0.05) and 0.91 (± 0.05) for TZ respectively with pairwise comparison and consensus reference. Variability was significantly (p < 0.05) lower for the mid-gland (DSC 0.95 (± 0.02)), higher for the apex (0.90 (± 0.06)) and the base (0.87 (± 0.06)), and higher for smaller prostates (p < 0.001) and when contrast between zones was low (p < 0.05). Impact of the other studied factors was non-significant. CONCLUSIONS: Variability is higher in the extreme parts of the gland, is influenced by changes in prostate morphology (volume, zone intensity ratio), and is relatively unaffected by the radiologist's level of expertise.