Components-of-variance models for random-effects ROC analysis: the case of unequal variance structures across modalities.

RATIONALE AND OBJECTIVES: Several of the authors have previously published an analysis of multiple sources of uncertainty in the receiver operating characteristic (ROC) assessment and comparison of diagnostic modalities. The analysis assumed that the components of variance were the same for the modalities under comparison. The purpose of the present work is to obtain a generalization that does not require that assumption. MATERIALS AND METHODS: The generalization is achieved by splitting three of the six components of variance in the previous model into modality-dependent contributions. Two distinct formulations of this approach can be obtained from alternative choices of the three components to be split; however, a one-to-one relationship exists between the magnitudes of the components estimated from these two formulations. RESULTS: The method is applied to a study of multiple readers, with and without the aid of a computer-assist modality. performing the task of discriminating between benign and malignant clusters of microcalcifications. Analysis according to the first method of splitting shows large decreases in the reader and reader-by-case components of variance when the computer assist is used by the readers. Analysis in terms of the alternative splitting shows large decreases in the corresponding modality-interaction components. CONCLUSION: A solution to the problem of multivariate ROC analysis without the assumption of equal variance structure across modalities has been provided. Alternative formulations lead to consistent results related by a one-to-one mapping. A surprising result is that estimates of confidence intervals and numbers of cases and readers required for a specified confidence interval remain the same in the more general model as in the restricted model.

Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis.

RATIONALE AND OBJECTIVES: Solutions have previously been presented to the problem of estimating the components of variance in the general linear model used for multivariate receiver operating characteristic (ROC) analysis. The case where the variance components do not change across the modalities under comparison was first treated, followed by the case where they are permitted to change. No analysis of uncertainties in these estimates has been presented previously. MATERIALS AND METHODS: For the case where the variance components do not change across modalities, the "jackknife-after-bootstrap" resampling procedure can be used together with conventional linear propagation of variance to solve for the uncertainties in estimates of the components. For the case where the components are permitted to change across modalities, a slight elaboration of this procedure is presented. RESULTS: The approach was validated by Monte Carlo simulations, where uncertainties in estimates of the variance components calculated by the jackknife-after-bootstrap procedure were found to converge in the mean to the Monte Carlo results over many independent trials. The method is exemplified with data from a study of readers-with and without the aid of a computer-assist modality-given the task of discriminating benign from malignant masses in mammography. CONCLUSION: The present approach is relevant to a broad class of problems where estimates of multiple contributions to the variance observed in ROC assessment of diagnostic modalities are desired, in particular, for the assessment of multiple-reader studies of computer-aided diagnosis in radiology where the variance components may change across reading modalities (eg, unaided vs computer-aided reading).

Continuous versus categorical data for ROC analysis: some quantitative considerations.

RATIONALE AND OBJECTIVES: Several authors have encouraged the use of a quasi-continuous rating scale for data collection in receiver operating characteristic (ROC) curve analysis of diagnostic modalities, rather than rating scales based on five to seven ordinal categories or levels of suspicion. Although many investigators have gone over to this method, a discussion of the issues continues. The present work provides a quantitative analysis from the viewpoint of measurement science. MATERIALS AND METHODS: A simple model of the effect of data discretization or quantization on the measurement of the variance of noisy data was developed. Then Monte Carlo simulations of multiple-reader, multiple-case ROC experiments were performed and analyzed in terms of components-of-variance models to investigate the effect of data quantization in that more complex setting. RESULTS: For single-reader studies, discretization into five categories can reduce the precision of ROC measurements by a large amount. The effect may be attenuated in multireader studies. CONCLUSION: More precise measurements of diagnostic detection performance and thus more efficient use of resources are served by good measurement methods. These are promoted by the use of a quasi-continuous rating scale in ROC studies.

Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis.

RATIONALE AND OBJECTIVES: The purpose of this study was to develop an alternative approach to random-effects, receiver operating characteristic analysis inspired by a general formulation of components-of-variance models. The alternative approach is a higher-order generalization of the Dorfman, Berbaum, and Metz (DBM) approach that yields additional information on the variance structure of the problem. MATERIALS AND METHODS: Six population experiments were designed to determine the six variance components in the DBM model. For practical problems, in which only a finite set of readers and patients are available, six analogous bootstrap experiments may be substituted for the population experiments to estimate the variance components. Monte Carlo simulations were performed on the population experiments, and those results were compared with the corresponding multiple-bootstrap estimates and those obtained with the DBM approach. Confidence intervals on the difference of ROC parameters for competing diagnostic modalities were estimated, and corresponding comparisons were made. RESULTS: For mean values, the agreement of present estimates of variance structures with population results was excellent and, when suitably weighted and mixed, similar to or closer than that with the DBM method. For many variance structures, the confidence intervals in this study for the difference in ROC area between modalities were comparable to those with the DBM method. When reader variability was large, however, mean confidence intervals from this study were tighter than those with the DBM method and closer to population results. CONCLUSION: The jackknife approach of DBM provides a linear approximation to receiver-operating-characteristic statistics that are intrinsically nonlinear. The multiple-bootstrap technique of this study, however, provides a more general, nonparametric, maximum-likelihood approach. It also yields estimates of the variance structure previously unavailable.