A fiducial-based confidence interval for the linear combination of multinomial probabilities.

Across a broad set of applications, system outcomes may be summarized as probabilities in confusion or contingency tables. In settings with more than two outcomes (e.g., stages of cancer), these outcomes represent multinomial experiments. Measures to summarize system performance have been presented as linear combinations of the resulting multinomial probabilities. Statistical inference on the linear combination of multinomial probabilities has been focused on large-sample and parametric settings and not small-sample settings. Such inference is valuable, however, especially in settings such as those resulting from pilot or low-cost studies. To address this gap, we leverage the fiducial approach to derive confidence intervals around the linear combination of multinomial parameters with desirable frequentist properties. One of the original arguments against the fiducial approach was its inability to extend to multiparameter settings. Therefore, the great novelty of this work is both the derived interval and the logical framework for applying the fiducial approach in multiparameter settings. Through simulation, we demonstrate that the proposed method maintains a minimum coverage of 1 - α $1 - \alpha$ , unlike the bootstrap and large-sample methods, at comparable interval lengths. Finally, we illustrate its use in a medical problem of selecting classifiers for diagnosing chronic allograph nephropathy in postkidney transplant patients.

A nonparametric fiducial interval for the Youden index in multi-state diagnostic settings.

The Youden index is a commonly employed metric to characterize the performance of a diagnostic test at its optimal point. For tests with three or more outcome classes, the Youden index has been extended; however, there are limited methods to compute a confidence interval (CI) about its value. Often, outcome classes are assumed to be normally distributed, which facilitates computational formulas for the CI bounds; however, many scenarios exist for which these assumptions cannot be made. In addition, many of these existing CI methods do not work well for small sample sizes. We propose a method to compute a nonparametric interval about the Youden index utilizing the fiducial argument. This fiducial interval ensures that CI coverage is met regardless of sample size, underlying distributional assumptions, or use of a complex classifier for diagnosis. Two alternate fiducial intervals are also considered. A simulation was conducted, which demonstrates the coverage and interval length for the proposed methods. Comparisons were made using no distributional assumptions on the outcome classes and for when outcomes were assumed to be normally distributed. In general, coverage probability was consistently met, and interval length was reasonable. The proposed fiducial method was also demonstrated in data examining biomarkers in subjects to predict diagnostic stages ranging from normal kidney function to chronic allograph nephropathy. Published 2015. This article is a U.S. Government work and is in the public domain in the USA.

Confidence intervals around Bayes Cost in multi-state diagnostic settings to estimate optimal performance.

A critical feature of diagnostic testing is correctly classifying subjects based upon specified thresholds of some measure. The commonly employed Youden index determines a test's optimal thresholds by maximizing the correct classification rates for a diagnostic scenario. An alternative to the Youden index is the cost function, Bayes Cost (BC). BC determines a test's optimal setting by minimizing the sum of all misclassification rates from the test. Unlike the Youden index, BC can consider a priori costs of all the diagnostic outcomes including class specific misclassifications regardless of the number of classes. Delta method approximate confidence intervals around BC are derived under the assumption of normally distributed classes as a means for quantifying a test's performance and comparing classifiers at their optimal settings in a multi-state diagnostic framework. A simulation study is conducted to demonstrate the performance of the derived confidence intervals that are found to perform well, especially for sample sizes of 50 or larger in each diagnostic class. Finally, the proposed methods are applied to a four-class breast tissue classification problem, where four possible discriminatory features are compared under varying decision cost structures. Using the confidence intervals around BC, the best feature for classification is selected, and the optimal thresholds and their 95% confidence intervals are determined.