Visualising and quantifying the usefulness of new predictors stratified by outcome class: The U-smile method.

Binary classification methods encompass various algorithms to categorize data points into two distinct classes. Binary prediction, in contrast, estimates the likelihood of a binary event occurring. We introduce a novel graphical and quantitative approach, the U-smile method, for assessing prediction improvement stratified by binary outcome class. The U-smile method utilizes a smile-like plot and novel coefficients to measure the relative and absolute change in prediction compared with the reference method. The likelihood-ratio test was used to assess the significance of the change in prediction. Logistic regression models using the Heart Disease dataset and generated random variables were employed to validate the U-smile method. The receiver operating characteristic (ROC) curve was used to compare the results of the U-smile method. The likelihood-ratio test demonstrated that the proposed coefficients consistently generated smile-shaped U-smile plots for the most informative predictors. The U-smile plot proved more effective than the ROC curve in comparing the effects of adding new predictors to the reference method. It effectively highlighted differences in model performance for both non-events and events. Visual analysis of the U-smile plots provided an immediate impression of the usefulness of different predictors at a glance. The U-smile method can guide the selection of the most valuable predictors. It can also be helpful in applications beyond prediction.

Cohen's Kappa Coefficient as a Measure to Assess Classification Improvement following the Addition of a New Marker to a Regression Model.

The need to search for new measures describing the classification of a logistic regression model stems from the difficulty in searching for previously unknown factors that predict the occurrence of a disease. A classification quality assessment can be performed by testing the change in the area under the receiver operating characteristic curve (AUC). Another approach is to use the Net Reclassification Improvement (NRI), which is based on a comparison between the predicted risk, determined on the basis of the basic model, and the predicted risk that comes from the model enriched with an additional factor. In this paper, we draw attention to Cohen's Kappa coefficient, which examines the actual agreement in the correction of a random agreement. We proposed to extend this coefficient so that it may be used to detect the quality of a logistic regression model reclassification. The results provided by Kappa's reclassification were compared with the results obtained using NRI. The random variables' distribution attached to the model on the classification change, measured by NRI, Kappa, and AUC, was presented. A simulation study was conducted on the basis of a cohort containing 3971 Poles obtained during the implementation of a lower limb atherosclerosis prevention program.