Closed-Form Solution of the Unit Normal Loss Integral in 2 Dimensions, with Application in Value-of-Information Analysis.

HIGHLIGHTS: The unit normal loss integral (UNLI) is widely used in decision analysis and risk modeling, including in the computation of various value-of-information metrics, but its closed-form solution is only applicable to comparisons of 2 strategies.We derive a closed-form solution for 2-dimensional UNLI, extending the applicability of the UNLI to 3-strategy comparisons.Such closed-form computation takes only a fraction of a second and is free from simulation errors that affect the hitherto available methods.In addition to the relevance in 3-strategy model-based and data-driven decision analyses, a particular application is in risk prediction modeling, where the net benefit of a classifier should always be compared with 2 default strategies of treating none and treating all.

Uncertainty and the Value of Information in Risk Prediction Modeling.

BACKGROUND: Because of the finite size of the development sample, predicted probabilities from a risk prediction model are inevitably uncertain. We apply value-of-information methodology to evaluate the decision-theoretic implications of prediction uncertainty. METHODS: Adopting a Bayesian perspective, we extend the definition of the expected value of perfect information (EVPI) from decision analysis to net benefit calculations in risk prediction. In the context of model development, EVPI is the expected gain in net benefit by using the correct predictions as opposed to predictions from a proposed model. We suggest bootstrap methods for sampling from the posterior distribution of predictions for EVPI calculation using Monte Carlo simulations. We used subsets of data of various sizes from a clinical trial for predicting mortality after myocardial infarction to show how EVPI changes with sample size. RESULTS: With a sample size of 1000 and at the prespecified threshold of 2% on predicted risks, the gains in net benefit using the proposed and the correct models were 0.0006 and 0.0011, respectively, resulting in an EVPI of 0.0005 and a relative EVPI of 87%. EVPI was zero only at unrealistically high thresholds (>85%). As expected, EVPI declined with larger samples. We summarize an algorithm for incorporating EVPI calculations into the commonly used bootstrap method for optimism correction. CONCLUSION: The development EVPI can be used to decide whether a model can advance to validation, whether it should be abandoned, or whether a larger development sample is needed. Value-of-information methods can be applied to explore decision-theoretic consequences of uncertainty in risk prediction and can complement inferential methods in predictive analytics. R code for implementing this method is provided.