New weighting methods when cases are only a subset of events in a nested case-control study.

Nested case control (NCC) is a sampling method widely used for developing and evaluating risk models with expensive biomarkers on large prospective cohort studies. In a typical NCC design, biomarker values are obtained on a subcohort, where cases consist of all the events (subjects who experience the event during the follow-up). However, when the number of events is not small, due to the cost and limited availability of biospecimen, one may select only a subset of events as cases. We refer to such a variation as the untypical NCC. Unfortunately, existing inverse probability weighted (IPW) estimators for the untypical NCC are biased, and they only focus on relative risk parameters under the proportional hazards (PH) model. In this manuscript, we propose new weighting methods that produce consistent IPW estimators for not only relative risk parameters but also several metrics that evaluate a risk model's predictive performance. We also provide the inference procedure via perturbation resampling, which captures all the variance and between-subject covariance induced by the sampling processes for both case and control selections. In addition, our methods are not limited to the PH model, and they can be applied to the time-specific generalized linear model. Under the typical NCC design, our new weights are equivalent to the weight proposed by Samuelsen; under the untypical NCC, the IPW estimators using our weights have smaller bias and variance than the existing methods. We will demonstrate this improved performance via both analytical and numerical investigations.

A relationship between the incremental values of area under the ROC curve and of area under the precision-recall curve.

BACKGROUND: Incremental value (IncV) evaluates the performance change between an existing risk model and a new model. Different IncV metrics do not always agree with each other. For example, compared with a prescribed-dose model, an ovarian-dose model for predicting acute ovarian failure has a slightly lower area under the receiver operating characteristic curve (AUC) but increases the area under the precision-recall curve (AP) by 48%. This phenomenon of disagreement is not uncommon, and can create confusion when assessing whether the added information improves the model prediction accuracy. METHODS: In this article, we examine the analytical connections and differences between the AUC IncV (ΔAUC) and AP IncV (ΔAP). We also compare the true values of these two IncV metrics in a numerical study. Additionally, as both are semi-proper scoring rules, we compare them with a strictly proper scoring rule: the IncV of the scaled Brier score (ΔsBrS) in the numerical study. RESULTS: We demonstrate that ΔAUC and ΔAP are both weighted averages of the changes (from the existing model to the new one) in separating the risk score distributions between events and non-events. However, ΔAP assigns heavier weights to the changes in higher-risk regions, whereas ΔAUC weights the changes equally. Due to this difference, the two IncV metrics can disagree, and the numerical study shows that their disagreement becomes more pronounced as the event rate decreases. In the numerical study, we also find that ΔAP has a wide range, from negative to positive, but the range of ΔAUC is much smaller. In addition, ΔAP and ΔsBrS are highly consistent, but ΔAUC is negatively correlated with ΔsBrS and ΔAP when the event rate is low. CONCLUSIONS: ΔAUC treats the wins and losses of a new risk model equally across different risk regions. When neither the existing or new model is the true model, this equality could attenuate a superior performance of the new model for a sub-region. In contrast, ΔAP accentuates the change in the prediction accuracy for higher-risk regions.

Gender-Specific Predictors of Recidivism Among DUI Offenders.

OBJECTIVE: Women are less likely than men to be arrested for driving under the influence (DUI) of alcohol or another drug, yet their proportion of DUI offenders is growing. Understanding how DUI recidivism risk varies for men and women is of practical utility for DUI assessment and intervention programs. The goals of the current study are to determine if there are different sets of predictors for men and women and whether gender-specific DUI recidivism risk scales perform better than a single recidivism scale for both men and women. METHOD: We rely on statistically driven techniques to develop gender-specific and total sample recidivism risk scales. We then test the ability of the scales to predict recidivism within 12 months among a large sample (N = 10,827, 22.3% female) of DUI offenders court mandated to a DUI intervention in Mississippi. RESULTS: Predictors of recidivism were drawn from measures of criminal history, substance use disorders, driving behaviors, and accidents. Gender-specific models yielded different sets of recidivism risk factors for men and women, with minimal overlap between the two. Male risk factors were criminal history and heavy alcohol consumption. For women, evidence of a substance use disorder was a unique predictor. Having a prior DUI arrest, driving behaviors, and a physical health consequence of alcohol or drug use (i.e., weight loss) were shared predictors for both sexes. CONCLUSIONS: Findings suggest that within broad categories of risk factors, the predictive validity of specific assessment items may vary by sex. Our methods represent progression toward more efficient prediction of DUI recidivists.

A threshold-free summary index of prediction accuracy for censored time to event data.

Prediction performance of a risk scoring system needs to be carefully assessed before its adoption in clinical practice. Clinical preventive care often uses risk scores to screen asymptomatic population. The primary clinical interest is to predict the risk of having an event by a prespecified future time t0 . Accuracy measures such as positive predictive values have been recommended for evaluating the predictive performance. However, for commonly used continuous or ordinal risk score systems, these measures require a subjective cutoff threshold value that dichotomizes the risk scores. The need for a cutoff value created barriers for practitioners and researchers. In this paper, we propose a threshold-free summary index of positive predictive values that accommodates time-dependent event status and competing risks. We develop a nonparametric estimator and provide an inference procedure for comparing this summary measure between 2 risk scores for censored time to event data. We conduct a simulation study to examine the finite-sample performance of the proposed estimation and inference procedures. Lastly, we illustrate the use of this measure on a real data example, comparing 2 risk score systems for predicting heart failure in childhood cancer survivors.

Robust Dynamic Risk Prediction with Longitudinal Studies.

Providing accurate and dynamic age-specific risk prediction is a crucial step in precision medicine. In this manuscript, we introduce an approach for estimating the τ-year age-specific absolute risk directly via a flexible varying coefficient model. The approach facilitates the utilization of predictors varying over an individual's lifetime. By using a nonparametric inverse probability weighted kernel estimating equation, the age-specific effects of risk factors are estimated without requiring the specification of the functional form. The approach allows borrowing information across individuals of similar ages, and therefore provides a practical solution for situations where the longitudinal information is only measured sparsely. We evaluate the performance of the proposed estimation and inference procedures with numerical studies, and make comparisons with existing methods in the literature. We illustrate the performance of our proposed approach by developing a dynamic prediction model using data from the Framingham Study.

Assessing incremental value of biomarkers with multi-phase nested case-control studies.

Accurate risk prediction models are needed to identify different risk groups for individualized prevention and treatment strategies. In the Nurses' Health Study, to examine the effects of several biomarkers and genetic markers on the risk of rheumatoid arthritis (RA), a three-phase nested case-control (NCC) design was conducted, in which two sequential NCC subcohorts were formed with one nested within the other, and one set of new markers measured on each of the subcohorts. One objective of the study is to evaluate clinical values of novel biomarkers in improving upon existing risk models because of potential cost associated with assaying biomarkers. In this paper, we develop robust statistical procedures for constructing risk prediction models for RA and estimating the incremental value (IncV) of new markers based on three-phase NCC studies. Our method also takes into account possible time-varying effects of biomarkers in risk modeling, which allows us to more robustly assess the biomarker utility and address the question of whether a marker is better suited for short-term or long-term risk prediction. The proposed procedures are shown to perform well in finite samples via simulation studies.

Assessment of biomarkers for risk prediction with nested case-control studies.

BACKGROUND: Accurate risk prediction plays a key role in disease prevention and disease management; emergence of new biomarkers may lead to an important question about how much improvement in prediction accuracy it would achieve by adding the new markers into the existing risk prediction tools. PURPOSE: In large prospective cohort studies, the standard full-cohort design, requiring marker measurement on the entire cohort, may be infeasible due to cost and low rate of the clinical condition of interest. To overcome such difficulties, nested case-control (NCC) studies provide cost-effective alternatives but bring about challenges in statistical analyses due to complex data sets generated. METHODS: To evaluate prognostic accuracy of a risk model, Cai and Zheng proposed a class of nonparametric inverse probability weighting (IPW) estimators for accuracy measures in the time-dependent receiver operating characteristic curve analysis. To accommodate a three-phase NCC design in Nurses' Health Study, we extend the double IPW estimators of Cai and Zheng to develop risk prediction models under time-dependent generalized linear models and evaluate the incremental values of new biomarkers and genetic markers. RESULTS: Our results suggest that aggregating the information from both the genetic markers and biomarkers substantially improves the accuracy for predicting 5-year and 10-year risks of rheumatoid arthritis. CONCLUSIONS: Our method provided robust procedures to evaluate the incremental value of new biomarkers allowing for complex sampling designs.

Subgroup specific incremental value of new markers for risk prediction.

In many clinical applications, understanding when measurement of new markers is necessary to provide added accuracy to existing prediction tools could lead to more cost effective disease management. Many statistical tools for evaluating the incremental value (IncV) of the novel markers over the routine clinical risk factors have been developed in recent years. However, most existing literature focuses primarily on global assessment. Since the IncVs of new markers often vary across subgroups, it would be of great interest to identify subgroups for which the new markers are most/least useful in improving risk prediction. In this paper we provide novel statistical procedures for systematically identifying potential traditional-marker based subgroups in whom it might be beneficial to apply a new model with measurements of both the novel and traditional markers. We consider various conditional time-dependent accuracy parameters for censored failure time outcome to assess the subgroup-specific IncVs. We provide non-parametric kernel-based estimation procedures to calculate the proposed parameters. Simultaneous interval estimation procedures are provided to account for sampling variation and adjust for multiple testing. Simulation studies suggest that our proposed procedures work well in finite samples. The proposed procedures are applied to the Framingham Offspring Study to examine the added value of an inflammation marker, C-reactive protein, on top of the traditional Framingham risk score for predicting 10-year risk of cardiovascular disease.

International multicenter psoriasis and psoriatic arthritis reliability trial for the assessment of skin, joints, nails, and dactylitis.

OBJECTIVE: Clinical trials in psoriasis and psoriatic arthritis (PsA) involve assessment of the skin and joints. This study aimed to determine whether assessment of the skin and joints in patients with PsA by rheumatologists and dermatologists is reproducible. METHODS: Ten rheumatologists and 9 dermatologists from 7 countries met for a combined physical examination exercise to assess 20 PsA patients (11 men, mean age 51 years, mean PsA duration 11 years). Each physician assessed 10 patients according to a modified Latin square design that enabled the assessment of patient, assessor, and order effect. Tender joint count (TJC), swollen joint count (SJC), dactylitis, physician's global assessment (PGA) of PsA disease activity (PGA-PsA), psoriasis body surface area (BSA), Psoriasis Area and Severity Index (PASI), Lattice System Physician's Global Assessment of psoriasis (LS-PGA), National Psoriasis Foundation Psoriasis Score (NPF-PS), modified Nail Psoriasis Severity Index (mNAPSI), number of fingernails with nail changes (NN), and PGA of psoriasis activity (PGA-Ps) were assessed. Variance components analyses were carried out to estimate the intraclass correlation coefficient (ICC), adjusted for the order of measurements. RESULTS: There is excellent agreement (ICC >/=0.80) on the mNAPSI, substantial agreement (0.6 >/= ICC < 0.80) on the TJC, PASI, and NN, moderate agreement (0.4 >/= ICC < 0.60) on the PGA-Ps, LS-PGA, NPF-PS, and BSA, and fair agreement (0.2 >/= ICC < 0.40) on the SJC, dactylitis, and PGA-PsA. The only measure that showed a significant difference between dermatologists and rheumatologists was dactylitis (P = 0.0005). CONCLUSION: There is substantial to excellent agreement on the TJC, PASI, NN, and mNAPSI among rheumatologists and dermatologists.