Development and validation of an international preoperative risk assessment model for postoperative delirium.

BACKGROUND: Postoperative delirium (POD) is a frequent complication in older adults, characterised by disturbances in attention, awareness and cognition, and associated with prolonged hospitalisation, poor functional recovery, cognitive decline, long-term dementia and increased mortality. Early identification of patients at risk of POD can considerably aid prevention. METHODS: We have developed a preoperative POD risk prediction algorithm using data from eight studies identified during a systematic review and providing individual-level data. Ten-fold cross-validation was used for predictor selection and internal validation of the final penalised logistic regression model. The external validation used data from university hospitals in Switzerland and Germany. RESULTS: Development included 2,250 surgical (excluding cardiac and intracranial) patients 60 years of age or older, 444 of whom developed POD. The final model included age, body mass index, American Society of Anaesthesiologists (ASA) score, history of delirium, cognitive impairment, medications, optional C-reactive protein (CRP), surgical risk and whether the operation is a laparotomy/thoracotomy. At internal validation, the algorithm had an AUC of 0.80 (95% CI: 0.77-0.82) with CRP and 0.79 (95% CI: 0.77-0.82) without CRP. The external validation consisted of 359 patients, 87 of whom developed POD. The external validation yielded an AUC of 0.74 (95% CI: 0.68-0.80). CONCLUSIONS: The algorithm is named PIPRA (Pre-Interventional Preventive Risk Assessment), has European conformity (ce) certification, is available at http://pipra.ch/ and is accepted for clinical use. It can be used to optimise patient care and prioritise interventions for vulnerable patients and presents an effective way to implement POD prevention strategies in clinical practice.

External validation of 87 clinical prediction models supporting clinical decisions for breast cancer patients.

INTRODUCTION: Numerous prediction models have been developed to support treatment-related decisions for breast cancer patients. External validation, a prerequisite for implementation in clinical practice, has been performed for only a few models. This study aims to externally validate published clinical prediction models using population-based Dutch data. METHODS: Patient-, tumor- and treatment-related data were derived from the Netherlands Cancer Registry (NCR). Model performance was assessed using the area under the receiver operating characteristic curve (AUC), scaled Brier score, and model calibration. Net benefit across applicable risk thresholds was evaluated with decision curve analysis. RESULTS: After assessing 922 models, 87 (9%) were included for validation. Models were excluded due to an incomplete model description (n = 262 (28%)), lack of required data (n = 521 (57%)), previously validated or developed with NCR data (n = 45 (5%)), or the associated NCR sample size was insufficient (n = 7 (1%)). The included models predicted survival (33 (38%) overall, 27 (31%) breast cancer-specific, and 3 (3%) other cause-specific), locoregional recurrence (n = 7 (8%)), disease free survival (n = 7 (8%)), metastases (n = 5 (6%)), lymph node involvement (n = 3 (3%)), pathologic complete response (n = 1 (1%)), and surgical margins (n = 1 (1%)). Seven models (8%) showed poor (AUC<0.6), 39 (45%) moderate (AUC:0.6-0.7), 38 (46%) good (AUC:0.7-0.9), and 3 (3%) excellent (AUC≥0.9) discrimination. Using the scaled Brier score, worse performance than an uninformative model was found in 34 (39%) models. CONCLUSION: Comprehensive registry data supports broad validation of published prediction models. Model performance varies considerably in new patient populations, affirming the importance of external validation studies before applying models in clinical practice. Well performing models could be clinically useful in a Dutch setting after careful impact evaluation.

The majority of 922 prediction models supporting breast cancer decision-making are at high risk of bias.

OBJECTIVES: To systematically review the currently available prediction models that may support treatment decision-making in breast cancer. STUDY DESIGN AND SETTING: Literature was systematically searched to identify studies reporting on development of prediction models aiming to support breast cancer treatment decision-making, published between January 2010 and December 2020. Quality and risk of bias were assessed using the Prediction model Risk Of Bias (ROB) Assessment Tool (PROBAST). RESULTS: After screening 20,460 studies, 534 studies were included, reporting on 922 models. The 922 models predicted: mortality (n = 417 45%), recurrence (n = 217, 24%), lymph node involvement (n = 141, 15%), adverse events (n = 58, 6%), treatment response (n = 56, 6%), or other outcomes (n = 33, 4%). In total, 285 models (31%) lacked a complete description of the final model and could not be applied to new patients. Most models (n = 878, 95%) were considered to contain high ROB. CONCLUSION: A substantial overlap in predictor variables and outcomes between the models was observed. Most models were not reported according to established reporting guidelines or showed methodological flaws during the development and/or validation of the model. Further development of prediction models with thorough quality and validity assessment is an essential first step for future clinical application.

Improved risk estimation of locoregional recurrence, secondary contralateral tumors and distant metastases in early breast cancer: the INFLUENCE 2.0 model.

PURPOSE: To extend the functionality of the existing INFLUENCE nomogram for locoregional recurrence (LRR) of breast cancer toward the prediction of secondary primary tumors (SP) and distant metastases (DM) using updated follow-up data and the best suitable statistical approaches. METHODS: Data on women diagnosed with non-metastatic invasive breast cancer were derived from the Netherlands Cancer Registry (n = 13,494). To provide flexible time-dependent individual risk predictions for LRR, SP, and DM, three statistical approaches were assessed; a Cox proportional hazard approach (COX), a parametric spline approach (PAR), and a random survival forest (RSF). These approaches were evaluated on their discrimination using the Area Under the Curve (AUC) statistic and on calibration using the Integrated Calibration Index (ICI). To correct for optimism, the performance measures were assessed by drawing 200 bootstrap samples. RESULTS: Age, tumor grade, pT, pN, multifocality, type of surgery, hormonal receptor status, HER2-status, and adjuvant therapy were included as predictors. While all three approaches showed adequate calibration, the RSF approach offers the best optimism-corrected 5-year AUC for LRR (0.75, 95%CI: 0.74-0.76) and SP (0.67, 95%CI: 0.65-0.68). For the prediction of DM, all three approaches showed equivalent discrimination (5-year AUC: 0.77-0.78), while COX seems to have an advantage concerning calibration (ICI < 0.01). Finally, an online calculator of INFLUENCE 2.0 was created. CONCLUSIONS: INFLUENCE 2.0 is a flexible model to predict time-dependent individual risks of LRR, SP and DM at a 5-year scale; it can support clinical decision-making regarding personalized follow-up strategies for curatively treated non-metastatic breast cancer patients.

External validation of the Memorial Sloan Kettering Cancer Centre and Briganti nomograms for the prediction of lymph node involvement of prostate cancer using clinical stage assessed by magnetic resonance imaging.

OBJECTIVES: To evaluate the impact of using clinical stage assessed by multiparametric magnetic resonance imaging (mpMRI) on the performance of two established nomograms for the prediction of pelvic lymph node involvement (LNI) in patients with prostate cancer. PATIENTS AND METHODS: Patients undergoing robot-assisted extended pelvic lymph node dissection (ePLND) from 2015 to 2019 at three teaching hospitals were retrospectively evaluated. Risk of LNI was calculated four times for each patient, using clinical tumour stage (T-stage) assessed by digital rectal examination (DRE) and by mpMRI, in the Memorial Sloan Kettering Cancer Centre (MSKCC; 2018) and Briganti (2012) nomograms. Discrimination (area under the curve [AUC]), calibration, and the net benefit of these four strategies were assessed and compared. RESULTS: A total of 1062 patients were included, of whom 301 (28%) had histologically proven LNI. Using DRE T-stage resulted in AUCs of 0.71 (95% confidence interval [CI] 0.70-0.72) for the MSKCC and 0.73 (95% CI 0.72-0.74) for the Briganti nomogram. Using mpMRI T-stage, the AUCs were 0.72 (95% CI 0.71-0.73) for the MSKCC and 0.75 (95% CI 0.74-0.76) for the Briganti nomogram. mpMRI T-stage resulted in equivalent calibration compared with DRE T-stage. Combined use of mpMRI T-stage and the Briganti 2012 nomogram was shown to be superior in terms of AUC, calibration, and net benefit. Use of mpMRI T-stage led to increased sensitivity for the detection of LNI for all risk thresholds in both models, countered by a decreased specificity, compared with DRE T-stage. CONCLUSION: T-stage as assessed by mpMRI is an appropriate alternative for T-stage assessed by DRE to determine nomogram-based risk of LNI in patients with prostate cancer, and was associated with improved model performance of both the MSKCC 2018 and Briganti 2012 nomograms.

Optimizing the risk threshold of lymph node involvement for performing extended pelvic lymph node dissection in prostate cancer patients: a cost-effectiveness analysis.

BACKGROUND: Extended pelvic lymph node dissection (ePLND) may be omitted in prostate cancer (CaP) patients with a low predicted risk of lymph node involvement (LNI). The aim of the current study was to quantify the cost-effectiveness of using different risk thresholds for predicted LNI in CaP patients to inform decision making on omitting ePLND. METHODS: Five different thresholds (2%, 5%, 10%, 20%, and 100%) used in practice for performing ePLND were compared using a decision analytic cohort model with the 100% threshold (i.e., no ePLND) as reference. Compared outcomes consisted of quality-adjusted life years (QALYs) and costs. Baseline characteristics for the hypothetical cohort were based on an actual Dutch patient cohort containing 925 patients who underwent ePLND with risks of LNI predicted by the Memorial Sloan Kettering Cancer Center web-calculator. The best strategy was selected based on the incremental cost effectiveness ratio when applying a willingness to pay (WTP) threshold of €20,000 per QALY gained. Probabilistic sensitivity analysis was performed with Monte Carlo simulation to assess the robustness of the results. RESULTS: Costs and health outcomes were lowest (€4,858 and 6.04 QALYs) for the 100% threshold, and highest (€10,939 and 6.21 QALYs) for the 2% threshold, respectively. The incremental cost effectiveness ratio for the 2%, 5%, 10%, and 20% threshold compared with the first threshold above (i.e., 5%, 10%, 20%, and 100%) were €189,222/QALY, €130,689/QALY, €51,920/QALY, and €23,187/QALY respectively. Applying a WTP threshold of €20.000 the probabilities for the 2%, 5%, 10%, 20%, and 100% threshold strategies being cost-effective were 0.0%, 0.3%, 4.9%, 30.3%, and 64.5% respectively. CONCLUSION: Applying a WTP threshold of €20.000, completely omitting ePLND in CaP patients is cost-effective compared to other risk-based strategies. However, applying a 20% threshold for probable LNI to the Briganti 2012 nomogram or the Memorial Sloan Kettering Cancer Center web-calculator, may be a feasible alternative, in particular when higher WTP values are considered.

Validation of the online prediction model CancerMath in the Dutch breast cancer population.

PURPOSE: CancerMath predicts the expected benefit of adjuvant systemic therapy on overall (OS) and breast cancer-specific survival (BCSS). Here, CancerMath was validated in Dutch breast cancer patients. METHODS: All operated women diagnosed with stage I-III primary invasive breast cancer in 2005 were identified from the Netherlands Cancer Registry. Calibration was assessed by comparing 5- and 10-year predicted and observed OS/BCSS using χ2 tests. A difference > 3% was considered as clinically relevant. Discrimination was assessed by area under the receiver operating characteristic (AUC) curves. RESULTS: Altogether, 8032 women were included. CancerMath underestimated 5- and 10-year OS by 2.2% and 1.9%, respectively. AUCs of 5- and 10-year OS were both 0.77. Divergence between predicted and observed OS was most pronounced in grade II, patients without positive nodes, tumours 1.01-2.00 cm, hormonal receptor positive disease and patients 60-69 years. CancerMath underestimated 5- and 10-year BCSS by 0.5% and 0.6%, respectively. AUCs were 0.78 and 0.73, respectively. No significant difference was found in any subgroup. CONCLUSION: CancerMath predicts OS accurately for most patients with early breast cancer although outcomes should be interpreted with care in some subgroups. BCSS is predicted accurately in all subgroups. Therefore, CancerMath can reliably be used in (Dutch) clinical practice.

Predicting the risk of locoregional recurrence after early breast cancer: an external validation of the Dutch INFLUENCE-nomogram with clinical cancer registry data from Germany.

PURPOSE: Follow-up after breast cancer treatment aims for an early detection of locoregional breast cancer recurrences (LRR) to improve the patients' outcome. By estimating individual's 5-year recurrence-risks, the Dutch INFLUENCE-nomogram can assist health professionals and patients in developing personalized risk-based follow-up pathways. The objective of this study is to validate the prediction tool on non-Dutch patients. MATERIAL AND METHODS: Data for this external validation derive from a large clinical cancer registry in southern Germany, covering a population of 1.1 million. Patients with curative resection of early-stage breast cancer, diagnosed between 2000 and 2012, were included in the analysis (n = 6520). For each of them, an individual LRR-risk was estimated by the INFLUENCE-nomogram. Its predictive ability was tested by comparing estimated and observed LRR-probabilities using the Hosmer-Lemeshow goodness-of-fit test and C-statistics. RESULTS: In the German validation-cohort, 2.8% of the patients developed an LRR within 5 years after primary surgery (n = 184). While the INFLUENCE-nomogram generally underestimates the actual LRR-risk of the German patients (p < 0.001), its discriminative ability is comparable to the one observed in the original Dutch modeling-cohort (C-statistic German validation-cohort: 0.73, CI 0.69-0.77 vs. C-statistic Dutch modeling-cohort: 0.71, CI 0.69-0.73). Similar results were obtained in most of the subgroup analyses stratified by age, type of surgery and intrinsic biological subtypes. CONCLUSION: The outcomes of this external validation underline the generalizability of the INFLUENCE-nomogram beyond the Dutch population. The model performance could be enhanced in future by incorporating additional risk factors for LRR.

External Validation of Models Predicting the Probability of Lymph Node Involvement in Prostate Cancer Patients.

BACKGROUND: Multiple statistical models predicting lymph node involvement (LNI) in prostate cancer (PCa) exist to support clinical decision-making regarding extended pelvic lymph node dissection (ePLND). OBJECTIVE: To validate models predicting LNI in Dutch PCa patients. DESIGN, SETTING, AND PARTICIPANTS: Sixteen prediction models were validated using a patient cohort of 1001 men who underwent ePLND. Patient characteristics included serum prostate specific antigen (PSA), cT stage, primary and secondary Gleason scores, number of biopsy cores taken, and number of positive biopsy cores. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: Model performance was assessed using the area under the receiver operating characteristic curve (AUC). Calibration plots were used to visualize over- or underestimation by the models. RESULTS AND LIMITATIONS: LNI was identified in 276 patients (28%). Patients with LNI had higher PSA, higher primary Gleason pattern, higher Gleason score, higher number of nodes harvested, higher number of positive biopsy cores, and higher cT stage compared to patients without LNI. Predictions generated by the 2012 Briganti nomogram (AUC 0.76) and the Memorial Sloan Kettering Cancer Center (MSKCC) web calculator (AUC 0.75) were the most accurate. Calibration had a decisive role in selecting the most accurate models because of overlapping confidence intervals for the AUCs. Underestimation of LNI probability in patients had a predicted probability of <20%. The omission of model updating was a limitation of the study. CONCLUSIONS: Models predicting LNI in PCa patients were externally validated in a Dutch patient cohort. The 2012 Briganti and MSKCC nomograms were identified as the most accurate prediction models available. PATIENT SUMMARY: In this report we looked at how well models were able to predict the risk of prostate cancer spreading to the pelvic lymph nodes. We found that two models performed similarly in predicting the most accurate probabilities.