Weighted metrics are required when evaluating the performance of prediction models in nested case-control studies.

BACKGROUND: Nested case-control (NCC) designs are efficient for developing and validating prediction models that use expensive or difficult-to-obtain predictors, especially when the outcome is rare. Previous research has focused on how to develop prediction models in this sampling design, but little attention has been given to model validation in this context. We therefore aimed to systematically characterize the key elements for the correct evaluation of the performance of prediction models in NCC data. METHODS: We proposed how to correctly evaluate prediction models in NCC data, by adjusting performance metrics with sampling weights to account for the NCC sampling. We included in this study the C-index, threshold-based metrics, Observed-to-expected events ratio (O/E ratio), calibration slope, and decision curve analysis. We illustrated the proposed metrics with a validation of the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA version 5) in data from the population-based Rotterdam study. We compared the metrics obtained in the full cohort with those obtained in NCC datasets sampled from the Rotterdam study, with and without a matched design. RESULTS: Performance metrics without weight adjustment were biased: the unweighted C-index in NCC datasets was 0.61 (0.58-0.63) for the unmatched design, while the C-index in the full cohort and the weighted C-index in the NCC datasets were similar: 0.65 (0.62-0.69) and 0.65 (0.61-0.69), respectively. The unweighted O/E ratio was 18.38 (17.67-19.06) in the NCC datasets, while it was 1.69 (1.42-1.93) in the full cohort and its weighted version in the NCC datasets was 1.68 (1.53-1.84). Similarly, weighted adjustments of threshold-based metrics and net benefit for decision curves were unbiased estimates of the corresponding metrics in the full cohort, while the corresponding unweighted metrics were biased. In the matched design, the bias of the unweighted metrics was larger, but it could also be compensated by the weight adjustment. CONCLUSIONS: Nested case-control studies are an efficient solution for evaluating the performance of prediction models that use expensive or difficult-to-obtain biomarkers, especially when the outcome is rare, but the performance metrics need to be adjusted to the sampling procedure.

Personalised decision making to predict absolute metastatic risk in cutaneous squamous cell carcinoma: development and validation of a clinico-pathological model.

Background: Cutaneous squamous cell carcinoma (cSCC) is a common skin cancer, affecting more than 2 million people worldwide yearly and metastasising in 2-5% of patients. However, current clinical staging systems do not provide estimates of absolute metastatic risk, hence missing the opportunity for more personalised treatment advice. We aimed to develop a clinico-pathological model that predicts the probability of metastasis in patients with cSCC. Methods: Nationwide cohorts from (1) all patients with a first primary cSCC in The Netherlands in 2007-2008 and (2) all patients with a cSCC in 2013-2015 in England were used to derive nested case-control cohorts. Pathology records of primary cSCCs that originated a loco-regional or distant metastasis were identified, and these cSCCs were matched to primary cSCCs of controls without metastasis (1:1 ratio). The model was developed on the Dutch cohort (n = 390) using a weighted Cox regression model with backward selection and validated on the English cohort (n = 696). Model performance was assessed using weighted versions of the C-index, calibration metrics, and decision curve analysis; and compared to the Brigham and Women's Hospital (BWH) and the American Joint Committee on Cancer (AJCC) staging systems. Members of the multidisciplinary Skin Cancer Outcomes (SCOUT) consortium were surveyed to interpret metastatic risk cutoffs in a clinical context. Findings: Eight out of eleven clinico-pathological variables were selected. The model showed good discriminative ability, with an optimism-corrected C-index of 0.80 (95% Confidence interval (CI) 0.75-0.85) in the development cohort and a C-index of 0.84 (95% CI 0.81-0.87) in the validation cohort. Model predictions were well-calibrated: the calibration slope was 0.96 (95% CI 0.76-1.16) in the validation cohort. Decision curve analysis showed improved net benefit compared to current staging systems, particularly for thresholds relevant for decisions on follow-up and adjuvant treatment. The model is available as an online web-based calculator (https://emc-dermatology.shinyapps.io/cscc-abs-met-risk/). Interpretation: This validated model assigns personalised metastatic risk predictions to patients with cSCC, using routinely reported histological and patient-specific risk factors. The model can empower clinicians and healthcare systems in identifying patients with high-risk cSCC and offering personalised care/treatment and follow-up. Use of the model for clinical decision-making in different patient populations must be further investigated. Funding: PPP Allowance made available by Health-Holland, Top Sector Life Sciences & Health, to stimulate public-private partnerships.

Design, validation and implementation of an automated e-alert for acute kidney injury: 6-month pilot study shows increased awareness.

BACKGROUND: Acute kidney injury (AKI) is defined as a sudden episode of kidney failure but is known to be under-recognized by healthcare professionals. The Kidney Disease Improving Global Outcome (KDIGO) guidelines have formulated criteria to facilitate AKI diagnosis by comparing changes in plasma creatinine measurements (PCr). To improve AKI awareness, we implemented these criteria as an electronic alert (e-alert), in our electronic health record (EHR) system. METHODS: For every new PCr measurement measured in the University Medical Center Utrecht that triggered the e-alert, we provided the physician with actionable insights in the form of a memo, to improve or stabilize kidney function. Since e-alerts qualify for software as a medical device (SaMD), we designed, implemented and validated the e-alert according to the European Union In Vitro Diagnostic Regulation (IVDR). RESULTS: We evaluated the impact of the e-alert using pilot data six months before and after implementation. 2,053 e-alerts of 866 patients were triggered in the before implementation, and 1,970 e-alerts of 853 patients were triggered after implementation. We found improvements in AKI awareness as measured by (1) 2 days PCr follow up (56.6-65.8%, p-value: 0.003), and (2) stop of nephrotoxic medication within 7 days of the e-alert (59.2-63.2%, p-value: 0.002). CONCLUSION: Here, we describe the design and implementation of the e-alert in line with the IVDR, leveraging a multi-disciplinary team consisting of physicians, clinical chemists, data managers and data scientists, and share our firsts results that indicate an improved awareness among treating physicians.

A machine learning approach using endpoint adjudication committee labels for the identification of sepsis predictors at the emergency department.

Accurate sepsis diagnosis is paramount for treatment decisions, especially at the emergency department (ED). To improve diagnosis, clinical decision support (CDS) tools are being developed with machine learning (ML) algorithms, using a wide range of variable groups. ML models can find patterns in Electronic Health Record (EHR) data that are unseen by the human eye. A prerequisite for a good model is the use of high-quality labels. Sepsis gold-standard labels are hard to define due to a lack of reliable diagnostic tools for sepsis at the ED. Therefore, standard clinical tools, such as clinical prediction scores (e.g. modified early warning score and quick sequential organ failure assessment), and claims-based methods (e.g. ICD-10) are used to generate suboptimal labels. As a consequence, models trained with these "silver" labels result in ill-trained models. In this study, we trained ML models for sepsis diagnosis at the ED with labels of 375 ED visits assigned by an endpoint adjudication committee (EAC) that consisted of 18 independent experts. Our objective was to evaluate which routinely measured variables show diagnostic value for sepsis. We performed univariate testing and trained multiple ML models with 95 routinely measured variables of three variable groups; demographic and vital, laboratory and advanced haematological variables. Apart from known diagnostic variables, we identified added diagnostic value for less conventional variables such as eosinophil count and platelet distribution width. In this explorative study, we show that the use of an EAC together with ML can identify new targets for future sepsis diagnosis research.

Clinicopathologic models predicting non-sentinel lymph node metastasis in cutaneous melanoma patients: Are they useful for patients with a single positive sentinel node?

BACKGROUND AND OBJECTIVES: Of clinically node-negative (cN0) cutaneous melanoma patients with sentinel lymph node (SLN) metastasis, between 10% and 30% harbor additional metastases in non-sentinel lymph nodes (NSLNs). Approximately 80% of SLN-positive patients have a single positive SLN. METHODS: To assess whether state-of-the-art clinicopathologic models predicting NSLN metastasis had adequate performance, we studied a single-institution cohort of 143 patients with cN0 SLN-positive primary melanoma who underwent subsequent completion lymph node dissection. We used sensitivity (SE) and positive predictive value (PPV) to characterize the ability of the models to identify patients at high risk for NSLN disease. RESULTS: Across Stage III patients, all clinicopathologic models tested had comparable performances. The best performing model identified 52% of NSLN-positive patients (SE = 52%, PPV = 37%). However, for the single SLN-positive subgroup (78% of cohort), none of the models identified high-risk patients (SE > 20%, PPV > 20%) irrespective of the chosen probability threshold used to define the binary risk labels. Thus, we designed a new model to identify high-risk patients with a single positive SLN, which achieved a sensitivity of 49% (PPV = 26%). CONCLUSION: For the largest SLN-positive subgroup, those with a single positive SLN, current model performance is inadequate. New approaches are needed to better estimate nodal disease burden of these patients.

Ambiguous definitions for baseline serum creatinine affect acute kidney diagnosis at the emergency department.

BACKGROUND: Acute kidney injury (AKI) incidence is increasing, however AKI is often missed at the emergency department (ED). AKI diagnosis depends on changes in kidney function by comparing a serum creatinine (SCr) measurement to a baseline value. However, it remains unclear to what extent different baseline values may affect AKI diagnosis at ED. METHODS: Routine care data from ED visits between 2012 and 2019 were extracted from the Utrecht Patient Oriented Database. We evaluated baseline definitions with criteria from the RIFLE, AKIN and KDIGO guidelines. We evaluated four baseline SCr definitions (lowest, most recent, mean, median), as well as five different time windows (up to 365 days prior to ED visit) to select a baseline and compared this to the first measured SCr at ED. As an outcome, we assessed AKI prevalence at ED. RESULTS: We included 47,373 ED visits with both SCr-ED and SCr-BL available. Of these, 46,100 visits had a SCr-BL from the - 365/- 7 days time window. Apart from the lowest value, AKI prevalence remained similar for the other definitions when varying the time window. The lowest value with the - 365/- 7 time window resulted in the highest prevalence (21.4%). Importantly, applying the guidelines with all criteria resulted in major differences in prevalence ranging from 5.9 to 24.0%. CONCLUSIONS: AKI prevalence varies with the use of different baseline definitions in ED patients. Clinicians, as well as researchers and developers of automatic diagnostic tools should take these considerations into account when aiming to diagnose AKI in clinical and research settings.

Transportability and Implementation Challenges of Early Warning Scores for Septic Shock in the ICU: A Perspective on the TREWScore.

The increased use of electronic health records (EHRs) has improved the availability of routine care data for medical research. Combined with machine learning techniques this has spurred the development of early warning scores (EWSs) in hospitals worldwide. EWSs are commonly used in the hospital where they have been developed, yet few have been transported to external settings and/or internationally. In this perspective, we describe our experiences in implementing the TREWScore, a septic shock EWS, and the transportability challenges regarding domain, predictors, and clinical outcome we faced. We used data of 53,330 ICU stays from Medical Information Mart for Intensive Care-III (MIMIC-III) and 18,013 ICU stays from the University Medical Center (UMC) Utrecht, including 17,023 (31.9%) and 2,557 (14.2%) cases of sepsis, respectively. The MIMIC-III and UMC populations differed significantly regarding the length of stay (6.9 vs. 9.0 days) and hospital mortality (11.6% vs. 13.6%). We mapped all 54 TREWScore predictors to the UMC database: 31 were readily available, seven required unit conversion, 14 had to be engineered, one predictor required text mining, and one predictor could not be mapped. Lastly, we classified sepsis cases for septic shock using the sepsis-2 criteria. Septic shock populations (UMC 31.3% and MIMIC-III 23.3%) and time to shock events showed significant differences between the two cohorts. In conclusion, we identified challenges to transportability and implementation regarding domain, predictors, and clinical outcome when transporting EWS between hospitals across two continents. These challenges need to be systematically addressed to improve model transportability between centers and unlock the potential clinical utility of EWS.

Identification of stage I/IIA melanoma patients at high risk for disease relapse using a clinicopathologic and gene expression model.

PURPOSE: Patients with stage I/IIA cutaneous melanoma (CM) are currently not eligible for adjuvant therapies despite uncertainty in relapse risk. Here, we studied the ability of a recently developed model which combines clinicopathologic and gene expression variables (CP-GEP) to identify stage I/IIA melanoma patients who have a high risk for disease relapse. PATIENTS AND METHODS: Archival specimens from a cohort of 837 consecutive primary CMs were used for assessing the prognostic performance of CP-GEP. The CP-GEP model combines Breslow thickness and patient age, with the expression of eight genes in the primary tumour. Our specific patient group, represented by 580 stage I/IIA patients, was stratified based on their risk of relapse: CP-GEP High Risk and CP-GEP Low Risk. The main clinical end-point of this study was five-year relapse-free survival (RFS). RESULTS: Within the stage I/IIA melanoma group, CP-GEP identified a high-risk patient group (47% of total stage I/IIA patients) which had a considerably worse five-year RFS than the low-risk patient group; 74% (95% confidence interval [CI]: 67%-80%) versus 89% (95% CI: 84%-93%); hazard ratio [HR] = 2.98 (95% CI: 1.78-4.98); P < 0.0001. Of patients in the high-risk group, those who relapsed were most likely to do so within the first 3 years. CONCLUSION: The CP-GEP model can be used to identify stage I/IIA patients who have a high risk for disease relapse. These patients may benefit from adjuvant therapy.

Model Combining Tumor Molecular and Clinicopathologic Risk Factors Predicts Sentinel Lymph Node Metastasis in Primary Cutaneous Melanoma.

PURPOSE: More than 80% of patients who undergo sentinel lymph node (SLN) biopsy have no nodal metastasis. Here we describe a model that combines clinicopathologic and molecular variables to identify patients with thin and intermediate thickness melanomas who may forgo the SLN biopsy procedure due to their low risk of nodal metastasis. PATIENTS AND METHODS: Genes with functional roles in melanoma metastasis were discovered by analysis of next generation sequencing data and case control studies. We then used PCR to quantify gene expression in diagnostic biopsy tissue across a prospectively designed archival cohort of 754 consecutive thin and intermediate thickness primary cutaneous melanomas. Outcome of interest was SLN biopsy metastasis within 90 days of melanoma diagnosis. A penalized maximum likelihood estimation algorithm was used to train logistic regression models in a repeated cross validation scheme to predict the presence of SLN metastasis from molecular, clinical and histologic variables. RESULTS: Expression of genes with roles in epithelial-to-mesenchymal transition (glia derived nexin, growth differentiation factor 15, integrin ß3, interleukin 8, lysyl oxidase homolog 4, TGFß receptor type 1 and tissue-type plasminogen activator) and melanosome function (melanoma antigen recognized by T cells 1) were associated with SLN metastasis. The predictive ability of a model that only considered clinicopathologic or gene expression variables was outperformed by a model which included molecular variables in combination with the clinicopathologic predictors Breslow thickness and patient age; AUC, 0.82; 95% CI, 0.78-0.86; SLN biopsy reduction rate of 42% at a negative predictive value of 96%. CONCLUSION: A combined model including clinicopathologic and gene expression variables improved the identification of melanoma patients who may forgo the SLN biopsy procedure due to their low risk of nodal metastasis.

Monte-Carlo modeling of the central carbon metabolism of Lactococcus lactis: insights into metabolic regulation.

Metabolic pathways are complex dynamic systems whose response to perturbations and environmental challenges are governed by multiple interdependencies between enzyme properties, reactions rates, and substrate levels. Understanding the dynamics arising from such a network can be greatly enhanced by the construction of a computational model that embodies the properties of the respective system. Such models aim to incorporate mechanistic details of cellular interactions to mimic the temporal behavior of the biochemical reaction system and usually require substantial knowledge of kinetic parameters to allow meaningful conclusions. Several approaches have been suggested to overcome the severe data requirements of kinetic modeling, including the use of approximative kinetics and Monte-Carlo sampling of reaction parameters. In this work, we employ a probabilistic approach to study the response of a complex metabolic system, the central metabolism of the lactic acid bacterium Lactococcus lactis, subject to perturbations and brief periods of starvation. Supplementing existing methodologies, we show that it is possible to acquire a detailed understanding of the control properties of a corresponding metabolic pathway model that is directly based on experimental observations. In particular, we delineate the role of enzymatic regulation to maintain metabolic stability and metabolic recovery after periods of starvation. It is shown that the feedforward activation of the pyruvate kinase by fructose-1,6-bisphosphate qualitatively alters the bifurcation structure of the corresponding pathway model, indicating a crucial role of enzymatic regulation to prevent metabolic collapse for low external concentrations of glucose. We argue that similar probabilistic methodologies will help our understanding of dynamic properties of small-, medium- and large-scale metabolic networks models.

Role of phosphate in the central metabolism of two lactic acid bacteria--a comparative systems biology approach.

Lactic acid-producing bacteria survive in distinct environments, but show common metabolic characteristics. Here we studied the dynamic interactions of the central metabolism in Lactococcus lactis, extensively used as a starter culture in the dairy industry, and Streptococcus pyogenes, a human pathogen. Glucose-pulse experiments and enzymatic measurements were performed to parameterize kinetic models of glycolysis. Significant improvements were made to existing kinetic models for L. lactis, which subsequently accelerated the development of the first kinetic model of S. pyogenes glycolysis. The models revealed an important role for extracellular phosphate in the regulation of central metabolism and the efficient use of glucose. Thus, phosphate, which is rarely taken into account as an independent species in models of central metabolism, should be considered more thoroughly in the analysis of metabolic systems in the future. Insufficient phosphate supply can lead to a strong inhibition of glycolysis at high glucose concentrations in both species, but this was more severe in S. pyogenes. S. pyogenes is more efficient at converting glucose to ATP, showing a higher tendency towards heterofermentative energy metabolism than L. lactis. Our comparative systems biology approach revealed that the glycolysis of L. lactis and S. pyogenes have similar characteristics, but are adapted to their individual natural habitats with respect to phosphate regulation.

Response rescaling in bacterial chemotaxis.

Sensory systems rescale their response sensitivity upon adaptation according to simple strategies that recur in processes as diverse as single-cell signaling, neural network responses, and whole-organism perception. Here, we study response rescaling in Escherichia coli chemotaxis, where adaptation dynamically tunes the cells' motile response during searches for nutrients. Using in vivo fluorescence resonance energy transfer (FRET) measurements on immobilized cells, we demonstrate that the design of this prokaryotic signaling network follows the fold-change detection (FCD) strategy, responding faithfully to the shape of the input profile irrespective of its absolute intensity. Using a microfluidics-based assay for free swimming cells, we confirm intensity-independent gradient responses at the behavioral level. By theoretical analysis, we identify a set of sufficient conditions for FCD in E. coli chemotaxis, which leads to the prediction that the adaptation timescale is invariant with respect to the background input level. Additional FRET experiments confirm that the adaptation timescale is invariant over an ∼10,000-fold range of background concentrations. These observations in a highly optimized bacterial system support the concept that FCD represents a robust sensing strategy for spatial searches. To our knowledge, these experiments provide a unique demonstration of FCD in any biological sensory system.

Developing optimal input design strategies in cancer systems biology with applications to microfluidic device engineering.

BACKGROUND: Mechanistic models are becoming more and more popular in Systems Biology; identification and control of models underlying biochemical pathways of interest in oncology is a primary goal in this field. Unfortunately the scarce availability of data still limits our understanding of the intrinsic characteristics of complex pathologies like cancer: acquiring information for a system understanding of complex reaction networks is time consuming and expensive. Stimulus response experiments (SRE) have been used to gain a deeper insight into the details of biochemical mechanisms underlying cell life and functioning. Optimisation of the input time-profile, however, still remains a major area of research due to the complexity of the problem and its relevance for the task of information retrieval in systems biology-related experiments. RESULTS: We have addressed the problem of quantifying the information associated to an experiment using the Fisher Information Matrix and we have proposed an optimal experimental design strategy based on evolutionary algorithm to cope with the problem of information gathering in Systems Biology. On the basis of the theoretical results obtained in the field of control systems theory, we have studied the dynamical properties of the signals to be used in cell stimulation. The results of this study have been used to develop a microfluidic device for the automation of the process of cell stimulation for system identification. CONCLUSION: We have applied the proposed approach to the Epidermal Growth Factor Receptor pathway and we observed that it minimises the amount of parametric uncertainty associated to the identified model. A statistical framework based on Monte-Carlo estimations of the uncertainty ellipsoid confirmed the superiority of optimally designed experiments over canonical inputs. The proposed approach can be easily extended to multiobjective formulations that can also take advantage of identifiability analysis. Moreover, the availability of fully automated microfluidic platforms explicitly developed for the task of biochemical model identification will hopefully reduce the effects of the 'data rich--data poor' paradox in Systems Biology.