Timing and prediction of secondary bacteremia in patients with COVID-19: A retrospective cohort study.

Background: We aimed to aid the appropriate use of antimicrobial agents by determining the timing of secondary bacteremia and validating and updating clinical prediction models for bacteremia in patients with COVID-19. Methods: We performed a retrospective cohort study on all hospitalized patients diagnosed with COVID-19 who underwent blood culture tests from January 1, 2020, and September 30, 2021, at an urban teaching hospital in Japan. The primary outcome measure was secondary bacteremia in patients with COVID-19. Results: Of the 507 patients hospitalized with COVID-19, 169 underwent blood culture tests. Eleven of them had secondary bacteremia. The majority of secondary bacteremia occurred on or later than the 9th day after symptom onset. Positive blood culture samples collected on day 9 or later after disease onset had an odds ratio of 22.4 (95% CI 2.76-181.2, p < 0.001) compared with those collected less than 9 days after onset. The area under the receiver operating characteristic curve of the modified Shapiro rule combined with blood culture collection on or after the 9th day from onset was 0.919 (95% CI, 0.843-0.995), and the net benefit was high according to the decision curve analysis. Conclusions: The timings of symptom onset and hospital admission may be valuable indicators for making a clinical decision to perform blood cultures in patients hospitalized with COVID-19.

Clinical prediction for surgical versus nonsurgical interventions in patients with vertebral osteomyelitis and discitis.

Background: Vertebral osteomyelitis and discitis (VOD), an infection of intervertebral discs, often requires spine surgical intervention and timely management to prevent adverse outcomes. Our study aims to develop a machine learning (ML) model to predict the indication for surgical intervention (during the same hospital stay) versus nonsurgical management in patients with VOD. Methods: This retrospective study included adult patients (≥18 years) with VOD (ICD-10 diagnosis codes M46.2,3,4,5) treated at a single institution between 01/01/2015 and 12/31/2019. The primary outcome studied was surgery. Candidate predictors were age, sex, race, Elixhauser comorbidity index, first-recorded lab values, first-recorded vital signs, and admit diagnosis. After splitting the dataset, XGBoost, logistic regression, and K-neighbor classifier algorithms were trained and tested for model development. Results: A total of 1,111 patients were included in this study, among which 30% (n=339) of patients underwent surgical intervention. Age and sex did not significantly differ between the two groups; however, race did significantly differ (P<0.0001), with the surgical group having a higher percentage of white patients. The top ten model features for the best-performing model (XGBoost) were as follows (in descending order of importance): admit diagnosis of fever, negative culture, Staphylococcus aureus culture, partial pressure of arterial oxygen to fractional inspired oxygen ratio (PaO2:FiO2), admit diagnosis of intraspinal abscess and granuloma, admit diagnosis of sepsis, race, troponin I, acid-fast bacillus culture, and alveolar-arterial gradient (A-a gradient). XGBoost model metrics were as follows: accuracy =0.7534, sensitivity =0.7436, specificity =0.7586, and area under the curve (AUC) =0.8210. Conclusions: The XGBoost model reliably predicts the indication for surgical intervention based on several readily available patient demographic information and clinical features. The interpretability of a supervised ML model provides robust insight into patient outcomes. Furthermore, it paves the way for the development of an efficient hospital resource allocation instrument, designed to guide clinical suggestions.

Establishment and validation of a bad outcomes prediction model based on EEG and clinical parameters in prolonged disorder of consciousness.

Objective: This study aimed to explore the electroencephalogram (EEG) indicators and clinical factors that may lead to poor prognosis in patients with prolonged disorder of consciousness (pDOC), and establish and verify a clinical predictive model based on these factors. Methods: This study included 134 patients suffering from prolonged disorder of consciousness enrolled in our department of neurosurgery. We collected the data of sex, age, etiology, coma recovery scales (CRS-R) score, complications, blood routine, liver function, coagulation and other laboratory tests, resting EEG data and follow-up after discharge. These patients were divided into two groups: training set (n = 107) and verification set (n = 27). These patients were divided into a training set of 107 and a validation set of 27 for this study. Univariate and multivariate regression analysis were used to determine the factors affecting the poor prognosis of pDOC and to establish nomogram model. We use the receiver operating characteristic (ROC) and calibration curves to quantitatively test the effectiveness of the training set and the verification set. In order to further verify the clinical practical value of the model, we use decision curve analysis (DCA) to evaluate the model. Result: The results from univariate and multivariate logistic regression analyses suggested that an increased frequency of occurrence microstate A, reduced CRS-R scores at the time of admission, the presence of episodes associated with paroxysmal sympathetic hyperactivity (PSH), and decreased fibrinogen levels all function as independent prognostic factors. These factors were used to construct the nomogram. The training and verification sets had areas under the curve of 0.854 and 0.920, respectively. Calibration curves and DCA demonstrated good model performance and significant clinical benefits in both sets. Conclusion: This study is based on the use of clinically available and low-cost clinical indicators combined with EEG to construct a highly applicable and accurate model for predicting the adverse prognosis of patients with prolonged disorder of consciousness. It provides an objective and reliable tool for clinicians to evaluate the prognosis of prolonged disorder of consciousness, and helps clinicians to provide personalized clinical care and decision-making for patients with prolonged disorder of consciousness and their families.

Prediction models for diagnosis and prognosis of the colonisation or infection of multidrug-resistant organisms in adults: A systematic review, critical appraisal, and meta-analysis.

BACKGROUND: Prediction models help target patients at risk of multidrug-resistant organism (MDRO) colonisation or infection and could serve as tools informing clinical practices to prevent MDRO transmission and inappropriate empiric antibiotic therapy. However, limited evidence identifies which among the available models are of low risk of bias and suitable for clinical application. OBJECTIVES: To identify, describe, appraise, and summarise the performance of all prognostic and diagnostic models developed or validated for predicting MDRO colonisation or infection. DATA SOURCES: Six electronic literature databases and clinical registration databases were searched until April 2022. STUDY ELIGIBILITY CRITERIA: Development and validation studies of any multivariable prognostic and diagnostic models to predict MDRO colonisation or infection in adults. ASSESSMENT OF RISK OF BIAS: The Prediction Model Risk of Bias Assessment Tool was used to assess risk of bias. Evidence certainty was assessed using the GRADE approach. METHODS OF DATA SYNTHESIS: Meta-analyses were conducted to summarise the discrimination and calibration of the models' external validations conducted in at least two non-overlapping datasets. RESULTS: We included 162 models (108 studies) developed for diagnosing (n=135) and predicting (n=27) MDRO colonisation or infection. Models exhibited a high risk of bias, especially in statistical analysis. High-frequency predictors were age, recent invasive procedures, antibiotic usage, and prior hospitalisation. Less than 25% of the models underwent external validations, with only seven by independent teams. Meta-analyses for one diagnostic and two prognostic models only produced very-low to low certainty of evidence. CONCLUSIONS: The review comprehensively described the models for identifying patients at risk of MDRO colonisation or infection. We cannot recommend which models are ready for application due to high risk of bias, limited validations, and low certainty of evidence from meta-analyses, indicating a clear need to improve the conducting and reporting of model development and external validation studies to facilitate clinical application.

Diagnostic accuracy of 3-item stroke scale for detection of cerebral large vessel occlusion: A systematic review and meta-analysis.

BACKGROUND: Prompt identification of large vessel occlusion (LVO) in acute ischemic stroke (AIS) is crucial for expedited endovascular therapy (EVT) and improved patient outcomes. Prehospital stroke scales, such as the 3-Item Stroke Scale (3I-SS), could be beneficial in detecting LVO in suspected patients. This meta-analysis evaluates the diagnostic accuracy of 3I-SS for LVO detection in AIS. METHODS: A systematic search was conducted in Medline, Embase, Scopus, and Web of Science databases until February 2024 with no time and language restrictions. Prehospital and in-hospital studies reporting diagnostic accuracy were included. Review articles, studies without reported 3I-SS cut-offs, and studies lacking the required data were excluded. Pooled effect sizes, including area under the curve (AUC), sensitivity, specificity, diagnostic odds ratio (DOR), positive and negative likelihood ratios (PLR and NLR) with 95% confidence intervals (CI) were calculated. RESULTS: Twenty-two studies were included in the present meta-analysis. A 3I-SS score of 2 or higher demonstrated sensitivity of 76% (95% CI: 52%-90%) and specificity of 74% (95% CI: 57%-86%) as the optimal cut-off, with an AUC of 0.81 (95% CI: 0.78-0.84). DOR, PLR, and NLR, were 9 (95% CI: 5-15), 2.9 (95% CI: 2.0-4.3) and 0.32 (95% CI: 0.17-0.61), respectively. Sensitivity analysis confirmed the analyses' robustness in suspected to stroke patients, anterior circulation LVO, assessment by paramedics, and pre-hospital settings. Meta-regression analyses pinpointed LVO definition (anterior circulation, posterior circulation) and patient setting (suspected stroke, confirmed stroke) as potential sources of heterogeneity. CONCLUSION: 3I-SS demonstrates good diagnostic accuracy in identifying LVO stroke and may be valuable in the prompt identification of patients for direct transfer to comprehensive stroke centers.

A novel clinical prediction model for in-hospital mortality in sepsis patients complicated by ARDS: A MIMIC IV database and external validation study.

Background: Sepsis complicated by ARDS significantly increases morbidity and mortality, underscoring the need for robust predictive models to enhance patient management. Methods: We collected data on 6390 patients with ARDS-complicated sepsis from the MIMIC IV database. Following rigorous data cleaning, including outlier management, handling missing values, and transforming variables, we conducted univariate analysis and logistic multivariate regression. We employed the LASSO machine learning algorithm to identify risk factors closely associated with patient outcomes. These factors were then used to develop a new clinical prediction model. The model underwent preliminary assessment and internal validation, and its performance was further tested through external validation using data from 225 patients at a major tertiary hospital in China. This validation assessed the model's discrimination, calibration, and net clinical benefits. Results: The model, illustrated by a concise nomogram, demonstrated significant discrimination with an area under the curve (AUC) of 0.711 in the internal validation set and 0.771 in the external validation set, outperforming conventional severity scores such as the SOFA and SAPS II. It also showed good calibration and net clinical benefits. Conclusions: Our model serves as a valuable tool for identifying sepsis patients with ARDS at high risk of in-hospital mortality. This could enable the implementation of personalized treatment strategies, potentially improving patient outcomes.

Development and Internal Validation of the Cleveland Clinic Bleeding Model to Predict Major Bleeding Risk at Admission in Medical Inpatients.

BACKGROUND: Guidelines recommend pharmacological VTE prophylaxis for acutely ill medical patients at acceptable bleeding risk, but only the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) model has been validated for bleeding risk assessment. OBJECTIVE: We developed and internally validated a risk assessment model (RAM) to predict major in-hospital bleeding using risk factors at admission and compared our model to IMPROVE. METHODS: We selected patients admitted to medical services at 10 hospitals in the Cleveland Clinic Health System from 2017 to 2020. We identified major bleeding according to the International Society on Thrombosis and Hemostasis criteria, using a combination of diagnostic codes and laboratory values, and confirmed events with chart review. We fit a LASSO logistic regression model in the training set and compared the discrimination and calibration of our model and IMPROVE in the validation set. RESULTS: Among 46,314 admissions, 268 (0.58%) had a major bleed. The final RAM included 16 risk factors, of which prior bleeding (OR = 4.83), peptic ulcer (OR = 3.82), history of sepsis (OR = 3.26), and steroid use (OR = 2.59) were the strongest. The Cleveland Clinic Bleeding Model (CCBM) had better discrimination than IMPROVE (AUC = 0.85 vs. 0.70, p < .001) and, at equivalent sensitivity (52%), categorized fewer patients as high-risk (7.2% vs. 11.8%, p < .001). Calibration was adequate (Brier score = 0.0057). CONCLUSION: Using a large population of medical inpatients with verified major bleeding events, we developed and internally validated a RAM for major bleeding whose performance surpassed the IMPROVE model.

Integrative nomogram model based on anoikis-related genes enhances prognostic evaluation in colorectal cancer.

Background: Revealing the role of anoikis resistance plays in CRC is significant for CRC diagnosis and treatment. This study integrated the CRC anoikis-related key genes (CRC-AKGs) and established a novel model for improving the efficiency and accuracy of the prognostic evaluation of CRC. Methods: CRC-ARGs were screened out by performing differential expression and univariate Cox analysis. CRC-AKGs were obtained through the LASSO machine learning algorithm and the LASSO Risk-Score was constructed to build a nomogram clinical prediction model combined with the clinical predictors. In parallel, this work developed a web-based dynamic nomogram to facilitate the generalization and practical application of our model. Results: We identified 10 CRC-AKGs and a risk-related prognostic Risk-Score was calculated. Multivariate COX regression analysis indicated that the Risk-Score, TNM stage, and age were independent risk factors that significantly associated with the CRC prognosis(p < 0.05). A prognostic model was built to predict the outcome with satisfied accuracy (3-year AUC = 0.815) for CRC individuals. The web interactive nomogram (https://yuexiaozhang.shinyapps.io/anoikisCRC/) showed strong generalizability of our model. In parallel, a substantial correlation between tumor microenvironment and Risk-Score was discovered in the present work. Conclusion: This study reveals the potential role of anoikis in CRC and sets new insights into clinical decision-making in colorectal cancer based on both clinical and sequencing data. Also, the interactive tool provides researchers with a user-friendly interface to input relevant clinical variables and obtain personalized risk predictions or prognostic assessments based on our established model.

Novel prediction model combining PET/CT metabolic parameters, inflammation markers, and TNM stage: prospects for personalizing prognosis in nasopharyngeal carcinoma.

PURPOSE: This study aims to develop a novel prediction model and risk stratification system that could accurately predict progression-free survival (PFS) in patients with nasopharyngeal carcinoma (NPC). METHODS: Herein, we included 106 individuals diagnosed with NPC, who underwent 18F-FDG PET/CT scanning before treatment. They were divided into training (n = 76) and validation (n = 30) sets. The prediction model was constructed based on multivariate Cox regression analysis results and its predictive performance was evaluated. Risk factor stratification was performed based on the nomogram scores of each case, and Kaplan-Meier curves were used to evaluate the model's discriminative ability for high- and low-risk groups. RESULTS: Multivariate Cox regression analysis showed that N stage, M stage, SUVmax, MTV, HI, and SIRI were independent factors affecting the prognosis of patients with NPC. In the training set, the model considerably outperformed the TNM stage in predicting PFS (AUCs of 0.931 vs. 0.841, 0.892 vs. 0.785, and 0.892 vs. 0.804 at 1-3 years, respectively). The calibration plots showed good agreement between actual observations and model predictions. The DCA curves further justified the effectiveness of the model in clinical practice. Between high- and low-risk group, 3-year PFS rates were significantly different (high- vs. low-risk group: 62.8% vs. 9.8%, p < 0.001). Adjuvant chemotherapy was also effective for prolonging survival in high-risk patients (p = 0.009). CONCLUSION: Herein, a novel prediction model was successfully developed and validated to improve the accuracy of prognostic prediction for patients with NPC, with the aim of facilitating personalized treatment.

[Development of a Prognostic Model for Overall Survival Adult Patients with Core Binding Factor Acute Myeloid Leukaemia].

OBJECTIVE: To analyze the factors affecting overall survival (OS) of adult patients with core-binding factor acute myeloid leukemia (CBF-AML) and establish a prediction model. METHODS: A total of 216 newly diagnosed patients with CBF-AML in the First Affiliated Hospital of Zhengzhou University from May 2015 to July 2021 were retrospectively analyzed. The 216 CBF-AML patients were divided into the training and the validation cohort at 7∶3 ratio. The Cox regression model was used to analyze the clinical factors affecting OS. Stepwise regression was used to establish the optimal model and the nomogram. Receiver operating characteristic (ROC) curve, calibration curve and decision curve analysis (DCA) were used to evaluate the model performance. RESULTS: Age(≥55 years old), peripheral blood blast(≥80%), fusion gene (AML1-ETO), KIT mutations were identified as independent adverse factors for OS. The area under the ROC curve at 3-year was 0.772 and 0.722 in the training cohort and validation cohort, respectively. The predicted value of the calibration curve is in good agreement with the measured value. DCA shows that this model performs better than a single factor. CONCLUSION: This prediction model is simple and feasible, and can effectively predict the OS of CBF-AML, and provide a basis for treatment decision.

Scoring System to Evaluate Risk of Nonoperative Management Failure in Children With Intussusception.

INTRODUCTION: Typical first-line management of children with intussusception is enema reduction; however, failure necessitates surgical intervention. The number of attempts varies by clinician, and predictors of failed nonoperative management are not routinely considered in practice. The purpose of this study is to create a scoring system that predicts risk of nonoperative failure and need for surgical intervention. METHODS: Children diagnosed with intussusception upon presentation to the emergency department of a tertiary children's hospital between 2019 and 2022 were retrospectively identified. Univariable logistic regression identified predictors of nonoperative failure used as starting covariates for multivariable logistic regression with final model determined by backwards elimination. Regression coefficients for final predictors were used to create the scoring system and optimal cut-points were delineated. RESULTS: We identified 143 instances of ultrasound-documented intussusception of which 28 (19.6%) required operative intervention. Predictors of failed nonoperative management included age ≥4 y (odds ratio [OR] 32.83, 95% confidence interval [CI]: 1.91-564.23), ≥1 failed enema reduction attempts (OR 189.53, 95% CI: 19.07-1884.11), presenting heart rate ≥128 (OR 3.38, 95% CI: 0.74-15.36), presenting systolic blood pressure ≥115 mmHg (OR 6.59, 95% CI: 0.93-46.66), and trapped fluid between intussuscepted loops on ultrasound (OR 17.54, 95% CI: 0.77-397.51). Employing these factors, a novel risk scoring system was developed (area under the curve 0.96, 95% CI: 0.93-0.99). Scores range from 0 to 8; ≤2 have low (1.1%), 3-4 moderate (50.0%), and ≥5 high (100%) failure risk. CONCLUSIONS: Using known risk factors for enema failure, we produced a risk scoring system with outstanding discriminate ability for children with intussusception necessitating surgical intervention. Prospective validation is warranted prior to clinical integration.

Assessing acutely ill children in general practice using the National PEWS and LqSOFA clinical scores: a retrospective cohort study.

BACKGROUND: Clinical tools are needed in general practice to help identify seriously ill children. The Liverpool quick Sequential Organ Failure Assessment (LqSOFA) was validated in an Emergency Department and performed well. The National Paediatric Early Warning score (PEWS) has been introduced in hospitals throughout England with hopes for implementation in general practice. AIM: To validate the LqSOFA and National PEWS in general practice. DESIGN/SETTING: Secondary analysis of 6,703 children <5 years presenting to 225 general practices in England and Wales with acute illnesses, linked to hospital data. METHOD: Variables from the LqSOFA and National PEWS were mapped onto study data to calculate score totals. A primary outcome of admission within two days of GP consultation was used to calculate sensitivity, specificity, negative predictive values (NPV), positive predictive values (PPV) and area-under-the-curve (AUC). RESULTS: 104/6,703 children were hospitalised within two days (pre-test probability 1.6%). The sensitivity of the LqSOFA was 30.6% (95% confidence interval 21.8% - 41.0%), with a specificity of 84.7% (83.7% - 85.6%), PPV of 3.0% (2.1% - 4.4%), NPV of 98.7% (98.4% - 99.0%), and AUC of 0.58 (0.53 - 0.63). The sensitivity of the National PEWS was 81.0% (71.0% - 88.1%), with a specificity of 32.5% (31.2% - 33.8%); PPV of 1.9% (1.5% - 2.5%); NPV of 99.1% (98.4% - 99.4%) and AUC of 0.66 (0.59 - 0.72). CONCLUSION: Although the NPVs appear useful, due to low pre-test probabilities rather than discriminative ability, neither tool accurately identified hospitalisations. Unconsidered use by GPs could result in unsustainable referrals.

Development and internal validation of a clinical prediction model to predict independence in daily living at discharge for patients with heart failure: analysis using a Japanese national inpatient database real-world dataset.

PURPOSE: To develop a clinical prediction model (CPM) to predict independence in activities of daily living (ADLs) in patients with heart failure. SUBJECTS AND METHODS: We collected the data of the individuals who were admitted and rehabilitated for heart failure from January 2017 to June 2022 from Japan's Diagnosis Procedure Combination database. We assessed the subjects' ADLs at discharge using the Barthel Index and classified them into independence, partial-independence, and total-dependence groups based on their ADLs at discharge. Two CPMs (an independence model and a partial-independence model) were developed by a binomial logistic regression analysis. The predictors included subject characteristics, treatment, and post-hospitalization disease onset. The CPMs' accuracy was validated by the area under the curve (AUC). Internal validation was performed using the bootstrap method. The final CPM is presented in a nomogram. RESULTS: We included 96,753 patients whose ADLs could be traced at discharge. The independence model had a 0.73 mean AUC and a 1.0 slope at bootstrapping. We thus developed a simplified model using nomograms, which also showed adequate predictive accuracy in the independence model. The partial-independence model had a 0.65 AUC and inadequate predictive accuracy. CONCLUSIONS: The independence model of ADLs in patients with heart failure is a useful CPM.

Risk factors for falls in Parkinson's disease: a cross-sectional observational and Mendelian randomization study.

Background: Patients with Parkinson's disease (PD) exhibit a heightened risk of falls and related fractures compared to the general population. This study aims to assess the clinical characteristics associated with falls in the patient with PD and to gain further insight into these factors through Mendelian randomization analysis. Methods: From January 2013 to December 2023, we included 591 patients diagnosed with Parkinson's disease at Shenzhen Baoan People's Hospital. Using univariate and multivariate logistic regression analyses, we identified clinical variables associated with falls. We constructed a nomogram based on these variables and evaluated the predictive efficacy of the model. Additionally, we employed summary statistics from genome-wide association studies to conduct two-sample Mendelian randomization (MR) analyses on key variables influencing falls. Results: Compared to the control group, we identified osteoporosis, motor dysfunction, higher Hoehn and Yahr scale as significant risk factors for falls in PD patients. Conversely, treatment with levodopa and a higher level of education exhibited a protective effect against the risk of falling. MR analysis further confirmed a causal relationship between osteoporosis, education level and falls in PD patients. Conclusion: Osteoporosis and educational attainment are correlated with falls in Parkinson's disease.

Framework for personalized prediction of treatment response in relapsing-remitting multiple sclerosis: a replication study in independent data.

BACKGROUND: Individualizing and optimizing treatment of relapsing-remitting multiple sclerosis patients is a challenging problem, which would benefit from a clinically valid decision support. Stühler et al. presented black box models for this aim which were developed and internally evaluated in a German registry but lacked external validation. METHODS: In patients from the French OFSEP registry, we independently built and validated models predicting being free of relapse and free of confirmed disability progression (CDP), following the methodological roadmap and predictors reported by Stühler. Hierarchical Bayesian models were fit to predict the outcomes under 6 disease-modifying treatments given the individual disease course up to the moment of treatment change. Data was temporally split on 2017, and models were developed in patients treated earlier (n = 5517). Calibration curves, discrimination, mean squared error (MSE) and relative percentage of root MSE (RMSE%) were assessed by external validation of models in more-recent patients (n = 3768). Non-Bayesian fixed-effects GLMs were also applied and their outcomes were compared to these of the Bayesian ones. For both, we modelled the number of on-therapy relapses with a negative binomial distribution, and CDP occurrence with a binomial distribution. RESULTS: The performance of our temporally-validated relapse model (MSE: 0.326, C-Index: 0.639) is potentially superior to that of Stühler's (MSE: 0.784, C-index: 0.608). Calibration plots revealed miscalibration. Our CDP model (MSE: 0.072, C-Index: 0.777) was also better than its counterpart (MSE: 0.131, C-index: 0.554). Results from non-Bayesian fixed-effects GLM models were similar to the Bayesian ones. CONCLUSIONS: The relapse and CDP models rebuilt and externally validated in independent data could compare and strengthen the credibility of the Stühler models. Their model-building strategy was replicable.

Don't be misled: 3 misconceptions about external validation of clinical prediction models.

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing 3 common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

Barriers to Implementing Registered Nurse-Driven Clinical Decision Support for Antibiotic Stewardship: Retrospective Case Study.

BACKGROUND: Up to 50% of antibiotic prescriptions for upper respiratory infections (URIs) are inappropriate. Clinical decision support (CDS) systems to mitigate unnecessary antibiotic prescriptions have been implemented into electronic health records, but their use by providers has been limited. OBJECTIVE: As a delegation protocol, we adapted a validated electronic health record-integrated clinical prediction rule (iCPR) CDS-based intervention for registered nurses (RNs), consisting of triage to identify patients with low-acuity URI followed by CDS-guided RN visits. It was implemented in February 2022 as a randomized controlled stepped-wedge trial in 43 primary and urgent care practices within 4 academic health systems in New York, Wisconsin, and Utah. While issues were pragmatically addressed as they arose, a systematic assessment of the barriers to implementation is needed to better understand and address these barriers. METHODS: We performed a retrospective case study, collecting quantitative and qualitative data regarding clinical workflows and triage-template use from expert interviews, study surveys, routine check-ins with practice personnel, and chart reviews over the first year of implementation of the iCPR intervention. Guided by the updated CFIR (Consolidated Framework for Implementation Research), we characterized the initial barriers to implementing a URI iCPR intervention for RNs in ambulatory care. CFIR constructs were coded as missing, neutral, weak, or strong implementation factors. RESULTS: Barriers were identified within all implementation domains. The strongest barriers were found in the outer setting, with those factors trickling down to impact the inner setting. Local conditions driven by COVID-19 served as one of the strongest barriers, impacting attitudes among practice staff and ultimately contributing to a work infrastructure characterized by staff changes, RN shortages and turnover, and competing responsibilities. Policies and laws regarding scope of practice of RNs varied by state and institutional application of those laws, with some allowing more clinical autonomy for RNs. This necessitated different study procedures at each study site to meet practice requirements, increasing innovation complexity. Similarly, institutional policies led to varying levels of compatibility with existing triage, rooming, and documentation workflows. These workflow conflicts were compounded by limited available resources, as well as an implementation climate of optional participation, few participation incentives, and thus low relative priority compared to other clinical duties. CONCLUSIONS: Both between and within health care systems, significant variability existed in workflows for patient intake and triage. Even in a relatively straightforward clinical workflow, workflow and cultural differences appreciably impacted intervention adoption. Takeaways from this study can be applied to other RN delegation protocol implementations of new and innovative CDS tools within existing workflows to support integration and improve uptake. When implementing a system-wide clinical care intervention, considerations must be made for variability in culture and workflows at the state, health system, practice, and individual levels. TRIAL REGISTRATION: ClinicalTrials.gov NCT04255303; https://clinicaltrials.gov/ct2/show/NCT04255303.

Calibration plots for multistate risk predictions models.

INTRODUCTION: There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation. METHODS: We studied pseudo-values based on the Aalen-Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR-IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW). The MLR-IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records. RESULTS: The pseudo-value, BLR-IPCW, and MLR-IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low-density regions of predicted transition probability. CONCLUSIONS: We recommend implementing either the pseudo-value or BLR-IPCW approaches to produce a calibration curve, combined with the MLR-IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the "calibmsm" R package available on CRAN.

External validation of the computer aided risk scoring system in predicting in-hospital mortality following emergency medical admissions.

BACKGROUND: Clinical prediction models have the potential to improve the quality of care and enhance patient safety outcomes. A Computer-aided Risk Scoring system (CARSS) was previously developed to predict in-hospital mortality following emergency admissions based on routinely collected blood tests and vitals. We aimed to externally validate the CARSS model. METHODS: In this retrospective external validation study, we considered all adult (≥18 years) emergency medical admissions discharged between 11/11/2020 and 11/11/2022 from The Rotherham Foundation Trust (TRFT), UK. We assessed the predictive performance of the CARSS model based on its discriminative (c-statistic) and calibration characteristics (calibration slope and calibration plots). RESULTS: Out of 32,774 admissions, 20,422 (62.3 %) admissions were included. The TRFT sample had similar demographic characteristics to the development sample but had higher mortality (6.1 % versus 5.7 %). The CARSS model demonstrated good discrimination (c-statistic 0.87 [95 % CI 0.86-0.88]) and good calibration to the TRFT dataset (slope = 1.03 [95 % CI 0.98-1.08] intercept = 0 [95 % CI -0.06-0.07]) after re-calibrating for differences in baseline mortality (intercept = 0.96 [95 % CI 0.90-1.03] before re-calibration). CONCLUSION: In summary, the CARSS model is externally validated after correcting the baseline risk of death between development and validation datasets. External validation of the CARSS model showed that it under-predicted in-hospital mortality. Re-calibration of this model showed adequate performance in the TRFT dataset.