Evaluation of guided reporting: quality and reading time of automatically generated radiology report in breast magnetic resonance imaging using a dedicated software solution.

PURPOSE: Unstructured, free-text dictation (FT), the current standard in breast magnetic resonance imaging (MRI) reporting, is considered time-consuming and prone to error. The purpose of this study is to assess the usability and performance of a novel, software-based guided reporting (GR) strategy in breast MRI. METHODS: Eighty examinations previously evaluated for a clinical indication (e.g., mass and focus/non-mass enhancement) with FT were reevaluated by three specialized radiologists using GR. Each radiologist had a different number of cases (R1, n = 24; R2, n = 20; R3, n = 36). Usability was assessed by subjective feedback, and quality was assessed by comparing the completeness of automatically generated GR reports with that of their FT counterparts. Errors in GR were categorized and analyzed for debugging with a final software version. Combined reading and reporting times and learning curves were analyzed. RESULTS: Usability was rated high by all readers. No non-sense, omission/commission, or translational errors were detected with the GR method. Spelling and grammar errors were observed in 3/80 patient reports (3.8%) with GR (exclusively in the discussion section) and in 36/80 patient reports (45%) with FT. Between FT and GR, 41 patient reports revealed no content differences, 33 revealed minor differences, and 6 revealed major differences that resulted in changes in treatment. The errors in all patient reports with major content differences were categorized as content omission errors caused by improper software operation (n = 2) or by missing content in software v. 0.8 displayable with v. 1.7 (n = 4). The mean combined reading and reporting time was 576 s (standard deviation: 327 s; min: 155 s; max: 1,517 s). The mean times for each reader were 485, 557, and 754 s, and the respective learning curves evaluated by regression models revealed statistically significant slopes (P = 0.002; P = 0.0002; P < 0.0001). Overall times were shorter compared with external references that used FT. The mean combined reading and reporting time of MRI examinations using FT was 1,043 s and decreased by 44.8% with GR. CONCLUSION: GR allows for complete reporting with minimized error rates and reduced combined reading and reporting times. The streamlining of the process (evidenced by lower reading times) for the readers in this study proves that GR can be learned quickly. Reducing reporting errors leads to fewer therapeutic faults and lawsuits against radiologists. It is known that delays in radiology reporting hinder early treatment and lead to poorer patient outcomes. CLINICAL SIGNIFICANCE: While the number of scans and images per examination is continuously rising, staff shortages create a bottleneck in radiology departments. The IT-based GR method can be a major boon, improving radiologist efficiency, report quality, and the quality of simultaneously generated data.

The right care in the right place: a scoping review of digital health education and training for rural healthcare workers.

BACKGROUND: Digital health offers unprecedented opportunities to enhance health service delivery across vast geographic regions. However, these benefits can only be realized with effective capabilities and clinical leadership of the rural healthcare workforce. Little is known about how rural healthcare workers acquire skills in digital health, how digital health education or training programs are evaluated and the barriers and enablers for high quality digital health education and training. OBJECTIVE: To conduct a scoping review to identify and synthesize existing evidence on digital health education and training of the rural healthcare workforce. INCLUSION CRITERIA: Sources that reported digital health and education or training in the healthcare workforce in any healthcare setting outside metropolitan areas. METHODS: We searched for published and unpublished studies written in English in the last decade to August 2023. The databases searched were PubMed, Embase, Scopus, CINAHL and Education Resources Information Centre. We also searched the grey literature (Google, Google Scholar), conducted citation searching and stakeholder engagement. The JBI Scoping Review methodology and PRISMA guidelines for scoping reviews were used. RESULTS: Five articles met the eligibility criteria. Two case studies, one feasibility study, one micro-credential and one fellowship were described. The mode of delivery was commonly modular online learning. Only one article described an evaluation, and findings showed the train-the-trainer model was technically and pedagogically feasible and well received. A limited number of barriers and enablers for high quality education or training of the rural healthcare workforce were reported across macro (legal, regulatory, economic), meso (local health service and community) and micro (day-to-day practice) levels. CONCLUSIONS: Upskilling rural healthcare workers in digital health appears rare. Current best practice points to flexible, blended training programs that are suitably embedded with interdisciplinary and collaborative rural healthcare improvement initiatives. Future work to advance the field could define rural health informatician career pathways, address concurrent rural workforce issues, and conduct training implementation evaluations. REVIEW REGISTRATION NUMBER: Open Science Framework: https://doi.org/10.17605/OSF.IO/N2RMX .

Optimizing Response Rates to Examine Health IT Maturity and Nurse Practitioner Care Environments in US Nursing Homes: Mixed Mode Survey Recruitment Protocol.

BACKGROUND: Survey-driven research is a reliable method for large-scale data collection. Investigators incorporating mixed-mode survey designs report benefits for survey research including greater engagement, improved survey access, and higher response rate. Mix-mode survey designs combine 2 or more modes for data collection including web, phone, face-to-face, and mail. Types of mixed-mode survey designs include simultaneous (ie, concurrent), sequential, delayed concurrent, and adaptive. This paper describes a research protocol using mixed-mode survey designs to explore health IT (HIT) maturity and care environments reported by administrators and nurse practitioners (NPs), respectively, in US nursing homes (NHs). OBJECTIVE: The aim of this study is to describe a research protocol using mixed-mode survey designs in research using 2 survey tools to explore HIT maturity and NP care environments in US NHs. METHODS: We are conducting a national survey of 1400 NH administrators and NPs. Two data sets (ie, Care Compare and IQVIA) were used to identify eligible facilities at random. The protocol incorporates 2 surveys to explore how HIT maturity (survey 1 collected by administrators) impacts care environments where NPs work (survey 2 collected by NPs). Higher HIT maturity collected by administrators indicates greater IT capabilities, use, and integration in resident care, clinical support, and administrative activities. The NP care environment survey measures relationships, independent practice, resource availability, and visibility. The research team conducted 3 iterative focus groups, including 14 clinicians (NP and NH experts) and recruiters from 2 national survey teams experienced with these populations to achieve consensus on which mixed-mode designs to use. During focus groups we identified the pros and cons of using mixed-mode designs in these settings. We determined that 2 mixed-mode designs with regular follow-up calls (Delayed Concurrent Mode and Sequential Mode) is effective for recruiting NH administrators while a concurrent mixed-mode design is best to recruit NPs. RESULTS: Participant recruitment for the project began in June 2023. As of April 22, 2024, a total of 98 HIT maturity surveys and 81 NP surveys have been returned. Recruitment of NH administrators and NPs is anticipated through July 2025. About 71% of the HIT maturity surveys have been submitted using the electronic link and 23% were submitted after a QR code was sent to the administrator. Approximately 95% of the NP surveys were returned with electronic survey links. CONCLUSIONS: Pros of mixed-mode designs for NH research identified by the team were that delayed concurrent, concurrent, and sequential mixed-mode methods of delivering surveys to potential participants save on recruitment time compared to single mode delivery methods. One disadvantage of single-mode strategies is decreased versatility and adaptability to different organizational capabilities (eg, access to email and firewalls), which could reduce response rates. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/56170.

Adapting a Patient-Centered Tuberculosis Digital Adherence Technology Intervention for New Patient Populations.

Tuberculosis (TB) intervention adaptation strategies can be optimized to inform digital health intervention refinement. With experience we improved our strategies during the refinement of tools to support individuals with active TB.

Muzima Mobile Application for Screening Hypertension and Diabetes: A User Experience of the App Among Community Health Workers in Rwanda.

Mobile technology has become the leading utility in the social and well-being of people especially in low-resource settings. The use of mobile applications in healthcare promise to improve care and treatment. This study explored the user experience of muzima mobile application among community health workers in Rwanda. We used three data collection methods: observation, Key informant interviews and focus group discussions. We analysed data using thematic content analysis. We found that users were able to complete tasks in the app although some less experienced and older participants struggled to complete the tasks. Users felt that the application helped them to screen and manage patients with diabetes and hypertension in the community which reduced frequent visits to the health centers. Users felt that the application needs improvements in the workflow to facilitate the ease of use. They suggested to digitse other health programs implemented by community health workers. To improve the use and ensure wider implementation, there is a need to consider users' needs and concerns as discussed in this paper.

Predictive Models for Long Term Survival of AML Patients Treated with Venetoclax and Azacitidine or 7+3 Based on Post Treatment Events and Responses: Retrospective Cohort Study.

BACKGROUND: The treatment of acute myeloid leukemia (AML) in older or unfit patients typically involves a regimen of venetoclax plus azacitidine (ven/aza). Toxicity and treatment responses are highly variable following treatment initiation and clinical decision-making continually evolves in response to these as treatment progresses. To improve clinical decision support (CDS) following treatment initiation, predictive models based on evolving and dynamic toxicities, disease responses, and other features should be developed. OBJECTIVE: This study aims to generate machine learning (ML)-based predictive models that incorporate individual predictors of overall survival (OS) for patients with AML, based on clinical events occurring after the initiation of ven/aza or 7+3 regimen. METHODS: Data from 221 patients with AML, who received either the ven/aza (n=101 patients) or 7+3 regimen (n=120 patients) as their initial induction therapy, were retrospectively analyzed. We performed stratified univariate and multivariate analyses to quantify the association between toxicities, hospital events, and short-term disease responses and OS for the 7+3 and ven/aza subgroups separately. We compared the estimates of confounders to assess potential effect modifications by treatment. 17 ML-based predictive models were developed. The optimal predictive models were selected based on their predictability and discriminability using cross-validation. Uncertainty in the estimation was assessed through bootstrapping. RESULTS: The cumulative incidence of posttreatment toxicities varies between the ven/aza and 7+3 regimen. A variety of laboratory features and clinical events during the first 30 days were differentially associated with OS for the two treatments. An initial transfer to intensive care unit (ICU) worsened OS for 7+3 patients (aHR 1.18, 95% CI 1.10-1.28), while ICU readmission adversely affected OS for those on ven/aza (aHR 1.24, 95% CI 1.12-1.37). At the initial follow-up, achieving a morphologic leukemia free state (MLFS) did not affect OS for ven/aza (aHR 0.99, 95% CI 0.94-1.05), but worsened OS following 7+3 (aHR 1.16, 95% CI 1.01-1.31) compared to that of complete remission (CR). Having blasts over 5% at the initial follow-up negatively impacted OS for both 7+3 (P<.001) and ven/aza (P<.001) treated patients. A best response of CR and CR with incomplete recovery (CRi) was superior to MLFS and refractory disease after ven/aza (P<.001), whereas for 7+3, CR was superior to CRi, MLFS, and refractory disease (P<.001), indicating unequal outcomes. Treatment-specific predictive models, trained on 120 7+3 and 101 ven/aza patients using over 114 features, achieved survival AUCs over 0.70. CONCLUSIONS: Our findings indicate that toxicities, clinical events, and responses evolve differently in patients receiving ven/aza compared with that of 7+3 regimen. ML-based predictive models were shown to be a feasible strategy for CDS in both forms of AML treatment. If validated with larger and more diverse data sets, these findings could offer valuable insights for developing AML-CDS tools that leverage posttreatment clinical data.

Machine learning-based Cerebral Venous Thrombosis diagnosis with clinical data.

OBJECTIVES: Cerebral Venous Thrombosis (CVT) poses diagnostic challenges due to the variability in disease course and symptoms. The prognosis of CVT relies on early diagnosis. Our study focuses on developing a machine learning-based screening algorithm using clinical data from a large neurology referral center in southern Iran. METHODS: The Iran Cerebral Venous Thrombosis Registry (ICVTR code: 9001013381) provided data on 382 CVT cases from Namazi Hospital. The control group comprised of adult headache patients without CVT as confirmed by neuroimaging and was retrospectively selected from those admitted to the same hospital. We collected 60 clinical and demographic features for model development and validation. Our modeling pipeline involved imputing missing values and evaluating four machine learning algorithms: generalized linear model, random forest, support vector machine, and extreme gradient boosting. RESULTS: A total of 314 CVT cases and 575 controls were included. The highest AUROC was reached when imputation was used to estimate missing values for all the variables, combined with the support vector machine model (AUROC = 0.910, Recall = 0.73, Precision = 0.88). The best recall was achieved also by the support vector machine model when only variables with less than 50 % missing rate were included (AUROC = 0.887, Recall = 0.77, Precision = 0.86). The random forest model yielded the best precision by using variables with less than 50 % missing rate (AUROC = 0.882, Recall = 0.61, Precision = 0.94). CONCLUSION: The application of machine learning techniques using clinical data showed promising results in accurately diagnosing CVT within our study population. This approach offers a valuable complementary assistive tool or an alternative to resource-intensive imaging methods.

A randomized, embedded, pragmatic, Bayesian clinical trial examining clinical decision support for high flow nasal cannula management in children with bronchiolitis: design and statistical analysis plan.

BACKGROUND: High flow nasal cannula (HFNC) has been increasingly adopted in the past 2 decades as a mode of respiratory support for children hospitalized with bronchiolitis. The growing use of HFNC despite a paucity of high-quality data regarding the therapy's efficacy has led to concerns about overutilization. We developed an electronic health record (EHR) embedded, quality improvement (QI) oriented clinical trial to determine whether standardized management of HFNC weaning guided by clinical decision support (CDS) results in a reduction in the duration of HFNC compared to usual care for children with bronchiolitis. METHODS: The design and summary of the statistical analysis plan for the REspiratory SupporT for Efficient and cost-Effective Care (REST EEC; "rest easy") trial are presented. The investigators hypothesize that CDS-coupled, standardized HFNC weaning will reduce the duration of HFNC, the trial's primary endpoint, for children with bronchiolitis compared to usual care. Data supporting trial design and eventual analyses are collected from the EHR and other real world data sources using existing informatics infrastructure and QI data sources. The trial workflow, including randomization and deployment of the intervention, is embedded within the EHR of a large children's hospital using existing vendor features. Trial simulations indicate that by assuming a true hazard ratio effect size of 1.27, equivalent to a 6-h reduction in the median duration of HFNC, and enrolling a maximum of 350 children, there will be a > 0.75 probability of declaring superiority (interim analysis posterior probability of intervention effect > 0.99 or final analysis posterior probability of intervention effect > 0.9) and a > 0.85 probability of declaring superiority or the CDS intervention showing promise (final analysis posterior probability of intervention effect > 0.8). Iterative plan-do-study-act cycles are used to monitor the trial and provide targeted education to the workforce. DISCUSSION: Through incorporation of the trial into usual care workflows, relying on QI tools and resources to support trial conduct, and relying on Bayesian inference to determine whether the intervention is superior to usual care, REST EEC is a learning health system intervention that blends health system operations with active evidence generation to optimize the use of HFNC and associated patient outcomes. TRIAL REGISTRATION: ClinicalTrials.gov NCT05909566. Registered on June 18, 2023.

The incremental design of a machine learning framework for medical records processing.

OBJECTIVES: This work presents the development and evaluation of coordn8, a web-based application that streamlines fax processing in outpatient clinics using a "human-in-the-loop" machine learning framework. We demonstrate the effectiveness of the platform at reducing fax processing time and producing accurate machine learning inferences across the tasks of patient identification, document classification, spam classification, and duplicate document detection. METHODS: We deployed coordn8 in 11 outpatient clinics and conducted a time savings analysis by observing users and measuring fax processing event logs. We used statistical methods to evaluate the machine learning components across different datasets to show generalizability. We conducted a time series analysis to show variations in model performance as new clinics were onboarded and to demonstrate our approach to mitigating model drift. RESULTS: Our observation analysis showed a mean reduction in individual fax processing time by 147.5 s, while our event log analysis of over 7000 faxes reinforced this finding. Document classification produced an accuracy of 81.6%, patient identification produced an accuracy of 83.7%, spam classification produced an accuracy of 98.4%, and duplicate document detection produced a precision of 81.0%. Retraining document classification increased accuracy by 10.2%. DISCUSSION: coordn8 significantly decreased fax-processing time and produced accurate machine learning inferences. Our human-in-the-loop framework facilitated the collection of high-quality data necessary for model training. Expanding to new clinics correlated with performance decline, which was mitigated through model retraining. CONCLUSION: Our framework for automating clinical tasks with machine learning offers a template for health systems looking to implement similar technologies.

Development and Optimization of Machine Learning Algorithms for Predicting In-hospital Patient Charges for Congestive Heart Failure Exacerbations, Chronic Obstructive Pulmonary Disease Exacerbations and Diabetic Ketoacidosis.

Background: Hospitalizations for exacerbations of congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD) and diabetic ketoacidosis (DKA) are costly in the United States. The purpose of this study was to predict in-hospital charges for each condition using machine learning (ML) models. Results: We conducted a retrospective cohort study on national discharge records of hospitalized adult patients from January 1st, 2016, to December 31st, 2019. We used numerous ML techniques to predict in-hospital total cost. We found that linear regression (LM), gradient boosting (GBM) and extreme gradient boosting (XGB) models had good predictive performance and were statistically equivalent, with training R-square values ranging from 0.49-0.95 for CHF, 0.56-0.95 for COPD, and 0.32-0.99 for DKA. We identified important key features driving costs, including patient age, length of stay, number of procedures. and elective/nonelective admission. Conclusions: ML methods may be used to accurately predict costs and identify drivers of high cost for COPD exacerbations, CHF exacerbations and DKA. Overall, our findings may inform future studies that seek to decrease the underlying high patient costs for these conditions.

Technology-Related Safety Event Analysis in Community Clinical Informatics: A Case Study.

This case study presents a process that was iteratively developed for clinical informaticians to identify, analyse, and respond to safety events related to health information technologies (HIT) in community care settings (This research was supported by the CIHR Health Systems Impact Fellowship Program. We would also like to thank Vancouver Coastal Health for their valuable contributions.). The goal was to build capacity within a clinical informatics team to integrate patient safety into their work and to help them recognize and respond to HIT-related safety events. The technology-related safety event analysis process that was ultimately developed included three key components: 1) an internal workflow to analyse voluntarily reported HIT-related safety events using a sociotechnical model, 2) safety huddles to amplify learnings from reviewed events, and 3) a cumulative analysis of all events over time to identify and respond to patterns. A systematic approach to quickly identify and understand HIT safety concerns enables informatics teams to proactively reduce risks and prevent harm.

Clinical Informatics in Pre-Registration Nurse Education.

This case study explores the pivotal role Clinical Informaticians in Nursing and Midwifery in Wales can have within pre-registration education. It underscores the necessity for nurses and midwives to adapt to digital transformations in healthcare delivery and discusses the potential digital career paths within the often-misunderstood domain of digital nursing. The initiative aimed to enhance awareness at both national and local levels, collaborating with educational institutions to incorporate digital education into pre-registration nursing programs. In partnership with the University of South Wales, sessions were tailored to the existing curriculum to highlight digital career opportunities and foster digital understanding among future nurses. The session design was aligned with course guidelines to emphasize the role of digital technology in quality improvement and leadership. Evaluations using interactive tools facilitated continuous improvement and provided insights, shaping the future of digital integration in nursing education.

Supporting Clinical Informatics Competency Development Among the Clinical Informatics Team at Providence Health Care.

Clinical informatics (CI) competencies are crucial for health care organizations to effectively use information communication technologies (ICTs) and deliver quality care. An interdisciplinary CI team can assist organizations with leveraging ICTs, but may also require support. This case study describes a peer-led knowledge translation project designed, delivered and implemented over two years by members of the CI team at Providence Health Care (PHC). The project included CI competencies assessment of CI team members, followed by tailored education for identified knowledge gaps. The Kirkpatrick evaluation model was used to assess three levels of learning among CI team members, including a satisfaction survey, pre-and post-cognitive retention of the education intervention using a validated tool for informatics specialists, and project partner feedback of CI team performance 12 weeks after education completion. This case study provides evidence-informed guidance on 'how to' implement peer-led, practice-based CI training for CI teams.

National rollout of a medication safety dashboard to improve testing for latent infections among biologic and targeted synthetic disease-modifying agent users within the Veterans Health Administration.

OBJECTIVE: To develop, deploy, and evaluate a national, electronic health record (EHR)-based dashboard to support safe prescribing of biologic and targeted synthetic disease-modifying agents (b/tsDMARDs) in the United States Veterans Affairs Healthcare System (VA). DATA SOURCES AND STUDY SETTING: We extracted and displayed hepatitis B (HBV), hepatitis C (HCV), and tuberculosis (TB) screening data from the EHR for users of b/tsDMARDs using PowerBI (Microsoft) and deployed the dashboard to VA facilities across the United States in 2022; we observed facilities for 44 weeks post-deployment. STUDY DESIGN: We examined the association between dashboard engagement by healthcare personnel and the percentage of patients with all screenings complete (HBV, HCV, and TB) at the facility level using an interrupted time series. Based on frequency of sessions, facilities were grouped into high- and low/none-engagement categories. We modeled changes in complete screening pre- and post-deployment of the dashboard. DATA COLLECTION METHODS: All VA facilities were eligible for inclusion; excluded facilities participated in design of the dashboard or had <20 patients receiving b/tsDMARDs. Session counts from facility personnel were captured using PowerBI audit log data. Outcomes were assessed weekly based on EHR data extracted via the dashboard itself. PRINCIPAL FINDINGS: Totally 117 facilities (serving a total of 41,224 Veterans prescribed b/tsDMARDs) were included. Before dashboard deployment, across all facilities, 61.5% of patients had all screenings complete, which improved to 66.3% over the course of the study period. The largest improvement (15 percentage points, 60.3%-75.3%) occurred among facilities with high engagement (post-intervention difference in outcome between high and low/none-engagement groups was 0.17 percentage points (pp) per week, 95% confidence interval (0.04 pp, 0.30 pp); p = 0.01). CONCLUSIONS: We observed significant improvements in screening for latent infections among facilities with high engagement with the dashboard, compared with those with fewer sessions.

Validating the Physician Documentation Quality Instrument for Intensive Care Unit-Ward Transfer Notes.

Background: Physician communication failures during transfers of patients from the intensive care unit (ICU) to the general ward are common and can lead to adverse events. Efforts to improve written handoffs during these transfers are increasingly prominent, but no instruments have been developed to assess the quality of physician ICU-ward transfer notes. Objective: To collect validity evidence for the modified nine-item Physician Documentation Quality Instrument (mPDQI-9) for assessing ICU-ward transfer note usefulness across several hospitals. Methods: Twenty-four physician raters independently used the mPDQI-9 to grade 12 notes collected from three academic hospitals. A priori, we excluded the "up-to-date" and "accurate" domains, because these could not be assessed without giving the rater access to the complete patient chart. Assessments therefore used the domains "thorough," "useful," "organized," "comprehensible," "succinct," "synthesized," and "consistent." Raters scored each domain on a Likert scale ranging from 1 (low) to 5 (high). The total mPDQI-9 was the sum of these domain scores. The primary outcome was the raters' perceived clinical utility of the notes, and the primary measures of interest were criterion validity (Spearman's ρ) and interrater reliability (intraclass correlation [ICC]). Results: Mean mPDQI-9 scores by note ranged from 19 (SD = 5.5) to 30 (SD = 4.2). Mean note ratings did not systematically differ by rater expertise (for interaction, P = 0.15). The proportion of raters perceiving each note as independently sufficient for patient care (the primary outcome) ranged from 33% to 100% across the set of notes. We found a moderately positive correlation between mPDQI-9 ratings and raters' overall assessments of each note's clinical utility (ρ = 0.48, P < 0.001). Interrater reliability was strong; the overall ICC was 0.89 (95% confidence interval [CI], 0.80-0.85), and ICCs were similar among reviewer groups. Finally, Cronbach's α was 0.87 (95% CI, 0.84-0.89), indicating good internal consistency. Conclusions: We report moderate validity evidence for the mPDQI-9 to assess the usefulness of ICU-ward transfer notes written by internal medicine residents.

Identifying Precise Targets to Improve Child Mental Health Care Equity: Leveraging Advances in Clinical Research Informatics and Lived Experience.

To reduce child mental health disparities, it is imperative to improve the precision of targets and to expand our vision of social determinants of health as modifiable. Advancements in clinical research informatics and please state accurate measurement of child mental health service use and quality. Participatory action research promotes representation of underserved groups in informatics research and practice and may improve the effectiveness of interventions by informing research across all stages, including the identification of key variables, risk and protective factors, and data interpretation.

Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room.

BACKGROUND: Artificial intelligence (AI) and large language models (LLMs) can play a critical role in emergency room operations by augmenting decision-making about patient admission. However, there are no studies for LLMs using real-world data and scenarios, in comparison to and being informed by traditional supervised machine learning (ML) models. We evaluated the performance of GPT-4 for predicting patient admissions from emergency department (ED) visits. We compared performance to traditional ML models both naively and when informed by few-shot examples and/or numerical probabilities. METHODS: We conducted a retrospective study using electronic health records across 7 NYC hospitals. We trained Bio-Clinical-BERT and XGBoost (XGB) models on unstructured and structured data, respectively, and created an ensemble model reflecting ML performance. We then assessed GPT-4 capabilities in many scenarios: through Zero-shot, Few-shot with and without retrieval-augmented generation (RAG), and with and without ML numerical probabilities. RESULTS: The Ensemble ML model achieved an area under the receiver operating characteristic curve (AUC) of 0.88, an area under the precision-recall curve (AUPRC) of 0.72 and an accuracy of 82.9%. The naïve GPT-4's performance (0.79 AUC, 0.48 AUPRC, and 77.5% accuracy) showed substantial improvement when given limited, relevant data to learn from (ie, RAG) and underlying ML probabilities (0.87 AUC, 0.71 AUPRC, and 83.1% accuracy). Interestingly, RAG alone boosted performance to near peak levels (0.82 AUC, 0.56 AUPRC, and 81.3% accuracy). CONCLUSIONS: The naïve LLM had limited performance but showed significant improvement in predicting ED admissions when supplemented with real-world examples to learn from, particularly through RAG, and/or numerical probabilities from traditional ML models. Its peak performance, although slightly lower than the pure ML model, is noteworthy given its potential for providing reasoning behind predictions. Further refinement of LLMs with real-world data is necessary for successful integration as decision-support tools in care settings.

Integrating clinical research into electronic health record workflows to support a learning health system.

Objective: Integrating clinical research into routine clinical care workflows within electronic health record systems (EHRs) can be challenging, expensive, and labor-intensive. This case study presents a large-scale clinical research project conducted entirely within a commercial EHR during the COVID-19 pandemic. Case Report: The UCSD and UCSDH COVID-19 NeutraliZing Antibody Project (ZAP) aimed to evaluate antibody levels to SARS-CoV-2 virus in a large population at an academic medical center and examine the association between antibody levels and subsequent infection diagnosis. Results: The project rapidly and successfully enrolled and consented over 2000 participants, integrating the research trial with standing COVID-19 testing operations, staff, lab, and mobile applications. EHR-integration increased enrollment, ease of scheduling, survey distribution, and return of research results at a low cost by utilizing existing resources. Conclusion: The case study highlights the potential benefits of EHR-integrated clinical research, expanding their reach across multiple health systems and facilitating rapid learning during a global health crisis.

Early Warning Systems for Critical Illness Outside the Intensive Care Unit.

Early warning systems (EWSs) are designed and deployed to create a rapid assessment and response for patients with clinical deterioration outside the intensive care unit (ICU). These models incorporate patient-level data such as vital signs and laboratory values to detect or prevent adverse clinical events, such as vital signs and laboratories to allow detection and prevention of adverse clinical events such as cardiac arrest, intensive care transfer, or sepsis. The applicability, development, clinical utility, and general perception of EWS in clinical practice vary widely. Here, we review the field as it has grown from early vital sign-based scoring systems to contemporary multidimensional algorithms and predictive technologies for clinical decompensation outside the ICU.

Montelukast as a repurposable additive drug for standard-efficacy multiple sclerosis treatment: Emulating clinical trials with retrospective administrative health claims data.

BACKGROUND: Effective and safe treatment options for multiple sclerosis (MS) are still needed. Montelukast, a leukotriene receptor antagonist (LTRA) currently indicated for asthma or allergic rhinitis, may provide an additional therapeutic approach. OBJECTIVE: The study aimed to evaluate the effects of montelukast on the relapses of people with MS (pwMS). METHODS: In this retrospective case-control study, two independent longitudinal claims datasets were used to emulate randomized clinical trials (RCTs). We identified pwMS aged 18-65 years, on MS disease-modifying therapies concomitantly, in de-identified claims from Optum's Clinformatics® Data Mart (CDM) and IQVIA PharMetrics® Plus for Academics. Cases included 483 pwMS on montelukast and with medication adherence in CDM and 208 in PharMetrics Plus for Academics. We randomly sampled controls from 35,330 pwMS without montelukast prescriptions in CDM and 10,128 in PharMetrics Plus for Academics. Relapses were measured over a 2-year period through inpatient hospitalization and corticosteroid claims. A doubly robust causal inference model estimated the effects of montelukast, adjusting for confounders and censored patients. RESULTS: pwMS treated with montelukast demonstrated a statistically significant 23.6% reduction in relapses compared to non-users in 67.3% of emulated RCTs. CONCLUSION: Real-world evidence suggested that montelukast reduces MS relapses, warranting future clinical trials and further research on LTRAs' potential mechanism in MS.