Retrieval Augmented Generation Enabled Generative Pre-Trained Transformer 4 (GPT-4) Performance for Clinical Trial Screening.

Background: Subject screening is a key aspect of all clinical trials; however, traditionally, it is a labor-intensive and error-prone task, demanding significant time and resources. With the advent of large language models (LLMs) and related technologies, a paradigm shift in natural language processing capabilities offers a promising avenue for increasing both quality and efficiency of screening efforts. This study aimed to test the Retrieval-Augmented Generation (RAG) process enabled Generative Pretrained Transformer Version 4 (GPT-4) to accurately identify and report on inclusion and exclusion criteria for a clinical trial. Methods: The Co-Operative Program for Implementation of Optimal Therapy in Heart Failure (COPILOT-HF) trial aims to recruit patients with symptomatic heart failure. As part of the screening process, a list of potentially eligible patients is created through an electronic health record (EHR) query. Currently, structured data in the EHR can only be used to determine 5 out of 6 inclusion and 5 out of 17 exclusion criteria. Trained, but non-licensed, study staff complete manual chart review to determine patient eligibility and record their assessment of the inclusion and exclusion criteria. We obtained the structured assessments completed by the study staff and clinical notes for the past two years and developed a workflow of clinical note-based question answering system powered by RAG architecture and GPT-4 that we named RECTIFIER (RAG-Enabled Clinical Trial Infrastructure for Inclusion Exclusion Review). We used notes from 100 patients as a development dataset, 282 patients as a validation dataset, and 1894 patients as a test set. An expert clinician completed a blinded review of patients' charts to answer the eligibility questions and determine the "gold standard" answers. We calculated the sensitivity, specificity, accuracy, and Matthews correlation coefficient (MCC) for each question and screening method. We also performed bootstrapping to calculate the confidence intervals for each statistic. Results: Both RECTIFIER and study staff answers closely aligned with the expert clinician answers across criteria with accuracy ranging between 97.9% and 100% (MCC 0.837 and 1) for RECTIFIER and 91.7% and 100% (MCC 0.644 and 1) for study staff. RECTIFIER performed better than study staff to determine the inclusion criteria of "symptomatic heart failure" with an accuracy of 97.9% vs 91.7% and an MCC of 0.924 vs 0.721, respectively. Overall, the sensitivity and specificity of determining eligibility for the RECTIFIER was 92.3% (CI) and 93.9% (CI), and study staff was 90.1% (CI) and 83.6% (CI), respectively. Conclusion: GPT-4 based solutions have the potential to improve efficiency and reduce costs in clinical trial screening. When incorporating new tools such as RECTIFIER, it is important to consider the potential hazards of automating the screening process and set up appropriate mitigation strategies such as final clinician review before patient engagement.

Results of a Remotely Delivered Hypertension and Lipid Program in More Than 10 000 Patients Across a Diverse Health Care Network.

Importance: Blood pressure (BP) and cholesterol control remain challenging. Remote care can deliver more effective care outside of traditional clinician-patient settings but scaling and ensuring access to care among diverse populations remains elusive. Objective: To implement and evaluate a remote hypertension and cholesterol management program across a diverse health care network. Design, Setting, and Participants: Between January 2018 and July 2021, 20 454 patients in a large integrated health network were screened; 18 444 were approached, and 10 803 were enrolled in a comprehensive remote hypertension and cholesterol program (3658 patients with hypertension, 8103 patients with cholesterol, and 958 patients with both). A total of 1266 patients requested education only without medication titration. Enrolled patients received education, home BP device integration, and medication titration. Nonlicensed navigators and pharmacists, supported by cardiovascular clinicians, coordinated care using standardized algorithms, task management and automation software, and omnichannel communication. BP and laboratory test results were actively monitored. Main Outcomes and Measures: Changes in BP and low-density lipoprotein cholesterol (LDL-C). Results: The mean (SD) age among 10 803 patients was 65 (11.4) years; 6009 participants (56%) were female; 1321 (12%) identified as Black, 1190 (11%) as Hispanic, 7758 (72%) as White, and 1727 (16%) as another or multiple races (including American Indian or Alaska Native, Asian, Native Hawaiian or Other Pacific Islander, unknown, other, and declined to respond; consolidated owing to small numbers); and 142 (11%) reported a preferred language other than English. A total of 424 482 BP readings and 139 263 laboratory reports were collected. In the hypertension program, the mean (SD) office BP prior to enrollment was 150/83 (18/10) mm Hg, and the mean (SD) home BP was 145/83 (20/12) mm Hg. For those engaged in remote medication management, the mean (SD) clinic BP 6 and 12 months after enrollment decreased by 8.7/3.8 (21.4/12.4) and 9.7/5.2 (22.2/12.6) mm Hg, respectively. In the education-only cohort, BP changed by a mean (SD) -1.5/-0.7 (23.0/11.1) and by +0.2/-1.9 (30.3/11.2) mm Hg, respectively (P < .001 for between cohort difference). In the lipids program, patients in remote medication management experienced a reduction in LDL-C by a mean (SD) 35.4 (43.1) and 37.5 (43.9) mg/dL at 6 and 12 months, respectively, while the education-only cohort experienced a mean (SD) reduction in LDL-C of 9.3 (34.3) and 10.2 (35.5) mg/dL at 6 and 12 months, respectively (P < .001). Similar rates of enrollment and reductions in BP and lipids were observed across different racial, ethnic, and primary language groups. Conclusions and Relevance: The results of this study indicate that a standardized remote BP and cholesterol management program may help optimize guideline-directed therapy at scale, reduce cardiovascular risk, and minimize the need for in-person visits among diverse populations.

Workflow Automation for a Virtual Hypertension Management Program.

OBJECTIVES: Hypertension is a modifiable risk factor for numerous comorbidities and treating hypertension can greatly improve health outcomes. We sought to increase the efficiency of a virtual hypertension management program through workflow automation processes. METHODS: We developed a customer relationship management (CRM) solution at our institution for the purpose of improving processes and workflow for a virtual hypertension management program and describe here the development, implementation, and initial experience of this CRM system. RESULTS: Notable system features include task automation, patient data capture, multi-channel communication, integration with our electronic health record (EHR), and device integration (for blood pressure cuffs). In the five stages of our program (intake and eligibility screening, enrollment, device configuration/setup, medication titration, and maintenance), we describe some of the key process improvements and workflow automations that are enabled using our CRM platform, like automatic reminders to capture blood pressure data and present these data to our clinical team when ready for clinical decision making. We also describe key limitations of CRM, like balancing out-of-the-box functionality with development flexibility. Among our first group of referred patients, 76% (39/51) preferred email as their communication method, 26/51 (51%) were able to enroll electronically, and 63% of those enrolled (32/51) were able to transmit blood pressure data without phone support. CONCLUSION: A CRM platform could improve clinical processes through multiple pathways, including workflow automation, multi-channel communication, and device integration. Future work will examine the operational improvements of this health information technology solution as well as assess clinical outcomes.