Distilling large language models for matching patients to clinical trials.

OBJECTIVE: The objective of this study is to systematically examine the efficacy of both proprietary (GPT-3.5, GPT-4) and open-source large language models (LLMs) (LLAMA 7B, 13B, 70B) in the context of matching patients to clinical trials in healthcare. MATERIALS AND METHODS: The study employs a multifaceted evaluation framework, incorporating extensive automated and human-centric assessments along with a detailed error analysis for each model, and assesses LLMs' capabilities in analyzing patient eligibility against clinical trial's inclusion and exclusion criteria. To improve the adaptability of open-source LLMs, a specialized synthetic dataset was created using GPT-4, facilitating effective fine-tuning under constrained data conditions. RESULTS: The findings indicate that open-source LLMs, when fine-tuned on this limited and synthetic dataset, achieve performance parity with their proprietary counterparts, such as GPT-3.5. DISCUSSION: This study highlights the recent success of LLMs in the high-stakes domain of healthcare, specifically in patient-trial matching. The research demonstrates the potential of open-source models to match the performance of proprietary models when fine-tuned appropriately, addressing challenges like cost, privacy, and reproducibility concerns associated with closed-source proprietary LLMs. CONCLUSION: The study underscores the opportunity for open-source LLMs in patient-trial matching. To encourage further research and applications in this field, the annotated evaluation dataset and the fine-tuned LLM, Trial-LLAMA, are released for public use.

Evaluation of an artificial intelligence-based clinical trial matching system in Chinese patients with hepatocellular carcinoma: a retrospective study.

BACKGROUND: Artificial intelligence (AI)-assisted clinical trial screening is a promising prospect, although previous matching systems were developed in English, and relevant studies have only been conducted in Western countries. Therefore, we evaluated an AI-based clinical trial matching system (CTMS) that extracts medical data from the electronic health record system and matches them to clinical trials automatically. METHODS: This study included 1,053 consecutive inpatients primarily diagnosed with hepatocellular carcinoma who were referred to the liver tumor center of an academic medical center in China between January and December 2019. The eligibility criteria extracted from two clinical trials, patient attributes, and gold standard were decided manually. We evaluated the performance of the CTMS against the established gold standard by measuring the accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and run time required. RESULTS: The manual reviewers demonstrated acceptable interrater reliability (Cohen's kappa 0.65-0.88). The performance results for the CTMS were as follows: accuracy, 92.9-98.0%; sensitivity, 51.9-83.5%; specificity, 99.0-99.1%; PPV, 75.7-85.1%; and NPV, 97.4-98.9%. The time required for eligibility determination by the CTMS and manual reviewers was 2 and 150 h, respectively. CONCLUSIONS: We found that the CTMS is particularly reliable in excluding ineligible patients in a significantly reduced amount of time. The CTMS excluded ineligible patients for clinical trials with good performance, reducing 98.7% of the work time. Thus, such AI-based systems with natural language processing and machine learning have potential utility in Chinese clinical trials.

Consensus recommendations for improving the cancer clinical trial matching environment.

Enrollment in cancer clinical trials cannot occur without first successfully identifying trials for which patients are a match based on their clinical characteristics. A lack of onsite matching trials has been identified as the single largest barrier preventing patients from participating in clinical trials. The site-agnostic cancer clinical trial matching environment is a mix of public and private tools and infrastructure that are not designed to work together to facilitate trial matching in an efficient manner. To identify policy and infrastructure solutions that could enable more effective and more frequent use of third-party site-agnostic matching, the American Cancer Society Cancer Action Network held a summit to examine challenges and propose consensus recommendations that could address those challenges. At this 2019 summit, stakeholders in this field examined these barriers and challenges and made a number of policy and infrastructure recommendations to improve the ability of this environment to work in a more coordinated and efficient manner.

Provider motivations and barriers to cancer clinical trial screening, referral, and operations: Findings from a survey.

BACKGROUND: Provider and institutional practices have been shown to have a large impact on cancer clinical trial enrollment. Understanding provider perspectives on screening for trial eligibility is necessary to improve enrollment. METHODS: A questionnaire about incentives, barriers, process tools, and infrastructure related to opening trials and referring patients to onsite and offsite trials was administered to diverse stakeholders, including professional societies, advocacy organizations, and industry networks. Descriptive statistics were used to summarize findings. RESULTS: Overall, 693 responses were received, primarily from physicians (42.7%) and nurses (35.6%) employed at hospital health systems (43.7%) and academic centers (36.5%). Approximately half (49.2%) screened all patients for onsite clinical trials with screening typically done by manual chart review (81.9%). The greatest incentive reported for offering trials was providing the best treatment options for patients (67.7%). Contracting and paperwork (48.5%) were the greatest barriers to opening more onsite trials. Offsite referrals were rare. CONCLUSIONS: Screening for trial eligibility is a largely manual and ad hoc process, with screening and referral to offsite trials occurring infrequently. Administrative and infrastructure barriers commonly prevent sites from opening more onsite trials. These findings suggest that automated trial screening tools built into workflows that screen in a site-agnostic manner could result in more frequent trial eligibility screening, especially for offsite trials. With recent momentum, in part in response to the COVID-19 pandemic, to improve clinical trial efficiencies and broaden access and participant diversity, implementing tools to improve screening and referral processes is timely and essential. PLAIN LANGUAGE SUMMARY: There are many factors that contribute to low adult enrollment in cancer clinical trials, but previous research has indicated that provider and institutional barriers are the largest contributors to low cancer clinical trial enrollment. In this survey, we sought to gain insight into cancer clinical trial enrollment practices from the perspective of health care providers such as physicians and nurses. We found that only approximately half of respondents indicated their institution systematically screens their patients for clinical trials and this process is manual and time consuming. Furthermore, we found that providers infrequently search for and refer patients to clinical trials at other sites. Creating better screening methods could improve enrollment in clinical trials.

Facilitating increased participation in clinical trials: what do consumers expect of clinical trial matching websites?

Clinical trials offer access to novel therapies and potential major benefits for patients, but identifying and accessing suitable trials remains a significant challenge for consumers. A burgeoning range of online services aims to meet this need; however, there is a paucity of data on whether these services are addressing the requirements and concerns of consumers. Here, we report our findings from a survey of cancer consumers, with results we believe are relevant to the broader research community.

Review of the Performance Metrics for Natural Language Systems for Clinical Trials Matching.

Natural Language Processing (NLP) has been adopted widely in clinical trial matching for its ability to process unstructured text that is often found in electronic health records. Despite the rise in the new tools that use NLP to match patients to eligible clinical trials, the comparison of these tools is difficult due to the lack of consistency in how these tools are evaluated. The ground truth or reference that the tools use to assess results varies, making it difficult to compare the robustness of the tools against each other. This paper alarms the lack of definition and consistency of ground truth data used to evaluate such tools and suggests two ways to define a gold standard for the ground truth in small and large-scale studies.

A systematic review on natural language processing systems for eligibility prescreening in clinical research.

OBJECTIVE: We conducted a systematic review to assess the effect of natural language processing (NLP) systems in improving the accuracy and efficiency of eligibility prescreening during the clinical research recruitment process. MATERIALS AND METHODS: Guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) standards of quality for reporting systematic reviews, a protocol for study eligibility was developed a priori and registered in the PROSPERO database. Using predetermined inclusion criteria, studies published from database inception through February 2021 were identified from 5 databases. The Joanna Briggs Institute Critical Appraisal Checklist for Quasi-experimental Studies was adapted to determine the study quality and the risk of bias of the included articles. RESULTS: Eleven studies representing 8 unique NLP systems met the inclusion criteria. These studies demonstrated moderate study quality and exhibited heterogeneity in the study design, setting, and intervention type. All 11 studies evaluated the NLP system's performance for identifying eligible participants; 7 studies evaluated the system's impact on time efficiency; 4 studies evaluated the system's impact on workload; and 2 studies evaluated the system's impact on recruitment. DISCUSSION: NLP systems in clinical research eligibility prescreening are an understudied but promising field that requires further research to assess its impact on real-world adoption. Future studies should be centered on continuing to develop and evaluate relevant NLP systems to improve enrollment into clinical studies. CONCLUSION: Understanding the role of NLP systems in improving eligibility prescreening is critical to the advancement of clinical research recruitment.

Accuracy of an Artificial Intelligence System for Cancer Clinical Trial Eligibility Screening: Retrospective Pilot Study.

BACKGROUND: Screening patients for eligibility for clinical trials is labor intensive. It requires abstraction of data elements from multiple components of the longitudinal health record and matching them to inclusion and exclusion criteria for each trial. Artificial intelligence (AI) systems have been developed to improve the efficiency and accuracy of this process. OBJECTIVE: This study aims to evaluate the ability of an AI clinical decision support system (CDSS) to identify eligible patients for a set of clinical trials. METHODS: This study included the deidentified data from a cohort of patients with breast cancer seen at the medical oncology clinic of an academic medical center between May and July 2017 and assessed patient eligibility for 4 breast cancer clinical trials. CDSS eligibility screening performance was validated against manual screening. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for eligibility determinations were calculated. Disagreements between manual screeners and the CDSS were examined to identify sources of discrepancies. Interrater reliability between manual reviewers was analyzed using Cohen (pairwise) and Fleiss (three-way) κ, and the significance of differences was determined by Wilcoxon signed-rank test. RESULTS: In total, 318 patients with breast cancer were included. Interrater reliability for manual screening ranged from 0.60-0.77, indicating substantial agreement. The overall accuracy of breast cancer trial eligibility determinations by the CDSS was 87.6%. CDSS sensitivity was 81.1% and specificity was 89%. CONCLUSIONS: The AI CDSS in this study demonstrated accuracy, sensitivity, and specificity of greater than 80% in determining the eligibility of patients for breast cancer clinical trials. CDSSs can accurately exclude ineligible patients for clinical trials and offer the potential to increase screening efficiency and accuracy. Additional research is needed to explore whether increased efficiency in screening and trial matching translates to improvements in trial enrollment, accruals, feasibility assessments, and cost.

Evaluation of an artificial intelligence clinical trial matching system in Australian lung cancer patients.

OBJECTIVE: The objective of this technical study was to evaluate the performance of an artificial intelligence (AI)-based system for clinical trials matching for a cohort of lung cancer patients in an Australian cancer hospital. METHODS: A lung cancer cohort was derived from clinical data from patients attending an Australian cancer hospital. Ten phases I-III clinical trials registered on clinicaltrials.gov and open to lung cancer patients at this institution were utilized for assessments. The trial matching system performance was compared to a gold standard established by clinician consensus for trial eligibility. RESULTS: The study included 102 lung cancer patients. The trial matching system evaluated 7252 patient attributes (per patient median 74, range 53-100) against 11 467 individual trial eligibility criteria (per trial median 597, range 243-4132). Median time for the system to run a query and return results was 15.5 s (range 7.2-37.8). In establishing the gold standard, clinician interrater agreement was high (Cohen's kappa 0.70-1.00). On a per-patient basis, the performance of the trial matching system for eligibility was as follows: accuracy, 91.6%; recall (sensitivity), 83.3%; precision (positive predictive value), 76.5%; negative predictive value, 95.7%; and specificity, 93.8%. DISCUSSION AND CONCLUSION: The AI-based clinical trial matching system allows efficient and reliable screening of cancer patients for clinical trials with 95.7% accuracy for exclusion and 91.6% accuracy for overall eligibility assessment; however, clinician input and oversight are still required. The automated system demonstrates promise as a clinical decision support tool to prescreen a large patient cohort to identify subjects suitable for further assessment.

The creation of an integrated health-information platform: building the framework to support personalized medicine.

To advance medicine toward better evidence-based, cost-effective and individualized treatment, a new model of discovery, translation and delivery of information must be developed. This requires the collaboration of the major constituents in the areas of healthcare (i.e., patients, clinicians, administrators and researchers) and the partnership of disciplines (i.e., bioinformatics, epidemiologists, statisticians, information technologists, physicians and scientists) and organizations (i.e., drug and device companies, healthcare agencies, academic and community medical centers and information technology firms) to develop an integrative platform. The over-riding goal of this platform is to improve patient care, with the developed system enabling this by providing each of the major constituents with evidence-based and tailored information at the individual patient level.