Performance of Phenotype Algorithms for the Identification of Opioid-Exposed Infants.

OBJECTIVE: Observational studies examining outcomes among opioid-exposed infants are limited by phenotype algorithms that may under identify opioid-exposed infants without neonatal opioid withdrawal syndrome (NOWS). We developed and validated the performance of different phenotype algorithms to identify opioid-exposed infants using electronic health record data. METHODS: We developed phenotype algorithms for the identification of opioid-exposed infants among a population of birthing person-infant dyads from an academic health care system (2010-2022). We derived phenotype algorithms from combinations of 6 unique indicators of in utero opioid exposure, including those from the infant record (NOWS or opioid-exposure diagnosis, positive toxicology) and birthing person record (opioid use disorder diagnosis, opioid drug exposure record, opioid listed on medication reconciliation, positive toxicology). We determined the positive predictive value (PPV) and 95% confidence interval for each phenotype algorithm using medical record review as the gold standard. RESULTS: Among 41 047 dyads meeting exclusion criteria, we identified 1558 infants (3.80%) with evidence of at least 1 indicator for opioid exposure and 32 (0.08%) meeting all 6 indicators of the phenotype algorithm. Among the sample of dyads randomly selected for review (n = 600), the PPV for the phenotype requiring only a single indicator was 95.4% (confidence interval: 93.3-96.8) with varying PPVs for the other phenotype algorithms derived from a combination of infant and birthing person indicators (PPV range: 95.4-100.0). CONCLUSIONS: Opioid-exposed infants can be accurately identified using electronic health record data. Our publicly available phenotype algorithms can be used to conduct research examining outcomes among opioid-exposed infants with and without NOWS.

Large language models facilitate the generation of electronic health record phenotyping algorithms.

OBJECTIVES: Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts. MATERIALS AND METHODS: We prompted four LLMs-GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard-in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (ie, type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network. RESULTS: GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values). CONCLUSION: GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms.

Objectives: Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts. Materials and Methods: We prompted four LLMs-GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard-in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (i.e., type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network. Results: GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values). Conclusion: GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.

A User-Centered Evaluation of a COVID-19 Intelligent Query System (COVID-IQS).

As the fight against COVID-19 continues, it is critical to discover and accumulate knowledge in scientific literature to combat the pandemic. In this work, we shared the experience in developing an intelligent query system on COVID-19 literature. We conducted a user-centered evaluation with 12 researchers in our institution and identified usability issues in four categories: distinct user needs, functionality errors, suboptimal information display, and implementation errors. Furthermore, we shared two lessons for building such a COVID-19 literature search engine. We will deploy the system and continue refining it through multiple phases of evaluation to aid in redesigning the system to accommodate different user roles as well as enhancing repository features to support collaborative information seeking. The successful implementation of the COVID-IQS can support knowledge discovery and hypothesis generation in our institution and can be shared with other institutions to make a broader impact.

Assessing the Readability of App Descriptions and Investigating its Role in the Choice of mHealth Apps: Retrospective and Prospective Analyses.

People with low health literacy are more likely to use mobile apps for health information. The choice of mHealth apps can affect health behaviors and outcomes. However, app descriptions may not be very readable to the target users, which can negatively impact app adoption and utilization. In this study, we assessed the readability of mHealth app descriptions and explored the relationship between description readability and other app metadata, as well as description writing styles. The results showed that app descriptions were at eleventh- to fifteenth-grade level, with only 6% of them meeting the readability recommendation (third- to seventh-grade level). The description readability played a vital role in predicting app installs when an app had no reviews. The content analysis showed copy-paste behaviors and identified two potential causes for low readability. More work is needed to improve the readability of app descriptions and optimize mHealth app adoption and utilization.

Assessing the Readability of Freely Available ICU Notes.

Unstructured data stored in an electronic health record (EHR) system can be very informative but require techniques such as natural language processing to extract the information. Developing such techniques requires shared data, but clinical data are often not easy to access. A freely available intensive care unit database, MIMIC-III, was released in 2016 to address this issue and benefit the informatics research community. While the database has been utilized by a few studies, the text characteristics of the notes have not been summarized. In this study, we present the summary of the basic text characteristics and the readability of the MIMIC-III ICU notes. We further compare the results with our previous study where proprietary EHR notes were used. The results show that the text characteristics of MIMIC-III notes were comparable with proprietary EHR notes, although the note readability index was slightly lower. The clinical notes in MIMIC-III can be a viable option for researchers who are interested in clinicians' language use but have no access to proprietary EHR systems.

A CMMI-based approach for medical software project life cycle study.

In terms of medical techniques, Taiwan has gained international recognition in recent years. However, the medical information system industry in Taiwan is still at a developing stage compared with the software industries in other nations. In addition, systematic development processes are indispensable elements of software development. They can help developers increase their productivity and efficiency and also avoid unnecessary risks arising during the development process. Thus, this paper presents an application of Light-Weight Capability Maturity Model Integration (LW-CMMI) to Chang Gung Medical Research Project (CMRP) in the Nuclear medicine field. This application was intended to integrate user requirements, system design and testing of software development processes into three layers (Domain, Concept and Instance) model. Then, expressing in structural System Modeling Language (SysML) diagrams and converts part of the manual effort necessary for project management maintenance into computational effort, for example: (semi-) automatic delivery of traceability management. In this application, it supports establishing artifacts of "requirement specification document", "project execution plan document", "system design document" and "system test document", and can deliver a prototype of lightweight project management tool on the Nuclear Medicine software project. The results of this application can be a reference for other medical institutions in developing medical information systems and support of project management to achieve the aim of patient safety.