Evaluating the validity of the nursing statements algorithmically generated based on the International Classifications of Nursing Practice for respiratory nursing care using large language models.

OBJECTIVE: This study aims to facilitate the creation of quality standardized nursing statements in South Korea's hospitals using algorithmic generation based on the International Classifications of Nursing Practice (ICNP) and evaluation through Large Language Models. MATERIALS AND METHODS: We algorithmically generated 15 972 statements related to acute respiratory care using 117 concepts and concept composition models of ICNP. Human reviewers, Generative Pre-trained Transformers 4.0 (GPT-4.0), and Bio_Clinical Bidirectional Encoder Representations from Transformers (BERT) evaluated the generated statements for validity. The evaluation by GPT-4.0 and Bio_ClinicalBERT was conducted with and without contextual information and training. RESULTS: Of the generated statements, 2207 were deemed valid by expert reviewers. GPT-4.0 showed a zero-shot  AUC of 0.857, which aggravated with contextual information. Bio_ClinicalBERT, after training, significantly improved, reaching an AUC of 0.998. CONCLUSION: Bio_ClinicalBERT effectively validates auto-generated nursing statements, offering a promising solution to enhance and streamline healthcare documentation processes.

Design considerations for a hierarchical semantic compositional framework for medical natural language understanding.

Medical natural language processing (NLP) systems are a key enabling technology for transforming Big Data from clinical report repositories to information used to support disease models and validate intervention methods. However, current medical NLP systems fall considerably short when faced with the task of logically interpreting clinical text. In this paper, we describe a framework inspired by mechanisms of human cognition in an attempt to jump the NLP performance curve. The design centers on a hierarchical semantic compositional model (HSCM), which provides an internal substrate for guiding the interpretation process. The paper describes insights from four key cognitive aspects: semantic memory, semantic composition, semantic activation, and hierarchical predictive coding. We discuss the design of a generative semantic model and an associated semantic parser used to transform a free-text sentence into a logical representation of its meaning. The paper discusses supportive and antagonistic arguments for the key features of the architecture as a long-term foundational framework.

Detecting diabetic retinopathy through machine learning on electronic health record data from an urban, safety net healthcare system.

OBJECTIVE: Clinical guidelines recommend annual eye examinations to detect diabetic retinopathy (DR) in patients with diabetes. However, timely DR detection remains a problem in medically underserved and under-resourced settings in the United States. Machine learning that identifies patients with latent/undiagnosed DR could help to address this problem. MATERIALS AND METHODS: Using electronic health record data from 40 631 unique diabetic patients seen at Los Angeles County Department of Health Services healthcare facilities between January 1, 2015 and December 31, 2017, we compared ten machine learning environments, including five classifier models, for assessing the presence or absence of DR. We also used data from a distinct set of 9300 diabetic patients seen between January 1, 2018 and December 31, 2018 as an external validation set. RESULTS: Following feature subset selection, the classifier with the best AUC on the external validation set was a deep neural network using majority class undersampling, with an AUC of 0.8, the sensitivity of 72.17%, and specificity of 74.2%. DISCUSSION: A deep neural network produced the best AUCs and sensitivity results on the test set and external validation set. Models are intended to be used to screen guideline noncompliant diabetic patients in an urban safety-net setting. CONCLUSION: Machine learning on diabetic patients' routinely collected clinical data could help clinicians in safety-net settings to identify and target unscreened diabetic patients who potentially have undiagnosed DR.

Ambiguity in Communicating Intensity of Physical Activity: Survey Study.

BACKGROUND: Communicating physical activity information with sufficient details, such as activity type, frequency, duration, and intensity, is vital to accurately delineate the attributes of physical activity that bring positive health impact. Unlike frequency and duration, intensity is a subjective concept that can be interpreted differently by people depending on demographics, health status, physical fitness, and exercise habits. However, activity intensity is often communicated using general degree modifiers, degree of physical exertion, and physical activity examples, which are the expressions that people may interpret differently. Lack of clarity in communicating the intensity level of physical activity is a potential barrier to an accurate assessment of exercise effect and effective imparting of exercise recommendations. OBJECTIVE: This study aimed to assess the variations in people's perceptions and interpretations of commonly used intensity descriptions of physical activities and to identify factors that may contribute to these variations. METHODS: A Web-based survey with a 25-item questionnaire was conducted using Amazon Mechanical Turk, targeting adults residing in the United States. The questionnaire included questions on participants' demographics, exercise habits, overall perceived health status, and perceived intensity of 10 physical activity examples. The survey responses were analyzed using the R statistical package. RESULTS: The analyses included 498 responses. The majority of respondents were females (276/498, 55.4%) and whites (399/498, 79.9%). Numeric ratings of physical exertion after exercise were relatively well associated with the 3 general degree descriptors of exercise intensity: light, moderate, and vigorous. However, there was no clear association between the intensity expressed with those degree descriptors and the degree of physical exertion the participants reported to have experienced after exercise. Intensity ratings of various examples of physical activity differed significantly according to respondents' characteristics. Regression analyses showed that those who reported good health or considered regular exercise was important for their health tended to rate the intensity levels of the activity examples significantly higher than their counterparts. The respondents' age and race (white vs nonwhite) were not significant predictors of the intensity rating. CONCLUSIONS: This survey showed significant variations in how people perceive and interpret the intensity levels of physical activities described with general severity modifiers, degrees of physical exertion, and physical activity examples. Considering that these are among the most widely used methods of communicating physical activity intensity in current practice, a possible miscommunication in assessing and promoting physical activity seems to be a real concern. We need to adopt a method that represents activity intensity in a quantifiable manner to avoid unintended miscommunication.

Developing a Physical Activity Ontology to Support the Interoperability of Physical Activity Data.

BACKGROUND: Physical activity data provides important information on disease onset, progression, and treatment outcomes. Although analyzing physical activity data in conjunction with other clinical and microbiological data will lead to new insights crucial for improving human health, it has been hampered partly because of the large variations in the way the data are collected and presented. OBJECTIVE: The aim of this study was to develop a Physical Activity Ontology (PACO) to support structuring and standardizing heterogeneous descriptions of physical activities. METHODS: We prepared a corpus of 1140 unique sentences collected from various physical activity questionnaires and scales as well as existing standardized terminologies and ontologies. We extracted concepts relevant to physical activity from the corpus using a natural language processing toolkit called Multipurpose Text Processing Tool. The target concepts were formalized into an ontology using Protégé (version 4). Evaluation of PACO was performed to ensure logical and structural consistency as well as adherence to the best practice principles of building an ontology. A use case application of PACO was demonstrated by structuring and standardizing 36 exercise habit statements and then automatically classifying them to a defined class of either sufficiently active or insufficiently active using FaCT++, an ontology reasoner available in Protégé. RESULTS: PACO was constructed using 268 unique concepts extracted from the questionnaires and assessment scales. PACO contains 225 classes including 9 defined classes, 20 object properties, 1 data property, and 23 instances (excluding 36 exercise statements). The maximum depth of classes is 4, and the maximum number of siblings is 38. The evaluations with ontology auditing tools confirmed that PACO is structurally and logically consistent and satisfies the majority of the best practice rules of ontology authoring. We showed in a small sample of 36 exercise habit statements that we could formally represent them using PACO concepts and object properties. The formal representation was used to infer a patient activity status category of sufficiently active or insufficiently active using the FaCT++ reasoner. CONCLUSIONS: As a first step toward standardizing and structuring heterogeneous descriptions of physical activities for integrative data analyses, PACO was constructed based on the concepts collected from physical activity questionnaires and assessment scales. PACO was evaluated to be structurally consistent and compliant to ontology authoring principles. PACO was also demonstrated to be potentially useful in standardizing heterogeneous physical activity descriptions and classifying them into clinically meaningful categories that reflect adequacy of exercise.

Evaluating Casama: Contextualized semantic maps for summarization of lung cancer studies.

OBJECTIVE: It is crucial for clinicians to stay up to date on current literature in order to apply recent evidence to clinical decision making. Automatic summarization systems can help clinicians quickly view an aggregated summary of literature on a topic. Casama, a representation and summarization system based on "contextualized semantic maps," captures the findings of biomedical studies as well as the contexts associated with patient population and study design. This paper presents a user-oriented evaluation of Casama in comparison to a context-free representation, SemRep. MATERIALS AND METHODS: The effectiveness of the representation was evaluated by presenting users with manually annotated Casama and SemRep summaries of ten articles on driver mutations in cancer. Automatic annotations were evaluated on a collection of articles on EGFR mutation in lung cancer. Seven users completed a questionnaire rating the summarization quality for various topics and applications. RESULTS: Casama had higher median scores than SemRep for the majority of the topics (p≤ 0.00032), all of the applications (p≤ 0.00089), and in overall summarization quality (p≤ 1.5e-05). Casama's manual annotations outperformed Casama's automatic annotations (p = 0.00061). DISCUSSION: Casama performed particularly well in the representation of strength of evidence, which was highly rated both quantitatively and qualitatively. Users noted that Casama's less granular, more targeted representation improved usability compared to SemRep. CONCLUSION: This evaluation demonstrated the benefits of a contextualized representation for summarizing biomedical literature on cancer. Iteration on specific areas of Casama's representation, further development of its algorithms, and a clinically-oriented evaluation are warranted.

Medical Imaging Informatics.

Imaging is one of the most important sources of clinically observable evidence that provides broad coverage, can provide insight on low-level scale properties, is noninvasive, has few side effects, and can be performed frequently. Thus, imaging data provides a viable observable that can facilitate the instantiation of a theoretical understanding of a disease for a particular patient context by connecting imaging findings to other biologic parameters in the model (e.g., genetic, molecular, symptoms, and patient survival). These connections can help inform their possible states and/or provide further coherent evidence. The field of radiomics is particularly dedicated to this task and seeks to extract quantifiable measures wherever possible. Example properties of investigation include genotype characterization, histopathology parameters, metabolite concentrations, vascular proliferation, necrosis, cellularity, and oxygenation. Important issues within the field include: signal calibration, spatial calibration, preprocessing methods (e.g., noise suppression, motion correction, and field bias correction), segmentation of target anatomic/pathologic entities, extraction of computed features, and inferencing methods connecting imaging features to biological states.

Automatic Classification of Ultrasound Screening Examinations of the Abdominal Aorta.

Our work facilitates the identification of veterans who may be at risk for abdominal aortic aneurysms (AAA) based on the 2007 mandate to screen all veteran patients that meet the screening criteria. The main research objective is to automatically index three clinical conditions: pertinent negative AAA, pertinent positive AAA, and visually unacceptable image exams. We developed and evaluated a ConText-based algorithm with the GATE (General Architecture for Text Engineering) development system to automatically classify 1402 ultrasound radiology reports for AAA screening. Using the results from JAPE (Java Annotation Pattern Engine) transducer rules, we developed a feature vector to classify the radiology reports with a decision table classifier. We found that ConText performed optimally on precision and recall for pertinent negative (0.99 (0.98-0.99), 0.99 (0.99-1.00)) and pertinent positive AAA detection (0.98 (0.95-1.00), 0.97 (0.92-1.00)), and respectably for determination of non-diagnostic image studies (0.85 (0.77-0.91), 0.96 (0.91-0.99)). In addition, our algorithm can determine the AAA size measurements for further characterization of abnormality. We developed and evaluated a regular expression based algorithm using GATE for determining the three contextual conditions: pertinent negative, pertinent positive, and non-diagnostic from radiology reports obtained for evaluating the presence or absence of abdominal aortic aneurysm. ConText performed very well at identifying the contextual features. Our study also discovered contextual trigger terms to detect sub-standard ultrasound image quality. Limitations of performance included unknown dictionary terms, complex sentences, and vague findings that were difficult to classify and properly code.

Evaluating a Novel Summary Visualization for Clinical Trial Reports: A Usability Study.

Contributions of clinical trials are captured in published reports that are unstructured and often require extensive manual review to gain a deeper understanding of the study itself. Our goal is to increase comprehension and decrease the time necessary to understand these reports through the use of visualization tools. In this paper, we specify and evaluate the visualization of a previously developed representation as well as gain insight from user input for further development. The usability experiment consisted of a two-arm study with users either having or not having access to the visualization. A user questionnaire was used to measure time spent and accuracy in comprehension; intuitiveness and reproducibility of the visualization; and preferences. We found that having the visualization required on average 28.1% less time (25.8 min vs. 35.8 min, p=0.01) while maintaining similar accuracy (73.7% vs. 67.0%). Users were then asked to create their own visualizations, with their visualizations averaging 86.1% similar to the gold standard. All participants either preferred the visualization over the status quo or preferred both equally. These results demonstrate that novel visualizations for trial reports could provide time savings and achieve similar accuracy as reviewing the paper itself. Understanding the strength and quality of clinical trials can be alleviated with a visualization that makes content explicit.

Assessing Variability in Brain Tumor Segmentation to Improve Volumetric Accuracy and Characterization of Change.

Brain tumor analysis is moving towards volumetric assessment of magnetic resonance imaging (MRI), providing a more precise description of disease progression to better inform clinical decision-making and treatment planning. While a multitude of segmentation approaches exist, inherent variability in the results of these algorithms may incorrectly indicate changes in tumor volume. In this work, we present a systematic approach to characterize variability in tumor boundaries that utilizes equivalence tests as a means to determine whether a tumor volume has significantly changed over time. To demonstrate these concepts, 32 MRI studies from 8 patients were segmented using four different approaches (statistical classifier, region-based, edge-based, knowledge-based) to generate different regions of interest representing tumor extent. We showed that across all studies, the average Dice coefficient for the superset of the different methods was 0.754 (95% confidence interval 0.701-0.808) when compared to a reference standard. We illustrate how variability obtained by different segmentations can be used to identify significant changes in tumor volume between sequential time points. Our study demonstrates that variability is an inherent part of interpreting tumor segmentation results and should be considered as part of the interpretation process.

Toward patient-tailored summarization of lung cancer literature.

As the volume of biomedical literature increases, it can be challenging for clinicians to stay up-to-date. Graphical summarization systems help by condensing knowledge into networks of entities and relations. However, existing systems present relations out of context, ignoring key details such as study population. To better support precision medicine, summarization systems should include such information to contextualize and tailor results to individual patients. This paper introduces "contextualized semantic maps" for patient-tailored graphical summarization of published literature. These efforts are demonstrated in the domain of driver mutations in non-small cell lung cancer (NSCLC). A representation for relations and study population context in NSCLC was developed. An annotated gold standard for this representation was created from a set of 135 abstracts; F1-score annotator agreement was 0.78 for context and 0.68 for relations. Visualizing the contextualized relations demonstrated that context facilitates the discovery of key findings that are relevant to patient-oriented queries.

A formal representation for numerical data presented in published clinical trial reports.

Assessing the quality of and integrating clinical trial reports are necessary to practice evidence-based medicine. In particular, the numerical data is essential to understanding the strength and quality of the clinical trial study. In this paper, we present a formal representation for standardizing numerical data in published clinical trial reports, and our efforts towards developing computational tools to capture and visualize this representation. The approach includes two aspects: a process model used to precisely define experimental context behind the numerical value; and a spreadsheet, an intuitive and familiar tool used to organize numerical data. We demonstrated this representation using clinical trial reports on non-small cell lung cancer (NSCLC). We performed a preliminary evaluation to determine the usefulness of this formalism for identifying the characteristics, quality and significance of a clinical trial. Our initial results demonstrate that the representation is sufficiently expressive to capture reported numerical information in published papers.

Hierarchical semantic structures for medical NLP.

We present a framework for building a medical natural language processing (NLP) system capable of deep understanding of clinical text reports. The framework helps developers understand how various NLP-related efforts and knowledge sources can be integrated. The aspects considered include: 1) computational issues dealing with defining layers of intermediate semantic structures to reduce the dimensionality of the NLP problem; 2) algorithmic issues in which we survey the NLP literature and discuss state-of-the-art procedures used to map between various levels of the hierarchy; and 3) implementation issues to software developers with available resources. The objective of this poster is to educate readers to the various levels of semantic representation (e.g., word level concepts, ontological concepts, logical relations, logical frames, discourse structures, etc.). The poster presents an architecture for which diverse efforts and resources in medical NLP can be integrated in a principled way.

Imaging-based observational databases for clinical problem solving: the role of informatics.

Imaging has become a prevalent tool in the diagnosis and treatment of many diseases, providing a unique in vivo, multi-scale view of anatomic and physiologic processes. With the increased use of imaging and its progressive technical advances, the role of imaging informatics is now evolving--from one of managing images, to one of integrating the full scope of clinical information needed to contextualize and link observations across phenotypic and genotypic scales. Several challenges exist for imaging informatics, including the need for methods to transform clinical imaging studies and associated data into structured information that can be organized and analyzed. We examine some of these challenges in establishing imaging-based observational databases that can support the creation of comprehensive disease models. The development of these databases and ensuing models can aid in medical decision making and knowledge discovery and ultimately, transform the use of imaging to support individually-tailored patient care.

Imaging informatics for consumer health: towards a radiology patient portal.

OBJECTIVE: With the increased routine use of advanced imaging in clinical diagnosis and treatment, it has become imperative to provide patients with a means to view and understand their imaging studies. We illustrate the feasibility of a patient portal that automatically structures and integrates radiology reports with corresponding imaging studies according to several information orientations tailored for the layperson. METHODS: The imaging patient portal is composed of an image processing module for the creation of a timeline that illustrates the progression of disease, a natural language processing module to extract salient concepts from radiology reports (73% accuracy, F1 score of 0.67), and an interactive user interface navigable by an imaging findings list. The portal was developed as a Java-based web application and is demonstrated for patients with brain cancer. RESULTS AND DISCUSSION: The system was exhibited at an international radiology conference to solicit feedback from a diverse group of healthcare professionals. There was wide support for educating patients about their imaging studies, and an appreciation for the informatics tools used to simplify images and reports for consumer interpretation. Primary concerns included the possibility of patients misunderstanding their results, as well as worries regarding accidental improper disclosure of medical information. CONCLUSIONS: Radiologic imaging composes a significant amount of the evidence used to make diagnostic and treatment decisions, yet there are few tools for explaining this information to patients. The proposed radiology patient portal provides a framework for organizing radiologic results into several information orientations to support patient education.

Context-based electronic health record: toward patient specific healthcare.

Due to the increasingly data-intensive clinical environment, physicians now have unprecedented access to detailed clinical information from a multitude of sources. However, applying this information to guide medical decisions for a specific patient case remains challenging. One issue is related to presenting information to the practitioner: displaying a large (irrelevant) amount of information often leads to information overload. Next-generation interfaces for the electronic health record (EHR) should not only make patient data easily searchable and accessible, but also synthesize fragments of evidence documented in the entire record to understand the etiology of a disease and its clinical manifestation in individual patients. In this paper, we describe our efforts toward creating a context-based EHR, which employs biomedical ontologies and (graphical) disease models as sources of domain knowledge to identify relevant parts of the record to display. We hypothesize that knowledge (e.g., variables, relationships) from these sources can be used to standardize, annotate, and contextualize information from the patient record, improving access to relevant parts of the record and informing medical decision making. To achieve this goal, we describe a framework that aggregates and extracts findings and attributes from free-text clinical reports, maps findings to concepts in available knowledge sources, and generates a tailored presentation of the record based on the information needs of the user. We have implemented this framework in a system called Adaptive EHR, demonstrating its capabilities to present and synthesize information from neurooncology patients. This paper highlights the challenges and potential applications of leveraging disease models to improve the access, integration, and interpretation of clinical patient data.

Automated extraction of reported statistical analyses: towards a logical representation of clinical trial literature.

Randomized controlled trials are an important source of evidence for guiding clinical decisions when treating a patient. However, given the large number of studies and their variability in quality, determining how to summarize reported results and formalize them as part of practice guidelines continues to be a challenge. We have developed a set of information extraction and annotation tools to automate the identification of key information from papers related to the hypothesis, sample size, statistical test, confidence interval, significance level, and conclusions. We adapted the Automated Sequence Annotation Pipeline to map extracted phrases to relevant knowledge sources. We trained and tested our system on a corpus of 42 full-text articles related to chemotherapy of non-small cell lung cancer. On our test set of 7 papers, we obtained an overall precision of 86%, recall of 78%, and an F-score of 0.82 for classifying sentences. This work represents our efforts towards utilizing this information for quality assessment, meta-analysis, and modeling.

Improving the accuracy of therapy descriptions in clinical trials using a bottom-up approach.

Randomized clinical trial (RCT) reports commonly have complicated therapy descriptions that are written in free-text. Drug therapy is difficult to describe due to the dynamic nature of how protocols change and the many ways drugs can be administered. Details regarding protocol changes and drug administration must be explained clearly for reproducibility and reliability. A process model supplemented with concept ontologies can clarify the dynamics of how therapies change and make knowledge more explicit. We demonstrated the process to develop a representation model to reveal specific context concerning drug therapies within clinical trial report literature. A PubMed search was conducted to identify RCTs on non-small-cell lung cancer (NSCLC) pertaining to epithelial growth factor receptor (EGFR) mutations. Twenty-seven clinical trials were used to develop the model using a bottom-up approach. This representation describes drug dosage, administration details, and drug cycles within different experimental arms and control groups. We then presented preliminary evaluation of the clarity and understandability of the representation.

Natural language processing with dynamic classification improves P300 speller accuracy and bit rate.

The P300 speller is an example of a brain-computer interface that can restore functionality to victims of neuromuscular disorders. Although the most common application of this system has been communicating language, the properties and constraints of the linguistic domain have not to date been exploited when decoding brain signals that pertain to language. We hypothesized that combining the standard stepwise linear discriminant analysis with a Naive Bayes classifier and a trigram language model would increase the speed and accuracy of typing with the P300 speller. With integration of natural language processing, we observed significant improvements in accuracy and 40-60% increases in bit rate for all six subjects in a pilot study. This study suggests that integrating information about the linguistic domain can significantly improve signal classification.

A Case-based Retrieval System using Natural Language Processing and Population-based Visualization.

Electronic medical records capture large quantities of patient data generated as a result of routine care. Secondary use of this data for clinical research could provide new insights into the evolution of diseases and help assess the effectiveness of available interventions. Unfortunately, the unstructured nature of clinical data hinders a user's ability to understand this data: tools are needed to structure, model, and visualize the data to elucidate patterns in a patient population. We present a case-based retrieval framework that incorporates an extraction tool to identify concepts from clinical reports, a disease model to capture necessary context for interpreting extracted concepts, and a model-driven visualization to facilitate querying and interpretation of the results. We describe how the model is used to group, filter, and retrieve similar cases. We present an application of the framework that aids users in exploring a population of intracranial aneurysm patients.