Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain.

BACKGROUND: Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. METHODS: LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. RESULTS: The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1-5 to 1-10) had a considerable impact on the performance. CONCLUSIONS: LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.

Exploring Capabilities of Large Language Models such as ChatGPT in Radiation Oncology.

Purpose: Technological progress of machine learning and natural language processing has led to the development of large language models (LLMs), capable of producing well-formed text responses and providing natural language access to knowledge. Modern conversational LLMs such as ChatGPT have shown remarkable capabilities across a variety of fields, including medicine. These models may assess even highly specialized medical knowledge within specific disciplines, such as radiation therapy. We conducted an exploratory study to examine the capabilities of ChatGPT to answer questions in radiation therapy. Methods and Materials: A set of multiple-choice questions about clinical, physics, and biology general knowledge in radiation oncology as well as a set of open-ended questions were created. These were given as prompts to the LLM ChatGPT, and the answers were collected and analyzed. For the multiple-choice questions, it was checked how many of the answers of the model could be clearly assigned to one of the allowed multiple-choice-answers, and the proportion of correct answers was determined. For the open-ended questions, independent blinded radiation oncologists evaluated the quality of the answers regarding correctness and usefulness on a 5-point Likert scale. Furthermore, the evaluators were asked to provide suggestions for improving the quality of the answers. Results: For 70 multiple-choice questions, ChatGPT gave valid answers in 66 cases (94.3%). In 60.61% of the valid answers, the selected answer was correct (50.0% of clinical questions, 78.6% of physics questions, and 58.3% of biology questions). For 25 open-ended questions, 12 answers of ChatGPT were considered as "acceptable," "good," or "very good" regarding both correctness and helpfulness by all 6 participating radiation oncologists. Overall, the answers were considered "very good" in 29.3% and 28%, "good" in 28% and 29.3%, "acceptable" in 19.3% and 19.3%, "bad" in 9.3% and 9.3%, and "very bad" in 14% and 14% regarding correctness/helpfulness. Conclusions: Modern conversational LLMs such as ChatGPT can provide satisfying answers to many relevant questions in radiation therapy. As they still fall short of consistently providing correct information, it is problematic to use them for obtaining medical information. As LLMs will further improve in the future, they are expected to have an increasing impact not only on general society, but also on clinical practice, including radiation oncology.

Extraction of Interoperable Data from Healthcare Documents by Identifying Common Data Elements: An Analysis of Radiation Therapy Planning CT Physician Order Entry Records.

INTRODUCTION: Documentation as well as IT-based management of medical data is of ever-increasing relevance in modern medicine. As radiation oncology is a rather technical, data-driven discipline, standardization, and data exchange are in principle possible. We examined electronic healthcare documents to extract structured information. Planning CT order entry documents were chosen for the analysis, as this covers a common and structured step in radiation oncology, for which standardized documentation may be achieved. The aim was to examine the extent to which relevant information may be exchanged among different institutions. MATERIALS AND METHODS: We contacted representatives of nine radiation oncology departments. Departments using standardized electronic documentation for planning CT were asked to provide templates of their records, which were analyzed in terms of form and content. Structured information was extracted by identifying definite common data elements, containing explicit information. Relevant common data elements were identified and classified. A quantitative analysis was performed to evaluate the possibility of data exchange. RESULTS: We received data of seven documents that were heterogeneous regarding form and content. 181 definite common data elements considered relevant for the planning CT were identified and assorted into five semantic groups. 139 data elements (76.8%) were present in only one document. The other 42 data elements were present in two to six documents, while none was shared among all seven documents. CONCLUSION: Structured and interoperable documentation of medical information can be achieved using common data elements. Our analysis showed that a lot of information recorded with healthcare documents can be presented with this approach. Yet, in the analyzed cohort of planning CT order entries, only a few common data elements were shared among the majority of documents. A common vocabulary and consensus upon relevant information is required to promote interoperability and standardization.

(Common) Data Elements in Radiation Oncology: A Systematic Literature Review.

PURPOSE: Structured medical data documentation is highly relevant in a data-driven discipline such as radiation oncology. Defined (common) data elements (CDEs) can be used to record data in clinical trials, health records, or computer systems for improved standardization and data exchange. The International Society for Radiation Oncology Informatics initiated a project for a scientific literature analysis of defined data elements for structured documentation in radiation oncology. METHODS: We performed a systematic literature review on both PubMed and Scopus to analyze publications relevant to the utilization of specified data elements for the documentation of radiation therapy (RT)-related information. Relevant publications were retrieved as full-text and searched for published data elements. Finally, the extracted data elements were quantitatively analyzed and classified. RESULTS: We found a total of 452 publications, of which 46 were considered relevant for structured data documentation. Twenty-nine publications addressed defined RT-specific data elements, of which 12 publications provided data elements. Only two publications focused on data elements in radiation oncology. The 29 analyzed publications were heterogeneous regarding the subject and usage of the defined data elements, and different concepts/terms for defined data elements were used. CONCLUSION: The literature about structured data documentation in radiation oncology using defined data elements is scarce. There is a need for a comprehensive list of RT-specific CDEs the radio-oncologic community can rely on. As it has been done in other medical fields, establishing such a list would be of great value for clinical practice and research as it would promote interoperability and standardization.

Parameters of the Lyman Model for Calculation of Normal-Tissue Complication Probability: A Systematic Literature Review.

PURPOSE: The Lyman model is one of the most used radiobiological models for calculation of normal-tissue complication probability (NTCP). Since its introduction in 1985, many authors have published parameter values for the model based on clinical data of different radiotherapeutic situations. This study attempted to collect the entirety of radiobiological parameter sets published to date and provide an overview of the data basis for different variations of the model. Furthermore, it sought to compare the parameter values and calculated NTCPs for selected endpoints with sufficient data available. METHODS AND MATERIALS: A systematic literature analysis was performed, searching for publications that provided parameters for the different variations of the Lyman model in the Medline database using PubMed. Parameter sets were grouped into 13 toxicity-related endpoint groups. For 3 selected endpoint groups (≤25% reduction of saliva 12 months after irradiation of the parotid, symptomatic pneumonitis after irradiation of the lung, and bleeding of grade 2 or less after irradiation of the rectum), parameter values were compared and differences in calculated NTCP values were analyzed. RESULTS: A total of 509 parameter sets from 130 publications were identified. Considerable heterogeneities were detected regarding the number of parameters available for different radio-oncological situations. Furthermore, for the 3 selected endpoints, large differences in published parameter values were found. These translated into great variations of calculated NTCPs, with maximum ranges of 35.2% to 93.4% for the saliva endpoint, of 39.4% to 90.4% for the pneumonitis endpoint, and of 5.4% to 99.3% for the rectal bleeding endpoint. CONCLUSIONS: The detected heterogeneity of the data as well as the large variations of published radiobiological parameters underline the necessity for careful interpretation when using such parameters for NTCP calculations. Appropriate selection of parameters and validation of values are essential when using the Lyman model.

Creation of clinical algorithms for decision-making in oncology: an example with dose prescription in radiation oncology.

In oncology, decision-making in individual situations is often very complex. To deal with such complexity, people tend to reduce it by relying on their initial intuition. The downside of this intuitive, subjective way of decision-making is that it is prone to cognitive and emotional biases such as overestimating the quality of its judgements or being influenced by one's current mood. Hence, clinical predictions based on intuition often turn out to be wrong and to be outperformed by statistical predictions. Structuring and objectivizing oncological decision-making may thus overcome some of these issues and have advantages such as avoidance of unwarranted clinical practice variance or error-prevention. Even for uncertain situations with limited medical evidence available or controversies about the best treatment option, structured decision-making approaches like clinical algorithms could outperform intuitive decision-making. However, the idea of such algorithms is not to prescribe the clinician which decision to make nor to abolish medical judgement, but to support physicians in making decisions in a systematic and structured manner. An example for a use-case scenario where such an approach may be feasible is the selection of treatment dose in radiation oncology. In this paper, we will describe how a clinical algorithm for selection of a fractionation scheme for palliative irradiation of bone metastases can be created. We explain which steps in the creation process of a clinical algorithm for supporting decision-making need to be  performed and which challenges and limitations have to be considered.

Inhibition of acid sphingomyelinase increases regulatory T cells in humans.

Genetic deficiency for acid sphingomyelinase or its pharmacological inhibition has been shown to increase Foxp3+ regulatory T-cell frequencies among CD4+ T cells in mice. We now investigated whether pharmacological targeting of the acid sphingomyelinase, which catalyzes the cleavage of sphingomyelin to ceramide and phosphorylcholine, also allows to manipulate relative CD4+ Foxp3+ regulatory T-cell frequencies in humans. Pharmacological acid sphingomyelinase inhibition with antidepressants like sertraline, but not those without an inhibitory effect on acid sphingomyelinase activity like citalopram, increased the frequency of Foxp3+ regulatory T cell among human CD4+ T cells in vitro. In an observational prospective clinical study with patients suffering from major depression, we observed that acid sphingomyelinase-inhibiting antidepressants induced a stronger relative increase in the frequency of CD4+ Foxp3+ regulatory T cells in peripheral blood than acid sphingomyelinase-non- or weakly inhibiting antidepressants. This was particularly true for CD45RA- CD25high effector CD4+ Foxp3+ regulatory T cells. Mechanistically, our data indicate that the positive effect of acid sphingomyelinase inhibition on CD4+ Foxp3+ regulatory T cells required CD28 co-stimulation, suggesting that enhanced CD28 co-stimulation was the driver of the observed increase in the frequency of Foxp3+ regulatory T cells among human CD4+ T cells. In summary, the widely induced pharmacological inhibition of acid sphingomyelinase activity in patients leads to an increase in Foxp3+ regulatory T-cell frequencies among CD4+ T cells in humans both in vivo and in vitro.

Translational Approaches Targeting Ceramide Generation From Sphingomyelin in T Cells to Modulate Immunity in Humans.

In T cells, as in all other cells of the body, sphingolipids form important structural components of membranes. Due to metabolic modifications, sphingolipids additionally play an active part in the signaling of cell surface receptors of T cells like the T cell receptor or the co-stimulatory molecule CD28. Moreover, the sphingolipid composition of their membranes crucially affects the integrity and function of subcellular compartments such as the lysosome. Previously, studying sphingolipid metabolism has been severely hampered by the limited number of analytical methods/model systems available. Besides well-established high resolution mass spectrometry new tools are now available like novel minimally modified sphingolipid subspecies for click chemistry as well as recently generated mouse mutants with deficiencies/overexpression of sphingolipid-modifying enzymes. Making use of these tools we and others discovered that the sphingolipid sphingomyelin is metabolized to ceramide to different degrees in distinct T cell subpopulations of mice and humans. This knowledge has already been translated into novel immunomodulatory approaches in mice and will in the future hopefully also be applicable to humans. In this paper we are, thus, summarizing the most recent findings on the impact of sphingolipid metabolism on T cell activation, differentiation, and effector functions. Moreover, we are discussing the therapeutic concepts arising from these insights and drugs or drug candidates which are already in clinical use or could be developed for clinical use in patients with diseases as distant as major depression and chronic viral infection.