Generative large language models are all-purpose text analytics engines: text-to-text learning is all your need.

OBJECTIVE: To solve major clinical natural language processing (NLP) tasks using a unified text-to-text learning architecture based on a generative large language model (LLM) via prompt tuning. METHODS: We formulated 7 key clinical NLP tasks as text-to-text learning and solved them using one unified generative clinical LLM, GatorTronGPT, developed using GPT-3 architecture and trained with up to 20 billion parameters. We adopted soft prompts (ie, trainable vectors) with frozen LLM, where the LLM parameters were not updated (ie, frozen) and only the vectors of soft prompts were updated, known as prompt tuning. We added additional soft prompts as a prefix to the input layer, which were optimized during the prompt tuning. We evaluated the proposed method using 7 clinical NLP tasks and compared them with previous task-specific solutions based on Transformer models. RESULTS AND CONCLUSION: The proposed approach achieved state-of-the-art performance for 5 out of 7 major clinical NLP tasks using one unified generative LLM. Our approach outperformed previous task-specific transformer models by ∼3% for concept extraction and 7% for relation extraction applied to social determinants of health, 3.4% for clinical concept normalization, 3.4%-10% for clinical abbreviation disambiguation, and 5.5%-9% for natural language inference. Our approach also outperformed a previously developed prompt-based machine reading comprehension (MRC) model, GatorTron-MRC, for clinical concept and relation extraction. The proposed approach can deliver the "one model for all" promise from training to deployment using a unified generative LLM.

Model tuning or prompt Tuning? a study of large language models for clinical concept and relation extraction.

OBJECTIVE: To develop soft prompt-based learning architecture for large language models (LLMs), examine prompt-tuning using frozen/unfrozen LLMs, and assess their abilities in transfer learning and few-shot learning. METHODS: We developed a soft prompt-based learning architecture and compared 4 strategies including (1) fine-tuning without prompts; (2) hard-prompting with unfrozen LLMs; (3) soft-prompting with unfrozen LLMs; and (4) soft-prompting with frozen LLMs. We evaluated GatorTron, a clinical LLM with up to 8.9 billion parameters, and compared GatorTron with 4 existing transformer models for clinical concept and relation extraction on 2 benchmark datasets for adverse drug events and social determinants of health (SDoH). We evaluated the few-shot learning ability and generalizability for cross-institution applications. RESULTS AND CONCLUSION: When LLMs are unfrozen, GatorTron-3.9B with soft prompting achieves the best strict F1-scores of 0.9118 and 0.8604 for concept extraction, outperforming the traditional fine-tuning and hard prompt-based models by 0.6 âˆ¼ 3.1 % and 1.2 âˆ¼ 2.9 %, respectively; GatorTron-345 M with soft prompting achieves the best F1-scores of 0.8332 and 0.7488 for end-to-end relation extraction, outperforming other two models by 0.2 âˆ¼ 2 % and 0.6 âˆ¼ 11.7 %, respectively. When LLMs are frozen, small LLMs have a big gap to be competitive with unfrozen models; scaling LLMs up to billions of parameters makes frozen LLMs competitive with unfrozen models. Soft prompting with a frozen GatorTron-8.9B model achieved the best performance for cross-institution evaluation. We demonstrate that (1) machines can learn soft prompts better than hard prompts composed by human, (2) frozen LLMs have good few-shot learning ability and generalizability for cross-institution applications, (3) frozen LLMs reduce computing cost to 2.5 âˆ¼ 6 % of previous methods using unfrozen LLMs, and (4) frozen LLMs require large models (e.g., over several billions of parameters) for good performance.

A study of generative large language model for medical research and healthcare.

There are enormous enthusiasm and concerns in applying large language models (LLMs) to healthcare. Yet current assumptions are based on general-purpose LLMs such as ChatGPT, which are not developed for medical use. This study develops a generative clinical LLM, GatorTronGPT, using 277 billion words of text including (1) 82 billion words of clinical text from 126 clinical departments and approximately 2 million patients at the University of Florida Health and (2) 195 billion words of diverse general English text. We train GatorTronGPT using a GPT-3 architecture with up to 20 billion parameters and evaluate its utility for biomedical natural language processing (NLP) and healthcare text generation. GatorTronGPT improves biomedical natural language processing. We apply GatorTronGPT to generate 20 billion words of synthetic text. Synthetic NLP models trained using synthetic text generated by GatorTronGPT outperform models trained using real-world clinical text. Physicians' Turing test using 1 (worst) to 9 (best) scale shows that there are no significant differences in linguistic readability (p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical relevance (p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that physicians cannot differentiate them (p < 0.001). This study provides insights into the opportunities and challenges of LLMs for medical research and healthcare.

Acarbose suppresses symptoms of mitochondrial disease in a mouse model of Leigh syndrome.

Mitochondrial diseases represent a spectrum of disorders caused by impaired mitochondrial function, ranging in severity from mortality during infancy to progressive adult-onset disease. Mitochondrial dysfunction is also recognized as a molecular hallmark of the biological ageing process. Rapamycin, a drug that increases lifespan and health during normative ageing, also increases survival and reduces neurological symptoms in a mouse model of the severe mitochondrial disease Leigh syndrome. The Ndufs4 knockout (Ndufs4-/-) mouse lacks the complex I subunit NDUFS4 and shows rapid onset and progression of neurodegeneration mimicking patients with Leigh syndrome. Here we show that another drug that extends lifespan and delays normative ageing in mice, acarbose, also suppresses symptoms of disease and improves survival of Ndufs4-/- mice. Unlike rapamycin, acarbose rescues disease phenotypes independently of inhibition of the mechanistic target of rapamycin. Furthermore, rapamycin and acarbose have additive effects in delaying neurological symptoms and increasing maximum lifespan in Ndufs4-/- mice. We find that acarbose remodels the intestinal microbiome and alters the production of short-chain fatty acids. Supplementation with tributyrin, a source of butyric acid, recapitulates some effects of acarbose on lifespan and disease progression, while depletion of the endogenous microbiome in Ndufs4-/- mice appears to fully recapitulate the effects of acarbose on healthspan and lifespan in these animals. To our knowledge, this study provides the first evidence that alteration of the gut microbiome plays a significant role in severe mitochondrial disease and provides further support for the model that biological ageing and severe mitochondrial disorders share underlying common mechanisms.

A large language model for electronic health records.

There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model-GatorTron-using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on five clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve five clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og .

FOXM1 drives HPV+ HNSCC sensitivity to WEE1 inhibition.

Head and neck squamous cell carcinoma (HNSCC) associated with high-risk human papilloma virus (HPV) infection is a growing clinical problem. The WEE1 kinase inhibitor AZD1775 (WEE1i) overrides cell cycle checkpoints and is being studied in HNSCC regimens. We show that the HPV16 E6/E7 oncoproteins sensitize HNSCC cells to single-agent WEE1i treatment through activation of a FOXM1-CDK1 circuit that drives mitotic gene expression and DNA damage. An isogenic cell system indicated that E6 largely accounts for these phenotypes in ways that extend beyond p53 inactivation. A targeted genomic analysis implicated FOXM1 signaling downstream of E6/E7 expression and analyses of primary tumors and The Cancer Genome Atlas (TCGA) data revealed an activated FOXM1-directed promitotic transcriptional signature in HPV+ versus HPV- HNSCCs. Finally, we demonstrate the causality of FOXM1 in driving WEE1i sensitivity. These data suggest that elevated basal FOXM1 activity predisposes HPV+ HNSCC to WEE1i-induced toxicity and provide mechanistic insights into WEE1i and HPV+ HNSCC therapies.

Perturbed maintenance of transcriptional repression on the inactive X-chromosome in the mouse brain after Xist deletion.

BACKGROUND: The long noncoding RNA Xist is critical for initiation and establishment of X-chromosome inactivation during embryogenesis in mammals, but it is unclear whether its continued expression is required for maintaining X-inactivation in vivo. RESULTS: By using an inactive X-chromosome-linked MeCP2-GFP reporter, which allowed us to enumerate reactivation events in the mouse brain even when they occur in very few cells, we found that deletion of Xist in the brain after establishment of X-chromosome inactivation leads to reactivation in 2-5% of neurons and in a smaller fraction of astrocytes. In contrast to global loss of both H3 lysine 27 trimethylation (H3K27m3) and histone H2A lysine 119 monoubiquitylation (H2AK119ub1) we observed upon Xist deletion, alterations in CpG methylation were subtle, and this was mirrored by only minor alterations in X-chromosome-wide gene expression levels, with highly expressed genes more prone to both derepression and demethylation compared to genes with low expression level. CONCLUSION: Our results demonstrate that Xist plays a role in the maintenance of histone repressive marks, DNA methylation and transcriptional repression on the inactive X-chromosome, but that partial loss of X-dosage compensation in the absence of Xist in the brain is well tolerated.

Transient rapamycin treatment can increase lifespan and healthspan in middle-aged mice.

The FDA approved drug rapamycin increases lifespan in rodents and delays age-related dysfunction in rodents and humans. Nevertheless, important questions remain regarding the optimal dose, duration, and mechanisms of action in the context of healthy aging. Here we show that 3 months of rapamycin treatment is sufficient to increase life expectancy by up to 60% and improve measures of healthspan in middle-aged mice. This transient treatment is also associated with a remodeling of the microbiome, including dramatically increased prevalence of segmented filamentous bacteria in the small intestine. We also define a dose in female mice that does not extend lifespan, but is associated with a striking shift in cancer prevalence toward aggressive hematopoietic cancers and away from non-hematopoietic malignancies. These data suggest that a short-term rapamycin treatment late in life has persistent effects that can robustly delay aging, influence cancer prevalence, and modulate the microbiome.