Investigating a neural language model's replicability of psycholinguistic experiments: A case study of NPI licensing.

The recent success of deep learning neural language models such as Bidirectional Encoder Representations from Transformers (BERT) has brought innovations to computational language research. The present study explores the possibility of using a language model in investigating human language processes, based on the case study of negative polarity items (NPIs). We first conducted an experiment with BERT to examine whether the model successfully captures the hierarchical structural relationship between an NPI and its licensor and whether it may lead to an error analogous to the grammatical illusion shown in the psycholinguistic experiment (Experiment 1). We also investigated whether the language model can capture the fine-grained semantic properties of NPI licensors and discriminate their subtle differences on the scale of licensing strengths (Experiment 2). The results of the two experiments suggest that overall, the neural language model is highly sensitive to both syntactic and semantic constraints in NPI processing. The model's processing patterns and sensitivities are shown to be very close to humans, suggesting their role as a research tool or object in the study of language.

Predicting medical specialty from text based on a domain-specific pre-trained BERT.

BACKGROUND: Owing to the prevalence of the coronavirus disease (COVID-19), coping with clinical issues at the individual level has become important to the healthcare system. Accordingly, precise initiation of treatment after a hospital visit is required for expedited processes and effective diagnoses of outpatients. To achieve this, artificial intelligence in medical natural language processing (NLP), such as a healthcare chatbot or a clinical decision support system, can be suitable tools for an advanced clinical system. Furthermore, support for decisions on the medical specialty from the initial visit can be helpful. MATERIALS AND METHODS: In this study, we propose a medical specialty prediction model from patient-side medical question text based on pre-trained bidirectional encoder representations from transformers (BERT). The dataset comprised pairs of medical question texts and labeled specialties scraped from a website for the medical question-and-answer service. The model was fine-tuned for predicting the required medical specialty labels among 27 labels from medical question texts. To demonstrate the feasibility, we conducted experiments on a real-world dataset and elaborately evaluated the predictive performance compared with four deep learning NLP models through cross-validation and test set evaluation. RESULTS: The proposed model showed improved performance compared with competitive models in terms of overall specialties. In addition, we demonstrate the usefulness of the proposed model by performing case studies for visualization applications. CONCLUSION: The proposed model can benefit hospital patient management and reasonable recommendations for specialties for patients.

A pre-trained BERT for Korean medical natural language processing.

With advances in deep learning and natural language processing (NLP), the analysis of medical texts is becoming increasingly important. Nonetheless, despite the importance of processing medical texts, no research on Korean medical-specific language models has been conducted. The Korean medical text is highly difficult to analyze because of the agglutinative characteristics of the language, as well as the complex terminologies in the medical domain. To solve this problem, we collected a Korean medical corpus and used it to train the language models. In this paper, we present a Korean medical language model based on deep learning NLP. The model was trained using the pre-training framework of BERT for the medical context based on a state-of-the-art Korean language model. The pre-trained model showed increased accuracies of 0.147 and 0.148 for the masked language model with next sentence prediction. In the intrinsic evaluation, the next sentence prediction accuracy improved by 0.258, which is a remarkable enhancement. In addition, the extrinsic evaluation of Korean medical semantic textual similarity data showed a 0.046 increase in the Pearson correlation, and the evaluation for the Korean medical named entity recognition showed a 0.053 increase in the F1-score.

A Word Pair Dataset for Semantic Similarity and Relatedness in Korean Medical Vocabulary: Reference Development and Validation.

BACKGROUND: The fact that medical terms require special expertise and are becoming increasingly complex makes it difficult to employ natural language processing techniques in medical informatics. Several human-validated reference standards for medical terms have been developed to evaluate word embedding models using the semantic similarity and relatedness of medical word pairs. However, there are very few reference standards in non-English languages. In addition, because the existing reference standards were developed a long time ago, there is a need to develop an updated standard to represent recent findings in medical sciences. OBJECTIVE: We propose a new Korean word pair reference set to verify embedding models. METHODS: From January 2010 to December 2020, 518 medical textbooks, 72,844 health information news, and 15,698 medical research articles were collected, and the top 10,000 medical terms were selected to develop medical word pairs. Attending physicians (n=16) participated in the verification of the developed set with 607 word pairs. RESULTS: The proportion of word pairs answered by all participants was 90.8% (551/607) for the similarity task and 86.5% (525/605) for the relatedness task. The similarity and relatedness of the word pair showed a high correlation (ρ=0.70, P<.001). The intraclass correlation coefficients to assess the interrater agreements of the word pair sets were 0.47 on the similarity task and 0.53 on the relatedness task. The final reference standard was 604 word pairs for the similarity task and 599 word pairs for relatedness, excluding word pairs with answers corresponding to outliers and word pairs that were answered by less than 50% of all the respondents. When FastText models were applied to the final reference standard word pair sets, the embedding models learning medical documents had a higher correlation between the calculated cosine similarity scores compared to human-judged similarity and relatedness scores (namu, ρ=0.12 vs with medical text for the similarity task, ρ=0.47; namu, ρ=0.02 vs with medical text for the relatedness task, ρ=0.30). CONCLUSIONS: Korean medical word pair reference standard sets for semantic similarity and relatedness were developed based on medical documents from the past 10 years. It is expected that our word pair reference sets will be actively utilized in the development of medical and multilingual natural language processing technology in the future.

Expression of Six Proteins Causes Reprogramming of Porcine Fibroblasts Into Induced Pluripotent Stem Cells With Both Active X Chromosomes.

In this study, we created porcine-induced pluripotent stem (iPS) cells with the expression of six reprogramming factors (Oct3/4, Klf4, Sox2, c-Myc, Lin28, and Nanog). The resulting cells showed growth dependent on LIF (leukemia inhibitory factor) and expression of multiple stem cell markers. Furthermore, the iPS cells caused teratoma formation with three layers of differentiation and had both active X chromosomes (XaXa). Our iPS cells satisfied the both of important characteristics of stem cells: teratoma formation and activation of both X chromosomes. Injection of these iPS cells into morula stage embryos showed that these cells participate in the early stage of porcine embryogenesis. Furthermore, the RNA-Seq analysis detected that expression levels of endogenous pluripotent related genes, NANOG, SOX2, ZFP42, OCT3/4, ESRRB, and ERAS were much higher in iPS with six factors than that with four reprogramming factors. We can conclude that the expression of six reprogramming factors enables the creation of porcine iPS cells, which is partially close to naive iPS state. J. Cell. Biochem. 118: 537-553, 2017. © 2016 Wiley Periodicals, Inc.

Effects of grass forage species and long-term period of low quality forage diet feeding on growth performance, nutrient utilization and microbial nitrogen yield in growing wether lambs.

Six growing lambs were used to evaluate the feeding value of two forage-based diets in a long-term feeding period by measuring body weight (BW) gain, digestibility, nitrogen (N) retention and microbial N (MBN) yield. The animals were fed imported low-quality timothy hay (TH) with concentrate diet (THD) or imported low-quality Italian ryegrass straw (IR) with concentrate diet (IRD) for 9 months. The forages were offered at 2% BW, and concentrate was fed at 40% of forage intake. The BW gain averaged 82.6 and 66.2 g/day for THD and IRD, respectively, without showing significant difference. Average forage intake (% BW) was significantly greater for IR than for TH, although it was not affected by feeding periods. The digestibility did not differ between diets or periods. The numerically greater (P = 0.06) ratio of retained N to absorbed N for IRD than that for THD was prominent. Neither diet nor period had significant effect on MBN supply and efficiency of MBN synthesis. The results suggest that the IR-based diet can be also used for long-term periods of feeding to growing ruminant animals as a grass hay-based diet without any detrimental effects on nutrient utilization and growth performance.

Effects of different roughage sources and feeding levels on adipogenesis of ovine adipocytes.

The objective of the present study was to conduct an adipogenic evaluation of different roughage sources and feeding levels during ruminant adipocyte differentiation in vitro. Six wether sheep were divided into a timothy hay feeding group (TFG, n = 3) and an Italian ryegrass straw feeding group (IFG, n = 3). The sheep were fed high-roughage (HR), medium roughage (MR) and low-roughage (LR) diets in a one-way layout design each over a 6-day period. Sheep serum samples collected on the last day of each dietary treatment were added to an adipogenic induction medium for differentiation of preadipocytes derived from sheep subcutaneous adipose tissue. The cytoplasmic lipid accumulations in the TFG serum-treated preadipocytes were significantly higher than those of the IFG-serum treated preadipocytes on day 12. Messenger RNA expression of CCAAT/enhancer-binding protein (C/EBP)-α, C/EBP-ß, C/EBP-δ, fatty-acid-binding protein (aP2) and stearoyl-coenzyme A desaturase (SCD) were regulated by each serum treatment. This study shows that different roughage source diets and roughage-to-concentrate ratio diets can regulate adipocyte differentiation via ruminant blood composition.

Changes in circulating adiponectin and metabolic hormone concentrations during periparturient and lactation periods in Holstein dairy cows.

Although our previous report demonstrated that adiponectin and AdipoR1 gene expressions changed among different lactation stages in the bovine mammary gland, its in vivo kinetics remain unclear in ruminant animals. In this study, we investigated the changes in circulating concentrations of adiponectin, as well as other metabolic hormones and metabolites, (i) during the periparturient period and (ii) among different lactation stages, in Holstein dairy cows. In experiment 1, serum adiponectin concentrations increased after parturition. Serum insulin concentrations were lower in the postpartum than prepartum period, whereas serum growth hormone (GH) concentrations increased in the postpartum period. Serum nonesterified fatty acids (NEFA) levels were increased during the postpartum period and were dependent on the parity. In experiment 2, there was no significant difference in plasma adiponectin concentrations among lactational stages. Plasma insulin concentrations tended to be lower in early lactation while plasma GH levels tended to be higher. Plasma NEFA concentrations were significantly lower in mid- and late-lactation stages than non-lactation stages. These findings indicate that elevation of serum adiponectin might be involved in energy metabolism just around parturition, and might exert its action through regulation of receptor expression levels in target tissues in each lactational stage in Holstein dairy cows.

Efficient establishment of pig embryonic fibroblast cell lines with conditional expression of the simian vacuolating virus 40 large T fragment.

The pig is an important animal for both agricultural and medical purposes. However, the number of pig-derived cell lines is relatively limited when compared with mouse- and human-derived lines. We established in this study a retroviral conditional expression system for the Simian vacuolating virus 40 large T fragment (SV40T) which allowed us to efficiently establish pig embryonic fibroblast cell lines. The established cell lines showed high levels of cell proliferation and resistance to cellular senescence. A chromosome analysis showed that 84% of the cells had the normal karyotype. Transient expression of the Cre recombinase allowed us to excise the SV40T fragment from the genome. The development of this research tool will enable us to quickly establish new cell lines derived from various animals.

Chemerin analog regulates energy metabolism in sheep.

Accumulating data suggest a relationship between chemerin and energy metabolism. Our group previously described gene cloning, expression analysis and the regulatory mechanism of chemerin and its own receptor in mice and cattle. The objective of the present study was to investigate the physiological effect of chemerin on endocrine changes and energy metabolism in sheep using a biologically stable chemerin analog. The chemerin analog was intravenously administrated (100 or 500 µg/head) to sheep, and plasma insulin and metabolites (glucose, nonesterified fatty acids (NEFA), triglyceride, total cholesterol and high-density lipoprotein (HDL) cholesterol) were analyzed. The chemerin analog dramatically increased the insulin levels, and glucose levels were decreased. NEFA levels were slightly decreased at 20 min but then increased gradually from 60 to 180 min after analog administration. In addition, injection of the chemerin analog immediately increased triglyceride and total cholesterol but not HDL levels. These results suggested that chemerin analog regulated insulin secretion related to glucose metabolism and the release of triglycerides in sheep in vivo. This study provides new information about endocrine and metabolic changes in response to chemerin in sheep.

Gene expression and hormonal regulation of adiponectin and its receptors in bovine mammary gland and mammary epithelial cells.

Although the functions of adiponectin, a differentiated adipocyte-derived hormone, in regulating glucose and fatty acid metabolism are regulated by two subtypes of adiponectin receptors (AdipoRs; AdipoR1 and AdipoR2), those in ruminants remain unclear. Therefore we examined the messenger RNA (mRNA) expression levels of adiponectin and its receptors in various bovine tissues and mammary glands among different lactation stages, and the effects of lactogenic hormones (insulin, dexamethasone and prolactin) and growth hormone (GH) on mRNA expression of the AdipoRs in cultured bovine mammary epithelial cells (BMEC). AdipoRs mRNAs were widely expressed in various bovine tissues, but adiponectin mRNA expression was significantly higher in adipose tissue than in other tissues. In the mammary gland, although adiponectin mRNA expression was significantly decreased at lactation, AdipoR1 mRNA expression was significantly higher at peak lactation than at the dry-off stage. In BMEC, lactogenic hormones and GH upregulated AdipoR2 mRNA expression but did not change that of AdipoR1. In conclusion, adiponectin and its receptor mRNA were expressed in various bovine tissues and the adiponectin mRNA level was decreased during lactation. These results suggest that adiponectin and its receptors ware changed in mammary glands by lactation and that AdipoRs mRNA expression was regulated by different pathways in BMEC.

Prostatic androgen-repressed message-1 as a regulator of adipocyte differentiation in the mouse.

Adipocyte differentiation is an important aspect in energy homeostasis. Although the regulation of adipocyte differentiation is relatively well understood, the underlying molecular mechanism remains unclear. In this study, subcutaneous and epididymal adipose tissues were used to study the differential expression of associated genes. We found that the expression level of mouse homologue of rat prostatic androgen-repressed message-1 (mPARM-1) gene was higher in subcutaneous, perirenal and mesenteric adipose tissues than in epididymal adipose tissue. In mouse subcutaneous, perirenal, and mesenteric adipose tissues, the expression level of this gene was higher in adipocytes than in non-adipocyte cells, i.e. stromal-vascular cells. Furthermore, mPARM-1 mRNA expression was up-regulated in subcutaneous, mesenteric, and epididymal adipose tissues of mice fed a high-fat diet compared to those fed a normal-fat diet. Expression level of mPARM-1 mRNA increased in the early stage of the chemically induced adipocyte differentiation, preceding the increase in peroxisome proliferator-activated receptor-gamma 2 (PPAR-gamma2) mRNA. Tumor necrosis factor-alpha (TNF-alpha), an inhibitor of adipocyte differentiation, reduced the expression of mPARM-1 mRNA in differentiated 3T3-L1 cells and subsequently down-regulated the expression of adipogenic genes, including PPAR-gamma2, leptin and adipogenin. Moreover, knockdown of mPARM-1 expression with siRNA reduced lipid accumulation and the expression levels of PPAR-gamma2 and adipocyte protein 2 mRNAs, which suggest that the degree of adipocyte differentiation of 3T3-L1 cells has been reduced. These results indicate that mPARM-1 might be involved in the regulation of fat accumulation and adipocyte differentiation.

Chemerin--a new adipokine that modulates adipogenesis via its own receptor.

Chemerin, an 18 kDa protein secreted by adipose tissue, was reported to modulate immune system function through its binding to the chemerin receptor (chemerinR). We herein demonstrate that chemerin also influences adipose cell function. Our data showed that chemerin and chemerinR mRNA expressions were highly expressed in adipose tissues, and that their expression levels were up-regulated in mice fed a high-fat diet. Both chemerin and chemerinR mRNA expression dramatically increased during the differentiation of 3T3-L1 cells and human preadipocytes into adipocytes. Furthermore, recombinant chemerin induced the phosphorylation of extracellular signal-regulated kinases 1/2 (ERK 1/2) and lipolysis in differentiated 3T3-L1 adipocytes. Thus, the adipokine chemerin likely regulates adipocyte function by autocrine/paracrine mechanisms.

The regulation of adipogenesis through GPR120.

Recently, it has been found that long-chain fatty acids activate the G protein-coupled receptors (GPRs), GPR120 and GPR40. However, there have been no reports to date on the possible physiological roles of these GPRs in adipose tissue development and adipocyte differentiation. GPR120 mRNA was highly expressed in the four different adipose tissues, and the amount of mRNA was elevated in adipose tissues of mice fed a high fat diet. However, GPR40 mRNA was not detected in any of the adipose tissues. The expression of GPR120 mRNA was higher in adipocytes compared to stromal-vascular (S-V) cells. The level of GPR120 mRNA increased during adipocyte differentiation in 3T3-L1 cells. Similar results were observed in human adipose tissue, human preadipocytes, and cultured adipocytes. Moreover, use of a small interference RNA (siRNA) to down-regulate GPR120 expression resulted in inhibition of adipocyte differentiation. Our results suggest that GPR120 regulates adipogenic processes such as adipocyte development and differentiation.

Up-regulation of the claudin-6 gene in adipogenesis.

To investigate the role of claudin-6 in adipogenesis, claudin-6 mRNA was examined in adipose tissues and adipocyte differentiation. Claudin-6 mRNA was found to be differentially expressed in four different adipose tissues, and up-regulated in each fat depot of mice fed a high-fat diet as compared to a normal-fat diet. Levels of claudin-6 transcripts were increased during differentiation of 3T3-L1 cells in vitro. Moreover, small interfering RNA (siRNA)-mediated reduction of claudin-6 mRNA inhibited differentiation of 3T3-L1 cells. These results suggest that claudin-6 is another important regulator in adipogenesis and fat deposition.