Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)-Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study.

BACKGROUND: The bidirectional encoder representations from transformers (BERT) model has achieved great success in many natural language processing (NLP) tasks, such as named entity recognition and question answering. However, little prior work has explored this model to be used for an important task in the biomedical and clinical domains, namely entity normalization. OBJECTIVE: We aim to investigate the effectiveness of BERT-based models for biomedical or clinical entity normalization. In addition, our second objective is to investigate whether the domains of training data influence the performances of BERT-based models as well as the degree of influence. METHODS: Our data was comprised of 1.5 million unlabeled electronic health record (EHR) notes. We first fine-tuned BioBERT on this large collection of unlabeled EHR notes. This generated our BERT-based model trained using 1.5 million electronic health record notes (EhrBERT). We then further fine-tuned EhrBERT, BioBERT, and BERT on three annotated corpora for biomedical and clinical entity normalization: the Medication, Indication, and Adverse Drug Events (MADE) 1.0 corpus, the National Center for Biotechnology Information (NCBI) disease corpus, and the Chemical-Disease Relations (CDR) corpus. We compared our models with two state-of-the-art normalization systems, namely MetaMap and disease name normalization (DNorm). RESULTS: EhrBERT achieved 40.95% F1 in the MADE 1.0 corpus for mapping named entities to the Medical Dictionary for Regulatory Activities and the Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT), which have about 380,000 terms. In this corpus, EhrBERT outperformed MetaMap by 2.36% in F1. For the NCBI disease corpus and CDR corpus, EhrBERT also outperformed DNorm by improving the F1 scores from 88.37% and 89.92% to 90.35% and 93.82%, respectively. Compared with BioBERT and BERT, EhrBERT outperformed them on the MADE 1.0 corpus and the CDR corpus. CONCLUSIONS: Our work shows that BERT-based models have achieved state-of-the-art performance for biomedical and clinical entity normalization. BERT-based models can be readily fine-tuned to normalize any kind of named entities.

Herbacetin treatment remitted LPS induced inhibition of osteoblast differentiation through blocking AKT/NF-κB signaling pathway.

Inflammation, a common situation during the process of bone healing, is reported to play a negative role in bone regeneration. Up to date, therapeutic strategies for inflammation triggered inhibition of osteoblast differentiation are still limited. The aim of this study was to explore the potential roles and molecular mechanisms of Herbacetin in the process of osteoblast differentiation under LPS-mediated inflammatory environment. By using MC3T3-E1, C2C12 and primary mouse calvarial osteoblast (PMCO) cells as experimental models, we observed that LPS stimulation suppressed osteoblast differentiation via inhibiting alkaline phosphatase (ALP) activity and the expression of several osteoblastic genes (osterix, runx2 and osteocalcin). However, the negative role of LPS during osteoblast differentiation could be restored by Herbacetin treatment. Mechanistical studies revealed that Herbacetin treatment suppressed AKT activation and in turn blocked NF-κB signaling pathway. Furthermore, reactivating AKT by a selective PTEN inhibitor SF1670 suppressed the effect of Herbacetin. These data suggested that Herbacetin might play a protective role in osteoblast differentiation in MC3T3-E1/C2C12/PMCO cells under LPS stimulation.

Generating Classical Chinese Poems from Vernacular Chinese.

Classical Chinese poetry is a jewel in the treasure house of Chinese culture. Previous poem generation models only allow users to employ keywords to interfere the meaning of generated poems, leaving the dominion of generation to the model. In this paper, we propose a novel task of generating classical Chinese poems from vernacular, which allows users to have more control over the semantic of generated poems. We adapt the approach of unsupervised machine translation (UMT) to our task. We use segmentation-based padding and reinforcement learning to address under-translation and over-translation respectively. According to experiments, our approach significantly improve the perplexity and BLEU compared with typical UMT models. Furthermore, we explored guidelines on how to write the input vernacular to generate better poems. Human evaluation showed our approach can generate high-quality poems which are comparable to amateur poems.

Protective effect of HO-1 transfection against ethanol-induced osteoblast damage.

Heme oxygenase-1 (HO-1) plays important roles in anti-oxidant, anti-inflammatory and immunoregulative activities. The aim of this study was to observe if HO-1 transfection could inhibit the damage of osteoblasts induced by ethanol. HO-1 was transfected into osteoblasts via constructed plasmid. After exposure to ethanol for 24 h, cytoactivity and apoptosis of osteoblasts were measured by MTT assay and flow cytometry, respectively. Furthermore, the oxidative stress and inflammatory factors in osteoblasts were measured. Compared to positive control group, the cytoactivity of transfected osteoblasts was significantly increased, and the apoptosis rate was significantly decreased (P<0.05). At the same time, the levels of reactive oxygen species (ROS), methane dicarboxylic aldehyde (MDA), tumor necrosis factor-α (TNF-α) and interleukin-1 (IL-1) were significantly decreased (P<0.05), and superoxide dismutase (SOD) level was increased (P<0.05) in the transfected osteoblasts as compared with positive controls. These results suggest that HO-1 plays a protective role in osteoblasts, and HO-1 transfection can effectively inhibit bone damage induced by ethanol.