PET: Parameter-efficient Knowledge Distillation on Transformer.

Given a large Transformer model, how can we obtain a small and computationally efficient model which maintains the performance of the original model? Transformer has shown significant performance improvements for many NLP tasks in recent years. However, their large size, expensive computational cost, and long inference time make it challenging to deploy them to resource-constrained devices. Existing Transformer compression methods mainly focus on reducing the size of the encoder ignoring the fact that the decoder takes the major portion of the long inference time. In this paper, we propose PET (Parameter-Efficient knowledge distillation on Transformer), an efficient Transformer compression method that reduces the size of both the encoder and decoder. In PET, we identify and exploit pairs of parameter groups for efficient weight sharing, and employ a warm-up process using a simplified task to increase the gain through Knowledge Distillation. Extensive experiments on five real-world datasets show that PET outperforms existing methods in machine translation tasks. Specifically, on the IWSLT'14 EN→DE task, PET reduces the memory usage by 81.20% and accelerates the inference speed by 45.15% compared to the uncompressed model, with a minor decrease in BLEU score of 0.27.

A comparative study of bone formation following grafting with different ratios of particle dentin and tricalcium phosphate combinations.

This study aimed to evaluate the bone regeneration relative to tooth powder and tricalcium phosphate (TCP) mixing ratios using the rabbit cranium defect model. The tooth powder was mixed with TCP in 1:1, 3:1, and 1:3 ratios, and the different ratios were implanted in the rabbit cranium defect for 4 and 8 weeks. Powders crystal structure evaluated using scanning electron microscopy (SEM), attenuated total reflectance-Fourier transform infrared spectroscopy (ATR-FTIR), and new bone formation (NBF) was analyzed using micro-computed tomography (CT) and histologic examination. NBF in the control group was restricted to the defect margins. More NBF was observed around the defect margins in the experimental groups compared with the control group. Specifically, active NBF was identified around the implant materials of the centrifugal part of the defect and defect margins in the 3:1 tooth powder: TCP group. Our results suggested that tooth powder and TCP may be useful in bone regeneration.

Effect of platelet-rich plasma and platelet-rich fibrin on peri-implant bone defects in dogs.

The purpose of this study was to evaluate the effect of tooth ash and platelet-rich plasma (PRP), and platelet-rich fibrin (PRF) grafts into bone defects around implants on bone formation. Six adult dogs were used as experimental subjects. Graft materials were used to create a particulate material. Forty-eight tapered-type implants, 3.7 mm in diameter, 10 mm in length, and with surface treated with hydroxyapatite (HA) coating, were used as implant fixtures. Using a trephine bur, four bone defects were formed and implants were placed in the femurs of the adult dogs. Bone grafts were not performed in the control group. Tooth ash was grafted into the defects in group 1. In group 2, a mixture of tooth ash and PRP (1:1 ratio by volume) was grafted into the defects. In group 3, a mixture of tooth ash and PRF (ratio of 1:1) was grafted in the defect area. Animals were sacrificed after 4 or 8 weeks. Based on histopathological examination, the amount and rate of new bone formation were evaluated. Histomorphometric examination revealed that the rate of new bone formation in group 3 of the 4-week group was significantly higher than that in the control group. In addition, in the 8-week group, a significant increase in new bone formation was confirmed in group 3. In this study, a bone graft method using a mixture of tooth ash and PRF was found to increase new bone formation compared to the method using PRP. In addition, it was confirmed that this effect was more prominent in the initial stage of the bone graft.

Novel insertion mutation of ABCB1 gene in an ivermectin-sensitive Border Collie.

P-glycoprotein (P-gp) is encoded by the ABCB1 gene and acts as an efflux pump for xenobiotics. In the Border Collie, a nonsense mutation caused by a 4-base pair deletion in the ABCB1 gene is associated with a premature stop to P-gp synthesis. In this study, we examined the full-length coding sequence of the ABCB1 gene in an ivermectin-sensitive Border Collie that lacked the aforementioned deletion mutation. The sequence was compared to the corresponding sequences of a wild-type Beagle and seven ivermectin-tolerant family members of the Border Collie. When compared to the wild-type Beagle sequence, that of the ivermectin-sensitive Border Collie was found to have one insertion mutation and eight single nucleotide polymorphisms (SNPs) in the coding sequence of the ABCB1 gene. While the eight SNPs were also found in the family members' sequences, the insertion mutation was found only in the ivermectin-sensitive dog. These results suggest the possibility that the SNPs are species-specific features of the ABCB1 gene in Border Collies, and that the insertion mutation may be related to ivermectin intolerance.

Gastrointestinal tract cancer screening using fecal carcinoembryonic antigen.

There is a great need to detect gastrointestinal tract cancer at an early stage. It is well known that most carcinoma tissues of the gastrointestinal tract contain carcinoembryonic antigen (CEA). Stools are a rich source of cells derived from the gastrointestinal tract. We analyzed total fecal CEA in 60 gastrointestinal tract cancer patients, 20 benign gastrointestinal tract disorder patients, and 240 normal controls, using a simple, reliable method. We compared the sensitivity and specificity of fecal CEA with those of serum CEA and fecal occult blood test (FOBT). The level of fecal CEA in gastrointestinal tract cancer was much higher than controls (44.1 +/- 70.1 ng/mg stool vs 3.7 +/- 3.5 ng/mg stool, p < 0.001) and was not increased in benign gastrointestinal disorders (4.5 +/- 8.2 ng/mg stool). Fecal CEA level was > 10 ng/mg stool in 22 of 32 samples (69%)from stomach cancer patients and 24 of 28 samples (86%)from colorectal cancer patients. The sensitivity of serum CEA (> 5 ng/ml) was 19% in stomach cancer and 39% in colorectal cancer, whereas the sensitivity of FOBT was 13% in stomach cancer and 21% in colorectal cancer. The specificity of fecal CEA was 90% in benign gastrointestinal tract disorders and 93% in normal controls. This specificity was similar to those of serum CEA and FOBT. In conclusion, fecal CEA measurement is superior to serum CEA or FOBT for detection of gastrointestinal tract cancer. Fecal CEA may become the screening test of choice for gastrointestinal tract cancer.