Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition.

The teacher-free online Knowledge Distillation (KD) aims to train an ensemble of multiple student models collaboratively and distill knowledge from each other. Although existing online KD methods achieve desirable performance, they often focus on class probabilities as the core knowledge type, ignoring the valuable feature representational information. We present a Mutual Contrastive Learning (MCL) framework for online KD. The core idea of MCL is to perform mutual interaction and transfer of contrastive distributions among a cohort of networks in an online manner. Our MCL can aggregate cross-network embedding information and maximize the lower bound to the mutual information between two networks. This enables each network to learn extra contrastive knowledge from others, leading to better feature representations, thus improving the performance of visual recognition tasks. Beyond the final layer, we extend MCL to intermediate layers and perform an adaptive layer-matching mechanism trained by meta-optimization. Experiments on image classification and transfer learning to visual recognition tasks show that layer-wise MCL can lead to consistent performance gains against state-of-the-art online KD approaches. The superiority demonstrates that layer-wise MCL can guide the network to generate better feature representations. Our code is publicly avaliable at https://github.com/winycg/L-MCL.

Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution.

Knowledge distillation (KD) is an effective framework that aims to transfer meaningful information from a large teacher to a smaller student. Generally, KD often involves how to define and transfer knowledge. Previous KD methods often focus on mining various forms of knowledge, for example, feature maps and refined information. However, the knowledge is derived from the primary supervised task, and thus, is highly task-specific. Motivated by the recent success of self-supervised representation learning, we propose an auxiliary self-supervision augmented task to guide networks to learn more meaningful features. Therefore, we can derive soft self-supervision augmented distributions as richer dark knowledge from this task for KD. Unlike previous knowledge, this distribution encodes joint knowledge from supervised and self-supervised feature learning. Beyond knowledge exploration, we propose to append several auxiliary branches at various hidden layers, to fully take advantage of hierarchical feature maps. Each auxiliary branch is guided to learn self-supervision augmented tasks and distill this distribution from teacher to student. Overall, we call our KD method a hierarchical self-supervision augmented KD (HSSAKD). Experiments on standard image classification show that both offline and online HSSAKD achieves state-of-the-art performance in the field of KD. Further transfer experiments on object detection further verify that HSSAKD can guide the network to learn better features. The code is available at https://github.com/winycg/HSAKD.

CircINTS4 Facilitates Chemoresistance of TNBC by Competitively Binding miR-129-5p/POM121 Axis.

Objectives: To detect the expression of circular RNA (circRNA) circINTS4 in triple-negative breast cancer (TNBC) and to analyze the relationship between the expression of circRNA circINTS4 and the clinicopathological characteristics and chemotherapy resistance of patients with TNBC. Methods: Bioinformatics was used to predict that circINTS4 and POM121 could bind to miR-129-5p, and dual luciferase reporter genes proved that circINTS4 could bind to miR-129-5p and miR-129-5p could bind to POM121. RNA immunoprecipitation (RIP) and RNA pull-down experiments confirmed that circINTS4 binds to miR-129-5p. The correlation among circINTS4, miR-129-5p, and POM121 was detected by qRT-PCR. Results: In ADR-resistant TNB cells, circINTS4 was significantly up-regulated, miR-129-5p was down-regulated, and POM121 protein expression was significantly up-regulated. Experimental results showed that circINTS4 knockdown inhibited proliferation, migration, invasion, and autophagy. Knocking down miR-129-5p or overexpression of POM121 reversed the inhibitory effect of sh-circints4 on the development of ADR-resistant TNBC cells. In addition, CIRCINTS4 regulates POM121 expression by sponge-adsorbed miR-129-5p. CIRCINTS4 knockdown prevents ADR-resistant tumor growth by regulating the miR-129-5p/POM121 axis in vivo. Conclusions: CircRNA circINTS4 may act as the ceRNA of miR-129-5p to regulate the expression of target gene POM121, thereby promoting the progress of TNBC molecular mechanism and providing scientific basis for circINTS4 as a new molecular target for clinical diagnosis and drug resistance therapy of TNBC.