Prototype-based sample-weighted distillation unified framework adapted to missing modality sentiment analysis.

Missing modality sentiment analysis is a prevalent and challenging issue in real life. Furthermore, the heterogeneity of multimodality often leads to an imbalance in optimization when attempting to optimize the same objective across all modalities in multimodal networks. Previous works have consistently overlooked the optimization imbalance of the network in cases when modalities are absent. This paper presents a Prototype-Based Sample-Weighted Distillation Unified Framework Adapted to Missing Modality Sentiment Analysis (PSWD). Specifically, it fuses features with a more efficient transformer-based cross-modal hierarchical cyclic fusion module. Subsequently, we propose two strategies, namely sample-weighted distillation and prototype regularization network, to address the issues of missing modality and optimization imbalance. The sample-weighted distillation strategy assigns higher weights to samples that are located closer to class boundaries. This facilitates the obtaining of complete knowledge by the student network from the teacher's network. The prototype regularization network calculates a balanced metric for each modality, which adaptively adjusts the gradient based on the prototype cross-entropy loss. Unlike conventional approaches, PSWD not only connects the sentiment analysis study in the missing modality to the full modality, but the proposed prototype regularization network is not reliant on the network structure and can be expanded to more multimodal studies. Massive experiments conducted on IEMOCAP and MSP-IMPROV show that our method achieves the best results compared to the latest baseline methods, which demonstrates its value for application in sentiment analysis.

Modality attention fusion model with hybrid multi-head self-attention for video understanding.

Video question answering (Video-QA) is a subject undergoing intense study in Artificial Intelligence, which is one of the tasks which can evaluate such AI abilities. In this paper, we propose a Modality Attention Fusion framework with Hybrid Multi-head Self-attention (MAF-HMS). MAF-HMS focuses on the task of answering multiple-choice questions regarding a video-subtitle-QA representation by fusion of attention and self-attention between each modality. We use BERT to extract text features, and use Faster R-CNN to ex-tract visual features to provide a useful input representation for our model to answer questions. In addition, we have constructed a Modality Attention Fusion (MAF) framework for the attention fusion matrix from different modalities (video, subtitles, QA), and use a Hybrid Multi-headed Self-attention (HMS) to further determine the correct answer. Experiments on three separate scene datasets show our overall model outperforms the baseline methods by a large margin. Finally, we conducted extensive ablation studies to verify the various components of the network and demonstrate the effectiveness and advantages of our method over existing methods through question type and required modality experimental results.