Exploiting Diverse Contextual Features through Transformers for Detecting Informative Tweets

ABSTRACT

Twitter is deemed the most reliable and convenient microblogging platform for getting real-time news and information. During the COVID-19 pandemic, people are keen to share various information ranging from new cases, healthcare guidelines, medication, and vaccine news on Twitter. However, a major portion of the shared tweets is uninformative and misleading which may create mass panic. Hence, it is an important task to distinguish and label a COVID-19 tweet as informative or uninformative. Prior works mostly focused on various pretrained transformer models and different types of contextual feature extractors to address this task. However, most of the works applied these models one at a time and didn't employ any effective neural layer at the bottom to distill the tweet contexts effectively. Since a tweet may contain a multifarious context, therefore, representing a tweet using only one kind of feature extractor may not work well. To overcome this limitation, we present an approach that leverages an ensemble of various cutting-edge transformer models to capture the diverse contextual dimension of the tweets. We exploit the BERT, CTBERT, BERTweet, RoBERTa, and XLM-RoBERTa models in our proposed method. Next, we perform a pooling operation on those extracted embedding features to transform them into document embedding vectors. Then, we utilize a feed-forward neural architecture with a linear activation function for the classification task. To generate final prediction, we utilize the majority voting-driven ensemble technique. Experiments on WNUT-2020 COVID-19 English Tweet dataset manifested the efficacy of our method over other state-of-the-art methods. © 2022 IEEE.