Assessing Deep Learning Approaches in Detecting Masked Facial Expressions

ABSTRACT

Automated facial expression recognition (FER) is an active research area due to its practical importance in a wide range of applications. In recent years, deep learning-based approaches have delivered promising performances in FER, leveraging the latest advances in computer vision. However, mask-wearing after the onset of the COVID-19 pandemic has posed challenges for the existing models when the salient features from the masked region are unavailable. This study investigates what effects facial masks will bring to expression detection using state-of-the-art deep learners. Specifically, we evaluate three deep neural networks in recognizing six emotional categories on masked facial images and compare the results to unmasked counterparts reported by prior studies. We based our work on the FER2013 dataset and augmented regular face images with artificial masks utilizing the Dlib and OpenCV libraries. Our experimental results indicate that deep learning models can be effective in recognizing some masked expressions (e.g., 'Happy', 'Surprise', and 'Neural') but fall short on the others (e.g., 'Angry', 'Fear', 'Sad'). Furthermore, with the presence of facial masks, angry faces are most likely to be misclassified as neural, and fear is the most challenging emotion to detect. © 2022 IEEE.