Rendering Personalized Real-Time Expressions While Speaking Under a Mask

ABSTRACT

During COVID-19, people often wear masks in daily activities or communication. To solve the problem of generating faces with expressions under masks, we propose a framework of methods, including detecting the shape or locations of masked faces, generating the facial expressions under masks. Further, due to synthesizing quality facial expressions, we propose to optimize the merging of sub-results with useful face information such as key points of face. Further, we propose a framework for customization or personalization of user-preferring AI-generation results. We showed the system capable of running real-time and discussed the development in multiple aspects of research, interface, and applications. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.