REAL-TIME MASKED FACE REVEALING FOR VIDEO CONFERENCE

ABSTRACT

Video conferencing is an essential way for contactless conversation, which conveys abundant multimedia signals. Especially under COVID-19, the video conference has been becoming a common way for daily communications. However, for the sake of plague prevention, it usually happens that the people attending the video conference are wearing a mouth mask, leading to inconvenient communication due to incomplete facial information. To tackle this problem, we develop a novel system that reveals the masked faces in real-time, making each participant feel like the others are mask-free. Moreover, we map the audio to 3DMM expression to guide the generation of various mouth shapes utilizing multi-modal information. Extensive experiments validate the revealing effectiveness and better user experience of the system. Furthermore, by applying lightweight networks design, the proposed system can run in real-time. © 2021 IEEE