Deepfakes for Video Conferencing Using General Adversarial Networks (GANs) and Multilingual Voice Cloning

ABSTRACT

Covid-19 pandemic led to remote working and hence resulting in more video conferences among all sectors. Even important international conferences between different nations are being conducted on online video conferencing platforms. Hence, a methodology capable of performing real-time end-to-end speech translation has become a necessity. In this paper, we have proposed a complete pipeline methodology, wherein the real-time video conferencing will become interactive, and it can be used in the educational section for generating videos of instructors from just their images and textual notes. We are using automatic voice translation (AVT), text-to-stream machine translation (MT), and text-to-voice generator for voice cloning and translation in real time. For video generation, we use general adversarial networks (GANs), encoder-decoder, and various other previously implemented generative models. The proposed methodology has been implemented and tested with some raw data and is quite effective for the specified application. © 2022, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.