A Two-Stream Model Based on 3D Convolutional Neural Networks for the Recognition of Brazilian Sign Language in the Health Context

ABSTRACT

Deaf people are a considerable part of the world population and communicate naturally using sign languages. However, although many countries adopt their sign language as an official language, there are linguistics barriers to accessing fundamental rights, especially access to health services, even more critical situation in the midst of the COVID-19 crisis. This situation has been the focus of some government policies that oblige essential service providers to provide sign language interpreters to assist Deaf people. However, this type of solution has high operating costs, mainly to serve the entire Deaf community in all environments. These setbacks motivate the investigation of methodologies and automated tools to support this type of problem. In this paper, we address this problem by proposing two-stream model for the recognition of the Brazilian Sign Language (Libras) in the health context. The proposed solution does not use any additional capture sensor or hardware, being entirely base on images or sequences of images (videos). The results show that the solution is able to recognize the Libras signs in the test dataset reasonably well, achieved an average accuracy of approximately 96,12% considering a scenario where the interpreter used in the test set was not used in the training set, which shows that there are good evidence that it can assist in the communication process with Deaf people. Besides, an additional contribution of this paper is the introduction of a new dataset in the Brazilian sign language (Libras) containing 5000 videos of 50 signs in the health context, which may assist the development and research of other solutions. © 2020 ACM.