Telemedicine System Based on Mixed Reality and Cognitive Speech Service Technologies

ABSTRACT

Because of the prevalence of COVID-19, medical capacity reduction, and inconvenience due to epidemic prevention policies, the development of telemedicine has been emphasized. An AI sensing device (i.e., Azure Kinect DK) with a mixed reality device (i.e., HoloLens 2) and window application are combined to design a system that completes the consultation process from a distance. This registering system uses an SQL server database, which reads the data of registered patients, selects the registration time and department on its own, and converts the data by PC/SC with an ISO7816 database. For the window application, the patient side screen and audio content for the window screen are captured through the lens and microphone array of Azure Kinect DK. HoloLens 2 is used as the main device on the doctor's side to design the window application for dictating case conversion and transmission, whereas the Unity engine is used to create mixed-reality scenes for viewing medical records and photos of patients. In the consultation part, doctor and patient devices use Dynamic 365 Assist to conduct video consultations via Microsoft Teams, transforming the traditional consultation form into a remote consultation service via the Internet. The consultation process records the consultation data by voice and then the prescription by Speech to Text and Azure cognitive voice service. It is expected to develop a hybrid-reality telemedicine system and combine metaverse-related technologies for more online interactive service functions. The future development goal is to integrate the virtual reality rehabilitation training system further to make the application of telemedicine more complete. © 2022 IEEE.