MagTrack: A Wearable Tongue Motion Tracking System for Silent Speech Interfaces.

PURPOSE: Current electromagnetic tongue tracking devices are not amenable for daily use and thus not suitable for silent speech interface and other applications. We have recently developed MagTrack, a novel wearable electromagnetic articulograph tongue tracking device. This study aimed to validate MagTrack for potential silent speech interface applications. METHOD: We conducted two experiments: (a) classification of eight isolated vowels in consonant-vowel-consonant form and (b) continuous silent speech recognition. In these experiments, we used data from healthy adult speakers collected with MagTrack. The performance of vowel classification was measured by accuracies. The continuous silent speech recognition was measured by phoneme error rates. The performance was then compared with results using data collected with commercial electromagnetic articulograph in a prior study. RESULTS: The isolated vowel classification using MagTrack achieved an average accuracy of 89.74% when leveraging all MagTrack signals (x, y, z coordinates; orientation; and magnetic signals), which outperformed the accuracy using commercial electromagnetic articulograph data (only y, z coordinates) in our previous study. The continuous speech recognition from two subjects using MagTrack achieved phoneme error rates of 73.92% and 66.73%, respectively. The commercial electromagnetic articulograph achieved 64.53% from the same subject (66.73% using MagTrack data). CONCLUSIONS: MagTrack showed comparable results with the commercial electromagnetic articulograph when using the same localized information. Adding raw magnetic signals would improve the performance of MagTrack. Our preliminary testing demonstrated the potential for silent speech interface as a lightweight wearable device. This work also lays the foundation to support MagTrack's potential for other applications including visual feedback-based speech therapy and second language learning.

Improving Reliability of Magnetic Localization Using Input Space Transformation.

Body motion tracking for medical applications has the potential to improve quality of life for people with physical or speech motor disorders. Current solutions available in the market are either inaccurate, not affordable, or are impractical for a medical setting or at home. Magnetic localization can address these issues thanks to its high accuracy, simplicity of use, wearability, and use of inexpensive sensors such as magnetometers. However, sources of unreliability affect magnetometers to such an extent that the localization model trained in a controlled environment might exhibit poor tracking accuracy when deployed to end users. Traditional magnetic calibration methods, such as ellipsoid fit (EF), do not sufficiently attenuate the effect of these sources of unreliability to reach a positional accuracy that is both consistent and satisfactory for our target applications. To improve reliability, we developed a calibration method called post-deployment input space transformation (PDIST) that reduces the distribution shift in the magnetic measurements between model training and deployment. In this paper, we focused on change in magnetization or magnetometer as sources of unreliability. Our results show that PDIST performs better than EF in decreasing positional errors by a factor of ~3 when magnetization is distorted, and up to ~7 when our localization model is tested on a different magnetometer than the one it was trained with. Furthermore, PDIST is shown to perform reliably by providing consistent results across all our data collection that tested various combinations of the sources of unreliability.

Evaluation of a Head-Tongue Controller for Power Wheelchair Driving by People With Quadriplegia.

The head-tongue controller (HTC) is a multimodal alternative controller designed for people with quadriplegia to access complex control capabilities by combining tongue and head tracking to offer both discrete and proportional controls in a single controller. In this human study, 17 patients with quadriplegia and current users of alternative controllers were asked to perform four trials of either simple driving tasks or advanced maneuvers in a custom-designed course. Completion time and accuracy were compared between their personal alternative controller (PAC) and various combinations of driving modalities with the HTC. Out of 8 subjects assigned to simple driving, the best HTC trial of 3 subjects was completed faster than their PAC for the tasks of rolling forward and turning around cones, and 5 subjects in rolling backward. Across all these subjects, the average completion time of their best HTC modality is 23 s for rolling forward, 15 s for rolling backward, and 70 s for turning around cones as compared to 19 s, 17 s, and 45 s with their PAC. For advanced driving, the course was completed faster with the HTC by 1 out of 9 subjects, while the best HTC trials of all subjects are less than 1.3 times of their best PAC completion time with an average of 170 s for the HTC and 140 s for their PAC. The qualitative feedback provided by all subjects to a post-study questionnaire scored to an average of 7.5 out of 10 which shows their interests in the HTC and acknowledgement of its usefulness for this population.

Inertial Measurements for Tongue Motion Tracking Based on Magnetic Localization with Orientation Compensation.

Permanent magnet localization (PML) is designed for applications requiring non-line-of-sight motion tracking with millimetric accuracy. Current PML-based tongue tracking is not only impractical for daily use due to many sensors being placed around the mouth, but also requires a large training set of tracer motion. Our method was designed to overcome these shortcomings by generating a local magnetic field and removing the need for the localization to be trained with tracer rotations. An inertial measurement unit (IMU) is used as a tracer that moves in a local magnetic field generated by a magnet strip. The magnetic strength can be optimized to enable the strip to be placed further away from the tracer, thus hidden from view. The tracer is small (6×6×0.8 mm3) to reduce hindrance to natural tongue movements, and the strip is designed to be worn as a neckband. The IMU's magnetometer measures the local magnetic field which is compensated for the tracer's orientation by using the IMU's accelerometer and gyroscope. The orientation-compensated magnetic measurements are then fed into a localization algorithm that estimates the tracer's 3D position. The objective of this study is to evaluate the tracking accuracy of our method. In a 8×8×5 cm3 volume, positional errors of 1.6 mm (median) and 2.4 mm (third quartile, Q3) were achieved on a tracer being rotated ±50° along both pitch and roll. These results indicate this technology is promising for tongue tracking applications.

Evaluation of a Wireless Tongue Tracking System on the Identification of Phoneme Landmarks.

OBJECTIVE: Evaluate the accuracy of a tongue tracking system based on the localization of a permanent magnet to generate a baseline of phoneme landmarks. The positional variability of the landmarks provides an indirect measure of the tracking errors to estimate the position of a small tracer attached on the tongue. The creation of a subject-independent (universal) baseline was also attempted for the first time. METHOD: 2,500 tongue trajectories were collected from 10 subjects tasked to utter 10 repetitions of 25 phonemes. A landmark was identified from each tongue trajectory, and tracking errors were calculated by comparing the distance of each repetition landmark to a final landmark set as their mean position. RESULTS: In the subject-dependent baseline, the tracking errors were found to be generally consistent across all phonemes, and subjects, with less than 25% of the errors reported to be greater than 5.8 mm (median: 3.9 mm). However, the inter-subject variability showed that current limitations of our system resulted in appreciable errors (median: 55 mm, Q3: 65 mm). CONCLUSION: The tracking errors reported in the subject-dependent case demonstrated the potential of our system to generate a baseline of phoneme landmarks. We have identified areas of improvement that will reduce the gap between the subject-dependent, and universal baseline, while lowering tracking errors to be comparable to the gold standard. SIGNIFICANCE: Creating a baseline of phoneme landmarks can help people affected by speech sound disorders to improve their intelligibility using visual feedback that guides their tongue placement to the proper position.

Multimodal Speech Capture System for Speech Rehabilitation and Learning.

Speech-language pathologists (SLPs) are trained to correct articulation of people diagnosed with motor speech disorders by analyzing articulators' motion and assessing speech outcome while patients speak. To assist SLPs in this task, we are presenting the multimodal speech capture system (MSCS) that records and displays kinematics of key speech articulators, the tongue and lips, along with voice, using unobtrusive methods. Collected speech modalities, tongue motion, lips gestures, and voice are visualized not only in real-time to provide patients with instant feedback but also offline to allow SLPs to perform post-analysis of articulators' motion, particularly the tongue, with its prominent but hardly visible role in articulation. We describe the MSCS hardware and software components, and demonstrate its basic visualization capabilities by a healthy individual repeating the words "Hello World." A proof-of-concept prototype has been successfully developed for this purpose, and will be used in future clinical studies to evaluate its potential impact on accelerating speech rehabilitation by enabling patients to speak naturally. Pattern matching algorithms to be applied to the collected data can provide patients with quantitative and objective feedback on their speech performance, unlike current methods that are mostly subjective, and may vary from one SLP to another.

A Vision-Based Respiration Monitoring System for Passive Airway Resistance Estimation.

OBJECTIVE: Airway resistance is the mechanical cause of most of the symptoms in obstructive pulmonary disease, and can be considered as the primary measure of disease severity. A low-cost and noninvasive method to measure the airway resistance that does not require patient effort could be of great benefit in evaluating the severity of lung diseases, especially in patient population that are unable to use spirometry, such as young children. METHODS: The Vision-Based Passive Airway Resistance Estimation (VB-PARE) technology is a passive method to measure airway resistance noninvasively. The airway resistance is estimated from: 1) airflow extracted from processing depth data captured by a Microsoft Kinect, and 2) Pulsus Paradoxus extracted from a pulse oximeter (SpO 2). RESULTS: To verify the validity and accuracy of the VB-PARE, two phases of experiment were conducted. In Phase I, spontaneous breathing data was collected from 14 healthy participants with externally induced airway obstruction, and the accuracy of 76.2±13.8% was achieved in predicting three levels of obstruction severity. In Phase II, VB-PARE outputs were compared with the clinical results from 14 patients. VB-PARE estimated the tidal volume with an average error of 0.07±0.06 liter. Also, patients with airway obstruction were detected with 80% accuracy. CONCLUSION: Using the information extracted from Kinect and SpO 2 , here, we present a quantitative method to measure the severity of airway obstruction without requiring active patient involvement. SIGNIFICANCE: The proposed VB-PARE system contributes to the state-of-art respiration monitoring methods by expanding the idea of passive and noninvasive airway resistance measurement.

Tongue-controlled robotic rehabilitation: A feasibility study in people with stroke.

Stroke survivors with severe upper limb (UL) impairment face years of therapy to recover function. Robot-assisted therapy (RT) is increasingly used in the field for goal-oriented rehabilitation as a means to improve function in ULs. To be used effectively for wrist and hand therapy, the current RT systems require the patient to have a minimal active range of movement in the UL, and those that do not have active voluntary movement cannot use these systems. We have overcome this limitation by harnessing tongue motion to allow patients to control a robot using synchronous tongue and hand movement. This novel RT device combines a commercially available UL exoskeleton, the Hand Mentor, and our custom-designed Tongue Drive System as its controller. We conducted a proof-of-concept study on six nondisabled participants to evaluate the system usability and a case series on three participants with movement limitations from poststroke hemiparesis. Data from two stroke survivors indicate that for patients with chronic, moderate UL impairment following stroke, a 15-session training regimen resulted in modest decreases in impairment, with functional improvement and improved quality of life. The improvement met the standard of minimal clinically important difference for activities of daily living, mobility, and strength assessments.