Visual and Haptic Feedback Fusion Based on Force Tracking in Upper-limb Rehabilitation Robot System / 中国康复理论与实践

ABSTRACT

Objective:To solve the issue regarding a low correlation between visual and haptic feedback provided by the current upper-limb rehabilitation training system, this study was implemented based on the end-effector based upper-limb rehabilitation robot developed in the lab. A novel visual and haptic feedback fusion technology based on force tracking was investigated and its effect on upper-limb training was also studied. Methods:Based on the force model constructed in a virtual environment, two types of haptic feedbacks correlated to the visual feedback were designed, including the repulsive force when two objects getting close and the friction force when the object moving above medium surfaces. The haptic feedback constructed in the virtual environment was delivered to the trainees by using force tracking based on robot controlling algorithm. Eight health subjects were recruited and trained with and without feedback fusion. In the training process, the actual and expected haptic feedbacks as well as the surface electromyography (EMG) signals from anterior deltoid, posterior deltoid, biceps, and triceps were collected. The root means square error (RMSE) between the actual and expected haptic feedback was calculated under the feedback fusion training mode to characterize the force tracking-based multi-sensory feedback fusion technology. The integrated EMG values (iEMG) and EMG amplitudes per unit time (EMG/T) under two training modes were measured to explore the effect of feedback fusion technology on the upper-limb motor training. Results:Under feedback fusion training mode, the RMSE between actual and expected haptic feedback was (0.757±0.171) N. The values of iEMG from four muscles were significantly higher (|t| > 7.965, P < 0.001), and the values of EMG/T from the biceps, triceps and anterior deltoid were significantly larger under feedback fusion training mode than under the training mode without feedback fusion. Conclusion:The proposed upper-limb rehabilitation robot training system could accurately transmit the haptic feedback constructed under the virtual environment to the trainees. This system could increase the stimulation to trainees' peripheral nervous function through visual and haptic feedback fusion as well as increase the trainees' training effort. The advantages of force tracking-based visual and haptic feedback fusion technology are to freely construct the force model under the virtual environment and the haptic feedback mode is not constrained by the spatial position. Moreover, two or more types of force models can be superimposed in the same spatial position by using this technology that could improve the matching effect between haptic feedback and visual feedback under a virtual environment. The trainees' motor rehabilitation interest could be stimulated and the experience feeling of human-robot interaction could also be enhanced.