Biomimetic learning of hand gestures in a humanoid robot.

Hand gestures are a natural and intuitive form of communication, and integrating this communication method into robotic systems presents significant potential to improve human-robot collaboration. Recent advances in motor neuroscience have focused on replicating human hand movements from synergies also known as movement primitives. Synergies, fundamental building blocks of movement, serve as a potential strategy adapted by the central nervous system to generate and control movements. Identifying how synergies contribute to movement can help in dexterous control of robotics, exoskeletons, prosthetics and extend its applications to rehabilitation. In this paper, 33 static hand gestures were recorded through a single RGB camera and identified in real-time through the MediaPipe framework as participants made various postures with their dominant hand. Assuming an open palm as initial posture, uniform joint angular velocities were obtained from all these gestures. By applying a dimensionality reduction method, kinematic synergies were obtained from these joint angular velocities. Kinematic synergies that explain 98% of variance of movements were utilized to reconstruct new hand gestures using convex optimization. Reconstructed hand gestures and selected kinematic synergies were translated onto a humanoid robot, Mitra, in real-time, as the participants demonstrated various hand gestures. The results showed that by using only few kinematic synergies it is possible to generate various hand gestures, with 95.7% accuracy. Furthermore, utilizing low-dimensional synergies in control of high dimensional end effectors holds promise to enable near-natural human-robot collaboration.

Data Fusion-Based Musculoskeletal Synergies in the Grasping Hand.

The hypothesis that the central nervous system (CNS) makes use of synergies or movement primitives in achieving simple to complex movements has inspired the investigation of different types of synergies. Kinematic and muscle synergies have been extensively studied in the literature, but only a few studies have compared and combined both types of synergies during the control and coordination of the human hand. In this paper, synergies were extracted first independently (called kinematic and muscle synergies) and then combined through data fusion (called musculoskeletal synergies) from 26 activities of daily living in 22 individuals using principal component analysis (PCA) and independent component analysis (ICA). By a weighted linear combination of musculoskeletal synergies, the recorded kinematics and the recorded muscle activities were reconstructed. The performances of musculoskeletal synergies in reconstructing the movements were compared to the synergies reported previously in the literature by us and others. The results indicate that the musculoskeletal synergies performed better than the synergies extracted without fusion. We attribute this improvement in performance to the musculoskeletal synergies that were generated on the basis of the cross-information between muscle and kinematic activities. Moreover, the synergies extracted using ICA performed better than the synergies extracted using PCA. These musculoskeletal synergies can possibly improve the capabilities of the current methodologies used to control high dimensional prosthetics and exoskeletons.

Musculoskeletal Synergies in the Grasping Hand.

Investigations on how the central nervous system (CNS) effortlessly conducts complex hand movements have led to an extensive study of synergies or movement primitives. Of the different types of hand synergies, kinematic and muscle synergies have been widely studied in literature, but only a few studies have fused both. In this paper kinematic and muscle activities recorded from the activities of daily living were first fused and then dimensionally reduced through principal component analysis (PCA). By using these principal components or musculoskeletal synergies in a weighted linear combination, the recorded kinematics and muscle activities were reconstructed. The performance of these musculoskeletal synergies in reconstructing the movements was compared to the kinematic and muscle synergies reported previously in the literature by us and others. The results from these findings indicate that musculoskeletal synergies perform better than the synergies extracted without fusion. These newly demonstrated musculoskeletal synergies might improve neural control of robotics, prosthetics and exoskeletons. Clinical Relevance- In this paper, musculoskeletal synergies were extracted from the fusion of kinematic and muscle activities recorded from the activities of daily living. These newly demonstrated musculoskeletal synergies might enhance our understanding of neural control of robotics, prosthetics and exoskeletons.

Dynamical Synergies in Multidigit Hand Prehension.

Hand prehension requires a highly coordinated control of contact forces. The high dimensional sensorimotor system of the human hand although operates at ease, poses several challenges when replicated for prosthetic control. This study investigates how the dynamical synergies, coordinated spatial patterns of contact forces, contribute to the contact forces in a grasp, and whether the dynamical synergies could potentially serve as candidates for feedforward and feedback mechanisms. Ten right-handed subjects were recruited to grasp and hold mass-varied objects. The contact forces during this multidigit prehension were recorded using an instrumented grip glove. The dynamical synergies were derived using principal component analysis (PCA). The contact force patterns during the grasps were reconstructed using the first few synergies. The significance of the dynamical synergies and the current challenges and possible applications of the dynamical synergies were discussed along with the integration of the dynamical synergies into prosthetics and exoskeletons that can possibly enable near-natural control. This research presents dynamical synergies observed in contact forces during hand grasps. These dynamical synergies could help in improving feedforward force control and sensory feedback in hand prosthetics and exoskeletons.

Reconstructing Synergy-Based Hand Grasp Kinematics from Electroencephalographic Signals.

Brain-machine interfaces (BMIs) have become increasingly popular in restoring the lost motor function in individuals with disabilities. Several research studies suggest that the CNS may employ synergies or movement primitives to reduce the complexity of control rather than controlling each DoF independently, and the synergies can be used as an optimal control mechanism by the CNS in simplifying and achieving complex movements. Our group has previously demonstrated neural decoding of synergy-based hand movements and used synergies effectively in driving hand exoskeletons. In this study, ten healthy right-handed participants were asked to perform six types of hand grasps representative of the activities of daily living while their neural activities were recorded using electroencephalography (EEG). From half of the participants, hand kinematic synergies were derived, and a neural decoder was developed, based on the correlation between hand synergies and corresponding cortical activity, using multivariate linear regression. Using the synergies and the neural decoder derived from the first half of the participants and only cortical activities from the remaining half of the participants, their hand kinematics were reconstructed with an average accuracy above 70%. Potential applications of synergy-based BMIs for controlling assistive devices in individuals with upper limb motor deficits, implications of the results in individuals with stroke and the limitations of the study were discussed.

Dynamical Synergies of Multidigit Hand Prehension.

Hand prehension requires highly coordinated control of contact forces. The high-dimensional sensorimotor system of the human hand operates at ease, but poses several challenges when replicated in artificial hands. This paper investigates how the dynamical synergies, coordinated spatiotemporal patterns of contact forces, contribute to the hand grasp, and whether they could potentially capture the force primitives in a low-dimensional space. Ten right-handed subjects were recruited to grasp and hold mass-varied objects. The contact forces during this multidigit prehension were recorded using an instrumented grip glove. The dynamical synergies were derived using principal component analysis (PCA). The contact force patterns during the grasps were reconstructed using the first few synergies. The significance of the dynamical synergies, the influence of load forces and task configurations on the synergies were explained. This study also discussed the contribution of biomechanical constraints on the first few synergies and the current challenges and possible applications of the dynamical synergies in the design and control of exoskeletons. The integration of the dynamical synergies into exoskeletons will be realized in the near future.