Progressive unsupervised control of myoelectric upper limbs.

Objective.Unsupervised myocontrol methods aim to create control models for myoelectric prostheses while avoiding the complications of acquiring reliable, regular, and sufficient labeled training data. A limitation of current unsupervised methods is that they fix the number of controlled prosthetic functions a priori, thus requiring an initial assessment of the user's motor skills and neglecting the development of novel motor skills over time.Approach.We developed a progressive unsupervised myocontrol (PUM) paradigm in which the user and the control model coadaptively identify distinct muscle synergies, which are then used to control arbitrarily associated myocontrol functions, each corresponding to a hand or wrist movement. The interaction starts with learning a single function and the user may request additional functions after mastering the available ones, which aligns the evolution of their motor skills with an increment in system complexity. We conducted a multi-session user study to evaluate PUM and compare it against a state-of-the-art non-progressive unsupervised alternative. Two participants with congenital upper-limb differences tested PUM, while ten non-disabled control participants tested either PUM or the non-progressive baseline. All participants engaged in myoelectric control of a virtual hand and wrist.Main results.PUM enabled autonomous learning of three myocontrol functions for participants with limb differences, and of all four available functions for non-disabled subjects, using both existing or newly identified muscle synergies. Participants with limb differences achieved similar success rates to non-disabled ones on myocontrol tests, but faced greater difficulties in internalizing new motor skills and exhibited slightly inferior movement quality. The performance was comparable with either PUM or the non-progressive baseline for the group of non-disabled participants.Significance.The PUM paradigm enables users to autonomously learn to operate the myocontrol system, adapts to the users' varied preexisting motor skills, and supports the further development of those skills throughout practice.

Unsupervised Myocontrol of a Virtual Hand Based on a Coadaptive Abstract Motor Mapping.

Applications of simultaneous and proportional control for upper-limb prostheses typically rely on supervised machine learning to map muscle activations to prosthesis movements. This scheme often poses problems for individuals with limb differences, as they may not be able to reliably reproduce the training activations required to construct a natural motor mapping. We propose an unsupervised myocontrol paradigm that eliminates the need for labeled data by mapping the most salient muscle synergies in arbitrary order to a number of predefined prosthesis actions. The paradigm is coadaptive, in the sense that while the user learns to control the system via interaction, the system continually refines the identification of the user's muscular synergies. Our evaluation consisted of eight subjects without limb-loss performing target achievement control tasks of four actions of the hand and wrist. The subjects achieved comparable performance using the proposed unsupervised myocontrol paradigm and a supervised benchmark method, despite reporting increased mental load with the former.

The Merits of Dynamic Data Acquisition for Realistic Myocontrol.

Natural myocontrol is the intuitive control of a prosthetic limb via the user's voluntary muscular activations. This type of control is usually implemented by means of pattern recognition, which uses a set of training data to create a model that can decipher these muscular activations. A consequence of this approach is that the reliability of a myocontrol system depends on how representative this training data is for all types of signal variability that may be encountered when the amputee puts the prosthesis into real use. Myoelectric signals are indeed known to vary according to the position and orientation of the limb, among other factors, which is why it has become common practice to take this variability into account by acquiring training data in multiple body postures. To shed further light on this problem, we compare two ways of collecting data: while the subjects hold their limb statically in several positions one at a time, which is the traditional way, or while they dynamically move their limb at a constant pace through those same positions. Since our interest is to investigate any differences when controlling an actual prosthetic device, we defined an evaluation protocol that consisted of a series of complex, bimanual daily-living tasks. Fourteen intact participants performed these tasks while wearing prosthetic hands mounted on splints, which were controlled via either a statically or dynamically built myocontrol model. In both cases all subjects managed to complete all tasks and participants without previous experience in myoelectric control manifested a significant learning effect; moreover, there was no significant difference in the task completion times achieved with either model. When evaluated in a simulated scenario with traditional offline performance evaluation, on the other hand, the dynamically-trained system showed significantly better accuracy. Regardless of the setting, the dynamic data acquisition was faster, less tiresome, and better accepted by the users. We conclude that dynamic data acquisition is advantageous and confirm the limited relevance of offline analyses for online myocontrol performance.

Publisher Correction: Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics.

A hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person's life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among them the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics.

On the Visuomotor Behavior of Amputees and Able-Bodied People During Grasping.

Visual attention is often predictive for future actions in humans. In manipulation tasks, the eyes tend to fixate an object of interest even before the reach-to-grasp is initiated. Some recent studies have proposed to exploit this anticipatory gaze behavior to improve the control of dexterous upper limb prostheses. This requires a detailed understanding of visuomotor coordination to determine in which temporal window gaze may provide helpful information. In this paper, we verify and quantify the gaze and motor behavior of 14 transradial amputees who were asked to grasp and manipulate common household objects with their missing limb. For comparison, we also include data from 30 able-bodied subjects who executed the same protocol with their right arm. The dataset contains gaze, first person video, angular velocities of the head, and electromyography and accelerometry of the forearm. To analyze the large amount of video, we developed a procedure based on recent deep learning methods to automatically detect and segment all objects of interest. This allowed us to accurately determine the pixel distances between the gaze point, the target object, and the limb in each individual frame. Our analysis shows a clear coordination between the eyes and the limb in the reach-to-grasp phase, confirming that both intact and amputated subjects precede the grasp with their eyes by more than 500 ms. Furthermore, we note that the gaze behavior of amputees was remarkably similar to that of the able-bodied control group, despite their inability to physically manipulate the objects.

Natural Myocontrol in a Realistic Setting: a Comparison Between Static and Dynamic Data Acquisition.

Natural myocontrol employs pattern recognition to allow users to control a robotic limb intuitively using their own voluntary muscular activations. The reliability of myocontrol strongly depends on the signals initially collected from the users, which must appropriately capture the variability encountered later on during operation. Since myoelectric signals can vary based on the position and orientation of the limb, it has become best practice to gather data in multiple body postures. We hereby concentrate on this acquisition protocol and investigate the relative merits of collecting data either statically or dynamically. In the static case, data for a desired hand configuration is collected while the users keep their hand still in certain positions, whereas in the dynamic case, data is collected while users move their limbs, passing through the required positions with a roughly constant velocity.Fourteen able-bodied subjects were asked to naturally control two dexterous hand prostheses mounted on splints, performing a set of complex, realistic bimanual activities of daily living. We could not find any significant difference between the protocols in terms of the total execution times, although the dynamic data acquisition was faster and less tiring. This would indicate that dynamic data acquisition should be preferred over the static one.

Adaptive learning to speed-up control of prosthetic hands: A few things everybody should know.

Domain adaptation methods have been proposed to reduce the training efforts needed to control an upper-limb prosthesis by adapting well performing models from previous subjects to the new subject. These studies generally reported impressive reductions in the required number of training samples to achieve a certain level of accuracy for intact subjects. We further investigate two popular methods in this field to verify whether this result also applies to amputees. Our findings show instead that this improvement can largely be attributed to a suboptimal hyperparameter configuration. When hyperparameters are appropriately tuned, the standard approach that does not exploit prior information performs on par with the more complicated transfer learning algorithms. Additionally, earlier studies erroneously assumed that the number of training samples relates proportionally to the efforts required from the subject. However, a repetition of a movement is the atomic unit for subjects and the total number of repetitions should therefore be used as reliable measure for training efforts. Also when correcting for this mistake, we do not find any performance increase due to the use of prior models.

Megane Pro: Myo-electricity, visual and gaze tracking data acquisitions to improve hand prosthetics.

During the past 60 years scientific research proposed many techniques to control robotic hand prostheses with surface electromyography (sEMG). Few of them have been implemented in commercial systems also due to limited robustness that may be improved with multimodal data. This paper presents the first acquisition setup, acquisition protocol and dataset including sEMG, eye tracking and computer vision to study robotic hand control. A data analysis on healthy controls gives a first idea of the capabilities and constraints of the acquisition procedure that will be applied to amputees in a next step. Different data sources are not fused together in the analysis. Nevertheless, the results support the use of the proposed multimodal data acquisition approach for prosthesis control. The sEMG movement classification results confirm that it is possible to classify several grasps with sEMG alone. sEMG can detect the grasp type and also small differences in the grasped object (accuracy: 95%). The simultaneous recording of eye tracking and scene camera data shows that these sensors allow performing object detection for grasp selection and that several neurocognitive parameters need to be taken into account for this. In conclusion, this work on intact subjects presents an innovative acquisition setup and protocol. The first results in terms of data analysis are promising and set the basis for future work on amputees, aiming to improve the robustness of prostheses with multimodal data.

Repeatability of grasp recognition for robotic hand prosthesis control based on sEMG data.

Control methods based on sEMG obtained promising results for hand prosthetics. Control system robustness is still often inadequate and does not allow the amputees to perform a large number of movements useful for everyday life. Only few studies analyzed the repeatability of sEMG classification of hand grasps. The main goals of this paper are to explore repeatability in sEMG data and to release a repeatability database with the recorded experiments. The data are recorded from 10 intact subjects repeating 7 grasps 12 times, twice a day for 5 days. The data are publicly available on the Ninapro web page. The analysis for the repeatability is based on the comparison of movement classification accuracy in several data acquisitions and for different subjects. The analysis is performed using mean absolute value and waveform length features and a Random Forest classifier. The accuracy obtained by training and testing on acquisitions at different times is on average 27.03% lower than training and testing on the same acquisition. The results obtained by training and testing on different acquisitions suggest that previous acquisitions can be used to train the classification algorithms. The inter-subject variability is remarkable, suggesting that specific characteristics of the subjects can affect repeatibility and sEMG classification accuracy. In conclusion, the results of this paper can contribute to develop more robust control systems for hand prostheses, while the presented data allows researchers to test repeatability in further analyses.

Effect of clinical parameters on the control of myoelectric robotic prosthetic hands.

Improving the functionality of prosthetic hands with noninvasive techniques is still a challenge. Surface electromyography (sEMG) currently gives limited control capabilities; however, the application of machine learning to the analysis of sEMG signals is promising and has recently been applied in practice, but many questions still remain. In this study, we recorded the sEMG activity of the forearm of 11 male subjects with transradial amputation who were mentally performing 40 hand and wrist movements. The classification performance and the number of independent movements (defined as the subset of movements that could be distinguished with >90% accuracy) were studied in relationship to clinical parameters related to the amputation. The analysis showed that classification accuracy and the number of independent movements increased significantly with phantom limb sensation intensity, remaining forearm percentage, and temporal distance to the amputation. The classification results suggest the possibility of naturally controlling up to 11 movements of a robotic prosthetic hand with almost no training. Knowledge of the relationship between classification accuracy and clinical parameters adds new information regarding the nature of phantom limb pain as well as other clinical parameters, and it can lay the foundations for future "functional amputation" procedures in surgery.

Characterization of a benchmark database for myoelectric movement classification.

In this paper, we characterize the Ninapro database and its use as a benchmark for hand prosthesis evaluation. The database is a publicly available resource that aims to support research on advanced myoelectric hand prostheses. The database is obtained by jointly recording surface electromyography signals from the forearm and kinematics of the hand and wrist while subjects perform a predefined set of actions and postures. Besides describing the acquisition protocol, overall features of the datasets and the processing procedures in detail, we present benchmark classification results using a variety of feature representations and classifiers. Our comparison shows that simple feature representations such as mean absolute value and waveform length can achieve similar performance to the computationally more demanding marginal discrete wavelet transform. With respect to classification methods, the nonlinear support vector machine was found to be the only method consistently achieving high performance regardless of the type of feature representation. Furthermore, statistical analysis of these results shows that classification accuracy is negatively correlated with the subject's Body Mass Index. The analysis and the results described in this paper aim to be a strong baseline for the Ninapro database. Thanks to the Ninapro database (and the characterization described in this paper), the scientific community has the opportunity to converge to a common position on hand movement recognition by surface electromyography, a field capable to strongly affect hand prosthesis capabilities.

Proceedings of the first workshop on Peripheral Machine Interfaces: going beyond traditional surface electromyography.

One of the hottest topics in rehabilitation robotics is that of proper control of prosthetic devices. Despite decades of research, the state of the art is dramatically behind the expectations. To shed light on this issue, in June, 2013 the first international workshop on Present and future of non-invasive peripheral nervous system (PNS)-Machine Interfaces (MI; PMI) was convened, hosted by the International Conference on Rehabilitation Robotics. The keyword PMI has been selected to denote human-machine interfaces targeted at the limb-deficient, mainly upper-limb amputees, dealing with signals gathered from the PNS in a non-invasive way, that is, from the surface of the residuum. The workshop was intended to provide an overview of the state of the art and future perspectives of such interfaces; this paper represents is a collection of opinions expressed by each and every researcher/group involved in it.

Movement error rate for evaluation of machine learning methods for sEMG-based hand movement classification.

There has been increasing interest in applying learning algorithms to improve the dexterity of myoelectric prostheses. In this work, we present a large-scale benchmark evaluation on the second iteration of the publicly released NinaPro database, which contains surface electromyography data for 6 DOF force activations as well as for 40 discrete hand movements. The evaluation involves a modern kernel method and compares performance of three feature representations and three kernel functions. Both the force regression and movement classification problems can be learned successfully when using a nonlinear kernel function, while the exp- χ(2) kernel outperforms the more popular radial basis function kernel in all cases. Furthermore, combining surface electromyography and accelerometry in a multimodal classifier results in significant increases in accuracy as compared to when either modality is used individually. Since window-based classification accuracy should not be considered in isolation to estimate prosthetic controllability, we also provide results in terms of classification mistakes and prediction delay. To this extent, we propose the movement error rate as an alternative to the standard window-based accuracy. This error rate is insensitive to prediction delays and it allows us therefore to quantify mistakes and delays as independent performance characteristics. This type of analysis confirms that the inclusion of accelerometry is superior, as it results in fewer mistakes while at the same time reducing prediction delay.

Stable myoelectric control of a hand prosthesis using non-linear incremental learning.

Stable myoelectric control of hand prostheses remains an open problem. The only successful human-machine interface is surface electromyography, typically allowing control of a few degrees of freedom. Machine learning techniques may have the potential to remove these limitations, but their performance is thus far inadequate: myoelectric signals change over time under the influence of various factors, deteriorating control performance. It is therefore necessary, in the standard approach, to regularly retrain a new model from scratch. We hereby propose a non-linear incremental learning method in which occasional updates with a modest amount of novel training data allow continual adaptation to the changes in the signals. In particular, Incremental Ridge Regression and an approximation of the Gaussian Kernel known as Random Fourier Features are combined to predict finger forces from myoelectric signals, both finger-by-finger and grouped in grasping patterns. We show that the approach is effective and practically applicable to this problem by first analyzing its performance while predicting single-finger forces. Surface electromyography and finger forces were collected from 10 intact subjects during four sessions spread over two different days; the results of the analysis show that small incremental updates are indeed effective to maintain a stable level of performance. Subsequently, we employed the same method on-line to teleoperate a humanoid robotic arm equipped with a state-of-the-art commercial prosthetic hand. The subject could reliably grasp, carry and release everyday-life objects, enforcing stable grasping irrespective of the signal changes, hand/arm movements and wrist pronation and supination.

Electromyography data for non-invasive naturally-controlled robotic hand prostheses.

Recent advances in rehabilitation robotics suggest that it may be possible for hand-amputated subjects to recover at least a significant part of the lost hand functionality. The control of robotic prosthetic hands using non-invasive techniques is still a challenge in real life: myoelectric prostheses give limited control capabilities, the control is often unnatural and must be learned through long training times. Meanwhile, scientific literature results are promising but they are still far from fulfilling real-life needs. This work aims to close this gap by allowing worldwide research groups to develop and test movement recognition and force control algorithms on a benchmark scientific database. The database is targeted at studying the relationship between surface electromyography, hand kinematics and hand forces, with the final goal of developing non-invasive, naturally controlled, robotic hand prostheses. The validation section verifies that the data are similar to data acquired in real-life conditions, and that recognition of different hand tasks by applying state-of-the-art signal features and machine-learning algorithms is possible.

Classification of hand movements in amputated subjects by sEMG and accelerometers.

Numerous recent studies have aimed to improve myoelectric control of prostheses. However, the majority of these studies is characterized by two problems that could be easily fulfilled with recent resources supplied by the scientific literature. First, the majority of these studies use only intact subjects, with the unproved assumption that the results apply equally to amputees. Second, usually only electromyography data are used, despite other sensors (e.g., accelerometers) being easy to include into a real life prosthesis control system. In this paper we analyze the mentioned problems by the classification of 40 hand movements in 5 amputated and 40 intact subjects, using both sEMG and accelerometry data and applying several different state of the art methods. The datasets come from the NinaPro database, which supplies publicly available sEMG data to develop and test machine learning algorithms for prosthetics. The number of subjects can seem small at first sight, but it is not considering the literature of the field (which has to face the difficulty of recruiting trans-radial hand amputated subjects). Our results indicate that the maximum average classification accuracy for amputated subjects is 61.14%, which is just 15.86% less than intact subjects, and they show that intact subjects results can be used as proxy measure for amputated subjects. Finally, our comparison shows that accelerometry as a modality is less affected by amputation than electromyography, suggesting that real life prosthetics performance may easily be improved by inclusion of accelerometers.

Natural control capabilities of robotic hands by hand amputated subjects.

People with transradial hand amputations who own a myoelectric prosthesis currently have some control capabilities via sEMG. However, the control systems are still limited and not natural. The Ninapro project is aiming at helping the scientific community to overcome these limits through the creation of publicly available electromyography data sources to develop and test machine learning algorithms. In this paper we describe the movement classification results gained from three subjects with an homogeneous level of amputation, and we compare them with the results of 40 intact subjects. The number of considered subjects can seem small at first sight, but it is not considering the literature of the field (which has to face the difficulty of recruiting trans-radial hand amputated subjects). The classification is performed with four different classifiers and the obtained balanced classification rates are up to 58.6% on 50 movements, which is an excellent result compared to the current literature. Successively, for each subject we find a subset of up to 9 highly independent movements, (defined as movements that can be distinguished with more than 90% accuracy), which is a deeply innovative step in literature. The natural control of a robotic hand in so many movements could lead to an immediate progress in robotic hand prosthetics and it could deeply change the quality of life of amputated subjects.

Exploiting accelerometers to improve movement classification for prosthetics.

Recent studies have explored the integration of additional input modalities to improve myoelectric control of prostheses. Arm dynamics in particular are an interesting option, as these can be measured easily by means of accelerometers. In this work, the benefit of accelerometer signals is demonstrated on a large scale movement classification task, consisting of 40 hand and wrist movements obtained from 20 subjects. The results demonstrate that the accelerometer modality is indeed highly informative and even outperforms surface electromyography in terms of classification accuracy. The highest accuracy, however, is obtained when both modalities are integrated in a multi-modal classifier.

Real-time model learning using Incremental Sparse Spectrum Gaussian Process Regression.

Novel applications in unstructured and non-stationary human environments require robots that learn from experience and adapt autonomously to changing conditions. Predictive models therefore not only need to be accurate, but should also be updated incrementally in real-time and require minimal human intervention. Incremental Sparse Spectrum Gaussian Process Regression is an algorithm that is targeted specifically for use in this context. Rather than developing a novel algorithm from the ground up, the method is based on the thoroughly studied Gaussian Process Regression algorithm, therefore ensuring a solid theoretical foundation. Non-linearity and a bounded update complexity are achieved simultaneously by means of a finite dimensional random feature mapping that approximates a kernel function. As a result, the computational cost for each update remains constant over time. Finally, algorithmic simplicity and support for automated hyperparameter optimization ensures convenience when employed in practice. Empirical validation on a number of synthetic and real-life learning problems confirms that the performance of Incremental Sparse Spectrum Gaussian Process Regression is superior with respect to the popular Locally Weighted Projection Regression, while computational requirements are found to be significantly lower. The method is therefore particularly suited for learning with real-time constraints or when computational resources are limited.