From monkeys to humans: observation-basedEMGbrain-computer interface decoders for humans with paralysis.

Objective. Intracortical brain-computer interfaces (iBCIs) aim to enable individuals with paralysis to control the movement of virtual limbs and robotic arms. Because patients' paralysis prevents training a direct neural activity to limb movement decoder, most iBCIs rely on 'observation-based' decoding in which the patient watches a moving cursor while mentally envisioning making the movement. However, this reliance on observed target motion for decoder development precludes its application to the prediction of unobservable motor output like muscle activity. Here, we ask whether recordings of muscle activity from a surrogate individual performing the same movement as the iBCI patient can be used as target for an iBCI decoder.Approach. We test two possible approaches, each using data from a human iBCI user and a monkey, both performing similar motor actions. In one approach, we trained a decoder to predict the electromyographic (EMG) activity of a monkey from neural signals recorded from a human. We then contrast this to a second approach, based on the hypothesis that the low-dimensional 'latent' neural representations of motor behavior, known to be preserved across time for a given behavior, might also be preserved across individuals. We 'transferred' an EMG decoder trained solely on monkey data to the human iBCI user after using Canonical Correlation Analysis to align the human latent signals to those of the monkey.Main results. We found that both direct and transfer decoding approaches allowed accurate EMG predictions between two monkeys and from a monkey to a human.Significance. Our findings suggest that these latent representations of behavior are consistent across animals and even primate species. These methods are an important initial step in the development of iBCI decoders that generate EMG predictions that could serve as signals for a biomimetic decoder controlling motion and impedance of a prosthetic arm, or even muscle force directly through functional electrical stimulation.

Using adversarial networks to extend brain computer interface decoding accuracy over time.

Existing intracortical brain computer interfaces (iBCIs) transform neural activity into control signals capable of restoring movement to persons with paralysis. However, the accuracy of the 'decoder' at the heart of the iBCI typically degrades over time due to turnover of recorded neurons. To compensate, decoders can be recalibrated, but this requires the user to spend extra time and effort to provide the necessary data, then learn the new dynamics. As the recorded neurons change, one can think of the underlying movement intent signal being expressed in changing coordinates. If a mapping can be computed between the different coordinate systems, it may be possible to stabilize the original decoder's mapping from brain to behavior without recalibration. We previously proposed a method based on Generalized Adversarial Networks (GANs), called 'Adversarial Domain Adaptation Network' (ADAN), which aligns the distributions of latent signals within underlying low-dimensional neural manifolds. However, we tested ADAN on only a very limited dataset. Here we propose a method based on Cycle-Consistent Adversarial Networks (Cycle-GAN), which aligns the distributions of the full-dimensional neural recordings. We tested both Cycle-GAN and ADAN on data from multiple monkeys and behaviors and compared them to a third, quite different method based on Procrustes alignment of axes provided by Factor Analysis. All three methods are unsupervised and require little data, making them practical in real life. Overall, Cycle-GAN had the best performance and was easier to train and more robust than ADAN, making it ideal for stabilizing iBCI systems over time.

Autoencoder-based myoelectric controller for prosthetic hands.

In the past, linear dimensionality-reduction techniques, such as Principal Component Analysis, have been used to simplify the myoelectric control of high-dimensional prosthetic hands. Nonetheless, their nonlinear counterparts, such as Autoencoders, have been shown to be more effective at compressing and reconstructing complex hand kinematics data. As a result, they have a potential of being a more accurate tool for prosthetic hand control. Here, we present a novel Autoencoder-based controller, in which the user is able to control a high-dimensional (17D) virtual hand via a low-dimensional (2D) space. We assess the efficacy of the controller via a validation experiment with four unimpaired participants. All the participants were able to significantly decrease the time it took for them to match a target gesture with a virtual hand to an average of 6.9s and three out of four participants significantly improved path efficiency. Our results suggest that the Autoencoder-based controller has the potential to be used to manipulate high-dimensional hand systems via a myoelectric interface with a higher accuracy than PCA; however, more exploration needs to be done on the most effective ways of learning such a controller.

Learning to operate a high-dimensional hand via a low-dimensional controller.

Dimensionality reduction techniques have proven useful in simplifying complex hand kinematics. They may allow for a low-dimensional kinematic or myoelectric interface to be used to control a high-dimensional hand. Controlling a high-dimensional hand, however, is difficult to learn since the relationship between the low-dimensional controls and the high-dimensional system can be hard to perceive. In this manuscript, we explore how training practices that make this relationship more explicit can aid learning. We outline three studies that explore different factors which affect learning of an autoencoder-based controller, in which a user is able to operate a high-dimensional virtual hand via a low-dimensional control space. We compare computer mouse and myoelectric control as one factor contributing to learning difficulty. We also compare training paradigms in which the dimensionality of the training task matched or did not match the true dimensionality of the low-dimensional controller (both 2D). The training paradigms were a) a full-dimensional task, in which the user was unaware of the underlying controller dimensionality, b) an implicit 2D training, which allowed the user to practice on a simple 2D reaching task before attempting the full-dimensional one, without establishing an explicit connection between the two, and c) an explicit 2D training, during which the user was able to observe the relationship between their 2D movements and the higher-dimensional hand. We found that operating a myoelectric interface did not pose a big challenge to learning the low-dimensional controller and was not the main reason for the poor performance. Implicit 2D training was found to be as good, but not better, as training directly on the high-dimensional hand. What truly aided the user's ability to learn the controller was the 2D training that established an explicit connection between the low-dimensional control space and the high-dimensional hand movements.

A Non-Linear Body Machine Interface for Controlling Assistive Robotic Arms.

OBJECTIVE: Body machine interfaces (BoMIs) enable individuals with paralysis to achieve a greater measure of independence in daily activities by assisting the control of devices such as robotic manipulators. The first BoMIs relied on Principal Component Analysis (PCA) to extract a lower dimensional control space from information in voluntary movement signals. Despite its widespread use, PCA might not be suited for controlling devices with a large number of degrees of freedom, as because of PCs' orthonormality the variance explained by successive components drops sharply after the first. METHODS: Here, we propose an alternative BoMI based on non-linear autoencoder (AE) networks that mapped arm kinematic signals into joint angles of a 4D virtual robotic manipulator. First, we performed a validation procedure that aimed at selecting an AE structure that would allow to distribute the input variance uniformly across the dimensions of the control space. Then, we assessed the users' proficiency practicing a 3D reaching task by operating the robot with the validated AE. RESULTS: All participants managed to acquire an adequate level of skill when operating the 4D robot. Moreover, they retained the performance across two non-consecutive days of training. CONCLUSION: While providing users with a fully continuous control of the robot, the entirely unsupervised nature of our approach makes it ideal for applications in a clinical context since it can be tailored to each user's residual movements. SIGNIFICANCE: We consider these findings as supporting a future implementation of our interface as an assistive tool for people with motor impairments.

Building an adaptive interface via unsupervised tracking of latent manifolds.

In human-machine interfaces, decoder calibration is critical to enable an effective and seamless interaction with the machine. However, recalibration is often necessary as the decoder off-line predictive power does not generally imply ease-of-use, due to closed loop dynamics and user adaptation that cannot be accounted for during the calibration procedure. Here, we propose an adaptive interface that makes use of a non-linear autoencoder trained iteratively to perform online manifold identification and tracking, with the dual goal of reducing the need for interface recalibration and enhancing human-machine joint performance. Importantly, the proposed approach avoids interrupting the operation of the device and it neither relies on information about the state of the task, nor on the existence of a stable neural or movement manifold, allowing it to be applied in the earliest stages of interface operation, when the formation of new neural strategies is still on-going. In order to more directly test the performance of our algorithm, we defined the autoencoder latent space as the control space of a body-machine interface. After an initial offline parameter tuning, we evaluated the performance of the adaptive interface versus that of a static decoder in approximating the evolving low-dimensional manifold of users simultaneously learning to perform reaching movements within the latent space. Results show that the adaptive approach increased the representational efficiency of the interface decoder. Concurrently, it significantly improved users' task-related performance, indicating that the development of a more accurate internal model is encouraged by the online co-adaptation process.

A hybrid Body-Machine Interface integrating signals from muscles and motions.

OBJECTIVE: Body-Machine Interfaces (BoMIs) establish a way to operate a variety of devices, allowing their users to extend the limits of their motor abilities by exploiting the redundancy of muscles and motions that remain available after spinal cord injury or stroke. Here, we considered the integration of two types of signals, motion signals derived from inertial measurement units (IMUs) and muscle activities recorded with electromyography (EMG), both contributing to the operation of the BoMI. APPROACH: A direct combination of IMU and EMG signals might result in inefficient control due to the differences in their nature. Accordingly, we used a nonlinear-regression-based approach to predict IMU from EMG signals, after which the predicted and actual IMU signals were combined into a hybrid control signal. The goal of this approach was to provide users with the possibility to switch seamlessly between movement and EMG control, using the BoMI as a tool for promoting the engagement of selected muscles. We tested the interface in three control modalities, EMG-only, IMU-only and hybrid, in a cohort of 15 unimpaired participants. Participants practiced reaching movements by guiding a computer cursor over a set of targets. MAIN RESULTS: We found that the proposed hybrid control led to comparable performance to IMU-based control and significantly outperformed the EMG-only control. Results also indicated that hybrid cursor control was predominantly influenced by EMG signals. SIGNIFICANCE: We concluded that combining EMG with IMU signals could be an efficient way to target muscle activations while overcoming the limitations of an EMG-only control.

Linear and Non-linear Dimensionality-Reduction Techniques on Full Hand Kinematics.

The purpose of this study was to find a parsimonious representation of hand kinematics data that could facilitate prosthetic hand control. Principal Component Analysis (PCA) and a non-linear Autoencoder Network (nAEN) were compared in their effectiveness at capturing the essential characteristics of a wide spectrum of hand gestures and actions. Performance of the two methods was compared on (a) the ability to accurately reconstruct hand kinematic data from a latent manifold of reduced dimension, (b) variance distribution across latent dimensions, and (c) the separability of hand movements in compressed and reconstructed representations derived using a linear classifier. The nAEN exhibited higher performance than PCA in its ability to more accurately reconstruct hand kinematic data from a latent manifold of reduced dimension. Whereas, for two dimensions in the latent manifold, PCA was able to account for 78% of input data variance, nAEN accounted for 94%. In addition, the nAEN latent manifold was spanned by coordinates with more uniform share of signal variance compared to PCA. Lastly, the nAEN was able to produce a manifold of more separable movements than PCA, as different tasks, when reconstructed, were more distinguishable by a linear classifier, SoftMax regression. It is concluded that non-linear dimensionality reduction may offer a more effective platform than linear methods to control prosthetic hands.

A Myoelectric Computer Interface for Reducing Abnormal Muscle Activations after Spinal Cord Injury.

Myoelectric Computer Interfaces (MCIs) are a viable option to promote the recovery of movements following spinal cord injury (SCI), stroke, or other neurological disorders that impair motor functions. We developed and tested a MCI interface with the goal of reducing abnormal muscular activations due to compensatory strategies or undesired co-contraction after SCI. The interface mapped surface electromyographic signals (sEMG) into the movement of a cursor on a computer monitor. First, we aimed to reduce the co-activation of muscles pairs: the activation of two muscles controlled orthogonal directions of the cursor movements. Furthermore, to decrease the undesired concurrent activation of a third muscle, we modulated the visual feedback related to the position of the cursor on the screen based on the activation of this muscle. We tested the interface with six unimpaired and two SCI participants. Participants were able to decrease the activity of the targeted muscle when it was associated with the visual feedback of the cursor, but, interestingly, after training, its activity increased again. As for the SCI participants, one successfully decreased the co-activation of arm muscles, while the other successfully improved the selective activation of leg muscles. This is a first proof of concept that people with SCI can acquire, through the proposed MCI, a greater awareness of their muscular activity, reducing abnormal muscle simultaneous activations.