Generalizing Upper Limb Force Modeling With Transfer Learning: A Multimodal Approach Using EMG and IMU for New Users and Conditions.

In the field of EMG-based force modeling, the ability to generalize models across individuals could play a significant role in its adoption across a range of applications, including assistive devices, robotic and rehabilitation devices. However, current studies have predominately focused on intra-subject modeling, largely neglecting the burden of end-user data acquisition. In this work, we propose the use of transfer learning (TL) to generalize force modeling to a new user by first establishing a baseline model trained using other users' data, and then adapting to the end-user using a small amount of new data (only 10% , 20% , and 40% of the new user data). Using a deep multimodal convolutional neural network, consisting of two CNN models, one with high-density (HD) EMG and one with motion data recorded by an Inertial Measurement Unit (IMU), our proposed TL technique significantly improved force modeling compared to leave-one-subject-out (LOSO) and even intra-subject scenarios. The TL approach increased the average R squared values of the force modeling task by 60.81%, 190.53%, and 199.79% compared to the LOSO case, and by 13.4%, 36.88%, and 45.51% compared to the intra-subject case for isotonic, isokinetic and dynamic conditions, respectively. These results show that it is possible to adapt to a new user with minimal data while improving performance significantly compared to the intra-subject scenario. We also show that TL can be used to generalize on a new experimental condition for a new user.

The Influence of Training With Visual Biofeedback on the Predictability of Myoelectric Control Usability.

Studies have shown that closed-loop myoelectric control schemes can lead to changes in user performance and behavior compared to open-loop systems. When users are placed within the control loop, such as during real-time use, they must correct for errors made by the controller and learn what behavior is necessary to produce desired outcomes. Augmented feedback, consequently, has been used to incorporate the user throughout the training process and to facilitate learning. This work explores the effect of visual feedback presented during user training on both the performance and predictability of a myoelectric classification-based control system. Our results suggest that properly designed feedback mechanisms and training tasks can influence the quality of the training data and the ability to predict usability using linear combinations of metrics derived from feature space. Furthermore, our results confirm that the most common in-lab training protocol, screen guided training, may yield training data that are less representative of online use than training protocols that incorporate the user in the loop. These results suggest that training protocols should be designed that better parallel the testing environment to more effectively prepare both the algorithms and users for real-time control.

Compression of EMG Signals Using Deep Convolutional Autoencoders.

Efficient storage and transmission of electromyogram (EMG) data are important for emerging applications such as telemedicine and big data, as a vital tool for further advancement of the field. However, due to limitations in internet speed and hardware resources, transmission and storage of EMG data are challenging. As a solution, this work proposes a new method for EMG data compression using deep convolutional autoencoders (CAE). Eight-channel EMG data from 10 subjects, and high-density EMG data from 18 subjects, were investigated for compression. The CAE architecture was designed to extract an abstract data representation that is heavily compressed, but from which the salient information for classification can be effectively reconstructed. The proposed method attained efficient compression; for CR = 1600, the average PRDN (percentage RMS difference normalized) was 31.5% and the wrist motions classification accuracy (CA) reduced roughly 5%. The CAE substantially outperformed the state-of-the-art high-efficiency video coding and a well-known wavelet-thresholding compression technique. Moreover, by reducing the bit-resolution of the CAE's compressed data from 24 bits to 6 bits, an additional 4-fold compression was achieved without significant degradation of the reconstruction performance. Furthermore, the CAE's inter-subject performance was promising; e.g., for CR = 1600, the PRDN for the inter-subject case was only 2.6% less than that of the within-subject performance. The powerful EMG compression performance with remarkable reconstruction results reflects the CAEs potential as an automatic end-to-end approach with the ability to learn the complete encoding and decoding process. Furthermore, the excellent inter-subject performance demonstrates the generalizability and usability of the proposed approach.

A Multi-Variate Approach to Predicting Myoelectric Control Usability.

Pattern recognition techniques leveraging the use of electromyography signals have become a popular approach to provide intuitive control of myoelectric devices. Performance of these control interfaces is commonly quantified using offline classification accuracy, despite studies having shown that this metric is a poor indicator of usability. Researchers have identified alternative offline metrics that better correlate with online performance; however, the relationship has yet to be fully defined in the literature. This has necessitated the continued trial-and-error-style online testing of algorithms developed using offline approaches. To bridge this information divide, we conducted an exploratory study where thirty-two different metrics from the offline training data were extracted. A correlation analysis and an ordinary least squares regression were implemented to investigate the relationship between the offline metrics and six aspects online use. The results indicate that the current offline standard, classification accuracy, is a poor indicator of usability and that other metrics may hold predictive power. The metrics identified in this work also may constitute more representative evaluation criteria when designing and reporting new control schemes. Furthermore, linear combinations of offline training metrics generate substantially more accurate predictions than using individual metrics. We found that the offline metric feature efficiency generated the best predictions for the usability metric throughput. A combination of two offline metrics (mean semi-principal axes and mean absolute value) significantly outperformed feature efficiency alone, with a 166% increase in the predicted R2 value (i.e., VEcv). These findings suggest that combinations of metrics could provide a more robust framework for predicting usability.

An analytical method reduces noise bias in motor adaptation analysis.

When a person makes a movement, a motor error is typically observed that then drives motor planning corrections on subsequent movements. This error correction, quantified as a trial-by-trial adaptation rate, provides insight into how the nervous system is operating, particularly regarding how much confidence a person places in different sources of information such as sensory feedback or motor command reproducibility. Traditional analysis has required carefully controlled laboratory conditions such as the application of perturbations or error clamping, limiting the usefulness of motor analysis in clinical and everyday environments. Here we focus on error adaptation during unperturbed and naturalistic movements. With increasing motor noise, we show that the conventional estimation of trial-by-trial adaptation increases, a counterintuitive finding that is the consequence of systematic bias in the estimate due to noise masking the learner's intention. We present an analytic solution relying on stochastic signal processing to reduce this effect of noise, producing an estimate of motor adaptation with reduced bias. The result is an improved estimate of trial-by-trial adaptation in a human learner compared to conventional methods. We demonstrate the effectiveness of the new method in analyzing simulated and empirical movement data under different noise conditions.

A Multiday Evaluation of Real-Time Intramuscular EMG Usability with ANN.

Recent developments in implantable technology, such as high-density recordings, wireless transmission of signals to a prosthetic hand, may pave the way for intramuscular electromyography (iEMG)-based myoelectric control in the future. This study aimed to investigate the real-time control performance of iEMG over time. A novel protocol was developed to quantify the robustness of the real-time performance parameters. Intramuscular wires were used to record EMG signals, which were kept inside the muscles for five consecutive days. Tests were performed on multiple days using Fitts' law. Throughput, completion rate, path efficiency and overshoot were evaluated as performance metrics using three train/test strategies. Each train/test scheme was categorized on the basis of data quantity and the time difference between training and testing data. An artificial neural network (ANN) classifier was trained and tested on (i) data from the same day (WDT), (ii) data collected from the previous day and tested on present-day (BDT) and (iii) trained on all previous days including the present day and tested on present-day (CDT). It was found that the completion rate (91.6 ± 3.6%) of CDT was significantly better (p < 0.01) than BDT (74.02 ± 5.8%) and WDT (88.16 ± 3.6%). For BDT, on average, the first session of each day was significantly better (p < 0.01) than the second and third sessions for completion rate (77.9 ± 14.0%) and path efficiency (88.9 ± 16.9%). Subjects demonstrated the ability to achieve targets successfully with wire electrodes. Results also suggest that time variations in the iEMG signal can be catered by concatenating the data over several days. This scheme can be helpful in attaining stable and robust performance.

A Deep Transfer Learning Approach to Reducing the Effect of Electrode Shift in EMG Pattern Recognition-Based Control.

An important barrier to commercialization of pattern recognition myoelectric control of prostheses is the lack of robustness to confounding factors such as electrode shift, skin impedance variations, and learning effects. To overcome this challenge, a novel supervised adaptation approach based on transfer learning (TL) with convolutional neural networks (CNNs) is proposed which requires only a short training session (a few seconds for each class) to recalibrate the system. TL is proposed as a solution to the problem of insufficient calibration data due to short training times for both classification and regression-based control schemes. This approach was validated for electrode shift of roughly 2.5cm with 13 able-bodied subjects to estimate individual and combined wrist motions. With this method, the original CNN (trained before the shift) was fine-tuned with the calibration data from after shifting. The results show that the proposed technique outperforms training a CNN from scratch (random initialization of weights) or a support vector machine (SVM) using the minimal calibration data. Moreover, it demonstrates superior performance than previous LDA and QDA-based adaptation approaches. As the outcomes confirm, the proposed CNN TL method provides a practical solution for adaptation to external factors, improving the robustness of electromyogram (EMG) pattern recognition systems.

Optimized control mapping through user-tuned cost of effort, time, and reliability.

Humans consistently coordinate their joints to perform a variety of tasks. Computational motor control theory explains these stereotypical behaviors using optimal control. Several cost functions have been used to explain specific movements, which suggests that the brain optimizes for a combination of costs and just varies their relative weights to perform different tasks. In the case of tunable human-machine interfaces, we hypothesize that the human-machine interface should be optimized according to the costs that the user cares about when making the movement. Here, we study how the relative weights of individual cost functions in a composite movement cost affect the optimal control signal produced by the user and the mapping between the user's control signals and the machine's output, using prosthesis control as a specific example. This framework was tested by building a hierarchical optimization model that independently optimized for the user control signal and the virtual dynamics of the device. Our results indicate the feasibility of the approach and show the potential for using such a model in prosthesis tuning. This method could be used to allow clinicians and users to tune their prosthesis based on costs they actually care about; and allow the platforms to be customized for the unique needs of every patient.

EMG Pattern Recognition for Persons with Cervical Spinal Cord Injury.

Pattern recognition based myoelectric control has been widely explored in the field of prosthetics, but little work has extended to other patient groups. Individuals with neurological injuries such as spinal cord injury may also benefit from more intuitive control that may facilitate more interactive treatments or improved control of functional electrical stimulation (FES) systems or assistive technologies. This work presents a pilot study with 10 individuals with cervical spinal cord injury between A and C on the American Spinal Injury Association Impairment Scale. Subjects attempted to elicit 10 classes of forearm and hand movements while their electromyogram (EMG) was recorded using a cuff of eight electrodes. Various well-known EMG features were evaluated using a linear discriminant analysis classifier, yielding classification error rates as low as 4.3% ± 3.9 across the 10 classes. Reducing the number of classes to five, those required to control a commercial therapeutic FES device, further reduced the error rates to (2.2% ± 4.4). Results from this study provide evidence supporting continued exploration of EMG pattern recognition techniques for use by high-level spinal cord injured populations as a method of intuitive control over interactive FES systems or assistive devices.

Regression convolutional neural network for improved simultaneous EMG control.

OBJECTIVE: Deep learning models can learn representations of data that extract useful information in order to perform prediction without feature engineering. In this paper, an electromyography (EMG) control scheme with a regression convolutional neural network (CNN) is proposed as a substitute of conventional regression models that use purposefully designed features. APPROACH: The usability of the regression CNN model is validated for the first time, using an online Fitts' law style test with both individual and simultaneous wrist motions. Results were compared to that of a support vector regression-based scheme with a group of widely used extracted features. MAIN RESULTS: In spite of the proven efficiency of these well-known features, the CNN-based system outperformed the support vector machine (SVM) based scheme in throughput, due to higher regression accuracies especially with high EMG amplitudes. SIGNIFICANCE: These results indicate that the CNN model can extract underlying motor control information from EMG signals during single and multiple degree-of-freedom (DoF) tasks. The advantage of regression CNN over classification CNN (studied previously) is that it allows independent and simultaneous control of motions.

FMG Versus EMG: A Comparison of Usability for Real-Time Pattern Recognition Based Control.

OBJECTIVE: Force myography (FMG), which measures the surface pressure profile exerted by contracting muscles, has been proposed as an alternative to electromyography (EMG) for human-machine interfaces. Although FMG pattern recognition-based control systems have yielded higher offline classification accuracy, comparatively few works have examined the usability of FMG for real-time control. In this work, we conduct a comprehensive comparison of EMG- and FMG-based schemes using both classification and regression controllers. METHODS: A total of 20 participants performed a two-degree-of-freedom Fitts' Law-style virtual target acquisition task using both FMG- and EMG-based classification and regression control schemes. Performance was evaluated based on the standard Fitts' law testing metrics throughput, path efficiency, average speed, number of timeouts, overshoot, stopping distance, and simultaneity. RESULTS: The FMG-based classification system significantly outperformed the EMG-based classification system in both throughput (0.902 ± 0.270) versus (0.751 ± 0.309), (ρ < 0.001) and path efficiency (87.2 ± 8.7) versus (83.2 ± 7.8), (ρ < 0.001). Similarly, FMG-based regression significantly outperformed EMG-based regression in throughput (0.871 ± 0.2) versus (0.69 ± 0.3), (ρ < 0.001) and path efficiency (64.8 ± 5.3) versus (58.8 ± 7.1), (ρ < 0.001). CONCLUSIONS: The FMG-based schemes outperformed the EMG-based schemes regardless of which controller was used. This provides further evidence for FMG as a viable alternative to EMG for human-machine interfaces. SIGNIFICANCE: This work describes a comprehensive evaluation of the online usability of FMG- and EMG-based control using both sequential classification and simultaneous regression control.

Multiday Evaluation of Techniques for EMG-Based Classification of Hand Motions.

Currently, most of the adopted myoelectric schemes for upper limb prostheses do not provide users with intuitive control. Higher accuracies have been reported using different classification algorithms but investigation on the reliability over time for these methods is very limited. In this study, we compared for the first time the longitudinal performance of selected state-of-the-art techniques for electromyography (EMG) based classification of hand motions. Experiments were conducted on ten able-bodied and six transradial amputees for seven continuous days. Linear discriminant analysis (LDA), artificial neural network (ANN), support vector machine (SVM), K-nearest neighbor (KNN), and decision trees (TREE) were compared. Comparative analysis showed that the ANN attained highest classification accuracy followed by LDA. Three-way repeated ANOVA test showed a significant difference (P < 0.001) between EMG types (surface, intramuscular, and combined), days (1-7), classifiers, and their interactions. Performance on the last day was significantly better (P < 0.05) than the first day for all classifiers and EMG types. Within-day, classification error (WCE) across all subject and days in ANN was: surface (9.12 ± 7.38%), intramuscular (11.86 ± 7.84%), and combined (6.11 ± 7.46%). The between-day analysis in a leave-one-day-out fashion showed that the ANN was the optimal classifier (surface (21.88 ± 4.14%), intramuscular (29.33 ± 2.58%), and combined (14.37 ± 3.10%). Results indicate that within day performances of classifiers may be similar but over time, it may lead to a substantially different outcome. Furthermore, training ANN on multiple days might allow capturing time-dependent variability in the EMG signals and thus minimizing the necessity for daily system recalibration.

Effects of Confidence-Based Rejection on Usability and Error in Pattern Recognition-Based Myoelectric Control.

Rejection of movements based on the confidence in the classification decision has previously been demonstrated to improve the usability of pattern recognition based myoelectric control. To this point, however, the optimal rejection threshold has been determined heuristically, and it is not known how different thresholds affect the tradeoff between error mitigation and false rejections in real-time closed-loop control. To answer this question, 24 able-bodied subjects completed a real-time Fitts' law-style virtual cursor control task using a support vector machine classifier. It was found that rejection improved information throughput at all thresholds, with the best performance coming at thresholds between 0.60 and 0.75. Two fundamental types of error were defined and identified: operator error (identifiable, repeatable behaviors, directly attributable to the user), and systemic error (other errors attributable to misclassification or noise). The incidence of both operator and systemic errors were found to decrease as rejection threshold increased. Moreover, while the incidence of all error types correlated strongly with path efficiency, only systemic errors correlated strongly with throughput and trial completion rate. Interestingly, more experienced users were found to commit as many errors as novice users, despite performing better in the Fitts' task, suggesting that there is more to usability than error prevention alone. Nevertheless, these results demonstrate the usability gains possible with rejection across a range of thresholds for both novice and experienced users alike.

On the robustness of real-time myoelectric control investigations: a multiday Fitts' law approach.

OBJECTIVE: Real-time myoelectric experimental protocol is considered as a means to quantify usability of myoelectric control schemes. While usability should be considered over time to assure clinical robustness, all real-time studies reported thus far are limited to a single session or day and thus the influence of time on real-time performance is still unexplored. In this study, the aim was to develop a novel experimental protocol to quantify the effect of time on real-time performance measures over multiple days using a Fitts' law approach. APPROACH: Four metrics: throughput, completion rate, path efficiency and overshoot, were assessed using three train-test strategies: (i) an artificial neural network (ANN) classifier was trained on data collected from the previous day and tested on present day (BDT) (ii) trained and tested on the same day (WDT) and (iii) trained on all previous days including present day and tested on present day (CDT) in a week-long experimental protocol. MAIN RESULTS: It was found that on average, the completion rate (98.37% ± 1.47%) of CDT was significantly better (P < 0.01) than that of BDT (86.25% ± 3.46%) and WDT (94.22% ± 2.74%). The throughput (0.40 ± 0.03 bits s-1) of CDT was significantly better (P = 0.001) than that of BDT (0.38 ± 0.03 bits s-1). Offline analysis showed a different trend due to the difference in the training strategies. SIGNIFICANCE: Results suggest that increasing the size of the training set over time can be beneficial to assure robust performance of the system over time.

Conventional analysis of trial-by-trial adaptation is biased: Empirical and theoretical support using a Bayesian estimator.

Research on human motor adaptation has often focused on how people adapt to self-generated or externally-influenced errors. Trial-by-trial adaptation is a person's response to self-generated errors. Externally-influenced errors applied as catch-trial perturbations are used to calculate a person's perturbation adaptation rate. Although these adaptation rates are sometimes compared to one another, we show through simulation and empirical data that the two metrics are distinct. We demonstrate that the trial-by-trial adaptation rate, often calculated as a coefficient in a linear regression, is biased under typical conditions. We tested 12 able-bodied subjects moving a cursor on a screen using a computer mouse. Statistically different adaptation rates arise when sub-sets of trials from different phases of learning are analyzed from within a sequence of movement results. We propose a new approach to identify when a person's learning has stabilized in order to identify steady-state movement trials from which to calculate a more reliable trial-by-trial adaptation rate. Using a Bayesian model of human movement, we show that this analysis approach is more consistent and provides a more confident estimate than alternative approaches. Constraining analyses to steady-state conditions will allow researchers to better decouple the multiple concurrent learning processes that occur while a person makes goal-directed movements. Streamlining this analysis may help broaden the impact of motor adaptation studies, perhaps even enhancing their clinical usefulness.

Rejection of Systemic and Operator Errors in a Real-Time Myoelectric Control Task.

In myoelectric pattern-recognition control, the rejection of movement decisions based on confidence - the likelihood of a correct classification - has been shown to improve system usability, however it is not known to what extent this is due directly to error mitigation, and to what extent this is due to users having opportunities to change the way they contract. To understand this, 24 subjects participated in a real-time pattern recognition control task with rejection at seven different confidence thresholds, and without rejection. Errors were classified into systemic errors (i.e., those produced by the classifier) and operator errors (i.e., those produced by user behavior). It was found that the error permitted by the rejection controller was reduced by about half at high rejection thresholds, with both systemic and operator errors significantly affected, while the errors produced by the user remained essentially constant throughout. Conversely, correct decisions were filtered out by the rejection controller at significantly greater rates at high rejection thresholds, which may be excessive enough to ultimately impair usability. While some subjects reported being experienced in myoelectric control, no significant differences were observed due to experience level.

Real-time, simultaneous myoelectric control using a convolutional neural network.

The evolution of deep learning techniques has been transformative as they have allowed complex mappings to be trained between control inputs and outputs without the need for feature engineering. In this work, a myoelectric control system based on convolutional neural networks (CNN) is proposed as a possible alternative to traditional approaches that rely on specifically designed features. This CNN-based system is validated using a real-time Fitts' law style target acquisition test requiring single and combined wrist motions. The performance of the proposed system is then compared to that of a standard support vector machine (SVM) based myoelectric system using a set of time-domain features. Despite the prevalence and demonstrated performance of these well-known features, no significant difference (p>0.05) was found between the two methods for any of the computed control metrics. This demonstrates the potential for automated learning approaches to extract complex and rich information from stochastic biological signals. This first evaluation of the usability of a CNN in a real-time myoelectric control environment provides a basis for further exploration.

A proportional control scheme for high density force myography.

OBJECTIVE: Force myography (FMG) has been shown to be a potentially higher accuracy alternative to electromyography for pattern recognition based prosthetic control. Classification accuracy, however, is just one factor that affects the usability of a control system. Others, like the ability to start and stop, to coordinate dynamic movements, and to control the velocity of the device through some proportional control scheme can be of equal importance. To impart effective fine control using FMG-based pattern recognition, it is important that a method of controlling the velocity of each motion be developed. METHODS: In this work force myography data were collected from 14 able bodied participants and one amputee participant as they performed a set of wrist and hand motions. The offline proportional control performance of a standard mean signal amplitude approach and a proposed regression-based alternative was compared. The impact of providing feedback during training, as well as the use of constrained or unconstrained hand and wrist contractions, were also evaluated. RESULTS: It is shown that the commonly used mean of rectified channel amplitudes approach commonly employed with electromyography does not translate to force myography. The proposed class-based regression proportional control approach is shown significantly outperform this standard approach (ρ < 0.001), yielding a R2 correlation coefficients of 0.837 and 0.830 for constrained and unconstrained forearm contractions, respectively for able bodied participants. No significant difference (ρ = 0.693) was found in R2 performance when feedback was provided during training or not. The amputee subject achieved a classification accuracy of 83.4% ± 3.47% demonstrating the ability to distinguish contractions well with FMG. In proportional control the amputee participant achieved an R2 of of 0.375 for regression based proportional control during unconstrained contractions. This is lower than the unconstrained case for able-bodied subjects for this particular amputee, possibly due to difficultly in visualizing contraction level modulation without feedback. This may be remedied in the use of a prosthetic limb that would provide real-time feedback in the form of device speed. CONCLUSION: A novel class-specific regression-based approach is proposed for multi-class control is described and shown to provide an effective means of providing FMG-based proportional control.

The effect of time on EMG classification of hand motions in able-bodied and transradial amputees.

While several studies have demonstrated the short-term performance of pattern recognition systems, long-term investigations are very limited. In this study, we investigated changes in classification performance over time. Ten able-bodied individuals and six amputees took part in this study. EMG signals were recorded concurrently from surface and intramuscular electrodes, with intramuscular electrodes kept in the muscles for seven days. Seven hand motions were evaluated daily using linear discriminant analysis and the classification error quantified within (WCE) and between (BCE) days. BCE was computed for all possible combinations between the days. For all subjects, surface sEMG (7.2 ±â€¯7.6%), iEMG (11.9 ±â€¯9.1%) and cEMG (4.6 ±â€¯4.8%) were significantly different (P < 0.001) from each other. A regression between WCE and days (1-7) was on average not significant implying that performance may be considered similar within each day. Regression between BCE and time difference (Df) in days was significant. The slope between BCE and Df (0-6) was significantly different from zero for sEMG (R2 = 89%) and iEMG (R2 = 95%) in amputees. Results indicate that performance continuously degrades as the time difference between training and testing day increases. Furthermore, for iEMG, performance in amputees was directly proportional to the size of the residual limb.

The effects of control signal noise on simultaneous submovements.

Understanding the stereotypical characteristics of human movement can better inform rehabilitation practices by providing a template of healthy and expected human motor control. Multiplicative noise is inherent in goal-directed movement, such as reaching to grasp an object. Multiplicative noise plays an important role in computational motor control models to help support phenomena such as stereotypical kinematic profiles in time-constrained and unconstrained tasks. Most tasks are not carried out along an isolated degree-of-freedom (DOF), and modelling the contribution of noise can be difficult. Here we add a noise term proportional to the degree of simultaneity for multi-DOF tasks to approximate the contribution of system noise. With this approach, we are able to explain previously observed motor phenomena including the presence of submovements in multi-DOF tasks, and the transition from simultaneous to sequential control of joints without the presence of feedback. Inclusion of a simultaneous multiplicative noise term presents a simple theory that expands on previous research in order to describe characteristics of multiple-DOF movements. This model can be used as a guide to compare healthy human motor control to the movements of patients receiving rehabilitation in an effort to improve their motor planning.