A multifaceted suite of metrics for comparative myoelectric prosthesis controller research.

Upper limb robotic (myoelectric) prostheses are technologically advanced, but challenging to use. In response, substantial research is being done to develop person-specific prosthesis controllers that can predict a user's intended movements. Most studies that test and compare new controllers rely on simple assessment measures such as task scores (e.g., number of objects moved across a barrier) or duration-based measures (e.g., overall task completion time). These assessment measures, however, fail to capture valuable details about: the quality of device arm movements; whether these movements match users' intentions; the timing of specific wrist and hand control functions; and users' opinions regarding overall device reliability and controller training requirements. In this work, we present a comprehensive and novel suite of myoelectric prosthesis control evaluation metrics that better facilitates analysis of device movement details-spanning measures of task performance, control characteristics, and user experience. As a case example of their use and research viability, we applied these metrics in real-time control experimentation. Here, eight participants without upper limb impairment compared device control offered by a deep learning-based controller (recurrent convolutional neural network-based classification with transfer learning, or RCNN-TL) to that of a commonly used controller (linear discriminant analysis, or LDA). The participants wore a simulated prosthesis and performed complex functional tasks across multiple limb positions. Analysis resulting from our suite of metrics identified 16 instances of a user-facing problem known as the "limb position effect". We determined that RCNN-TL performed the same as or significantly better than LDA in four such problem instances. We also confirmed that transfer learning can minimize user training burden. Overall, this study contributes a multifaceted new suite of control evaluation metrics, along with a guide to their application, for use in research and testing of myoelectric controllers today, and potentially for use in broader rehabilitation technologies of the future.

A Case Series in Position-Aware Myoelectric Prosthesis Control Using Recurrent Convolutional Neural Network Classification with Transfer Learning.

Position-aware myoelectric prosthesis controllers require long, data-intensive training routines. Transfer Learning (TL) might reduce training burden. A TL model can be pre-trained using forearm muscle signal data from many individuals to become the starting point for a new user. A recurrent convolutional neural network (RCNN)-based classifier has already been shown to benefit from TL in offline analysis (95% accuracy). The present real-time study tested whether an RCNN-based classification controller with TL (RCNN-TL) could reduce training burden, offer improved device control (per functional task performance metrics), and mitigate what is known as the "limb position effect". 27 participants without amputation were recruited. 19 participants performed wrist/hand movements across multiple limb positions, with resulting forearm muscle signal data used to pre-train RCNN-TL. 8 other participants donned a simulated prosthesis, retrained (calibrated) and tested RCNN-TL, plus trained and tested a conventional linear discriminant analysis classification controller (LDA-Baseline). Results confirmed that TL reduces user training burden. RCNN-TL yielded improved task performance durations over LDA-Baseline (in specific Grasp and Release phases), yet other metrics worsened. Overall, this work contributes training condition factors necessary for TL success, identifies metrics needed for comprehensive control analysis, and contributes insights towards improved position-aware control.

Limb loading enhances skill transfer between augmented and physical reality tasks during limb loss rehabilitation.

BACKGROUND: Virtual and augmented reality (AR) have become popular modalities for training myoelectric prosthesis control with upper-limb amputees. While some systems have shown moderate success, it is unclear how well the complex motor skills learned in an AR simulation transfer to completing the same tasks in physical reality. Limb loading is a possible dimension of motor skill execution that is absent in current AR solutions that may help to increase skill transfer between the virtual and physical domains. METHODS: We implemented an immersive AR environment where individuals could operate a myoelectric virtual prosthesis to accomplish a variety of object relocation manipulations. Intact limb participants were separated into three groups, the load control (CGLD; [Formula: see text]), the AR control (CGAR; [Formula: see text]), and the experimental group (EG; [Formula: see text]). Both the CGAR and EG completed a 5-session prosthesis training protocol in AR while the CGLD performed simple muscle training. The EG attempted manipulations in AR while undergoing limb loading. The CGAR attempted the same manipulations without loading. All participants performed the same manipulations in physical reality while operating a real prosthesis pre- and post-training. The main outcome measure was the change in the number of manipulations completed during the physical reality assessments (i.e. completion rate). Secondary outcomes included movement kinematics and visuomotor behavior. RESULTS: The EG experienced a greater increase in completion rate post-training than both the CGAR and CGLD. This performance increase was accompanied by a shorter motor learning phase, the EG's performance saturating in less sessions of AR training than the CGAR. CONCLUSION: The results demonstrated that limb loading plays an important role in transferring complex motor skills learned in virtual spaces to their physical reality analogs. While participants who did not receive limb loading were able to receive some functional benefit from AR training, participants who received the loading experienced a greater positive change in motor performance with their performance saturating in fewer training sessions.

Composite Recurrent Convolutional Neural Networks Offer a Position-Aware Prosthesis Control Alternative While Balancing Predictive Accuracy with Training Burden.

To mitigate the "limb position effect" that hinders myoelectric upper limb prosthesis control, pattern recognition-based models must accurately predict user-intended movements across a multitude of limb positions. Such models can use electromyography (EMG) and inertial measurement units to capture necessary multi-position data. However, this data capture solution requires lengthy user-performed model training routines, with movements in many limb positions, plus retraining thereafter due to inherent signal variations over time. While a general-purpose control model (trained with a dataset that represents numerous device users) eliminates the user-training requirement altogether, it yields low movement predictive accuracy. Conversely, a user-specific control model (trained with a smaller dataset from an individual) yields high predictive accuracy, but requires retraining over time. This study capitalizes on the benefits offered by both such control options, and contributes an alternative control solution-a novel recurrent convolutional neural network (RCNN)-based Composite Model that combines the representation portion of a general-purpose model, with the decision portion of a user-specific model. The resulting Composite Model offers moderate movement predictive accuracy across various limb positions and a reduction in user training routine requirements, suggesting a new research direction to help mitigate the limb position effect along with model training burden.

Preliminary Evaluation of the Effect of Mechanotactile Feedback Location on Myoelectric Prosthesis Performance Using a Sensorized Prosthetic Hand.

A commonly cited reason for the high abandonment rate of myoelectric prostheses is a lack of grip force sensory feedback. Researchers have attempted to restore grip force sensory feedback by stimulating the residual limb's skin surface in response to the prosthetic hand's measured grip force. Recent work has focused on restoring natural feedback to the missing digits directly through invasive surgical procedures. However, the functional benefit of utilizing somatotopically matching feedback has not been evaluated. In this paper, we propose an experimental protocol centered on a fragile object grasp and lift task using a sensorized myoelectric prosthesis to evaluate sensory feedback techniques. We formalized a suite of outcome measures related to task success, timing, and strategy. A pilot study (n = 3) evaluating the effect of utilizing a somatotopically accurate feedback stimulation location in able-bodied participants was conducted to evaluate the feasibility of the standardized platform, and to inform future studies on the role of feedback stimulation location in prosthesis use. Large between-participant effect sizes were observed in all outcome measures, indicating that the feedback location likely plays a role in myoelectric prosthesis performance. The success rate decreased, and task timing and task focus metrics increased, when using somatotopically-matched feedback compared to non-somatotopically-matched feedback. These results were used to conduct a power analysis, revealing that a sample size of n = 8 would be sufficient to achieve significance in all outcome measures.

Recurrent Convolutional Neural Networks as an Approach to Position-Aware Myoelectric Prosthesis Control.

OBJECTIVE: Persons with normal arm function can perform complex wrist and hand movements over a wide range of limb positions. However, for those with transradial amputation who use myoelectric prostheses, control across multiple limb positions can be challenging, frustrating, and can increase the likelihood of device abandonment. In response, the goal of this research was to investigate convolutional neural network (RCNN)-based position-aware myoelectric prosthesis control strategies. METHODS: Surface electromyographic (EMG) and inertial measurement unit (IMU) signals, obtained from 16 non-disabled participants wearing two Myo armbands, served as inputs to RCNN classification and regression models. Such models predicted movements (wrist flexion/extension and forearm pronation/supination), based on a multi-limb-position training routine. RCNN classifiers and RCNN regressors were compared to linear discriminant analysis (LDA) classifiers and support vector regression (SVR) regressors, respectively. Outcomes were examined to determine whether RCNN-based control strategies could yield accurate movement predictions, while using the fewest number of available Myo armband data streams. RESULTS: An RCNN classifier (trained with forearm EMG data, and forearm and upper arm IMU data) predicted movements with 99.00% accuracy (versus the LDA's 97.67%). An RCNN regressor (trained with forearm EMG and IMU data) predicted movements with R2 values of 84.93% for wrist flexion/extension and 84.97% for forearm pronation/supination (versus the SVR's 77.26% and 60.73%, respectively). The control strategies that employed these models required fewer than all available data streams. CONCLUSION: RCNN-based control strategies offer novel means of mitigating limb position challenges. SIGNIFICANCE: This research furthers the development of improved position-aware myoelectric prosthesis control.

Neurorobotic fusion of prosthetic touch, kinesthesia, and movement in bionic upper limbs promotes intrinsic brain behaviors.

Bionic prostheses have restorative potential. However, the complex interplay between intuitive motor control, proprioception, and touch that represents the hallmark of human upper limb function has not been revealed. Here, we show that the neurorobotic fusion of touch, grip kinesthesia, and intuitive motor control promotes levels of behavioral performance that are stratified toward able-bodied function and away from standard-of-care prosthetic users. This was achieved through targeted motor and sensory reinnervation, a closed-loop neural-machine interface, coupled to a noninvasive robotic architecture. Adding touch to motor control improves the ability to reach intended target grasp forces, find target durometers among distractors, and promote prosthetic ownership. Touch, kinesthesia, and motor control restore balanced decision strategies when identifying target durometers and intrinsic visuomotor behaviors that reduce the need to watch the prosthetic hand during object interactions, which frees the eyes to look ahead to the next planned action. The combination of these three modalities also enhances error correction performance. We applied our unified theoretical, functional, and clinical analyses, enabling us to define the relative contributions of the sensory and motor modalities operating simultaneously in this neural-machine interface. This multiperspective framework provides the necessary evidence to show that bionic prostheses attain more human-like function with effective sensory-motor restoration.

A Cost-Effective Inertial Measurement System for Tracking Movement and Triggering Kinesthetic Feedback in Lower-Limb Prosthesis Users.

Advances in lower-limb prosthetic technologies have facilitated the restoration of ambulation; however, users of such technologies still experience reduced balance control, also due to the absence of proprioceptive feedback. Recent efforts have demonstrated the ability to restore kinesthetic feedback in upper-limb prosthesis applications; however, technical solutions to trigger the required muscle vibration and provide automated feedback have not been explored for lower-limb prostheses. The study's first objective was therefore to develop a feedback system capable of tracking lower-limb movement and automatically triggering a muscle vibrator to induce the kinesthetic illusion. The second objective was to investigate the developed system's ability to provide kinesthetic feedback in a case participant. A low-cost, wireless feedback system, incorporating two inertial measurement units to trigger a muscle vibrator, was developed and tested in an individual with limb loss above the knee. Our system had a maximum communication delay of 50 ms and showed good tracking of Gaussian and sinusoidal movement profiles for velocities below 180 degrees per second (error < 8 degrees), mimicking stepping and walking, respectively. We demonstrated in the case participant that the developed feedback system can successfully elicit the kinesthetic illusion. Our work contributes to the integration of sensory feedback in lower-limb prostheses, to increase their use and functionality.

An analytical method reduces noise bias in motor adaptation analysis.

When a person makes a movement, a motor error is typically observed that then drives motor planning corrections on subsequent movements. This error correction, quantified as a trial-by-trial adaptation rate, provides insight into how the nervous system is operating, particularly regarding how much confidence a person places in different sources of information such as sensory feedback or motor command reproducibility. Traditional analysis has required carefully controlled laboratory conditions such as the application of perturbations or error clamping, limiting the usefulness of motor analysis in clinical and everyday environments. Here we focus on error adaptation during unperturbed and naturalistic movements. With increasing motor noise, we show that the conventional estimation of trial-by-trial adaptation increases, a counterintuitive finding that is the consequence of systematic bias in the estimate due to noise masking the learner's intention. We present an analytic solution relying on stochastic signal processing to reduce this effect of noise, producing an estimate of motor adaptation with reduced bias. The result is an improved estimate of trial-by-trial adaptation in a human learner compared to conventional methods. We demonstrate the effectiveness of the new method in analyzing simulated and empirical movement data under different noise conditions.

A Modular Adjustable Transhumeral Prosthetic Socket for Evaluating Myoelectric Control.

Novel myoelectric control strategies may yield more robust, capable prostheses which improve quality of life for those affected by upper-limb loss; however, the development and translation of such strategies from an experimental setting towards daily use by persons with limb loss is a slow and costly process. Since prosthesis functionality is highly dependent on the physical interface between the user's prosthetic socket and residual limb, assessment of such controllers under realistic (noisy) environmental conditions, integrated into prosthetic sockets, and with participants with amputation is essential for obtaining representative results. Unfortunately, this step is particularly difficult as participant- and control strategy-specific prosthetic sockets must be custom-designed and manufactured. There is thus a need for a system to reduce these burdens and facilitate this crucial phase of the development pipeline. This study aims to address this gap through the design and assessment of an inexpensive and easy-to-use 3D-printed Modular-Adjustable transhumeral Prosthetic Socket (MAPS). This 3D-printed, open-source socket was developed in consultation with prosthetists and compared with a participant-specific suction socket in a single-participant case-study. We conducted mechanical and functional assessments to ensure that the developed socket enabled similar performance compared to participant-specific sockets. Both socket systems yielded similar results in mechanical and functional assessments, as well as in self-reported user feedback. The MAPS system shows promise as a research tool which catalyzes the development and deployment of novel myoelectric control strategies by better-enabling comprehensive assessment involving participants with amputations.

Mechanotactile Sensory Feedback Improves Embodiment of a Prosthetic Hand During Active Use.

There have been several advancements in the field of myoelectric prostheses to improve dexterity and restore hand grasp patterns for persons with upper limb loss, including robust control strategies, novel sensory feedback, and multifunction prosthetic terminal devices. Although these advancements have shown to improve prosthesis performance, a key element that may further improve acceptance is often overlooked. Embodiment, which encompasses the feeling of owning, controlling and locating the device without the need to constantly look at it, has been shown to be affected by sensory feedback. However, the specific aspects of embodiment that are influenced are not clearly understood, particularly when a prosthesis is actively controlled. In this work, we used a sensorized simulated prosthesis in able-bodied participants to investigate the contribution of sensory feedback, active motor control, and the combination of both to the components of embodiment; using a common methodology in the literature, namely the rubber hand illusion (RHI). Our results indicate that (1) the sensorized simulated prosthesis may be embodied by able-bodied users in a similar fashion as prosthetic devices embodied by persons with upper limb amputation, and (2) mechanotactile sensory feedback might not only be useful for improving certain aspects of embodiment, i.e., ownership and location, but also may have a modulating effect on other aspects, namely sense of agency, when provided asynchronously during active motor control tasks. This work may allow us to further investigate and manipulate factors contributing to the complex phenomenon of embodiment in relation to active motor control of a device, enabling future study of more precise quantitative measures of embodiment that do not rely as much on subjective perception.

Skin Stretch Enhances Illusory Movement in Persons with Lower-Limb Amputation.

Performance of lower limb prostheses is related not only to the mechanical design and the control scheme, but also to the feedback provided to the user. Proprioceptive feedback, which is the sense of position and movement of one's body parts, can improve the utility as well as facilitate the embodiment of the prosthetic device. Recent studies have shown that proprioceptive kinesthetic (movement) sense can be elicited when non-invasively vibrating a muscle tendon proximal to the targeted joint. However, consistency and quality of the elicited sensation depend on several parameters and muscle tendons after lower limb amputation may not always be accessible. In this study, we developed an experimental protocol to quantitatively and qualitatively assess the elicited proprioceptive kinesthetic illusion when non-invasively vibrating a muscle belly. Furthermore, we explored ways to improve consistency and strength of the illusion by integrating another non-invasive feedback method, namely cutaneous information manipulation via skin stretch. Our preliminary results from tests conducted with a person with transtibial (below knee) amputation show that stretching skin while vibrating a muscle belly on the residual limb provided a stronger and more consistent kinesthetic illusion (90%) than only vibrating the muscle (50%). In addition, we found that stretching skin enhances the range (1.5 times) and speed (3.5 times) of the illusory movement triggered by muscle vibration. These findings may enable the development of mechanisms for controlling feedback parameters in closing the control loop for various walking routines, which may improve performance of lower limb prostheses.

When Less Is More - Discrete Tactile Feedback Dominates Continuous Audio Biofeedback in the Integrated Percept While Controlling a Myoelectric Prosthetic Hand.

State of the art myoelectric hand prostheses can restore some feedforward motor function to their users, but they cannot yet restore sensory feedback. It has been shown, using psychophysical tests, that multi-modal sensory feedback is readily used in the formation of the users' representation of the control task in their central nervous system - their internal model. Hence, to fully describe the effect of providing feedback to prosthesis users, not only should functional outcomes be assessed, but so should the internal model. In this study, we compare the complex interactions between two different feedback types, as well as a combination of the two, on the internal model, and the functional performance of naïve participants without limb difference. We show that adding complementary audio biofeedback to visual feedback enables the development of a significantly stronger internal model for controlling a myoelectric hand compared to visual feedback alone, but adding discrete vibrotactile feedback to vision does not. Both types of feedback, however, improved the functional grasping abilities to a similar degree. Contrary to our expectations, when both types of feedback are combined, the discrete vibrotactile feedback seems to dominate the continuous audio feedback. This finding indicates that simply adding sensory information may not necessarily enhance the formation of the internal model in the short term. In fact, it could even degrade it. These results support our argument that assessment of the internal model is crucial to understanding the effects of any type of feedback, although we cannot be sure that the metrics used here describe the internal model exhaustively. Furthermore, all the feedback types tested herein have been proven to provide significant functional benefits to the participants using a myoelectrically controlled robotic hand. This article, therefore, proposes a crucial conceptual and methodological addition to the evaluation of sensory feedback for upper limb prostheses - the internal model - as well as new types of feedback that promise to significantly and considerably improve functional prosthesis control.

Improving internal model strength and performance of prosthetic hands using augmented feedback.

BACKGROUND: The loss of an arm presents a substantial challenge for upper limb amputees when performing activities of daily living. Myoelectric prosthetic devices partially replace lost hand functions; however, lack of sensory feedback and strong understanding of the myoelectric control system prevent prosthesis users from interacting with their environment effectively. Although most research in augmented sensory feedback has focused on real-time regulation, sensory feedback is also essential for enabling the development and correction of internal models, which in turn are used for planning movements and reacting to control variability faster than otherwise possible in the presence of sensory delays. METHODS: Our recent work has demonstrated that audio-augmented feedback can improve both performance and internal model strength for an abstract target acquisition task. Here we use this concept in controlling a robotic hand, which has inherent dynamics and variability, and apply it to a more functional grasp-and-lift task. We assessed internal model strength using psychophysical tests and used an instrumented Virtual Egg to assess performance. RESULTS: Results obtained from 14 able-bodied subjects show that a classifier-based controller augmented with audio feedback enabled stronger internal model (p = 0.018) and better performance (p = 0.028) than a controller without this feedback. CONCLUSIONS: We extended our previous work and accomplished the first steps on a path towards bridging the gap between research and clinical usability of a hand prosthesis. The main goal was to assess whether the ability to decouple internal model strength and motion variability using the continuous audio-augmented feedback extended to real-world use, where the inherent mechanical variability and dynamics in the mechanisms may contribute to a more complicated interplay between internal model formation and motion variability. We concluded that benefits of using audio-augmented feedback for improving internal model strength of myoelectric controllers extend beyond a virtual target acquisition task to include control of a prosthetic hand.

Audible Feedback Improves Internal Model Strength and Performance of Myoelectric Prosthesis Control.

Myoelectric prosthetic devices are commonly used to help upper limb amputees perform activities of daily living, however amputees still lack the sensory feedback required to facilitate reliable and precise control. Augmented feedback may play an important role in affecting both short-term performance, through real-time regulation, and long-term performance, through the development of stronger internal models. In this work, we investigate the potential tradeoff between controllers that enable better short-term performance and those that provide sufficient feedback to develop a strong internal model. We hypothesize that augmented feedback may be used to mitigate this tradeoff, ultimately improving both short and long-term control. We used psychometric measures to assess the internal model developed while using a filtered myoelectric controller with augmented audio feedback, imitating classification-based control but with augmented regression-based feedback. In addition, we evaluated the short-term performance using a multi degree-of-freedom constrained-time target acquisition task. Results obtained from 24 able-bodied subjects show that an augmented feedback control strategy using audio cues enables the development of a stronger internal model than the filtered control with filtered feedback, and significantly better path efficiency than both raw and filtered control strategies. These results suggest that the use of augmented feedback control strategies may improve both short-term and long-term performance.

Evaluating Internal Model Strength and Performance of Myoelectric Prosthesis Control Strategies.

On-going developments in myoelectric prosthesis control have provided prosthesis users with an assortment of control strategies that vary in reliability and performance. Many studies have focused on improving performance by providing feedback to the user but have overlooked the effect of this feedback on internal model development, which is key to improve long-term performance. In this paper, the strength of internal models developed for two commonly used myoelectric control strategies: raw control with raw feedback (using a regression-based approach) and filtered control with filtered feedback (using a classifier-based approach), were evaluated using two psychometric measures: trial-by-trial adaptation and just-noticeable difference. The performance of both strategies was also evaluated using Schmidt's style target acquisition task. Results obtained from 24 able-bodied subjects showed that although filtered control with filtered feedback had better short-term performance in path efficiency ( ), raw control with raw feedback resulted in stronger internal model development ( ), which may lead to better long-term performance. Despite inherent noise in the control signals of the regression controller, these findings suggest that rich feedback associated with regression control may be used to improve human understanding of the myoelectric control system.

The effect of myoelectric prosthesis control strategies and feedback level on adaptation rate for a target acquisition task.

The long-term performance of myoelectric prostheses is related not only to the short-term performance of the controller, but also to the user's ability to learn and adapt to the system. Different control architectures may have inherent tradeoffs between their short-term performance and the amount of relevant feedback that informs this adaptation. In this study we focused on the ability of two common types of myoelectric control interfaces: raw control with raw feedback, such as a regression, and filtered control with filtered feedback, such as a classifier, to affect user adaptation. We evaluated trial-by-trial adaptation to self-generated errors during a multi degree-of-freedom target acquisition task by fitting a linear regression model to data collected from 24 able-bodied subjects. Subjects showed significantly higher adaptation behavior to self-generated errors when using raw control with a raw feedback strategy than when using filtered control with a filtered feedback strategy, which suggests that control strategies with more feedback allow for higher adaptation. These results support our hypothesis that feedback-rich control strategies allow users to better understand the myoelectric control system, which may enable better long-term performance.