A Real-Time Hand Gesture Recognition System for Low-Latency HMI via Transient HD-sEMG and In-Sensor Computing.

In real-time human-machine interaction (HMI) applications, hand gesture recognition (HGR) requires high accuracy with low latency. Surface electromyography (sEMG), a physiological electrical signal reflecting muscle activation, is extensively used in HMI. Recently, transient sEMG, generated during the gesture transitions, has been employed in HGR to achieve lower observational latency compared to steady-state sEMG. However, the use of long feature windows (up to 200 ms) still make it less desirable in low-latency HMI. In addition, most studies have relied on remote computing, where remote data processing and large data transfer result in high computation and network latency. In this paper, we proposed a method leveraging transient high density sEMG (HD-sEMG) and in-sensor computing to achieve low-latency HGR. An sEMG contrastive convolution network (sCCN) was proposed for HGR. The mean absolute value and its average integration were used to train the sCCN in a contrastive learning manner. In addition, all signal acquisition, data processing, and pattern recognition processes were deployed within designed sensor for in-sensor computing. Compared to the state-of-the-art study using multi-channel 200-ms transient sEMG, our proposed method achieved a comparable HGR accuracy of 0.963, and a 58% lower observational latency of only 84 ms. In-sensor computing realizes a 4 times lower computation latency of 3 ms, and significantly reduces the network latency to 2 ms. The proposed method offers a promising approach to achieving low-latency HGR without compromising accuracy. This facilitates real-time HMI in biomedical applications such as prostheses, exoskeletons, virtual reality, and video games.

Extended residual learning with one-shot imitation learning for robotic assembly in semi-structured environment.

Introduction: Robotic assembly tasks require precise manipulation and coordination, often necessitating advanced learning techniques to achieve efficient and effective performance. While residual reinforcement learning with a base policy has shown promise in this domain, existing base policy approaches often rely on hand-designed full-state features and policies or extensive demonstrations, limiting their applicability in semi-structured environments. Methods: In this study, we propose an innovative Object-Embodiment-Centric Imitation and Residual Reinforcement Learning (OEC-IRRL) approach that leverages an object-embodiment-centric (OEC) task representation to integrate vision models with imitation and residual learning. By utilizing a single demonstration and minimizing interactions with the environment, our method aims to enhance learning efficiency and effectiveness. The proposed method involves three key steps: creating an object-embodiment-centric task representation, employing imitation learning for a base policy using via-point movement primitives for generalization to different settings, and utilizing residual RL for uncertainty-aware policy refinement during the assembly phase. Results: Through a series of comprehensive experiments, we investigate the impact of the OEC task representation on base and residual policy learning and demonstrate the effectiveness of the method in semi-structured environments. Our results indicate that the approach, requiring only a single demonstration and less than 1.2 h of interaction, improves success rates by 46% and reduces assembly time by 25%. Discussion: This research presents a promising avenue for robotic assembly tasks, providing a viable solution without the need for specialized expertise or custom fixtures.

Attention-guided graph structure learning network for EEG-enabled auditory attention detection.

Objective: Decoding auditory attention from brain signals is essential for the development of neuro-steered hearing aids. This study aims to overcome the challenges of extracting discriminative feature representations from electroencephalography (EEG) signals for auditory attention detection (AAD) tasks, particularly focusing on the intrinsic relationships between different EEG channels.Approach: We propose a novel attention-guided graph structure learning network, AGSLnet, which leverages potential relationships between EEG channels to improve AAD performance. Specifically, AGSLnet is designed to dynamically capture latent relationships between channels and construct a graph structure of EEG signals.Main result: We evaluated AGSLnet on two publicly available AAD datasets and demonstrated its superiority and robustness over state-of-the-art models. Visualization of the graph structure trained by AGSLnet supports previous neuroscience findings, enhancing our understanding of the underlying neural mechanisms.Significance: This study presents a novel approach for examining brain functional connections, improving AAD performance in low-latency settings, and supporting the development of neuro-steered hearing aids.

Correction: Stretchable and durable HD-sEMG electrodes for accurate recognition of swallowing activities on complex epidermal surfaces.

[This corrects the article DOI: 10.1038/s41378-023-00591-3.].

Stretchable and durable HD-sEMG electrodes for accurate recognition of swallowing activities on complex epidermal surfaces.

Surface electromyography (sEMG) is widely used in monitoring human health. Nonetheless, it is challenging to capture high-fidelity sEMG recordings in regions with intricate curved surfaces such as the larynx, because regular sEMG electrodes have stiff structures. In this study, we developed a stretchable, high-density sEMG electrode array via layer-by-layer printing and lamination. The electrode offered a series of excellent human‒machine interface features, including conformal adhesion to the skin, high electron-to-ion conductivity (and thus lower contact impedance), prolonged environmental adaptability to resist water evaporation, and epidermal biocompatibility. This made the electrode more appropriate than commercial electrodes for long-term wearable, high-fidelity sEMG recording devices at complicated skin interfaces. Systematic in vivo studies were used to investigate its ability to classify swallowing activities, which was accomplished with high accuracy by decoding the sEMG signals from the chin via integration with an ear-mounted wearable system and machine learning algorithms. The results demonstrated the clinical feasibility of the system for noninvasive and comfortable recognition of swallowing motions for comfortable dysphagia rehabilitation.

Compensatory movement detection by using near-infrared spectroscopy technology based on signal improvement method.

Introduction: Compensatory movements usually occur in stroke survivors with hemiplegia, which is detrimental to recovery. This paper proposes a compensatory movement detection method based on near-infrared spectroscopy (NIRS) technology and verifies its feasibility using a machine learning algorithm. We present a differential-based signal improvement (DBSI) method to enhance NIRS signal quality and discuss its effect on improving detection performance. Method: Ten healthy subjects and six stroke survivors performed three common rehabilitation training tasks while the activation of six trunk muscles was recorded using NIRS sensors. After data preprocessing, DBSI was applied to the NIRS signals, and two time-domain features (mean and variance) were extracted. An SVM algorithm was used to test the effect of the NIRS signal on compensatory behavior detection. Results: Classification results show that NIRS signals have good performance in compensatory detection, with accuracy rates of 97.76% in healthy subjects and 97.95% in stroke survivors. After using the DBSI method, the accuracy improved to 98.52% and 99.47%, respectively. Discussion: Compared with other compensatory motion detection methods, our proposed method based on NIRS technology has better classification performance. The study highlights the potential of NIRS technology for improving stroke rehabilitation and warrants further investigation.

LiDAR-Based Hand Contralateral Controlled Functional Electrical Stimulation System.

Contralateral controlled functional electrical stimulation (CCFES) can induce simultaneous movements in patients' bilateral hands. It has been clinically proven to be effective in improving hand motor control and dexterity. sEMG and bending sensor-based data gloves for detecting patients' motor intent have been developed with limitations. sEMG sensor signals are unstable and susceptible to noise. Data gloves composed of bending sensors require complicated calibration and tend to have data drift. In this paper, a LiDAR-based system for hand CCFES is proposed. The method utilized LiDAR to detect the patient's motion intention without contact in CCFES systems. It has been clinically proven that LiDARs can effectively distinguish the different motion amplitudes of hand gestures as quantitative evaluation sensors of functional electrical stimulation (FES). Training data for classifiers were collected from 9 healthy individuals and 15 stroke patients performing 4 gestures, including hand opening, fist clenching, wrist extension, and wrist flexion. The support vector machine (SVM), linear discriminant analysis (LDA), and k-nearest neighbor (kNN) were verified for their classification performance in offline hand gesture recognition tests. Experiments were also conducted on 6 stroke volunteers to evaluate gestures triggered by FES. The SVM classifier showed excellent classification performance for four hand gestures, with an average F1-score of 0.97 ± 0.05 in offline tests. As for online gesture recognition, an average F1-score of 0.92 ± 0.09 was obtained. In the evaluation experiments, between data from 50% and 100% movement amplitude, paired t-tests showed significant differences. The experimental results indicated that the proposed system showed promise for hand rehabilitation.

Design and Testing of a Dynamic Orthosis to Reduce Glenohumeral Subluxation With Omnidirectional Shoulder Motion.

OBJECTIVE: This paper aimed to develop an orthosis to apply a compensating force to improve the stability of the glenohumeral joint without resisting arm movement. METHODS: The proposed orthosis was based on a parallelogram structure to provide a pair of compensating forces to the glenohumeral joint center. Theoretical analysis was used to evaluate the additional moments caused by glenohumeral joint center shifting. Then, an experimental evaluation platform, composed of a torque sensor, a force sensor, and a 3D printed arm, was set up to assess the additional moments and compensating force. Finally, the proposed orthosis was compared with the traditional orthosis to compare the subluxation reduction and the movement restriction when worn by stroke patients. RESULTS: There was only a maximum additional moment of 0.87 Nm for the glenohumeral center shifting. During 3D printed arm movement, the moment correlation coefficient between with and without the proposed orthosis was greater than 0.98, and the compensating force was larger than 90% of the arm weight. The proposed orthosis reduced subluxation by 12.5±3.5 mm, and the traditional orthosis reduced subluxation by 7.7±2.2 mm, indicating that the subluxation reduction of the proposed orthosis was more effective ( ). Meanwhile, the proposed orthosis's motion restriction joint was significantly smaller than traditional orthosis ( ). CONCLUSION: The proposed orthosis provided sufficient gravity compensation without resisting arm movement. SIGNIFICANCE: The proposed orthosis can improve the shoulder's stability during shoulder movement, potentially improving the rehabilitation effect of patients with shoulder subluxation.

Interpretable evaluation for the Brunnstrom recovery stage of the lower limb based on wearable sensors.

With the increasing number of stroke patients, there is an urgent need for an accessible, scientific, and reliable evaluation method for stroke rehabilitation. Although many rehabilitation stage evaluation methods based on the wearable sensors and machine learning algorithm have been developed, the interpretable evaluation of the Brunnstrom recovery stage of the lower limb (BRS-L) is still lacking. The paper propose an interpretable BRS-L evaluation method based on wearable sensors. We collected lower limb motion data and plantar pressure data of 20 hemiplegic patients and 10 healthy individuals using seven Inertial Measurement Units (IMUs) and two plantar pressure insoles. Then we extracted gait features from the motion data and pressure data. By using feature selection based on feature importance, we improved the interpretability of the machine learning-based evaluation method. Several machine learning models are evaluated on the dataset, the results show that k-Nearest Neighbor has the best prediction performance and achieves 94.2% accuracy with an input of 18 features. Our method provides a feasible solution for precise rehabilitation and home-based rehabilitation of hemiplegic patients.

STAnet: A Spatiotemporal Attention Network for Decoding Auditory Spatial Attention From EEG.

OBJECTIVE: Humans are able to localize the source of a sound. This enables them to direct attention to a particular speaker in a cocktail party. Psycho-acoustic studies show that the sensory cortices of the human brain respond to the location of sound sources differently, and the auditory attention itself is a dynamic and temporally based brain activity. In this work, we seek to build a computational model which uses both spatial and temporal information manifested in EEG signals for auditory spatial attention detection (ASAD). METHODS: We propose an end-to-end spatiotemporal attention network, denoted as STAnet, to detect auditory spatial attention from EEG. The STAnet is designed to assign differentiated weights dynamically to EEG channels through a spatial attention mechanism, and to temporal patterns in EEG signals through a temporal attention mechanism. RESULTS: We report the ASAD experiments on two publicly available datasets. The STAnet outperforms other competitive models by a large margin under various experimental conditions. Its attention decision for 1-second decision window outperforms that of the state-of-the-art techniques for 10-second decision window. Experimental results also demonstrate that the STAnet achieves competitive performance on EEG signals ranging from 64 to as few as 16 channels. CONCLUSION: This study provides evidence suggesting that efficient low-density EEG online decoding is within reach. SIGNIFICANCE: This study also marks an important step towards the practical implementation of ASAD in real life applications.

A model for predicting ground reaction force and energetics of human locomotion with an elastically suspended backpack.

This paper presents an actuated spring-loaded inverted pendulum model with a vertically constrained suspended load mass to predict the vertical GRF and energetics of walking and running. Experiments were performed to validate the model prediction accuracy of vertical GRF. The average correlation coefficient was greater than 0.97 during walking and 0.98 during running. The model's predictions of energy cost reduction were compared with experimental data from the literature, and the difference between the experimental and predicted results was less than 7%. The predicted results of characteristic forces and energy cost under different suspension stiffness and damping conditions showed a tradeoff when selecting the suspension parameters of elastically suspended backpacks.

Evaluation of the Load Reduction Performance Via a Suspended Backpack With Adjustable Stiffness.

Backpacks are essential for travel but carrying a load during a long journey can easily cause muscle fatigue and joint injuries. Previous studies have suggested that suspended backpacks can effectively reduce the energy cost while carrying loads. Researchers have found that adjusting the stiffness of a suspended backpack can optimize its performance. Therefore, this paper proposes a stiffness-adjustable suspended backpack; the system stiffness can be adjusted to suitable values at different speeds. The stiffness of the suspended backpack with a 5-kg load was designed to be 690 N/m for a speed of 4.5 km/h, and it was adjusted to 870 and 1050 N/m at speeds of 5.5 and 6.5 km/h, respectively. The goal of this study was to determine how carrying a stiffness-adjustable suspended backpack affected performance while carrying a load. Six healthy participants participated in experiments where they wore two backpacks under three conditions: the adjustable-stiffness suspended backpack condition (S_A), the unadjustable-stiffness suspended backpack condition (S_UA), and the ordinary backpack condition (ORB). Our results showed that the peak accelerations, muscle activities, and peak ground reaction forces in the S_A condition were reduced effectively by adjusting the stiffness to adapt to different walking speeds; this adjustment decreased the metabolic cost by 4.21 ± 1.21% and 2.68 ± 0.88% at 5.5 km/h and 4.27 ± 1.35% and 3.38 ± 1.31% at 6.5 km/h compared to the ORB and S_UA, respectively.

Auditory Attention Detection with EEG Channel Attention.

Auditory attention detection (AAD) seeks to detect the attended speech from EEG signals in a multi-talker scenario, i.e. cocktail party. As the EEG channels reflect the activities of different brain areas, a task-oriented channel selection technique improves the performance of brain-computer interface applications. In this study, we propose a soft channel attention mechanism, instead of hard channel selection, that derives an EEG channel mask by optimizing the auditory attention detection task. The neural AAD system consists of a neural channel attention mechanism and a convolutional neural network (CNN) classifier. We evaluate the proposed framework on a publicly available database. We achieve 88.3% and 77.2% for 2-second and 0.1-second decision windows with 64-channel EEG; and 86.1% and 83.9% for 2-second decision windows with 32-channel and 16-channel EEG, respectively. The proposed framework outperforms other competitive models by a large margin across all test cases.

Auditory Attention Detection via Cross-Modal Attention.

Humans show a remarkable perceptual ability to select the speech stream of interest among multiple competing speakers. Previous studies demonstrated that auditory attention detection (AAD) can infer which speaker is attended by analyzing a listener's electroencephalography (EEG) activities. However, previous AAD approaches perform poorly on short signal segments, more advanced decoding strategies are needed to realize robust real-time AAD. In this study, we propose a novel approach, i.e., cross-modal attention-based AAD (CMAA), to exploit the discriminative features and the correlation between audio and EEG signals. With this mechanism, we hope to dynamically adapt the interactions and fuse cross-modal information by directly attending to audio and EEG features, thereby detecting the auditory attention activities manifested in brain signals. We also validate the CMAA model through data visualization and comprehensive experiments on a publicly available database. Experiments show that the CMAA achieves accuracy values of 82.8, 86.4, and 87.6% for 1-, 2-, and 5-s decision windows under anechoic conditions, respectively; for a 2-s decision window, it achieves an average of 84.1% under real-world reverberant conditions. The proposed CMAA network not only achieves better performance than the conventional linear model, but also outperforms the state-of-the-art non-linear approaches. These results and data visualization suggest that the CMAA model can dynamically adapt the interactions and fuse cross-modal information by directly attending to audio and EEG features in order to improve the AAD performance.

Two-branch 3D convolutional neural network for motor imagery EEG decoding.

Objective.The original motor imagery electroencephalography (MI-EEG) data contains not only temporal features but also a large number of spatial features related to the distribution of electrodes on the brain. However, in the process of MI-EEG decoding, most of the current convolutional neural network (CNN) based methods do not make the utmost of the spatial features related to electrode distribution.Approach.In this study, we adopt a concise 3D representation for the MI-EEG data to take full advantage of the spatial features and propose a two-branch 3D CNN (TB-3D CNN) for the 3D representation of MI-EEG data. First, the spatial and temporal features of the input 3D samples are extracted by the spatial and temporal feature learning branches, respectively, to avoid the mutual interference between the temporal and spatial features. Then, the central loss is introduced into the TB-3D CNN framework to further improve the MI-EEG decoding accuracy. And a 3D data augmentation method based on the cyclic translation of time dimension is proposed for the 3D representation method to alleviate the overfitting problem.Main results.Some experiments are conducted on the famous BCI competition IV 2a dataset to evaluate the effectiveness of the proposed MI-EEG decoding method. The experimental results comparison with some state-of-the-art methods demonstrates that the average accuracy of our method is 4.42% higher than that of the best of the comparative methods.Significance.The proposed MI-EEG decoding method has great promise to improve the performance of motor imagery brain-computer interface system.

Mechanical Efficiency Investigation of an Ankle-Assisted Robot for Human Walking With a Backpack-Load.

The purpose of this work is to investigate the efficiency of wearable assistive devices under different load-carriage walking. We designed an experimental platform with a lightweight ankle-assisted robot. Eight subjects were tested in three experimental conditions: free walk with load (FWL), power-off with load (POFL), and power-on with load for different levels of force at a walking speed of 3.6 km/h. We recorded the metabolic expenditure and kinematics of the subjects under three levels of load-carried (10%, 20%, and 30% of body mass). We define the critical force, where at a certain load, the robot inputs a certain force to the human body, and with the assistance of this force, the positive effect of the robot on the human body exactly compensates for the negative effect. The critical forces from the fit of the assistive force and metabolic cost curves were 130 N, 160 N, and 215 N at three different load levels. The intrinsic weight of our device increases mechanical work at the ankle as the load weight rises with 2.08 J, 2.43 J, and 2.73 J for one leg during a gait cycle. With weight bearing increasing, the ratio of the mechanical work input by the robot to the mechanical work output by the weight of the device decreases (from 0.904 to 0.717 and 0.513), verifying that the walking assistance efficiency of such devices decreases as the weight rises.

A novel motor imagery EEG decoding method based on feature separation.

Objective. Motor imagery electroencephalography (EEG) decoding is a vital technology for the brain-computer interface (BCI) systems and has been widely studied in recent years. However, the original EEG signals usually contain a lot of class-independent information, and the existing motor imagery EEG decoding methods are easily interfered by this irrelevant information, which greatly limits the decoding accuracy of these methods.Approach. To overcome the interference of the class-independent information, a motor imagery EEG decoding method based on feature separation is proposed in this paper. Furthermore, a feature separation network based on adversarial learning (FSNAL) is designed for the feature separation of the original EEG samples. First, the class-related features and class-independent features are separated by the proposed FSNAL framework, and then motor imagery EEG decoding is performed only according to the class-related features to avoid the adverse effects of class-independent features.Main results. To validate the effectiveness of the proposed motor imagery EEG decoding method, we conduct some experiments on two public EEG datasets (the BCI competition IV 2a and 2b datasets). The experimental results comparison between our method and some state-of-the-art methods demonstrates that our motor imagery EEG decoding method outperforms all the compared methods on the two experimental datasets.Significance. Our motor imagery EEG decoding method can alleviate the interference of class-independent features, and it has great application potential for improving the performance of motor imagery BCI systems in the near future.

Motor Imagery EEG Decoding Method Based on a Discriminative Feature Learning Strategy.

With the rapid development of deep learning, more and more deep learning-based motor imagery electroencephalograph (EEG) decoding methods have emerged in recent years. However, the existing deep learning-based methods usually only adopt the constraint of classification loss, which hardly obtains the features with high discrimination and limits the improvement of EEG decoding accuracy. In this paper, a discriminative feature learning strategy is proposed to improve the discrimination of features, which includes the central distance loss (CD-loss), the central vector shift strategy, and the central vector update process. First, the CD-loss is proposed to make the same class of samples converge to the corresponding central vector. Then, the central vector shift strategy extends the distance between different classes of samples in the feature space. Finally, the central vector update process is adopted to avoid the non-convergence of CD-loss and weaken the influence of the initial value of central vectors on the final results. In addition, overfitting is another severe challenge for deep learning-based EEG decoding methods. To deal with this problem, a data augmentation method based on circular translation strategy is proposed to expand the experimental datasets without introducing any extra noise or losing any information of the original data. To validate the effectiveness of the proposed method, we conduct some experiments on two public motor imagery EEG datasets (BCI competition IV 2a and 2b dataset), respectively. The comparison with current state-of-the-art methods indicates that our method achieves the highest average accuracy and good stability on the two experimental datasets.

Individualized Gait Generation for Rehabilitation Robots Based on Recurrent Neural Networks.

Individualized reference gait patterns for lower limb rehabilitation robots can greatly improve the effectiveness of rehabilitation. However, previous methods can only generate customized gait patterns at several fixed discrete walking speeds and generating gaits at continuously varying speeds and stride lengths remains unsolved. This work proposes an individualized gait pattern generation method based on a recurrent neural network (RNN), which is proficient in series modeling. We collected the largest gait data set of this kind, which consists of 4,425 gait patterns from 137 subjects. Using this data set, we trained an RNN to create a function mapping from body parameters and gait parameters to a gait pattern. The experimental results indicate that our model is able to generate gait patterns at continuously varying walking speeds and stride lengths while also reducing the errors in the ankle, knee, and hip measurements by 12.83%, 20.95%, and 28.25%, respectively, compared to previous state-of-the-art method.

Computer-aided diagnosis of pectus excavatum using CT images and deep learning methods.

Pectus excavatum (PE) is one of the most common chest wall defects. Accurate assessment of PE deformities is critical for effective surgical intervention. Index-based evaluations have become the standard for objectively estimating PE, however, these indexes cannot represent the whole information of chest CT images and may associated with significant error due to the individual differences. To overcome these limitations, this paper developed a computer-aided diagnosis (CAD) system based on the convolutional neural network (CNN) to automatically learn discriminative features and classify PE images. We also adopted block-wise fine-tuning methods based on the transfer learning strategy to reduce the potential risk of overfitting caused by limited data and experimentally explored the best fine-tuning degree. Our method achieved a high level of classification accuracy with 94.76% for PE diagnosis. Furthermore, we proposed a majority rule-based voting method to provide a comprehensively diagnostic results for each patient, which integrated the classification results of the whole thorax. The promising results support the feasibility of our proposed CNN-based CAD system for automatic PE diagnosis, which paves a way for comprehensive assessments of PE in clinics.