An auditory brain-computer interface to detect changes in sound pressure level for automatic volume control.

Volume control is necessary to adjust sound levels for a comfortable audio or video listening experience. This study aims to develop an automatic volume control system based on a brain-computer interface (BCI). We thus focused on a BCI using an auditory oddball paradigm, and conducted two types of experiments. In the first experiment, the participant was asked to pay attention to a target sound where the sound level was high (70 dB) compared with the other sounds (60 dB). The brain activity measured by electroencephalogram showed large positive activity (P300) for the target sound, and classification of the target and nontarget sounds achieved an accuracy of 0.90. The second experiment adopted a two-target paradigm where a low sound level (50 dB) was introduced as the second target sound. P300 was also observed in the second experiment, and a value of 0.76 was obtained for the binary classification of the target and nontarget sounds. Further, we found that better accuracy was observed in large sound levels compared to small ones. These results suggest the possibility of using BCI for automatic volume control; however, it is necessary to improve its accuracy for application in daily life.

Sliding-Window Normalization to Improve the Performance of Machine-Learning Models for Real-Time Motion Prediction Using Electromyography.

Many researchers have used machine learning models to control artificial hands, walking aids, assistance suits, etc., using the biological signal of electromyography (EMG). The use of such devices requires high classification accuracy. One method for improving the classification performance of machine learning models is normalization, such as z-score. However, normalization is not used in most EMG-based motion prediction studies because of the need for calibration and fluctuation of reference value for calibration (cannot re-use). Therefore, in this study, we proposed a normalization method that combines sliding-window and z-score normalization that can be implemented in real-time processing without need for calibration. The effectiveness of this normalization method was confirmed by conducting a single-joint movement experiment of the elbow and predicting its rest, flexion, and extension movements from the EMG signal. The proposed method achieved 77.7% accuracy, an improvement of 21.5% compared to the non-normalization (56.2%). Furthermore, when using a model trained by other people's data for application without calibration, the proposed method achieved 63.1% accuracy, an improvement of 8.8% compared to the z-score (54.4%). These results showed the effectiveness of the simple and easy-to-implement method, and that the classification performance of the machine learning model could be improved.

Verification of Normalization Method to Improve Usability and Versatility among Users of Applications that Predict Continuous Motion Using Electromyography.

In applications using electromyography (EMG), it is important to ensure high performance for all users (versatility among users) and to enable use without prior preparation (usability). Some of the current applications that use EMG normalize the signal through methods based on the measured maximum absolute value of EMG (maEMG), such as dynamic contraction (DC). However, usability is low when using DC because the reference value must be measured first every time the application is used. Further, the versatility among users is low because of the nonlinearity of EMG and the fact that maEMG varies among users. This study aimed to improve usability and versatility among users for continuous classification tasks using EMG. To this end, we developed a normalization method using sliding-window and z-score normalization techniques. The results reveal that the proposed method exhibits higher usability and versatility among users than DC. The proposed method does not require any calibration time, suggesting improved usability, while yielding the same classification accuracy as DC (57% for three target tasks) for a model trained using a subject's own data. Further, for a model trained with other users' data, the proposed method yields a classification accuracy of 53%, which is 18% higher than that of DC (35%), suggesting versatility among users. These results demonstrate that the proposed normalization method improves usability and versatility for users of practical applications that use EMG and perform continuous classification, such as prosthetic hands.

Wrist and finger motor representations embedded in the cerebral and cerebellar resting-state activation.

Several functional magnetic resonance imaging (fMRI) studies have demonstrated that resting-state brain activity consists of multiple components, each corresponding to the spatial pattern of brain activity induced by performing a task. Especially in a movement task, such components have been shown to correspond to the brain activity pattern of the relevant anatomical region, meaning that the voxels of pattern that are cooperatively activated while using a body part (e.g., foot, hand, and tongue) also behave cooperatively in the resting state. However, it is unclear whether the components involved in resting-state brain activity correspond to those induced by the movement of discrete body parts. To address this issue, in the present study, we focused on wrist and finger movements in the hand, and a cross-decoding technique trained to discriminate between the multi-voxel patterns induced by wrist and finger movement was applied to the resting-state fMRI. We found that the multi-voxel pattern in resting-state brain activity corresponds to either wrist or finger movements in the motor-related areas of each hemisphere of the cerebrum and cerebellum. These results suggest that resting-state brain activity in the motor-related areas consists of the components corresponding to the elementary movements of individual body parts. Therefore, the resting-state brain activity possibly has a finer structure than considered previously.

Forward Inverse Relaxation Model Incorporating Movement Duration Optimization.

A computational trajectory formation model based on the optimization principle, which introduces the forward inverse relaxation model (FIRM) as the hardware and algorithm, represents the features of human arm movements well. However, in this model, the movement duration was defined as a given value and not as a planned value. According to considerable empirical facts, movement duration changes depending on task factors, such as required accuracy and movement distance thus, it is considered that there are some criteria that optimize the cost function. Therefore, we propose a FIRM that incorporates a movement duration optimization module. The movement duration optimization module minimizes the weighted sum of the commanded torque change term as the trajectory cost, and the tolerance term as the cost of time. We conducted a behavioral experiment to examine how well the movement duration obtained by the model reproduces the true movement. The results suggested that the model movement duration was close to the true movement. In addition, the trajectory generated by inputting the obtained movement duration to the FIRM reproduced the features of the actual trajectory well. These findings verify the use of this computational model in measuring human arm movements.

Relations between Speed-Accuracy Trade-Off and Muscle Synergy in Isometric Contraction Tasks.

Muscle synergy is the theory that movements are controlled by a module of coordinated combined muscles. This theory is thought to solve the degrees-of-freedom problem in the musculoskeletal system. Previous studies have investigated the robustness of muscle synergies under conditions such as varying speeds and required degrees of accuracy. One of the principles of human movement is that when movement becomes faster, spatial accuracy is reduced. This is called the "speed-accuracy trade-off" (SAT), and many models have been proposed to explain this phenomenon. Studies on muscle synergies have shown that muscle synergy modules are robust against changes in speed; however, the relationship between SAT and motor control by muscle synergies remains unclear. Therefore, we investigated the relationship between changes in spatial accuracy and changes in speed and muscle synergies from measured behavioral data and surface electromyography. This was achieved by performing an isometric contraction task in which subjects exerted a horizontal force with various movement speeds. The results showed that the module structures of muscle synergies were robust against speed changes, and that the neural commands to muscle synergies changed in response to speed changes. In addition, changes in spatial accuracy with variations in speed tended to increase when movement was performed with a single muscle synergy. These results suggest that the number of muscle synergies used for movement may affect movement accuracy.Clinical Relevance-The results of this study suggest that the number of muscle synergies used for movement affects spatial accuracy.

Developing a Method to Control an Arm-Assist-Suit by Predicting Arm-Trajectory Using Electromyography.

Many researchers have developed assist-suits to support repetitive and strenuous physical labor, but existing suits show unsatisfactory responsiveness and restrict arm motions. Therefore, we propose a method for an arm-assist-suit that synchronizes arm motions by using electromyography (EMG) to predict arm trajectory. EMG is used to measure and record electrical signals while muscles are active. Further, predicted arm-joint motions and estimated arm-joint angles are used for arm trajectory predictions. In this study, we attempted the prediction of elbow-joint motions and the timing of motion changes. Two subjects executed twelve types of elbow-joint movements that had four start and endpoints. We measured seven muscle types with EMG points on the right arm(hand, elbow, and shoulder) a motion capture system, respectively. After processing these data, we applied a multiclass logistic regression, which is a machine-learning technique, to predict elbow-joint motions, namely, rest, flexion, and extension. The precision in elbow joint motion prediction shows a difference between the two subjects for the three motions analyzed. Additionally, the rest prediction accuracy is lower than both flexion and extension for each subject. The prediction of elbow-joint motion change timing does not correlate with the elbow-joint motion predictions, with the timing prediction precision being very low and thus, causing some difficulties. To overcome these difficulties, and improve precision in future work, we plan to apply an independent component analysis to eliminate noise and add or change features.Clinical Relevance- This study aims to establish a benchmark for future research on the improvement of responsiveness and range-of-motion of arm-assist-suits.

fNIRS-GANs: data augmentation using generative adversarial networks for classifying motor tasks from functional near-infrared spectroscopy.

OBJECTIVE: Functional near-infrared spectroscopy (fNIRS) is expected to be applied to brain-computer interface (BCI) technologies. Since lengthy fNIRS measurements are uncomfortable for participants, it is difficult to obtain enough data to train classification models; hence, the fNIRS-BCI accuracy decreases. APPROACH: In this study, to improve the fNIRS-BCI accuracy, we examined an fNIRS data augmentation method using Wasserstein generative adversarial networks (WGANs). Using fNIRS data during hand-grasping tasks, we evaluated whether the proposed data augmentation method could generate artificial fNIRS data and improve the classification performance using support vector machines and simple neural networks. MAIN RESULTS: Trial-averaged temporal profiles of WGAN-generated fNIRS data were similar to those of the measured data except that they contained an extra noise component. By augmenting the generated data to training data, the accuracies for classifying four different task types were improved irrespective of the classifiers. SIGNIFICANCE: This result suggests that the artificial fNIRS data generated by the proposed data augmentation method is useful for improving BCI performance.

Explanation of Fitts' law in Reaching Movement based on Human Arm Dynamics.

Why does Fitts' law fit various human behavioural data well even though it is not a model based on human physical dynamics? To clarify this, we derived the relationships among the factors applied in Fitts' law-movement duration and spatial endpoint error-based on a multi-joint forward- and inverse-dynamics models in the presence of signal-dependent noise. As a result, the relationship between them was modelled as an inverse proportion. To validate whether the endpoint error calculated by the model can represent the endpoint error of actual movements, we conducted a behavioural experiment in which centre-out reaching movements were performed under temporal constraints in four directions using the shoulder and elbow joints. The result showed that the distributions of model endpoint error closely expressed the observed endpoint error distributions. Furthermore, the model was found to be nearly consistent with Fitts' law. Further analysis revealed that the coefficients of Fitts' law could be expressed by arm dynamics and signal-dependent noise parameters. Consequently, our answer to the question above is: Fitts' law for reaching movements can be expressed based on human arm dynamics; thus, Fitts' law closely fits human's behavioural data under various conditions.

Classification of Movement Direction From Electroencephalogram During Working Memory Time.

When humans perform cognitive tasks, it is necessary to hold information temporarily. This is done by a brain function called working memory (WM). Since WM is active during the whole time range from stimulus presentation to task execution, onset detection is unnecessary, in contrast to readiness potentials for movement. Therefore, it is possible to realize application in a brain-computer interface (BCI) in various tasks without onset detection by performing single-trial classification of electroencephalogram (EEG) signals during WM. The purpose of this research was to examine the possibility of WM application to BCI. We classified the EEG signals during WM-time that required the retention of movement direction information when performing a right arm movement in order of two instructed sequential target directions using a 3-layer neural network (3-NN). In classification based on the signal immediately after presentation of 1st target (WM1), the classification accuracy was significantly higher (62%) than chance level (50%). In addition, the accuracy was higher when providing the phase of the fast Fourier transform to the classifier as information rather than the spectrum. However, it could not be classified by WM requiring the retention of information regarding two tasks (WM2). In summary, these results suggest a possibility that single trial classification of EEG during the first WM (WM1) is possible, and that the WM information is included mainly in the phase. Future studies should aim at improving the classification accuracy by using other feature quantities and classifiers, and to examine classification of EEG in tasks other than arm movement. Furthermore, the relationship between WM and EEG distribution also needs to be investigated.

Cortical activation associated with motor preparation can be used to predict the freely chosen effector of an upcoming movement and reflects response time: An fMRI decoding study.

Motor action is prepared in the human brain for rapid initiation at the appropriate time. Recent non-invasive decoding techniques have shown that brain activity for action preparation represents various parameters of an upcoming action. In the present study, we demonstrated that a freely chosen effector can be predicted from brain activity measured using functional magnetic resonance imaging (fMRI) before initiation of the action. Furthermore, the activity was related to response time (RT). We measured brain activity with fMRI while 12 participants performed a finger-tapping task using either the left or right hand, which was freely chosen by them. Using fMRI decoding, we identified brain regions in which activity during the preparatory period could predict the hand used for the upcoming action. We subsequently evaluated the relationship between brain activity and the RT of the upcoming action to determine whether correct decoding was associated with short RT. We observed that activity in the supplementary motor area, dorsal premotor cortex (PMd), and primary motor cortex (M1) measured before action execution predicted the hand used to perform the action with significantly above-chance accuracy (approximately 70%). Furthermore, in most participants, the RT was shorter in trials for which the used hand was correctly predicted by activity in the PMd and M1. The present study showed that preparatory activity in cortical motor areas represents information about the effector used for an upcoming action, and that well-formed motor representations in these regions are associated with reduced response times.

Alpha Phase Synchronization of Parietal Areas Reflects Switch-Specific Activity During Mental Rotation: An EEG Study.

Action selection is typically influenced by the history of previously selected actions (the immediate motor history), which is apparent when a selected action is switched from a previously selected one to a new one. This history dependency of the action selection is even observable during a mental hand rotation task. Thus, we hypothesized that the history-dependent interaction of actions might share the same neural mechanisms among different types of action switching tasks. An alternative hypothesis is that the history dependency of the mental hand rotation task might involve a distinctive neural mechanism from the general action selection tasks so that the reported observation with the mental hand rotation task in the previously published literature might lack generality. To refute this possibility, we compared neural activity during action switching in the mental hand rotation with the general action switching task which is triggered by a simple visual stimulus. In the experiment, to focus on temporal changes in whole brain oscillatory activity, we recorded electroencephalographic (EEG) signals while 25 healthy subjects performed the two tasks. For analysis, we examined functional connectivity reflected in EEG phase synchronization and analyzed temporal changes in brain activity when subjects switched from a previously selected action to a new action. Using a clustering-based method to identify functional connectivity reflected in time-varying phase synchronization, we identified alpha-power inter-parietal synchronization that appears only during switching of the selected action, regardless of the hand laterality in the presented image. Moreover, the current study revealed that for both tasks the extent of this alpha-power inter-parietal synchronization was altered by the history of the selected actions. These findings suggest that alpha-power inter-parietal synchronization is engaged as a form of switching-specific functional connectivity, and that switching-related activity is independent of the task paradigm.

Investigation of isochrony phenomenon based on the computational theory of human arm trajectory planning.

The isochrony principle is a well-known phenomenon whereby the speed of human arm movement is regulated to increase as its trajectory distance increases. However, the relationship between the trajectory planning and the isochrony phenomenon has never been sufficiently explained. One computational study derived the algorithm for estimating the optimal movement segmentation and its duration based on the framework of the minimum commanded torque change criterion. By extending this finding, we can consider the hypothesis that the human arm trajectory is generated based on the minimum commanded torque change criterion to ensure that the duration average of the commanded torque changes (DCTCs) are equivalent between certain movement segmentations, rather than to satisfy the isochrony phenomenon. To test this hypothesis, we measured the behavioral performance of hand movement tasks in which subjects write eight-shaped and double-elliptical-shaped trajectories including two similar shaped arcs of different sizes (hereafter called large and small loops). Our results indicate that the human arm movement is planned in such a manner that the DCTCs for the large and small loops are equivalent during writing of the double-elliptical-shaped trajectories regardless of the arc size. A similar tendency was also observed for the data during the eight-shaped movements, although the ratio of the DCTCs slightly changed depending on the arc size conditions. Thus, our study provides experimental evidence that the isochrony phenomenon is ensured through the computational process of trajectory planning.

Improving the Performance of an Auditory Brain-Computer Interface Using Virtual Sound Sources by Shortening Stimulus Onset Asynchrony.

Recently, a brain-computer interface (BCI) using virtual sound sources has been proposed for estimating user intention via electroencephalogram (EEG) in an oddball task. However, its performance is still insufficient for practical use. In this study, we examine the impact that shortening the stimulus onset asynchrony (SOA) has on this auditory BCI. While very short SOA might improve its performance, sound perception and task performance become difficult, and event-related potentials (ERPs) may not be induced if the SOA is too short. Therefore, we carried out behavioral and EEG experiments to determine the optimal SOA. In the experiments, participants were instructed to direct attention to one of six virtual sounds (target direction). We used eight different SOA conditions: 200, 300, 400, 500, 600, 700, 800, and 1,100 ms. In the behavioral experiment, we recorded participant behavioral responses to target direction and evaluated recognition performance of the stimuli. In all SOA conditions, recognition accuracy was over 85%, indicating that participants could recognize the target stimuli correctly. Next, using a silent counting task in the EEG experiment, we found significant differences between target and non-target sound directions in all but the 200-ms SOA condition. When we calculated an identification accuracy using Fisher discriminant analysis (FDA), the SOA could be shortened by 400 ms without decreasing the identification accuracies. Thus, improvements in performance (evaluated by BCI utility) could be achieved. On average, higher BCI utilities were obtained in the 400 and 500-ms SOA conditions. Thus, auditory BCI performance can be optimized for both behavioral and neurophysiological responses by shortening the SOA.

Convolutional Neural Networks with 3D Input for P300 Identification in Auditory Brain-Computer Interfaces.

From allowing basic communication to move through an environment, several attempts are being made in the field of brain-computer interfaces (BCI) to assist people that somehow find it difficult or impossible to perform certain activities. Focusing on these people as potential users of BCI, we obtained electroencephalogram (EEG) readings from nine healthy subjects who were presented with auditory stimuli via earphones from six different virtual directions. We presented the stimuli following the oddball paradigm to elicit P300 waves within the subject's brain activity for later identification and classification using convolutional neural networks (CNN). The CNN models are given a novel single trial three-dimensional (3D) representation of the EEG data as an input, maintaining temporal and spatial information as close to the experimental setup as possible, a relevant characteristic as eliciting P300 has been shown to cause stronger activity in certain brain regions. Here, we present the results of CNN models using the proposed 3D input for three different stimuli presentation time intervals (500, 400, and 300 ms) and compare them to previous studies and other common classifiers. Our results show >80% accuracy for all the CNN models using the proposed 3D input in single trial P300 classification.

Transient increase in systemic interferences in the superficial layer and its influence on event-related motor tasks: a functional near-infrared spectroscopy study.

Functional near-infrared spectroscopy (fNIRS) is a widely utilized neuroimaging tool in fundamental neuroscience research and clinical investigation. Previous research has revealed that task-evoked systemic artifacts mainly originating from the superficial-tissue may preclude the identification of cerebral activation during a given task. We examined the influence of such artifacts on event-related brain activity during a brisk squeezing movement. We estimated task-evoked superficial-tissue hemodynamics from short source­detector distance channels (15 mm) by applying principal component analysis. The estimated superficial-tissue hemodynamics exhibited temporal profiles similar to the canonical cerebral hemodynamic model. Importantly, this task-evoked profile was also observed in data from a block design motor experiment, suggesting a transient increase in superficial-tissue hemodynamics occurs following motor behavior, irrespective of task design. We also confirmed that estimation of event-related cerebral hemodynamics was improved by a simple superficial-tissue hemodynamic artifact removal process using 15-mm short distance channels, compared to the results when no artifact removal was applied. Thus, our results elucidate task design-independent characteristics of superficial-tissue hemodynamics and highlight the need for the application of superficial-tissue hemodynamic artifact removal methods when analyzing fNIRS data obtained during event-related motor tasks.

Reduction of global interference of scalp-hemodynamics in functional near-infrared spectroscopy using short distance probes.

Functional near-infrared spectroscopy (fNIRS) is used to measure cerebral activity because it is simple and portable. However, scalp-hemodynamics often contaminates fNIRS signals, leading to detection of cortical activity in regions that are actually inactive. Methods for removing these artifacts using standard source-detector distance channels (Long-channel) tend to over-estimate the artifacts, while methods using additional short source-detector distance channels (Short-channel) require numerous probes to cover broad cortical areas, which leads to a high cost and prolonged experimental time. Here, we propose a new method that effectively combines the existing techniques, preserving the accuracy of estimating cerebral activity and avoiding the disadvantages inherent when applying the techniques individually. Our new method accomplishes this by estimating a global scalp-hemodynamic component from a small number of Short-channels, and removing its influence from the Long-channels using a general linear model (GLM). To demonstrate the feasibility of this method, we collected fNIRS and functional magnetic resonance imaging (fMRI) measurements during a motor task. First, we measured changes in oxygenated hemoglobin concentration (∆Oxy-Hb) from 18 Short-channels placed over motor-related areas, and confirmed that the majority of scalp-hemodynamics was globally consistent and could be estimated from as few as four Short-channels using principal component analysis. We then measured ∆Oxy-Hb from 4 Short- and 43 Long-channels. The GLM identified cerebral activity comparable to that measured separately by fMRI, even when scalp-hemodynamics exhibited substantial task-related modulation. These results suggest that combining measurements from four Short-channels with a GLM provides robust estimation of cerebral activity at a low cost.

Decoding sequential finger movements from preparatory activity in higher-order motor regions: a functional magnetic resonance imaging multi-voxel pattern analysis.

Performing a complex sequential finger movement requires the temporally well-ordered organization of individual finger movements. Previous behavioural studies have suggested that the brain prepares a whole sequence of movements as a single set, rather than the movements of individual fingers. However, direct neuroimaging support for this hypothesis is lacking and, assuming it to be true, it remains unclear which brain regions represent the information of a prepared sequence. Here, we measured brain activity with functional magnetic resonance imaging while 14 right-handed healthy participants performed two types of well-learned sequential finger movements with their right hands. Using multi-voxel pattern analysis, we examined whether the types of the forthcoming sequence could be predicted from the preparatory activities of nine regions of interest, which included the motor, somatosensory and posterior parietal regions in each hemisphere, bilateral visual cortices, cerebellum and basal ganglia. We found that, during preparation, the activity of the contralateral motor regions could predict which of the two sequences would be executed. Further detailed analysis revealed that the contralateral dorsal premotor cortex and supplementary motor area were the key areas that contributed to the prediction consistently across participants. These contrasted with results from execution-related brain activity where a performed sequence was successfully predicted from the activities in the broad cortical sensory-motor network, including the bilateral motor, parietal and ipsilateral somatosensory cortices. Our study supports the hypothesis that temporary well-organized sequences of movements are represented as a set in the brain, and that preparatory activity in higher-order motor regions represents information about upcoming motor actions.

Resting-state brain activity in the motor cortex reflects task-induced activity: A multi-voxel pattern analysis.

It has been suggested that resting-state brain activity reflects task-induced brain activity patterns. In this study, we examined whether neural representations of specific movements can be observed in the resting-state brain activity patterns of motor areas. First, we defined two regions of interest (ROIs) to examine brain activity associated with two different behavioral tasks. Using multi-voxel pattern analysis with regularized logistic regression, we designed a decoder to detect voxel-level neural representations corresponding to the tasks in each ROI. Next, we applied the decoder to resting-state brain activity. We found that the decoder discriminated resting-state neural activity with accuracy comparable to that associated with task-induced neural activity. The distribution of learned weighted parameters for each ROI was similar for resting-state and task-induced activities. Large weighted parameters were mainly located on conjunctive areas. Moreover, the accuracy of detection was higher than that for a decoder whose weights were randomly shuffled, indicating that the resting-state brain activity includes multi-voxel patterns similar to the neural representation for the tasks. Therefore, these results suggest that the neural representation of resting-state brain activity is more finely organized and more complex than conventionally considered.