Evaluating the Feasibility of Visual Imagery for an EEG-Based Brain-Computer Interface.

Visual imagery, or the mental simulation of visual information from memory, could serve as an effective control paradigm for a brain-computer interface (BCI) due to its ability to directly convey the user's intention with many natural ways of envisioning an intended action. However, multiple initial investigations into using visual imagery as a BCI control strategies have been unable to fully evaluate the capabilities of true spontaneous visual mental imagery. One major limitation in these prior works is that the target image is typically displayed immediately preceding the imagery period. This paradigm does not capture spontaneous mental imagery as would be necessary in an actual BCI application but something more akin to short-term retention in visual working memory. Results from the present study show that short-term visual imagery following the presentation of a specific target image provides a stronger, more easily classifiable neural signature in EEG than spontaneous visual imagery from long-term memory following an auditory cue for the image. We also show that short-term visual imagery and visual perception share commonalities in the most predictive electrodes and spectral features. However, visual imagery received greater influence from frontal electrodes whereas perception was mostly confined to occipital electrodes. This suggests that visual perception is primarily driven by sensory information whereas visual imagery has greater contributions from areas associated with memory and attention. This work provides the first direct comparison of short-term and long-term visual imagery tasks and provides greater insight into the feasibility of using visual imagery as a BCI control strategy.

Transfer learning promotes acquisition of individual BCI skills.

Subject training is crucial for acquiring brain-computer interface (BCI) control. Typically, this requires collecting user-specific calibration data due to high inter-subject neural variability that limits the usability of generic decoders. However, calibration is cumbersome and may produce inadequate data for building decoders, especially with naïve subjects. Here, we show that a decoder trained on the data of a single expert is readily transferrable to inexperienced users via domain adaptation techniques allowing calibration-free BCI training. We introduce two real-time frameworks, (i) Generic Recentering (GR) through unsupervised adaptation and (ii) Personally Assisted Recentering (PAR) that extends GR by employing supervised recalibration of the decoder parameters. We evaluated our frameworks on 18 healthy naïve subjects over five online sessions, who operated a customary synchronous bar task with continuous feedback and a more challenging car racing game with asynchronous control and discrete feedback. We show that along with improved task-oriented BCI performance in both tasks, our frameworks promoted subjects' ability to acquire individual BCI skills, as the initial neurophysiological control features of an expert subject evolved and became subject specific. Furthermore, those features were task-specific and were learned in parallel as participants practiced the two tasks in every session. Contrary to previous findings implying that supervised methods lead to improved online BCI control, we observed that longitudinal training coupled with unsupervised domain matching (GR) achieved similar performance to supervised recalibration (PAR). Therefore, our presented frameworks facilitate calibration-free BCIs and have immediate implications for broader populations-such as patients with neurological pathologies-who might struggle to provide suitable initial calibration data.

Combining detrended cross-correlation analysis with Riemannian geometry-based classification for improved brain-computer interface performance.

Riemannian geometry-based classification (RGBC) gained popularity in the field of brain-computer interfaces (BCIs) lately, due to its ability to deal with non-stationarities arising in electroencephalography (EEG) data. Domain adaptation, however, is most often performed on sample covariance matrices (SCMs) obtained from EEG data, and thus might not fully account for components affecting covariance estimation itself, such as regional trends. Detrended cross-correlation analysis (DCCA) can be utilized to estimate the covariance structure of such signals, yet it is computationally expensive in its original form. A recently proposed online implementation of DCCA, however, allows for its fast computation and thus makes it possible to employ DCCA in real-time applications. In this study we propose to replace the SCM with the DCCA matrix as input to RGBC and assess its effect on offline and online BCI performance. First we evaluated the proposed decoding pipeline offline on previously recorded EEG data from 18 individuals performing left and right hand motor imagery (MI), and benchmarked it against vanilla RGBC and popular MI-detection approaches. Subsequently, we recruited eight participants (with previous BCI experience) who operated an MI-based BCI (MI-BCI) online using the DCCA-enhanced Riemannian decoder. Finally, we tested the proposed method on a public, multi-class MI-BCI dataset. During offline evaluations the DCCA-based decoder consistently and significantly outperformed the other approaches. Online evaluation confirmed that the DCCA matrix could be computed in real-time even for 22-channel EEG, as well as subjects could control the MI-BCI with high command delivery (normalized Cohen's κ: 0.7409 ± 0.1515) and sample-wise MI detection (normalized Cohen's κ: 0.5200 ± 0.1610). Post-hoc analysis indicated characteristic connectivity patterns under both MI conditions, with stronger connectivity in the hemisphere contralateral to the MI task. Additionally, fractal scaling exponent of neural activity was found increased in the contralateral compared to the ipsilateral motor cortices (C4 and C3 for left and right MI, respectively) in both classes. Combining DCCA with Riemannian geometry-based decoding yields a robust and effective decoder, that not only improves upon the SCM-based approach but can also provide relevant information on the neurophysiological processes behind MI.

Inferring individual evaluation criteria for reaching trajectories with obstacle avoidance from EEG signals.

During reaching actions, the human central nerve system (CNS) generates the trajectories that optimize effort and time. When there is an obstacle in the path, we make sure that our arm passes the obstacle with a sufficient margin. This comfort margin varies between individuals. When passing a fragile object, risk-averse individuals may adopt a larger margin by following the longer path than risk-prone people do. However, it is not known whether this variation is associated with a personalized cost function used for the individual optimal control policies and how it is represented in our brain activity. This study investigates whether such individual variations in evaluation criteria during reaching results from differentiated weighting given to energy minimization versus comfort, and monitors brain error-related potentials (ErrPs) evoked when subjects observe a robot moving dangerously close to a fragile object. Seventeen healthy participants monitored a robot performing safe, daring and unsafe trajectories around a wine glass. Each participant displayed distinct evaluation criteria on the energy efficiency and comfort of robot trajectories. The ErrP-BCI outputs successfully inferred such individual variation. This study suggests that ErrPs could be used in conjunction with an optimal control approach to identify the personalized cost used by CNS. It further opens new avenues for the use of brain-evoked potential to train assistive robotic devices through the use of neuroprosthetic interfaces.

EEG error-related potentials encode magnitude of errors and individual perceptual thresholds.

Error-related potentials (ErrPs) are a prominent electroencephalogram (EEG) correlate of performance monitoring, and so crucial for learning and adapting our behavior. It is poorly understood whether ErrPs encode further information beyond error awareness. We report an experiment with sixteen participants over three sessions in which occasional visual rotations of varying magnitude occurred during a cursor reaching task. We designed a brain-computer interface (BCI) to detect ErrPs that provided real-time feedback. The individual ErrP-BCI decoders exhibited good transfer across sessions and scalability over the magnitude of errors. A non-linear relationship between the ErrP-BCI output and the magnitude of errors predicts individual perceptual thresholds to detect errors. We also reveal theta-gamma oscillatory coupling that co-varied with the magnitude of the required adjustment. Our findings open new avenues to probe and extend current theories of performance monitoring by incorporating continuous human interaction tasks and analysis of the ErrP complex rather than individual peaks.

EEG signature of breaks in embodiment in VR.

The brain mechanism of embodiment in a virtual body has grown a scientific interest recently, with a particular focus on providing optimal virtual reality (VR) experiences. Disruptions from an embodied state to a less- or non-embodied state, denominated Breaks in Embodiment (BiE), are however rarely studied despite their importance for designing interactions in VR. Here we use electroencephalography (EEG) to monitor the brain's reaction to a BiE, and investigate how this reaction depends on previous embodiment conditions. The experimental protocol consisted of two sequential steps; an induction step where participants were either embodied or non-embodied in an avatar, and a monitoring step where, in some cases, participants saw the avatar's hand move while their hand remained still. Our results show the occurrence of error-related potentials linked to observation of the BiE event in the monitoring step. Importantly, this EEG signature shows amplified potentials following the non-embodied condition, which is indicative of an accumulation of errors across steps. These results provide neurophysiological indications on how progressive disruptions impact the expectation of embodiment for a virtual body.

A highly stable electrode with low electrode-skin impedance for wearable brain-computer interface.

To date, brain-computer interfaces (BCIs) have proved to play a key role in many medical applications, for example, the rehabilitation of stroke patients. For post-stroke rehabilitation, the BCIs require the EEG electrodes to precisely translate the brain signals of patients into intended movements of the paralyzed limb for months. However, the gold standard silver/silver-chloride electrodes cannot satisfy the requirements for long-term stability and preparation-free recording capability in wearable EEG devices, thus limiting the versatility of EEG in wearable BCI applications over time outside the rehabilitation center. Here, we design a long-term stable and low electrode-skin interfacial impedance conductive polymer-hydrogel EEG electrode that maintains a lower impedance value than gel-based electrodes for 29 days. With this technology, EEG-based long-term and wearable BCIs could be realized in the near future. To demonstrate this, our designed electrode is applied for a wireless single-channel EEG device that detects changes in alpha rhythms in eye-open/eye-close conditions. In addition, we validate that the designed electrodes could capture oscillatory rhythms in motor imagery protocols as well as low-frequency time-locked event-related potentials from healthy subjects, with similar or better performance than gel-based electrodes. Finally, we demonstrate the use of the designed electrode in online BCI-based functional electrical stimulation, which could be used for post-stroke rehabilitation.

2020 International brain-computer interface competition: A review.

The brain-computer interface (BCI) has been investigated as a form of communication tool between the brain and external devices. BCIs have been extended beyond communication and control over the years. The 2020 international BCI competition aimed to provide high-quality neuroscientific data for open access that could be used to evaluate the current degree of technical advances in BCI. Although there are a variety of remaining challenges for future BCI advances, we discuss some of more recent application directions: (i) few-shot EEG learning, (ii) micro-sleep detection (iii) imagined speech decoding, (iv) cross-session classification, and (v) EEG(+ear-EEG) detection in an ambulatory environment. Not only did scientists from the BCI field compete, but scholars with a broad variety of backgrounds and nationalities participated in the competition to address these challenges. Each dataset was prepared and separated into three data that were released to the competitors in the form of training and validation sets followed by a test set. Remarkable BCI advances were identified through the 2020 competition and indicated some trends of interest to BCI researchers.

Brain-Computer Interfaces for Treatment of Focal Dystonia.

Task-specificity in isolated focal dystonias is a powerful feature that may successfully be targeted with therapeutic brain-computer interfaces. While performing a symptomatic task, the patient actively modulates momentary brain activity (disorder signature) to match activity during an asymptomatic task (target signature), which is expected to translate into symptom reduction.

Machine-Learning Based Monitoring of Cognitive Workload in Rescue Missions With Drones.

In search and rescue missions, drone operations are challenging and cognitively demanding. High levels of cognitive workload can affect rescuers' performance, leading to failure with catastrophic outcomes. To face this problem, we propose a machine learning algorithm for real-time cognitive workload monitoring to understand if a search and rescue operator has to be replaced or if more resources are required. Our multimodal cognitive workload monitoring model combines the information of 25 features extracted from physiological signals, such as respiration, electrocardiogram, photoplethysmogram, and skin temperature, acquired in a noninvasive way. To reduce both subject and day inter-variability of the signals, we explore different feature normalization techniques, and introduce a novel weighted-learning method based on support vector machines suitable for subject-specific optimizations. On an unseen test set acquired from 34 volunteers, our proposed subject-specific model is able to distinguish between low and high cognitive workloads with an average accuracy of 87.3% and 91.2% while controlling a drone simulator using both a traditional controller and a new-generation controller, respectively.

Learning to control a BMI-driven wheelchair for people with severe tetraplegia.

Mind-controlled wheelchairs are an intriguing assistive mobility solution applicable in complete paralysis. Despite progress in brain-machine interface (BMI) technology, its translation remains elusive. The primary objective of this study is to probe the hypothesis that BMI skill acquisition by end-users is fundamental to control a non-invasive brain-actuated intelligent wheelchair in real-world settings. We demonstrate that three tetraplegic spinal-cord injury users could be trained to operate a non-invasive, self-paced thought-controlled wheelchair and execute complex navigation tasks. However, only the two users exhibiting increasing decoding performance and feature discriminancy, significant neuroplasticity changes and improved BMI command latency, achieved high navigation performance. In addition, we show that dexterous, continuous control of robots is possible through low-degree of freedom, discrete and uncertain control channels like a motor imagery BMI, by blending human and artificial intelligence through shared-control methodologies. We posit that subject learning and shared-control are the key components paving the way for translational non-invasive BMI.

Customizing skills for assistive robotic manipulators, an inverse reinforcement learning approach with error-related potentials.

Robotic assistance via motorized robotic arm manipulators can be of valuable assistance to individuals with upper-limb motor disabilities. Brain-computer interfaces (BCI) offer an intuitive means to control such assistive robotic manipulators. However, BCI performance may vary due to the non-stationary nature of the electroencephalogram (EEG) signals. It, hence, cannot be used safely for controlling tasks where errors may be detrimental to the user. Avoiding obstacles is one such task. As there exist many techniques to avoid obstacles in robotics, we propose to give the control to the robot to avoid obstacles and to leave to the user the choice of the robot behavior to do so a matter of personal preference as some users may be more daring while others more careful. We enable the users to train the robot controller to adapt its way to approach obstacles relying on BCI that detects error-related potentials (ErrP), indicative of the user's error expectation of the robot's current strategy to meet their preferences. Gaussian process-based inverse reinforcement learning, in combination with the ErrP-BCI, infers the user's preference and updates the obstacle avoidance controller so as to generate personalized robot trajectories. We validate the approach in experiments with thirteen able-bodied subjects using a robotic arm that picks up, places and avoids real-life objects. Results show that the algorithm can learn user's preference and adapt the robot behavior rapidly using less than five demonstrations not necessarily optimal.

Brain Signals to Rescue Aphasia, Apraxia and Dysarthria Speech Recognition.

In this paper, we propose a deep learning-based algorithm to improve the performance of automatic speech recognition (ASR) systems for aphasia, apraxia, and dysarthria speech by utilizing electroencephalography (EEG) features recorded synchronously with aphasia, apraxia, and dysarthria speech. We demonstrate a significant decoding performance improvement by more than 50% during test time for isolated speech recognition task and we also provide preliminary results indicating performance improvement for the more challenging continuous speech recognition task by utilizing EEG features. The results presented in this paper show the first step towards demonstrating the possibility of utilizing non-invasive neural signals to design a real-time robust speech prosthetic for stroke survivors recovering from aphasia, apraxia, and dysarthria. Our aphasia, apraxia, and dysarthria speech-EEG data set will be released to the public to help further advance this interesting and crucial research.

Invariability of EEG error-related potentials during continuous feedback protocols elicited by erroneous actions at predicted or unpredicted states.

Objective.When humans perceive an erroneous action, an EEG error-related potential (ErrP) is elicited as a neural response. ErrPs have been largely investigated in discrete feedback protocols, where actions are executed at discrete steps, to enable seamless brain-computer interaction. However, there are only a few studies that investigate ErrPs in continuous feedback protocols. The objective of the present study is to better understand the differences between two types of ErrPs elicited during continuous feedback protocols, where errors may occur either at predicted or unpredicted states. We hypothesize that ErrPs of the unpredicted state is associated with longer latency as it requires higher cognitive workload to evaluate actions compared to the predicted states.Approach.Participants monitored the trajectory of an autonomous cursor that occasionally made erroneous actions on its way to the target in two conditions, namely, predicted or unpredicted states. After characterizing the ErrP waveform elicited by erroneous actions in the two conditions, we performed single-trial decoding of ErrPs in both synchronous (i.e. time-locked to the onset of the erroneous action) and asynchronous manner. Furthermore, we explored the possibility to transfer decoders built with data of one of the conditions to the other condition.Main results.As hypothesized, erroneous actions at unpredicted states gave rise to ErrPs with higher latency than erroneous actions at predicted states, a correlate of higher cognitive effort in the former condition. Moreover, ErrP decoders trained in a given condition successfully transferred to the other condition with a slight loss of classification performance. This was the case for synchronous as well as asynchronous ErrP decoding, showing the invariability of ErrPs across conditions.Significance.These results advance the characterization of ErrPs during continuous feedback protocols, enlarging the potential use of ErrPs during natural operation of brain-controlled devices as it is not necessary to have different decoders for each kind of erroneous conditions.

Context-Aware Learning for Generative Models.

This work studies the class of algorithms for learning with side-information that emerges by extending generative models with embedded context-related variables. Using finite mixture models (FMMs) as the prototypical Bayesian network, we show that maximum-likelihood estimation (MLE) of parameters through expectation-maximization (EM) improves over the regular unsupervised case and can approach the performances of supervised learning, despite the absence of any explicit ground-truth data labeling. By direct application of the missing information principle (MIP), the algorithms' performances are proven to range between the conventional supervised and unsupervised MLE extremities proportionally to the information content of the contextual assistance provided. The acquired benefits regard higher estimation precision, smaller standard errors, faster convergence rates, and improved classification accuracy or regression fitness shown in various scenarios while also highlighting important properties and differences among the outlined situations. Applicability is showcased with three real-world unsupervised classification scenarios employing Gaussian mixture models. Importantly, we exemplify the natural extension of this methodology to any type of generative model by deriving an equivalent context-aware algorithm for variational autoencoders (VAs), thus broadening the spectrum of applicability to unsupervised deep learning with artificial neural networks. The latter is contrasted with a neural-symbolic algorithm exploiting side information.

User Adaptation to Closed-Loop Decoding of Motor Imagery Termination.

One of the most popular methods in non-invasive brain machine interfaces (BMI) relies on the decoding of sensorimotor rhythms associated to sustained motor imagery. Although motor imagery has been intensively studied, its termination is mostly neglected. OBJECTIVE: Here, we provide insights in the decoding of motor imagery termination and investigate the use of such decoder in closed-loop BMI. METHODS: Participants (N = 9) were asked to perform kinesthetic motor imagery of both hands simultaneously cued with a clock indicating the initiation and termination of the action. Using electroencephalogram (EEG) signals, we built a decoder to detect the transition between event-related desynchronization and event-related synchronization. Features for this decoder were correlates of motor termination in the upper µ and ß bands. RESULTS: The decoder reached an accuracy of 76.2% (N = 9), revealing the high robustness of our approach. More importantly, this paper shows that the decoding of motor termination has an intrinsic latency mainly due to the delayed appearance of its correlates. Because the latency was consistent and thus predictable, users were able to compensate it after training. CONCLUSION: Using our decoding system, BMI users were able to adapt their behavior and modulate their sensorimotor rhythm to stop the device (clock) accurately on time. SIGNIFICANCE: These results show the importance of closed-loop evaluations of BMI decoders and open new possibilities for BMI control using decoding of movement termination.

Disentangling the origins of confidence in speeded perceptual judgments through multimodal imaging.

The human capacity to compute the likelihood that a decision is correct-known as metacognition-has proven difficult to study in isolation as it usually cooccurs with decision making. Here, we isolated postdecisional from decisional contributions to metacognition by analyzing neural correlates of confidence with multimodal imaging. Healthy volunteers reported their confidence in the accuracy of decisions they made or decisions they observed. We found better metacognitive performance for committed vs. observed decisions, indicating that committing to a decision may improve confidence. Relying on concurrent electroencephalography and hemodynamic recordings, we found a common correlate of confidence following committed and observed decisions in the inferior frontal gyrus and a dissociation in the anterior prefrontal cortex and anterior insula. We discuss these results in light of decisional and postdecisional accounts of confidence and propose a computational model of confidence in which metacognitive performance naturally improves when evidence accumulation is constrained upon committing a decision.

Brain-computer interfaces: Definitions and principles.

Throughout life, the central nervous system (CNS) interacts with the world and with the body by activating muscles and excreting hormones. In contrast, brain-computer interfaces (BCIs) quantify CNS activity and translate it into new artificial outputs that replace, restore, enhance, supplement, or improve the natural CNS outputs. BCIs thereby modify the interactions between the CNS and the environment. Unlike the natural CNS outputs that come from spinal and brainstem motoneurons, BCI outputs come from brain signals that represent activity in other CNS areas, such as the sensorimotor cortex. If BCIs are to be useful for important communication and control tasks in real life, the CNS must control these brain signals nearly as reliably and accurately as it controls spinal motoneurons. To do this, they might, for example, need to incorporate software that mimics the function of the subcortical and spinal mechanisms that participate in normal movement control. The realization of high reliability and accuracy is perhaps the most difficult and critical challenge now facing BCI research and development. The ongoing adaptive modifications that maintain effective natural CNS outputs take place primarily in the CNS. The adaptive modifications that maintain effective BCI outputs can also take place in the BCI. This means that the BCI operation depends on the effective collaboration of two adaptive controllers, the CNS and the BCI. Realization of this second adaptive controller, the BCI, and management of its interactions with concurrent adaptations in the CNS comprise another complex and critical challenge for BCI development. BCIs can use different kinds of brain signals recorded in different ways from different brain areas. Decisions about which signals recorded in which ways from which brain areas should be selected for which applications are empirical questions that can only be properly answered by experiments. BCIs, like other communication and control technologies, often face artifacts that contaminate or imitate their chosen signals. Noninvasive BCIs (e.g., EEG- or fNIRS-based) need to take special care to avoid interpreting nonbrain signals (e.g., cranial EMG) as brain signals. This typically requires comprehensive topographical and spectral evaluations. In theory, the outputs of BCIs can select a goal or control a process. In the future, the most effective BCIs will probably be those that combine goal selection and process control so as to distribute control between the BCI and the application in a fashion suited to the current action. Through such distribution, BCIs may most effectively imitate natural CNS operation. The primary measure of BCI development is the extent to which BCI systems benefit people with neuromuscular disorders. Thus, BCI clinical evaluation, validation, and dissemination is a key step. It is at the same time a complex and difficult process that depends on multidisciplinary collaboration and management of the demanding requirements of clinical studies. Twenty-five years ago, BCI research was an esoteric endeavor pursued in only a few isolated laboratories. It is now a steadily growing field that engages many hundreds of scientists, engineers, and clinicians throughout the world in an increasingly interconnected community that is addressing the key issues and pursuing the high potential of BCI technology.

General principles of machine learning for brain-computer interfacing.

Brain-computer interfaces (BCIs) are systems that translate brain activity patterns into commands that can be executed by an artificial device. This enables the possibility of controlling devices such as a prosthetic arm or exoskeleton, a wheelchair, typewriting applications, or games directly by modulating our brain activity. For this purpose, BCI systems rely on signal processing and machine learning algorithms to decode the brain activity. This chapter provides an overview of the main steps required to do such a process, including signal preprocessing, feature extraction and selection, and decoding. Given the large amount of possible methods that can be used for these processes, a comprehensive review of them is beyond the scope of this chapter, and it is focused instead on the general principles that should be taken into account, as well as discussing good practices on how these methods should be applied and evaluated for proper design of reliable BCI systems.