Multimodal fusion for anticipating human decision performance.

Anticipating human decisions while performing complex tasks remains a formidable challenge. This study proposes a multimodal machine-learning approach that leverages image features and electroencephalography (EEG) data to predict human response correctness in a demanding visual searching task. Notably, we extract a novel set of image features pertaining to object relationships using the Segment Anything Model (SAM), which enhances prediction accuracy compared to traditional features. Additionally, our approach effectively utilizes a combination of EEG signals and image features to streamline the feature set required for the Random Forest Classifier (RFC) while maintaining high accuracy. The findings of this research hold substantial potential for developing advanced fault alert systems, particularly in critical decision-making environments such as the medical and defence sectors.

Understanding the effects of stress on the P300 response during naturalistic simulation of heights exposure.

Stress is a prevalent bodily response universally experienced and significantly affects a person's mental and cognitive state. The P300 response is a commonly observed brain behaviour that provides insight into a person's cognitive state. Previous works have documented the effects of stress on the P300 behaviour; however, only a few have explored the performance in a mobile and naturalistic experimental setup. Our study examined the effects of stress on the human brain's P300 behaviour through a height exposure experiment that incorporates complex visual, vestibular, and proprioceptive stimuli. A more complex sensory environment could produce translatable findings toward real-world behaviour and benefit emerging technologies such as brain-computer interfaces. Seventeen participants experienced our experiment that elicited the stress response through physical and virtual height exposure. We found two unique groups within our participants that exhibited contrasting behavioural performance and P300 target reaction response when exposed to stressors (from walking at heights). One group performed worse when exposed to heights and exhibited a significant decrease in parietal P300 peak amplitude and increased beta and gamma power. On the other hand, the group less affected by stress exhibited a change in their N170 peak amplitude and alpha/mu rhythm desynchronisation. The findings of our study suggest that a more individualised approach to assessing a person's behaviour performance under stress can aid in understanding P300 performance when experiencing stress.

EEG Information Transfer Changes in Different Daily Fatigue Levels During Drowsy Driving.

A significant issue for traffic safety has been drowsy driving for decades. A number of studies have investigated the effects of acute fatigue on spectral power; and recent research has revealed that drowsy driving is associated with a variety of brain connections in a specific cortico-cortical pathway. In spite of this, it is still unclear how different brain regions are connected in drowsy driving at different levels of daily fatigue. This study identified the brain connectivity-behavior relationship among three different daily fatigue levels (low-, median- and high-fatigue) with the EEG data transfer entropy. According to the results, only low- and medium-fatigue groups demonstrated an inverted U-shaped change in connectivity from high performance to poor behavioral performance. In addition, from low- to high-fatigue groups, connectivity magnitude decreased in the frontal region and increased in the occipital region. These study results suggest that brain connectivity and driving behavior would be affected by different levels of daily fatigue.

Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human-robot collaboration.

Objective. Brain-computer interface (BCI) technology is poised to play a prominent role in modern work environments, especially a collaborative environment where humans and machines work in close proximity, often with physical contact. In a physical human robot collaboration (pHRC), the robot performs complex motion sequences. Any unexpected robot behavior or faulty interaction might raise safety concerns. Error-related potentials, naturally generated by the brain when a human partner perceives an error, have been extensively employed in BCI as implicit human feedback to adapt robot behavior to facilitate a safe and intuitive interaction. However, the integration of BCI technology with error-related potential for robot control demands failure-free integration of highly uncertain electroencephalography (EEG) signals, particularly influenced by the physical and cognitive state of the user. As a higher workload on the user compromises their access to cognitive resources needed for error awareness, it is crucial to study how mental workload variations impact the error awareness as it might raise safety concerns in pHRC. In this study, we aim to study how cognitive workload affects the error awareness of a human user engaged in a pHRC.Approach. We designed a blasting task with an abrasive industrial robot and manipulated the mental workload with a secondary arithmetic task of varying difficulty. EEG data, perceived workload, task and physical performance were recorded from 24 participants moving the robot arm. The error condition was achieved by the unexpected stopping of the robot in 33% of trials.Main results. We observed a diminished amplitude for the prediction error negativity (PEN) and error positivity (Pe), indicating reduced error awareness with increasing mental workload. We further observed an increased frontal theta power and increasing trend in the central alpha and central beta power after the unexpected robot stopping compared to when the robot stopped correctly at the target. We also demonstrate that a popular convolution neural network model, EEGNet, could predict the amplitudes of PEN and Pe from the EEG data prior to the error.Significance. This prediction model could be instrumental in developing an online prediction model that could forewarn the system and operators of the diminished error awareness of the user, alluding to a potential safety breach in error-related potential-based BCI system for pHRC. Therefore, our work paves the way for embracing BCI technology in pHRC to optimally adapt the robot behavior for personalized user experience using real-time brain activity, enriching the quality of the interaction.

Robust Fuzzy Neural Network With an Adaptive Inference Engine.

Fuzzy neural networks (FNNs) have been very successful at handling uncertainty in data using fuzzy mappings and if-then rules. However, they suffer from generalization and dimensionality issues. Although deep neural networks (DNNs) represent a step toward processing high-dimensional data, their capacity to address data uncertainty is limited. Furthermore, deep learning algorithms designed to improve robustness are either time consuming or yield unsatisfactory performance. In this article, we propose a robust fuzzy neural network (RFNN) to overcome these problems. The network contains an adaptive inference engine that is capable of handling samples with high-level uncertainty and high dimensions. Unlike traditional FNNs that use a fuzzy AND operation to calculate the firing strength for each rule, our inference engine is able to learn the firing strength adaptively. It also further processes the uncertainty in membership function values. Taking advantage of the learning ability of neural networks, the acquired fuzzy sets can be learned from training inputs automatically to cover the input space well. Furthermore, the consequent layer uses neural network structures to enhance the reasoning ability of the fuzzy rules when dealing with complex inputs. Experiments on a range of datasets show that RFNN delivers state-of-the-art accuracy even at very high levels of uncertainty. Our code is available online. https://github.com/leijiezhang/RFNN.

A Connectivity-Aware Graph Neural Network for Real-Time Drowsiness Classification.

Drowsy driving is one of the primary causes of driving fatalities. Electroencephalography (EEG), a method for detecting drowsiness directly from brain activity, has been widely used for detecting driver drowsiness in real-time. Recent studies have revealed the great potential of using brain connectivity graphs constructed based on EEG data for drowsy state predictions. However, traditional brain connectivity networks are irrelevant to the downstream prediction tasks. This article proposes a connectivity-aware graph neural network (CAGNN) using a self-attention mechanism that can generate task-relevant connectivity networks via end-to-end training. Our method achieved an accuracy of 72.6% and outperformed other convolutional neural networks (CNNs) and graph generation methods based on a drowsy driving dataset. In addition, we introduced a squeeze-and-excitation (SE) block to capture important features and demonstrated that the SE attention score can reveal the most important feature band. We compared our generated connectivity graphs in the drowsy and alert states and found drowsiness connectivity patterns, including significantly reduced occipital connectivity and interregional connectivity. Additionally, we performed a post hoc interpretability analysis and found that our method could identify drowsiness features such as alpha spindles. Our code is available online at https://github.com/ALEX95GOGO/CAGNN.

Early Detection of Human Decision-Making in Concealed Object Visual Searching Tasks: An EEG-BiLSTM Study.

Detecting concealed objects presents a significant challenge for human and artificial intelligent systems. Detecting concealed objects task necessitates a high level of human attention and cognitive effort to complete the task successfully. Thus, in this study, we use concealed objects as stimuli for our decision-making experimental paradigms to quantify participants' decision-making performance. We applied a deep learning model, Bi-directional Long Short Term Memory (BiLSTM), to predict the participant's decision accuracy by using their electroencephalogram (EEG) signals as input. The classifier model demonstrated high accuracy, reaching 96.1% with an epoching time range of 500 ms following the stimulus event onset. The results revealed that the parietal-occipital brain region provides highly informative information for the classifier in the concealed visual searching tasks. Furthermore, the neural mechanism underlying the concealed visual-searching and decision-making process was explained by analyzing serial EEG components. The findings of this study could contribute to the development of a fault alert system, which has the potential to improve human decision-making performance.

Classification of inattentional blindness using brain dynamics of ERPs.

Situational awareness (SA) is vital for understanding our surroundings. Multiple variables, including inattentive blindness (IB), contribute to the deterioration of SA, which may have detrimental effects on individuals' cognitive performance. IB occurs due to attentional limitations, ignoring critical information and resulting in a loss of SA and a decline in general performance, particularly in complicated situations requiring substantial cognitive resources. To the best of our knowledge, however, past research has not fully uncovered the neurological characteristics of IB nor classified these characteristics in life-alike virtual situations. Therefore, the purpose of this study is to determine whether ERP dynamics in the brain may be utilised as a neural feature to predict the occurrence of IB using machine learning (ML) algorithms. In a virtual reality simulation of an IB experiment, 30 participants' behaviour and Electroencephalography (EEG) measurements were obtained. Participants were given a target detection task in the IB experiment without knowing the unattended shapes displayed on the background building. The targets were presented in three different sensory modalities (auditory, visual, and visual-auditory). On the post-experiment questionnaire, participants who claimed not to have noticed the unattended shapes were assigned to the IB group. Subsequently, the Aware group was formed from individuals who reported seeing the unattended shapes. Using EEGNet to classify IB and Aware groups demonstrated a high classification performance. According to the research, ERP brain dynamics are associated with the awareness of unattended shapes and have the potential to serve as a reliable indication for predicting the visual consciousness of unexpected objects.(p/)(p)Clinical relevance- This research offers a potential brain marker for the mixed-reality and BCI systems that will be used in the future to identify cognitive deterioration, maintain attentional capacity, and prevent disasters.

Distinction of the object recognition and object identification in the brain-computer interfaces applications.

Object recognition is a complex cognitive process in which information is integrated and processed by various brain regions. Previous studies have shown that both the visual and temporal cortices are active during object recognition and identification. However, although object recognition and object identification are similar, these processes are considered distinct functions in the brain. Despite this, the differentiation between object recognition and identification has yet to be clearly defined for use in brain-computer interface (BCI) applications. This research aims to utilize neural features related to object recognition and identification and classify these features to differentiate between the two processes. The results demonstrate that several classifiers achieved high levels of accuracy, with the XGBoost classifier using a Linear Booster achieving the highest accuracy at 96% and a F1 score of 0.97. This ability to distinguish between object recognition and identification can be a beneficial aspect of a BCI object recognition system as it could help determine the intended target object for a user.

Ventral and Dorsal Stream EEG Channels: Key Features for EEG-Based Object Recognition and Identification.

Object recognition and object identification are multifaceted cognitive operations that require various brain regions to synthesize and process information. Prior research has evidenced the activity of both visual and temporal cortices during these tasks. Notwithstanding their similarities, object recognition and identification are recognized as separate brain functions. Drawing from the two-stream hypothesis, our investigation aims to understand whether the channels within the ventral and dorsal streams contain pertinent information for effective model learning regarding object recognition and identification tasks. By utilizing the data we collected during the object recognition and identification experiment, we scrutinized EEGNet models, trained using channels that replicate the two-stream hypothesis pathways, against a model trained using all available channels. The outcomes reveal that the model trained solely using the temporal region delivered a high accuracy level in classifying four distinct object categories. Specifically, the object recognition and object identification models achieved an accuracy of 89% and 85%, respectively. By incorporating the channels that mimic the ventral stream, the model's accuracy was further improved, with the object recognition model and object identification model achieving an accuracy of 95% and 94%, respectively. Furthermore, the Grad-CAM result of the trained models revealed a significant contribution from the ventral and dorsal stream channels toward the training of the EEGNet model. The aim of our study is to pinpoint the optimal channel configuration that provides a swift and accurate brain-computer interface system for object recognition and identification.

MGRW-Transformer: Multigranularity Random Walk Transformer Model for Interpretable Learning.

Deep-learning models have been widely used in image recognition tasks due to their strong feature-learning ability. However, most of the current deep-learning models are "black box" systems that lack a semantic explanation of how they reached their conclusions. This makes it difficult to apply these methods to complex medical image recognition tasks. The vision transformer (ViT) model is the most commonly used deep-learning model with a self-attention mechanism that shows the region of influence as compared to traditional convolutional networks. Thus, ViT offers greater interpretability. However, medical images often contain lesions of variable size in different locations, which makes it difficult for a deep-learning model with a self-attention module to reach correct and explainable conclusions. We propose a multigranularity random walk transformer (MGRW-Transformer) model guided by an attention mechanism to find the regions that influence the recognition task. Our method divides the image into multiple subimage blocks and transfers them to the ViT module for classification. Simultaneously, the attention matrix output from the multiattention layer is fused with the multigranularity random walk module. Within the multigranularity random walk module, the segmented image blocks are used as nodes to construct an undirected graph using the attention node as a starting node and guiding the coarse-grained random walk. We appropriately divide the coarse blocks into finer ones to manage the computational cost and combine the results based on the importance of the discovered features. The result is that the model offers a semantic interpretation of the input image, a visualization of the interpretation, and insight into how the decision was reached. Experimental results show that our method improves classification performance with medical images while presenting an understandable interpretation for use by medical professionals.

An investigation into the effectiveness of using acoustic touch to assist people who are blind.

Wearable smart glasses are an emerging technology gaining popularity in the assistive technologies industry. Smart glasses aids typically leverage computer vision and other sensory information to translate the wearer's surrounding into computer-synthesized speech. In this work, we explored the potential of a new technique known as "acoustic touch" to provide a wearable spatial audio solution for assisting people who are blind in finding objects. In contrast to traditional systems, this technique uses smart glasses to sonify objects into distinct sound auditory icons when the object enters the device's field of view. We developed a wearable Foveated Audio Device to study the efficacy and usability of using acoustic touch to search, memorize, and reach items. Our evaluation study involved 14 participants, 7 blind or low-visioned and 7 blindfolded sighted (as a control group) participants. We compared the wearable device to two idealized conditions, a verbal clock face description and a sequential audio presentation through external speakers. We found that the wearable device can effectively aid the recognition and reaching of an object. We also observed that the device does not significantly increase the user's cognitive workload. These promising results suggest that acoustic touch can provide a wearable and effective method of sensory augmentation.

Higher Order Fractal Belief Rényi Divergence With Its Applications in Pattern Classification.

Information can be quantified and expressed by uncertainty, and improving the decision level of uncertain information is vital in modeling and processing uncertain information. Dempster-Shafer evidence theory can model and process uncertain information effectively. However, the Dempster combination rule may provide counter-intuitive results when dealing with highly conflicting information, leading to a decline in decision level. Thus, measuring conflict is significant in the improvement of decision level. Motivated by this issue, this paper proposes a novel method to measure the discrepancy between bodies of evidence. First, the model of dynamic fractal probability transformation is proposed to effectively obtain more information about the non-specificity of basic belief assignments (BBAs). Then, we propose the higher-order fractal belief Rényi divergence (HOFBReD). HOFBReD can effectively measure the discrepancy between BBAs. Moreover, it is the first belief Rényi divergence that can measure the discrepancy between BBAs with dynamic fractal probability transformation. HoFBReD has several properties in terms of probability transformation as well as measurement. When the dynamic fractal probability transformation ends, HoFBReD is equivalent to measuring the Rényi divergence between the pignistic probability transformations of BBAs. When the BBAs degenerate to the probability distributions, HoFBReD will also degenerate to or be related to several well-known divergences. In addition, based on HoFBReD, a novel multisource information fusion algorithm is proposed. A pattern classification experiment with real-world datasets is presented to compare the proposed algorithm with other methods. The experiment results indicate that the proposed algorithm has a higher average pattern recognition accuracy with all datasets than other methods. The proposed discrepancy measurement method and multisource information algorithm contribute to the improvement of decision level.

Design and verification of a wearable wireless 64-channel high-resolution EEG acquisition system with wi-fi transmission.

Brain-computer interfaces (BCIs) allow communication between the brain and the external world. This type of technology has been extensively studied. However, BCI instruments with high signal quality are typically heavy and large. Thus, recording electroencephalography (EEG) signals is an inconvenient task. In recent years, system-on-chip (SoC) approaches have been integrated into BCI research, and sensors for wireless portable devices have been developed; however, there is still considerable work to be done. As neuroscience research has advanced, EEG signal analyses have come to require more accurate data. Due to the limited bandwidth of Bluetooth wireless transmission technology, EEG measurement systems with more than 16 channels must be used to reduce the sampling rate and prevent data loss. Therefore, the goal of this study was to develop a multichannel, high-resolution (24-bit), high-sampling-rate EEG BCI device that transmits signals via Wi-Fi. We believe that this system can be used in neuroscience research. The EEG acquisition system proposed in this work is based on a Cortex-M4 microcontroller with a Wi-Fi subsystem, providing a multichannel design and improved signal quality. This system is compatible with wet sensors, Ag/AgCl electrodes, and dry sensors. A LabVIEW-based user interface receives EEG data via Wi-Fi transmission and saves the raw data for offline analysis. In previous cognitive experiments, event tags have been communicated using Recommended Standard 232 (RS-232). The developed system was validated through event-related potential (ERP) and steady-state visually evoked potential (SSVEP) experiments. Our experimental results demonstrate that this system is suitable for recording EEG measurements and has potential in practical applications. The advantages of the developed system include its high sampling rate and high amplification. Additionally, in the future, Internet of Things (IoT) technology can be integrated into the system for remote real-time analysis or edge computing.

FedTP: Federated Learning by Transformer Personalization.

Federated learning is an emerging learning paradigm where multiple clients collaboratively train a machine learning model in a privacy-preserving manner. Personalized federated learning extends this paradigm to overcome heterogeneity across clients by learning personalized models. Recently, there have been some initial attempts to apply transformers to federated learning. However, the impacts of federated learning algorithms on self-attention have not yet been studied. In this article, we investigate this relationship and reveal that federated averaging (FedAvg) algorithms actually have a negative impact on self-attention in cases of data heterogeneity, which limits the capabilities of the transformer model in federated learning settings. To address this issue, we propose FedTP, a novel transformer-based federated learning framework that learns personalized self-attention for each client while aggregating the other parameters among the clients. Instead of using a vanilla personalization mechanism that maintains personalized self-attention layers of each client locally, we develop a learn-to-personalize mechanism to further encourage the cooperation among clients and to increase the scalability and generalization of FedTP. Specifically, we achieve this by learning a hypernetwork on the server that outputs the personalized projection matrices of self-attention layers to generate clientwise queries, keys, and values. Furthermore, we present the generalization bound for FedTP with the learn-to-personalize mechanism. Extensive experiments verify that FedTP with the learn-to-personalize mechanism yields state-of-the-art performance in the non-IID scenarios. Our code is available online https://github.com/zhyczy/FedTP.

EEG-Based Brain-Computer Interfaces are Vulnerable to Backdoor Attacks.

Research and development of electroencephalogram (EEG) based brain-computer interfaces (BCIs) have advanced rapidly, partly due to deeper understanding of the brain and wide adoption of sophisticated machine learning approaches for decoding the EEG signals. However, recent studies have shown that machine learning algorithms are vulnerable to adversarial attacks. This paper proposes to use narrow period pulse for poisoning attack of EEG-based BCIs, which makes adversarial attacks much easier to implement. One can create dangerous backdoors in the machine learning model by injecting poisoning samples into the training set. Test samples with the backdoor key will then be classified into the target class specified by the attacker. What most distinguishes our approach from previous ones is that the backdoor key does not need to be synchronized with the EEG trials, making it very easy to implement. The effectiveness and robustness of the backdoor attack approach is demonstrated, highlighting a critical security concern for EEG-based BCIs and calling for urgent attention to address it.

Speech2EEG: Leveraging Pretrained Speech Model for EEG Signal Recognition.

Identifying meaningful brain activities is critical in brain-computer interface (BCI) applications. Recently, an increasing number of neural network approaches have been proposed to recognize EEG signals. However, these approaches depend heavily on using complex network structures to improve the performance of EEG recognition and suffer from the deficit of training data. Inspired by the waveform characteristics and processing methods shared between EEG and speech signals, we propose Speech2EEG, a novel EEG recognition method that leverages pretrained speech features to improve the accuracy of EEG recognition. Specifically, a pretrained speech processing model is adapted to the EEG domain to extract multichannel temporal embeddings. Then, several aggregation methods, including the weighted average, channelwise aggregation, and channel-and-depthwise aggregation, are implemented to exploit and integrate the multichannel temporal embeddings. Finally, a classification network is used to predict EEG categories based on the integrated features. Our work is the first to explore the use of pretrained speech models for EEG signal analysis as well as the effective ways to integrate the multichannel temporal embeddings from the EEG signal. Extensive experimental results suggest that the proposed Speech2EEG method achieves state-of-the-art performance on two challenging motor imagery (MI) datasets, the BCI IV-2a and BCI IV-2b datasets, with accuracies of 89.5% and 84.07% , respectively. Visualization analysis of the multichannel temporal embeddings show that the Speech2EEG architecture can capture useful patterns related to MI categories, which can provide a novel solution for subsequent research under the constraints of a limited dataset scale.

The Effects of Virtual and Physical Elevation on Physiological Stress During Virtual Reality Height Exposure.

Advances in virtual reality technology have greatly benefited the acrophobia research field. Virtual reality height exposure is a reliable method of inducing stress with low variance across ages and demographics. When creating a virtual height exposure environment, researchers have often used haptic feedback elements to improve the sense of realism of a virtual environment. While the quality of the rendered for the virtual environment increases over time, the physical environment is often simplified to a conservative passive haptic feedback platform. The impact of the increasing disparity between the virtual and physical environment on the induced stress levels is unclear. This article presents an experiment that explored the effect of combining an elevated physical platform with different levels of virtual heights to induce stress. Eighteen participants experienced four different conditions of varying physical and virtual heights. The measurements included gait parameters, heart rate, heart rate variability, and electrodermal activity. The results show that the added physical elevation at a low virtual height shifts the participant's walking behaviour and increases the perception of danger. However, the virtual environment still plays an essential role in manipulating height exposure and inducing physiological stress. Another finding is that a person's behaviour always corresponds to the more significant perceived threat, whether from the physical or virtual environment.

Reinforcement Learning From Hierarchical Critics.

In this study, we investigate the use of global information to speed up the learning process and increase the cumulative rewards of reinforcement learning (RL) in competition tasks. Within the framework of actor-critic RL, we introduce multiple cooperative critics from two levels of a hierarchy and propose an RL from the hierarchical critics (RLHC) algorithm. In our approach, each agent receives value information from local and global critics regarding a competition task and accesses multiple cooperative critics in a top-down hierarchy. Thus, each agent not only receives low-level details, but also considers coordination from higher levels, thereby obtaining global information to improve the training performance. Then, we test the proposed RLHC algorithm against a benchmark algorithm, that is, proximal policy optimization (PPO), under four experimental scenarios consisting of tennis, soccer, banana collection, and crawler competitions within the Unity environment. The results show that RLHC outperforms the benchmark on these four competitive tasks.

Error-related potential-based shared autonomy via deep recurrent reinforcement learning.

Objective.Error-related potential (ErrP)-based brain-computer interfaces (BCIs) have received a considerable amount of attention in the human-robot interaction community. In contrast to traditional BCI, which requires continuous and explicit commands from an operator, ErrP-based BCI leverages the ErrP, which is evoked when an operator observes unexpected behaviours from the robot counterpart. This paper proposes a novel shared autonomy model for ErrP-based human-robot interaction.Approach.We incorporate ErrP information provided by a BCI as useful observations for an agent and formulate the shared autonomy problem as a partially observable Markov decision process. A recurrent neural network-based actor-critic model is used to address the uncertainty in the ErrP signal. We evaluate the proposed framework in a simulated human-in-the-loop robot navigation task with both simulated users and real users.Main results.The results show that the proposed ErrP-based shared autonomy model enables an autonomous robot to complete navigation tasks more efficiently. In a simulation with 70% ErrP accuracy, agents completed the task 14.1% faster than in the no ErrP condition, while with real users, agents completed the navigation task 14.9% faster.Significance.The evaluation results confirmed that the shared autonomy via deep recurrent reinforcement learning is an effective way to deal with uncertain human feedback in a complex human-robot interaction task.