A cross-scenario and cross-subject domain adaptation method for driving fatigue detection.

Objective.The scarcity of electroencephalogram (EEG) data, coupled with individual and scenario variations, leads to considerable challenges in real-world EEG-based driver fatigue detection. We propose a domain adaptation method that utilizes EEG data collected from a laboratory to supplement real-world EEG data and constructs a cross-scenario and cross-subject driver fatigue detection model for real-world scenarios.Approach.First, we collect EEG data from subjects participating in a driving experiment conducted in both laboratory and real-world scenarios. To address the issue of data scarcity, we build a real-world fatigued driving detection model by integrating the real-world data with the laboratory data. Then, we propose a method named cross-scenario and cross-subject domain adaptation (CS2DA), which aims to eliminate the domain shift problem caused by individual variances and scenario differences. Adversarial learning is adopted to extract the common features observed across different subjects within the same scenario. The multikernel maximum mean discrepancy (MK-MMD) method is applied to further minimize scenario differences. Additionally, we propose a conditional MK-MMD constraint to better utilize label information. Finally, we use seven rules to fuse the predicted labels.Main results.We evaluate the CS2DA method through extensive experiments conducted on the two EEG datasets created in this work: the SEED-VLA and the SEED-VRW datasets. Different domain adaptation methods are used to construct a real-world fatigued driving detection model using data from laboratory and real-world scenarios, as well as a combination of both. Our findings show that the proposed CS2DA method outperforms the existing traditional and adversarial learning-based domain adaptation approaches. We also find that combining data from both laboratory and real-world scenarios improves the performance of the model.Significance.This study contributes two EEG-based fatigue driving datasets and demonstrates that the proposed CS2DA method can effectively enhance the performance of a real-world fatigued driving detection model.

Investigating Emotion EEG Patterns for Depression Detection with Attentive Simple Graph Convolutional Network.

Depression severely limits daily functioning, diminishes quality of life and possibly leads to self-harm and suicide. Noninvasive electroencephalography (EEG) has been shown effective as biomarkers for objective depression diagnose and treatment response prediction, and dry EEG electrodes further extend its availability for clinical use. Even though many efforts have been made to identify depression biomarkers, searching reliable EEG biomarkers for depression detection remains challenging. This work presents a systematic investigation of capabilities of emotion EEG patterns for depression detection using a dry EEG electrode system. We design an emotion elicitation paradigm with happy, neutral and sad emotions and collect EEG signals during film watching from 33 depressed patients and 40 healthy controls. The mean activation levels at frontal and temporal sites in the alpha, beta and gamma bands of the depressed group are different to those of the healthy group, indicating the impacts of depressive symptoms on the emotion experiences. To leverage the topology information among EEG channels for emotion recognition and depression detection, an Attentive Simple Graph Convolutional network is built. The deep depression-health classifier achieves a sensitivity of 81.93% and a specificity of 91.69% on the happy emotions, suggesting the promising use of the emotion neural patterns for distinguishing the depressed patients from the healthy controls.

Objective Depression Detection Using EEG and Eye Movement Signals Induced by Oil Paintings.

Depression is a mental disorder characterized by persistent sadness and loss of interest, which has become one of the leading causes of disability worldwide. There are currently no objective diagnostic standards for depression in clinical practice. Previous studies have shown that depression causes both brain abnormalities and behavioral disorders. In this study, both electroencephalography (EEG) and eye movement signals were used to objectively detect depression. By presenting 40 carefully selected oil paintings-20 positive and 20 negative-as stimuli, we were able to successfully evoke emotions in 48 depressed patients (DPs) and 40 healthy controls (HCs) from three centers. We then used Transformer, a deep learning model, to conduct emotion recognition and depression detection. The experimental results demonstrate that: a) Transformer achieves the best accuracies of 89.21% and 92.19% in emotion recognition and depression detection, respectively; b) The HC group has higher accuracies than the DP group in emotion recognition for both subject-dependent and subject-independent experiments; c) The neural pattern differences do exist between DPs and HCs, and we find the consistent asymmetry of the neural patterns in DPs; d) For depression detection, using single oil painting achieves the best accuracies, and using negative oil paintings has higher accuracies than using positive oil paintings. These findings suggest that EEG and eye movement signals induced by oil paintings can be used to objectively identify depression.

Tagging Continuous Labels for EEG-based Emotion Classification.

EEG-based emotion classification has long been a critical task in the field of affective brain-computer interface (aBCI). The majority of leading researches construct supervised learning models based on labeled datasets. Several datasets have been released, including different kinds of emotions while utilizing various forms of stimulus materials. However, they adopt discrete labeling methods, in which the EEG data collected during the same stimulus material are given a same label. These methods neglect the fact that emotion changes continuously, and mislabeled data possibly exist. The imprecision of discrete labels may hinder the progress of emotion classification in concerned works. Therefore, we develop an efficient system in this paper to support continuous labeling by giving each sample a unique label, and construct a continuously labeled EEG emotion dataset. Using our dataset with continuous labels, we demonstrate the superiority of continuous labeling in emotion classification through experiments on several classification models. We further utilize the continuous labels to identify the EEG features under induced and non-induced emotions in both our dataset and a public dataset. Our experimental results reveal the learnability and generality of the relation between the EEG features and their continuous labels.

Deep Depression Detection with Resting-State and Cognitive-Task EEG.

Depression is a common mental disorder that negatively affects physical health and personal, social and occupational functioning. Currently, accurate and objective diagnosis of depression remains challenging, and electroencephalography (EEG) provides promising clinical practice or home use due to considerable performance and low cost. This work investigates the capabilities of deep neural networks with EEG-based neural patterns from both resting states and cognitive tasks for depression detection. We collect EEG signals from 33 depressed patients and 40 healthy controls using wearable dry electrodes and build Attentive Simple Graph Convolutional network and Transformer neural network for objective depression detection. Four experiment stages, including two resting states and two cognitive tasks, are designed to characterize the alteration of relevant neural patterns in the depressed patients, in terms of decreased energy and impaired performance in sustained attention and response inhibition. The Transformer model achieves an AUC of 0.94 on the Continuous Performance Test-Identical Pairs version (sensitivity: 0.87, specificity: 0.91) and the Stroop Color Word Test (sensitivity: 0.93, specificity: 0.88), and an AUC of 0.89 on the two resting states (sensitivity: 0.85 and 0.87, specificity: 0.88 and 0.90, respectively), indicating the potential of EEG-based neural patterns in identifying depression. These findings provide new insights into the research of depression mechanisms and EEG-based depression biomarkers.

Identifying sex differences in EEG-based emotion recognition using graph convolutional network with attention mechanism.

OBJECTIVE: Sex differences in emotions have been widely perceived via self-reports, peripheral physiological signals and brain imaging techniques. However, how sex differences are reflected in the EEG neural patterns of emotions remains unresolved. In this paper, we detect sex differences in emotional EEG patterns, investigate the consistency of such differences in various emotion datasets across cultures, and study how sex as a factor affects the performance of EEG-based emotion recognition models. APPROACH: We thoroughly assess sex differences in emotional EEG patterns on five public datasets, including SEED, SEED-IV, SEED-V, DEAP and DREAMER, systematically examine the sex-specific EEG patterns for happy, sad, fearful, disgusted and neutral emotions, and implement deep learning models for sex-specific emotion recognition. MAIN RESULTS: (1) Sex differences exist in various emotion types and both Western and Eastern cultures; (2) The emotion patterns of females are more stable than those of males, and the patterns of happiness from females are in sharp contrast with the patterns of sadness, fear and disgust, while the energy levels are more balanced for males; (3) The key features for emotion recognition are mainly located at the frontal and temporal sites for females and distributed more evenly over the whole brain for males, and (4) The same-sex emotion recognition models outperform the corresponding cross-sex models. SIGNIFICANCE: These findings extend efforts to characterize sex differences in emotional brain activation, provide new physiological evidence for sex-specific emotion processing, and reinforce the message that sex differences should be carefully considered in affective research and precision medicine.

Multimodal Emotion Recognition in Response to Oil Paintings.

Most previous affective studies use facial expression pictures, music or movie clips as emotional stimuli, which are either too simplified without contexts or too dynamic for emotion annotations. In this work, we evaluate the effectiveness of oil paintings as stimuli. We develop an emotion stimuli dataset with 114 oil paintings selected from subject ratings to evoke three emotional states (i.e., negative, neutral and positive), and acquire both EEG and eye tracking data from 20 subjects while watching the oil paintings. Furthermore, we propose a novel affective model for multimodal emotion recognition by 1) extracting informative features of EEG signals from both the time domain and the frequency domain, 2) exploring topological information embedded in EEG channels with graph neural networks (GNNs), and 3) combining EEG and eye tracking data with a deep autoencoder neural network. From the exper-iments, our model obtains an averaged classification accuracy of 94.72 % ± 1.47 %, which demonstrates the feasibility of using oil paintings as emotion elicitation material.

Exploring Sex Differences in Key Frequency Bands and Channel Connections for EEG-based Emotion Recognition.

Previous studies have demonstrated the existence of sex differences in emotion recognition by comparing the performance of same-sex and cross-sex training strategies. However, the EEG properties behind the sex differences have not been fully explored. To fill this research gap, we aim to investigate the sex differences in key frequency bands and channel connections of EEG signals. The single-modality attentive simple graph convolutional network (ASGC) is applied to three datasets SEED, SEED-IV and SEED-V under subject-dependence conditions. The classification rates are 90.86 ±4.84%, 83.14 ± 8.84% and 78.33±7.83%, respectively. The adjacency matrices learned by ASGC indicate that females and males have similar channel-connection patterns, but the degree of importance of channel connections varies by sex. Additionally, by comparing the classification results of 5 frequency bands, we find that males and females represent similar frequency band characteristics, i.e., high-frequency bands achieve better performance, indicating that these frequency bands are more related to emotion processing. Finally, we conduct the cross-subject experiment using ASGC and find that the same-sex strategy outperforms the cross-sex strategy, which is consistent with previous studies. The results also imply that males may be more suitable for sex generalization. However, this finding needs the support of more samples and advanced algorithms.

A Cross-modality Deep Learning Method for Measuring Decision Confidence from Eye Movement Signals.

Electroencephalography (EEG) signals can effectively measure the level of human decision confidence. However, it is difficult to acquire EEG signals in practice due to the ex-pensive cost and complex operation, while eye movement signals are much easier to acquire and process. To tackle this problem, we propose a cross-modality deep learning method based on deep canoncial correlation analysis (CDCCA) to transform each modality separately and coordinate different modalities into a hyperspace by using specific canonical correlation analysis constraints. In our proposed method, only eye movement signals are used as inputs in the test phase and the knowledge from EEG signals is learned in the training stage. Experimental results on two human decision confidence datasets demonstrate that our proposed method achieves advanced performance compared with the existing single-modal approaches trained and tested on eye movement signals and maintains a competitive accuracy in comparison with multimodal models.

OGSSL: A Semi-Supervised Classification Model Coupled With Optimal Graph Learning for EEG Emotion Recognition.

Electroencephalogram(EEG) signals are generated from central nervous system which are difficult to disguise, leading to its popularity in emotion recognition. Recently,semi-supervisedlearning exhibits promisingemotion recognition performance by involving unlabeled EEG data into model training. However, if we first build a graph to characterize the sample similarities and then perform label propagation on this graph, these two steps cannotwell collaborate with each other. In this paper, we propose an OptimalGraph coupledSemi-Supervised Learning (OGSSL) model for EEG emotion recognition by unifying the adaptive graph learning and emotion recognition into a single objective. Besides, we improve the label indicator matrix of unlabeledsamples in order to directly obtain theiremotional states. Moreover, the key EEG frequency bands and brain regions in emotion expression are automatically recognized by the projectionmatrix of OGSSL. Experimental results on the SEED-IV data set demonstrate that 1) OGSSL achieves excellent average accuracies of 76.51%, 77.08% and 81.29% in three cross-sessionemotion recognition tasks, 2) OGSSL is competent for discriminative EEG feature selection in emotion recognition, and 3) the Gamma frequency band, the left/righttemporal, prefrontal,and (central) parietal lobes are identified to be more correlated with the occurrence of emotions.

Identifying similarities and differences in emotion recognition with EEG and eye movements among Chinese, German, and French People.

Objective.Cultures have essential influences on emotions. However, most studies on cultural influences on emotions are in the areas of psychology and neuroscience, while the existing affective models are mostly built with data from the same culture. In this paper, we identify the similarities and differences among Chinese, German, and French individuals in emotion recognition with electroencephalogram (EEG) and eye movements from an affective computing perspective.Approach.Three experimental settings were designed: intraculture subject dependent, intraculture subject independent, and cross-culture subject independent. EEG and eye movements are acquired simultaneously from Chinese, German, and French subjects while watching positive, neutral, and negative movie clips. The affective models for Chinese, German, and French subjects are constructed by using machine learning algorithms. A systematic analysis is performed from four aspects: affective model performance, neural patterns, complementary information from different modalities, and cross-cultural emotion recognition.Main results.From emotion recognition accuracies, we find that EEG and eye movements can adapt to Chinese, German, and French cultural diversities and that a cultural in-group advantage phenomenon does exist in emotion recognition with EEG. From the topomaps of EEG, we find that theγandßbands exhibit decreasing activities for Chinese, while for German and French,θandαbands exhibit increasing activities. From confusion matrices and attentional weights, we find that EEG and eye movements have complementary characteristics. From a cross-cultural emotion recognition perspective, we observe that German and French people share more similarities in topographical patterns and attentional weight distributions than Chinese people while the data from Chinese are a good fit for test data but not suitable for training data for the other two cultures.Significance.Our experimental results provide concrete evidence of the in-group advantage phenomenon, cultural influences on emotion recognition, and different neural patterns among Chinese, German, and French individuals.

Investigating EEG-based functional connectivity patterns for multimodal emotion recognition.

Objective.Previous studies on emotion recognition from electroencephalography (EEG) mainly rely on single-channel-based feature extraction methods, which ignore the functional connectivity between brain regions. Hence, in this paper, we propose a novel emotion-relevant critical subnetwork selection algorithm and investigate three EEG functional connectivity network features: strength, clustering coefficient, and eigenvector centrality.Approach.After constructing the brain networks by the correlations between pairs of EEG signals, we calculated critical subnetworks through the average of brain network matrices with the same emotion label to eliminate the weak associations. Then, three network features were conveyed to a multimodal emotion recognition model using deep canonical correlation analysis along with eye movement features. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public datasets: SEED, SEED-V, and DEAP.Main results. The experimental results reveal that the strength feature outperforms the state-of-the-art features based on single-channel analysis. The classification accuracies of multimodal emotion recognition are95.08±6.42%on the SEED dataset,84.51±5.11%on the SEED-V dataset, and85.34±2.90%and86.61±3.76%for arousal and valence on the DEAP dataset, respectively, which all achieved the best performance. In addition, the brain networks constructed with 18 channels achieve comparable performance with that of the 62-channel network and enable easier setups in real scenarios.Significance.The EEG functional connectivity networks combined with emotion-relevant critical subnetworks selection algorithm we proposed is a successful exploration to excavate the information between channels.

Cross-session Emotion Recognition by Joint Label-common and Label-specific EEG Features Exploration.

Since Electroencephalogram (EEG) is resistant to camouflage, it has been a reliable data source for objective emotion recognition. EEG is naturally multi-rhythm and multi-channel, based on which we can extract multiple features for further processing. In EEG-based emotion recognition, it is important to investigate whether there exist some common features shared by different emotional states, and the specific features associated with each emotional state. However, such fundamental problem is ignored by most of the existing studies. To this end, we propose a Joint label-Common and label-Specific Features Exploration (JCSFE) model for semi-supervised cross-session EEG emotion recognition in this paper. To be specific, JCSFE imposes the ℓ2,1-norm on the projection matrix to explore the label-common EEG features and simultaneously the ℓ1-norm is used to explore the label-specific EEG features. Besides, a graph regularization term is introduced to enforce the data local invariance property, i.e., similar EEG samples are encouraged to have the same emotional state. Results obtained from the SEED-IV and SEED-V emotional data sets experimentally demonstrate that JCSFE not only achieves superior emotion recognition performance in comparison with the state-of-the-art models but also provides us with a quantitative method to identify the label-common and label-specific EEG features in emotion recognition.

Multimodal Vigilance Estimation Using Deep Learning.

The phenomenon of increasing accidents caused by reduced vigilance does exist. In the future, the high accuracy of vigilance estimation will play a significant role in public transportation safety. We propose a multimodal regression network that consists of multichannel deep autoencoders with subnetwork neurons (MCDAE sn ). After we define two thresholds of "0.35" and "0.70" from the percentage of eye closure, the output values are in the continuous range of 0-0.35, 0.36-0.70, and 0.71-1 representing the awake state, the tired state, and the drowsy state, respectively. To verify the efficiency of our strategy, we first applied the proposed approach to a single modality. Then, for the multimodality, since the complementary information between forehead electrooculography and electroencephalography features, we found the performance of the proposed approach using features fusion significantly improved, demonstrating the effectiveness and efficiency of our method.

A Novel Experiment Setting for Cross-subject Emotion Recognition.

Recently, cross-subject emotion recognition attracts widespread attention. The current emotional experiments mainly use video clips of different emotions as stimulus materials, but the videos watched by different subjects are the same, which may introduce the same noise pattern in the collected data. However, the traditional experiment settings for cross-subject emotion recognition models couldn't eliminate the impact of same video clips on recognition results, which may lead to a bias on classification. In this paper, we propose a novel experiment setting for cross-subject emotion recognition. We evaluate different experiment settings on four public emotion datasets, DEAP, SEED, SEED-IV and SEED-V. The experimental results demonstrate the deficiencies of the traditional experiment settings and the advantages of our proposed experiment setting.

Sex Difference in Emotion Recognition under Sleep Deprivation: Evidence from EEG and Eye-tracking.

Many psychiatric disorders are accompanied with sleep abnormalities, having significant influence on emotions which might worsen the disorder conditions. Previous studies discovered that the emotion recognition task with objective physiological signals, such as electroencephalography (EEG) and eye movements, provides a reliable way to figure out the complicated relationship between emotion and sleep. However, both of the emotion and EEG signals are affected by sex. This study aims to investigate how sex differences influence emotion recognition under three different sleep conditions. We firstly developed a four-class emotion recognition task based on various sleep conditions to augment the existing dataset. Then we improved the current state-of-the-art deep-learning model with the attention mechanism. It outperforms the best model with higher accuracy about 91.3% and more stabilization. After that, we compared the results of the male and the female group given by this model. The classification accuracy of happy emotion obviously decreases under sleep deprivation for both males and females, which indicates that sleep deprivation impairs the stimulation of happy emotion. Sleep deprivation also notably weakens the discrimination ability of sad emotion for males while females maintain the same as under common sleep. Our study is instructively beneficial to the real application of emotion recognition in disorder diagnosis.

Machine learning-based personalized subthalamic biomarkers predict ON-OFF levodopa states in Parkinson patients.

Objective.Adaptive deep brain stimulation (aDBS) based on subthalamic nucleus (STN) electrophysiology has recently been proposed to improve clinical outcomes of DBS for Parkinson's disease (PD) patients. Many current models for aDBS are based on one or two electrophysiological features of STN activity, such as beta or gamma activity. Although these models have shown interesting results, we hypothesized that an aDBS model that includes many STN activity parameters will yield better clinical results. The objective of this study was to investigate the most appropriate STN neurophysiological biomarkers, detectable over long periods of time, that can predict OFF and ON levodopa states in PD patients.Approach.Long-term local field potentials (LFPs) were recorded from eight STNs (four PD patients) during 92 recording sessions (44 OFF and 48 ON levodopa states), over a period of 3-12 months. Electrophysiological analysis included the power of frequency bands, band power ratio and burst features. A total of 140 engineered features was extracted for 20 040 epochs (each epoch lasting 5 s). Based on these engineered features, machine learning (ML) models classified LFPs as OFF vs ON levodopa states.Main results.Beta and gamma band activity alone poorly predicts OFF vs ON levodopa states, with an accuracy of 0.66 and 0.64, respectively. Group ML analysis slightly improved prediction rates, but personalized ML analysis, based on individualized engineered electrophysiological features, were markedly better, predicting OFF vs ON levodopa states with an accuracy of 0.8 for support vector machine learning models.Significance.We showed that individual patients have unique sets of STN neurophysiological biomarkers that can be detected over long periods of time. ML models revealed that personally classified engineered features most accurately predict OFF vs ON levodopa states. Future development of aDBS for PD patients might include personalized ML algorithms.

Examining Four Experimental Paradigms for EEG-Based Sleep Quality Evaluation with Domain Adaptation.

With the quick development of dry electrode electroencephalography (EEG) acquisition technology, EEG-based sleep quality evaluation attracts more attention for its objective and quantitative merits. However, there hasn't been a standard experimental paradigm. This situation hinders the development of sleep quality evaluation method and technique. In this paper, we experimentally examine the performance of four typical experimental paradigms for EEG-based sleep quality evaluation and develop a new EEG dataset recorded by dry-electrode headset. To eliminate individual variation caused by subjects, we evaluate the four experimental paradigms using domain adaptation (DA) methods. Experimental results demonstrate that a relaxing paradigm is more effective than other attention concentration paradigms and achieves the average accuracy of 76.01%. Domain Adversarial Neural Network outperforms other DA methods and obtains 18.69% improvement on accuracy compared with transfer component analysis.

Data augmentation for enhancing EEG-based emotion recognition with deep generative models.

OBJECTIVE: The data scarcity problem in emotion recognition from electroencephalography (EEG) leads to difficulty in building an affective model with high accuracy using machine learning algorithms or deep neural networks. Inspired by emerging deep generative models, we propose three methods for augmenting EEG training data to enhance the performance of emotion recognition models. APPROACH: Our proposed methods are based on two deep generative models, variational autoencoder (VAE) and generative adversarial network (GAN), and two data augmentation ways, full and partial usage strategies. For the full usage strategy, all of the generated data are augmented to the training dataset without judging the quality of the generated data, while for the partial usage, only high-quality data are selected and appended to the training dataset. These three methods are called conditional Wasserstein GAN (cWGAN), selective VAE (sVAE), and selective WGAN (sWGAN). MAIN RESULTS: To evaluate the effectiveness of these proposed methods, we perform a systematic experimental study on two public EEG datasets for emotion recognition, namely, SEED and DEAP. We first generate realistic-like EEG training data in two forms: power spectral density and differential entropy. Then, we augment the original training datasets with a different number of generated realistic-like EEG data. Finally, we train support vector machines and deep neural networks with shortcut layers to build affective models using the original and augmented training datasets. The experimental results demonstrate that our proposed data augmentation methods based on generative models outperform the existing data augmentation approaches such as conditional VAE, Gaussian noise, and rotational data augmentation. We also observe that the number of generated data should be less than 10 times of the original training dataset to achieve the best performance. SIGNIFICANCE: The augmented training datasets produced by our proposed sWGAN method significantly enhance the performance of EEG-based emotion recognition models.

Multimodal Emotion Recognition from Eye Image, Eye Movement and EEG Using Deep Neural Networks.

In consideration of the complexity of recording electroencephalography(EEG), some researchers are trying to find new features of emotion recognition. In order to investigate the potential of eye tracking glasses for multimodal emotion recognition, we collect and use eye images to classify five emotions along with eye movements and EEG. We compare four combinations of the three different types of data and two kinds of fusion methods, feature level fusion and Bimodal Deep AutoEncoder (BDAE). According to the three-modality fusion features generated by BDAE, the best mean accuracy of 79.63% is achieved. By analyzing the confusion matrices, we find that the three modalities can provide complementary information for recognizing five emotions. Meanwhile, the experimental results indicate that the classifiers with eye image and eye movement fusion features can achieve a comparable classification accuracy of 71.99%.