Emotion Recognition from Spatio-Temporal Representation of EEG Signals via 3D-CNN with Ensemble Learning Techniques.

The recognition of emotions is one of the most challenging issues in human-computer interaction (HCI). EEG signals are widely adopted as a method for recognizing emotions because of their ease of acquisition, mobility, and convenience. Deep neural networks (DNN) have provided excellent results in emotion recognition studies. Most studies, however, use other methods to extract handcrafted features, such as Pearson correlation coefficient (PCC), Principal Component Analysis, Higuchi Fractal Dimension (HFD), etc., even though DNN is capable of generating meaningful features. Furthermore, most earlier studies largely ignored spatial information between the different channels, focusing mainly on time domain and frequency domain representations. This study utilizes a pre-trained 3D-CNN MobileNet model with transfer learning on the spatio-temporal representation of EEG signals to extract features for emotion recognition. In addition to fully connected layers, hybrid models were explored using other decision layers such as multilayer perceptron (MLP), k-nearest neighbor (KNN), extreme learning machine (ELM), XGBoost (XGB), random forest (RF), and support vector machine (SVM). Additionally, this study investigates the effects of post-processing or filtering output labels. Extensive experiments were conducted on the SJTU Emotion EEG Dataset (SEED) (three classes) and SEED-IV (four classes) datasets, and the results obtained were comparable to the state-of-the-art. Based on the conventional 3D-CNN with ELM classifier, SEED and SEED-IV datasets showed a maximum accuracy of 89.18% and 81.60%, respectively. Post-filtering improved the emotional classification performance in the hybrid 3D-CNN with ELM model for SEED and SEED-IV datasets to 90.85% and 83.71%, respectively. Accordingly, spatial-temporal features extracted from the EEG, along with ensemble classifiers, were found to be the most effective in recognizing emotions compared to state-of-the-art methods.

Comprehensive Analysis of Feature Extraction Methods for Emotion Recognition from Multichannel EEG Recordings.

Advances in signal processing and machine learning have expedited electroencephalogram (EEG)-based emotion recognition research, and numerous EEG signal features have been investigated to detect or characterize human emotions. However, most studies in this area have used relatively small monocentric data and focused on a limited range of EEG features, making it difficult to compare the utility of different sets of EEG features for emotion recognition. This study addressed that by comparing the classification accuracy (performance) of a comprehensive range of EEG feature sets for identifying emotional states, in terms of valence and arousal. The classification accuracy of five EEG feature sets were investigated, including statistical features, fractal dimension (FD), Hjorth parameters, higher order spectra (HOS), and those derived using wavelet analysis. Performance was evaluated using two classifier methods, support vector machine (SVM) and classification and regression tree (CART), across five independent and publicly available datasets linking EEG to emotional states: MAHNOB-HCI, DEAP, SEED, AMIGOS, and DREAMER. The FD-CART feature-classification method attained the best mean classification accuracy for valence (85.06%) and arousal (84.55%) across the five datasets. The stability of these findings across the five different datasets also indicate that FD features derived from EEG data are reliable for emotion recognition. The results may lead to the possible development of an online feature extraction framework, thereby enabling the development of an EEG-based emotion recognition system in real time.

Improving automated diagnosis of epilepsy from EEGs beyond IEDs.

Objective.Clinical diagnosis of epilepsy relies partially on identifying interictal epileptiform discharges (IEDs) in scalp electroencephalograms (EEGs). This process is expert-biased, tedious, and can delay the diagnosis procedure. Beyond automatically detecting IEDs, there are far fewer studies on automated methods to differentiate epileptic EEGs (potentially without IEDs) from normal EEGs. In addition, the diagnosis of epilepsy based on a single EEG tends to be low. Consequently, there is a strong need for automated systems for EEG interpretation. Traditionally, epilepsy diagnosis relies heavily on IEDs. However, since not all epileptic EEGs exhibit IEDs, it is essential to explore IED-independent EEG measures for epilepsy diagnosis. The main objective is to develop an automated system for detecting epileptic EEGs, both with or without IEDs. In order to detect epileptic EEGs without IEDs, it is crucial to include EEG features in the algorithm that are not directly related to IEDs.Approach.In this study, we explore the background characteristics of interictal EEG for automated and more reliable diagnosis of epilepsy. Specifically, we investigate features based on univariate temporal measures (UTMs), spectral, wavelet, Stockwell, connectivity, and graph metrics of EEGs, besides patient-related information (age and vigilance state). The evaluation is performed on a sizeable cohort of routine scalp EEGs (685 epileptic EEGs and 1229 normal EEGs) from five centers across Singapore, USA, and India.Main results.In comparison with the current literature, we obtained an improved Leave-One-Subject-Out (LOSO) cross-validation (CV) area under the curve (AUC) of 0.871 (Balanced Accuracy (BAC) of 80.9%) with a combination of three features (IED rate, and Daubechies and Morlet wavelets) for the classification of EEGs with IEDs vs. normal EEGs. The IED-independent feature UTM achieved a LOSO CV AUC of 0.809 (BAC of 74.4%). The inclusion of IED-independent features also helps to improve the EEG-level classification of epileptic EEGs with and without IEDs vs. normal EEGs, achieving an AUC of 0.822 (BAC of 77.6%) compared to 0.688 (BAC of 59.6%) for classification only based on the IED rate. Specifically, the addition of IED-independent features improved the BAC by 21% in detecting epileptic EEGs that do not contain IEDs.Significance.These results pave the way towards automated detection of epilepsy. We are one of the first to analyze epileptic EEGs without IEDs, thereby opening up an underexplored option in epilepsy diagnosis.

EEG-based emotion charting for Parkinson's disease patients using Convolutional Recurrent Neural Networks and cross dataset learning.

Electroencephalogram (EEG) based emotion classification reflects the actual and intrinsic emotional state, resulting in more reliable, natural, and meaningful human-computer interaction with applications in entertainment consumption behavior, interactive brain-computer interface, and monitoring of psychological health of patients in the domain of e-healthcare. Challenges of EEG-based emotion recognition in real-world applications are variations among experimental settings and cognitive health conditions. Parkinson's Disease (PD) is the second most common neurodegenerative disorder, resulting in impaired recognition and expression of emotions. The deficit of emotional expression poses challenges for the healthcare services provided to PD patients. This study proposes 1D-CRNN-ELM architecture, which combines one-dimensional Convolutional Recurrent Neural Network (1D-CRNN) with an Extreme Learning Machine (ELM), robust for the emotion detection of PD patients, also available for cross dataset learning with various emotions and experimental settings. In the proposed framework, after EEG preprocessing, the trained CRNN can use as a feature extractor with ELM as the classifier, and again this trained CRNN can be used for learning of new emotions set with fine-tuning of other datasets. This paper also applied cross dataset learning of emotions by training with PD patients datasets and fine-tuning with publicly available datasets of AMIGOS and SEED-IV, and vice versa. Random splitting of train and test data with 80 - 20 ratio resulted in an accuracy of 97.75% for AMIGOS, 83.20% for PD, and 86.00% for HC with six basic emotion classes. Fine-tuning of trained architecture with four emotions of the SEED-IV dataset results in 92.5% accuracy. To validate the generalization of our results, leave one subject (patient) out cross-validation is also incorporated with mean accuracies of 95.84% for AMIGOS, 75.09% for PD, 77.85% for HC, and 84.97% for SEED-IV is achieved. Only a 1 - sec segment of EEG signal from 14 channels is enough to detect emotions with this performance. The proposed method outperforms state-of-the-art studies to classify EEG-based emotions with publicly available datasets, provide cross dataset learning, and validate the robustness of the deep learning framework for real-world application of psychological healthcare monitoring of Parkinson's disease patients.

Time-Frequency Decomposition of Scalp Electroencephalograms Improves Deep Learning-Based Epilepsy Diagnosis.

Epilepsy diagnosis based on Interictal Epileptiform Discharges (IEDs) in scalp electroencephalograms (EEGs) is laborious and often subjective. Therefore, it is necessary to build an effective IED detector and an automatic method to classify IED-free versus IED EEGs. In this study, we evaluate features that may provide reliable IED detection and EEG classification. Specifically, we investigate the IED detector based on convolutional neural network (ConvNet) with different input features (temporal, spectral, and wavelet features). We explore different ConvNet architectures and types, including 1D (one-dimensional) ConvNet, 2D (two-dimensional) ConvNet, and noise injection at various layers. We evaluate the EEG classification performance on five independent datasets. The 1D ConvNet with preprocessed full-frequency EEG signal and frequency bands (delta, theta, alpha, beta) with Gaussian additive noise at the output layer achieved the best IED detection results with a false detection rate of 0.23/min at 90% sensitivity. The EEG classification system obtained a mean EEG classification Leave-One-Institution-Out (LOIO) cross-validation (CV) balanced accuracy (BAC) of 78.1% (area under the curve (AUC) of 0.839) and Leave-One-Subject-Out (LOSO) CV BAC of 79.5% (AUC of 0.856). Since the proposed classification system only takes a few seconds to analyze a 30-min routine EEG, it may help in reducing the human effort required for epilepsy diagnosis.

Automated EEG pathology detection based on different convolutional neural network models: Deep learning approach.

The brain electrical activity, recorded and materialized as electroencephalogram (EEG) signals, is known to be very useful in the diagnosis of brain-related pathology. However, manual examination of these EEG signals has various limitations, including time-consuming inspections, the need for highly trained neurologists, and the subjectiveness of the evaluation. Thus, an automated EEG pathology detection system would be helpful to assist neurologists to enhance the treatment procedure by making a quicker diagnosis and reducing error due to the human element. This work proposes the application of a time-frequency spectrum to convert the EEG signals onto the image domain. The spectrum images are then applied to the Convolutional Neural Network (CNN) to learn robust features that can aid the automatic detection of pathology and normal EEG signals. Three popular CNN in the form of the DenseNet, Inception-ResNet v2, and SeizureNet were employed. The extracted deep-learned features from the spectrum images are then passed onto the support vector machine (SVM) classifier. The effectiveness of the proposed approach was assessed using the publicly available Temple University Hospital (TUH) abnormal EEG corpus dataset, which is demographically balanced. The proposed SeizureNet-SVM-based system achieved state-of-the-art performance: accuracy, sensitivity, and specificity of 96.65%, 90.48%, and 100%, respectively. The results show that the proposed framework may serve as a diagnostic tool to assist clinicians in the detection of EEG pathology for early treatment.

Automated Adult Epilepsy Diagnostic Tool Based on Interictal Scalp Electroencephalogram Characteristics: A Six-Center Study.

The diagnosis of epilepsy often relies on a reading of routine scalp electroencephalograms (EEGs). Since seizures are highly unlikely to be detected in a routine scalp EEG, the primary diagnosis depends heavily on the visual evaluation of Interictal Epileptiform Discharges (IEDs). This process is tedious, expert-centered, and delays the treatment plan. Consequently, the development of an automated, fast, and reliable epileptic EEG diagnostic system is essential. In this study, we propose a system to classify EEG as epileptic or normal based on multiple modalities extracted from the interictal EEG. The ensemble system consists of three components: a Convolutional Neural Network (CNN)-based IED detector, a Template Matching (TM)-based IED detector, and a spectral feature-based classifier. We evaluate the system on datasets from six centers from the USA, Singapore, and India. The system yields a mean Leave-One-Institution-Out (LOIO) cross-validation (CV) area under curve (AUC) of 0.826 (balanced accuracy (BAC) of 76.1%) and Leave-One-Subject-Out (LOSO) CV AUC of 0.812 (BAC of 74.8%). The LOIO results are found to be similar to the interrater agreement (IRA) reported in the literature for epileptic EEG classification. Moreover, as the proposed system can process routine EEGs in a few seconds, it may aid the clinicians in diagnosing epilepsy efficiently.

Automated Detection of Interictal Epileptiform Discharges from Scalp Electroencephalograms by Convolutional Neural Networks.

Visual evaluation of electroencephalogram (EEG) for Interictal Epileptiform Discharges (IEDs) as distinctive biomarkers of epilepsy has various limitations, including time-consuming reviews, steep learning curves, interobserver variability, and the need for specialized experts. The development of an automated IED detector is necessary to provide a faster and reliable diagnosis of epilepsy. In this paper, we propose an automated IED detector based on Convolutional Neural Networks (CNNs). We have evaluated the proposed IED detector on a sizable database of 554 scalp EEG recordings (84 epileptic patients and 461 nonepileptic subjects) recorded at Massachusetts General Hospital (MGH), Boston. The proposed CNN IED detector has achieved superior performance in comparison with conventional methods with a mean cross-validation area under the precision-recall curve (AUPRC) of 0.838[Formula: see text]±[Formula: see text]0.040 and false detection rate of 0.2[Formula: see text]±[Formula: see text]0.11 per minute for a sensitivity of 80%. We demonstrated the proposed system to be noninferior to 30 neurologists on a dataset from the Medical University of South Carolina (MUSC). Further, we clinically validated the system at National University Hospital (NUH), Singapore, with an agreement accuracy of 81.41% with a clinical expert. Moreover, the proposed system can be applied to EEG recordings with any arbitrary number of channels.

On the analysis of EEG power, frequency and asymmetry in Parkinson's disease during emotion processing.

OBJECTIVE: While Parkinson's disease (PD) has traditionally been described as a movement disorder, there is growing evidence of disruption in emotion information processing associated with the disease. The aim of this study was to investigate whether there are specific electroencephalographic (EEG) characteristics that discriminate PD patients and normal controls during emotion information processing. METHOD: EEG recordings from 14 scalp sites were collected from 20 PD patients and 30 age-matched normal controls. Multimodal (audio-visual) stimuli were presented to evoke specific targeted emotional states such as happiness, sadness, fear, anger, surprise and disgust. Absolute and relative power, frequency and asymmetry measures derived from spectrally analyzed EEGs were subjected to repeated ANOVA measures for group comparisons as well as to discriminate function analysis to examine their utility as classification indices. In addition, subjective ratings were obtained for the used emotional stimuli. RESULTS: Behaviorally, PD patients showed no impairments in emotion recognition as measured by subjective ratings. Compared with normal controls, PD patients evidenced smaller overall relative delta, theta, alpha and beta power, and at bilateral anterior regions smaller absolute theta, alpha, and beta power and higher mean total spectrum frequency across different emotional states. Inter-hemispheric theta, alpha, and beta power asymmetry index differences were noted, with controls exhibiting greater right than left hemisphere activation. Whereas intra-hemispheric alpha power asymmetry reduction was exhibited in patients bilaterally at all regions. Discriminant analysis correctly classified 95.0% of the patients and controls during emotional stimuli. CONCLUSION: These distributed spectral powers in different frequency bands might provide meaningful information about emotional processing in PD patients.

Review of emotion recognition in stroke patients.

OBJECTIVE: Patients suffering from stroke have a diminished ability to recognize emotions. This paper presents a review of neuropsychological studies that investigated the basic emotion processing deficits involved in individuals with interhemispheric brain (right, left) damage and normal controls, including processing mode (perception) and communication channels (facial, prosodic-intonational, lexical-verbal). METHODS: An electronic search was conducted using specific keywords for studies investigating emotion recognition in brain damage patients. The PubMed database was searched until March 2012 as well as citations and reference lists. 92 potential articles were identified. RESULTS: The findings showed that deficits in emotion perception were more frequently observed in individuals with right brain damage than those with left brain damage when processing facial, prosodic and lexical emotional stimuli. CONCLUSION: These findings suggest that the right hemisphere has a unique contribution in emotional processing and provide support for the right hemisphere emotion hypothesis. SIGNIFICANCE: This robust deficit in emotion recognition has clinical significance. The extent of emotion recognition deficit in brain damage patients appears to be correlated with a variety of interpersonal difficulties such as complaints of frustration in social relations, feelings of social discomfort, desire to connect with others, feelings of social disconnection and use of controlling behaviors.