Context-aware Multimodal Auditory BCI Classification through Graph Neural Networks.

The prospect of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) in the presence of topological information of participants is often left unexplored in most of the brain-computer interface (BCI) systems. Additionally, the usage of these modalities together in the field of multimodality analysis to support multiple brain signals toward improving BCI performance is not fully examined. This study first presents a multimodal data fusion framework to exploit and decode the complementary synergistic properties in multimodal neural signals. Moreover, the relations among different subjects and their observations also play critical roles in classifying unknown subjects. We developed a context-aware graph neural network (GNN) model utilizing the pairwise relationship among participants to investigate the performance on an auditory task classification. We explored standard and deviant auditory EEG and fNIRS data where each subject was asked to perform an auditory oddball task and has multiple trials regarded as context-aware nodes in our graph construction. In experiments, our multimodal data fusion strategy showed an improvement up to 8.40% via SVM and 2.02% via GNN, compared to the single-modal EEG or fNIRS. In addition, our context-aware GNN achieved 5.3%, 4.07% and 4.53% higher accuracy for EEG, fNIRS and multimodal data based experiments, compared to the baseline models.

A dynamical graph-based feature extraction approach to enhance mental task classification in brain-computer interfaces.

Graph theoretic approaches in analyzing spatiotemporal dynamics of brain activities are under-studied but could be very promising directions in developing effective brain-computer interfaces (BCIs). Many existing BCI systems use electroencephalogram (EEG) signals to record and decode human neural activities noninvasively. Often, however, the features extracted from the EEG signals ignore the topological information hidden in the EEG temporal dynamics. Moreover, existing graph theoretic approaches are mostly used to reveal the topological patterns of brain functional networks based on synchronization between signals from distinctive spatial regions, instead of interdependence between states at different timestamps. In this study, we present a robust fold-wise hyperparameter optimization framework utilizing a series of conventional graph-based measurements combined with spectral graph features and investigate its discriminative performance on classification of a designed mental task in 6 participants with amyotrophic lateral sclerosis (ALS). Across all of our participants, we reached an average accuracy of 71.1%±4.5% for mental task classification by combining the global graph-based measurements and the spectral graph features, higher than the conventional non-graph based feature performance (67.1%±7.5%). Compared to using either one of the graphic features (66.3%±6.5% for the eigenvalues and 65.9%±5.2% for the global graph features), our feature combination strategy shows considerable improvement in both accuracy and robustness performance. Our results indicate the feasibility and advantage of the presented fold-wise optimization framework utilizing graph-based features in BCI systems targeted at end-users.

A Graph-Based Feature Extraction Algorithm Towards a Robust Data Fusion Framework for Brain-Computer Interfaces.

OBJECTIVE: The topological information hidden in the EEG spectral dynamics is often ignored in the majority of the existing brain-computer interface (BCI) systems. Moreover, a systematic multimodal fusion of EEG with other informative brain signals such as functional near-infrared spectroscopy (fNIRS) towards enhancing the performance of the BCI systems is not fully investigated. In this study, we present a robust EEG-fNIRS data fusion framework utilizing a series of graph-based EEG features to investigate their performance on a motor imaginary (MI) classification task. METHOD: We first extract the amplitude and phase sequences of users' multi-channel EEG signals based on the complex Morlet wavelet time-frequency maps, and then convert them into an undirected graph to extract EEG topological features. The graph-based features from EEG are then selected by a thresholding method and fused with the temporal features from fNIRS signals after each being selected by the least absolute shrinkage and selection operator (LASSO) algorithm. The fused features were then classified as MI task vs. baseline by a linear support vector machine (SVM) classifier. RESULTS: The time-frequency graphs of EEG signals improved the MI classification accuracy by ∼5% compared to the graphs built on the band-pass filtered temporal EEG signals. Our proposed graph-based method also showed comparable performance to the classical EEG features based on power spectral density (PSD), however with a much smaller standard deviation, showing its robustness for potential use in a practical BCI system. Our fusion analysis revealed a considerable improvement of ∼17% as opposed to the highest average accuracy of EEG only and ∼3% compared with the highest fNIRS only accuracy demonstrating an enhanced performance when modality fusion is used relative to single modal outcomes. SIGNIFICANCE: Our findings indicate the potential use of the proposed data fusion framework utilizing the graph-based features in the hybrid BCI systems by making the motor imaginary inference more accurate and more robust.

Electrical and Hemodynamic Neural Functions in People With ALS: An EEG-fNIRS Resting-State Study.

OBJECTIVE: Amyotrophic lateral sclerosis (ALS) is a complex neurodegenerative disease that causes the progressive loss of voluntary muscle control. Recent studies have reported conflicting results on alterations in resting-state functional brain networks in ALS by adopting unimodal techniques that measure either electrophysiological or vascular-hemodynamic neural functions. However, no study to date has explored simultaneous electrical and vascular-hemodynamic changes in the resting-state brain in ALS. Using complementary multimodal electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) recording and analysis techniques, we explored the underlying multidimensional neural contributions to altered oscillations and functional connectivity in people with ALS. METHODS: 10 ALS patients and 9 age-matched controls underwent multimodal EEG-fNIRS recording in the resting state. Resting-state functional connectivity (RSFC) and power spectra of both modalities in both groups were analyzed and compared statistically. RESULTS: Increased fronto-parietal EEG connectivity in the alpha and beta bands and increased interhemispheric and right intra-hemispheric fNIRS connectivity in the frontal and prefrontal regions were observed in ALS. Frontal, central, and temporal theta and alpha EEG power decreased in ALS, as did parietal and occipital alpha EEG power, while frontal and parietal hemodynamic spectral power increased in ALS. SIGNIFICANCE: These results suggest that electro-vascular disruption in neuronal networks extends to the extra-motor regions in ALS patients, which can ultimately introduce novel neural markers of ALS that can be exploited further as diagnostic and prognostic tools.

Enhancing Communication for People in Late-Stage ALS Using an fNIRS-Based BCI System.

OBJECTIVE: Brain-computer interface (BCI) based communication remains a challenge for people with later-stage amyotrophic lateral sclerosis (ALS) who lose all voluntary muscle control. Although recent studies have demonstrated the feasibility of functional near-infrared spectroscopy (fNIRS) to successfully control BCIs primarily for healthy cohorts, these systems are yet inefficient for people with severe motor disabilities like ALS. In this study, we developed a new fNIRS-based BCI system in concert with a single-trial Visuo-Mental (VM) paradigm to investigate the feasibility of enhanced communication for ALS patients, particularly those in the later stages of the disease. METHODS: In the first part of the study, we recorded data from six ALS patients using our proposed protocol (fNIRS-VM) and compared the results with the conventional electroencephalography (EEG)-based multi-trial P3Speller (P3S). In the second part, we recorded longitudinal data from one patient in the late locked-in state (LIS) who had fully lost eye-gaze control. Using statistical parametric mapping (SPM) and correlation analysis, the optimal channels and hemodynamic features were selected and used in linear discriminant analysis (LDA). RESULTS: Over all the subjects, we obtained an average accuracy of 81.3%±5.7% within comparatively short times (< 4 sec) in the fNIRS-VM protocol relative to an average accuracy of 74.0%±8.9% in the P3S, though not competitive in patients with no substantial visual problems. Our longitudinal analysis showed substantially superior accuracy using the proposed fNIRS-VM protocol (73.2%±2.0%) over the P3S (61.8%±1.5%). SIGNIFICANCE: Our findings indicate the potential efficacy of our proposed system for communication and control for late-stage ALS patients.