An autonomous hybrid brain-computer interface system combined with eye-tracking in virtual environment.

BACKGROUND: Brain-computer interface (BCI) has become an effective human-machine interactive way. However, the performance of the traditional BCI system needs to be further improved, such as flexibility, robustness, and accuracy. We aim to develop an autonomous hybrid BCI system combined with eye-tracking for the control tasks in the virtual environment. NEW METHOD: This work developed an autonomous control strategy and proposed an effective fusion method for electroencephalogram (EEG) and eye tracking. For the autonomous control, the sliding window method was adopted to analyze the user's eye-gaze data. When the variance of eye-gaze data was less than the threshold, target recognition was triggered. EEG and eye-gaze data were synchronously collected and fused for classification. In addition, a fusion method based on particle swarm optimization (PSO) was proposed, which can find the best fusion weights to adapt to the differences of single modalities. RESULTS: EEG data and eye-gaze data of 15 subjects in steady-state visual evoked potentials (SSVEP) tasks were collected to evaluate the effectiveness of the hybrid BCI system. The results showed that the PSO fusion method performed best in all fusion methods. And the proposed hybrid BCI system obtained higher accuracy and information transfer rate (ITR) than the single-modality. COMPARISON WITH EXISTING METHODS: The PSO fusion method was compared with average weighting fusion, prior weighting fusion, support vector machine, decision tree, random forest, and extreme random tree. CONCLUSION: The proposed methods of autonomous control and dual-modal fusion can improve the flexibility, robustness and classification performance of the hybrid BCI system.

A deep learning method for single-trial EEG classification in RSVP task based on spatiotemporal features of ERPs.

Objective. Single-trial electroencephalography (EEG) classification is of great importance in the rapid serial visual presentation (RSVP) task. Convolutional neural networks (CNNs), as one of the mainstream deep learning methods, have been proven to be effective in extracting RSVP EEG features. However, most existing CNN models for EEG classification do not consider the phase-locked characteristic of event-related potential (ERP) components very well in the architecture design. Here, we propose a novel CNN model to make better use of the phase-locked characteristic to extract spatiotemporal features for single-trial RSVP EEG classification. Based on the phase-locked characteristic, the spatial distributions of the main ERP component in different periods can be learned separately.Approach.In this work, we propose a novel CNN model to achieve superior performance on single-trial RSVP EEG classification. We introduce the combination of the standard convolutional layer, the permute layer and the depthwise convolutional layer to separately operate the spatial convolution in different periods, which more fully utilizes the phase-locked characteristic of ERPs for classification. We compare our model with several traditional and deep-learning methods in the classification performance. Moreover, we use spatial topography and saliency map to visually analyze the ERP features extracted by our model.Main results. The results show that our model obtains better classification performance than those of reference methods. The spatial topographies of each subject exhibit the typical ERP spatial distribution in different time periods. And the saliency map of each subject illustrates the discriminant electrodes and the meaningful temporal features.Significance. Our model is designed with better consideration of the phase-locked ERP characteristic and reaches excellent performance on single-trial RSVP EEG classification.