Continual Learning of a Transformer-Based Deep Learning Classifier Using an Initial Model from Action Observation EEG Data to Online Motor Imagery Classification.

The motor imagery (MI)-based brain computer interface (BCI) is an intuitive interface that enables users to communicate with external environments through their minds. However, current MI-BCI systems ask naïve subjects to perform unfamiliar MI tasks with simple textual instruction or a visual/auditory cue. The unclear instruction for MI execution not only results in large inter-subject variability in the measured EEG patterns but also causes the difficulty of grouping cross-subject data for big-data training. In this study, we designed an BCI training method in a virtual reality (VR) environment. Subjects wore a head-mounted device (HMD) and executed action observation (AO) concurrently with MI (i.e., AO + MI) in VR environments. EEG signals recorded in AO + MI task were used to train an initial model, and the initial model was continually improved by the provision of EEG data in the following BCI training sessions. We recruited five healthy subjects, and each subject was requested to participate in three kinds of tasks, including an AO + MI task, an MI task, and the task of MI with visual feedback (MI-FB) three times. This study adopted a transformer- based spatial-temporal network (TSTN) to decode the user's MI intentions. In contrast to other convolutional neural network (CNN) or recurrent neural network (RNN) approaches, the TSTN extracts spatial and temporal features, and applies attention mechanisms along spatial and temporal dimensions to perceive the global dependencies. The mean detection accuracies of TSTN were 0.63, 0.68, 0.75, and 0.77 in the MI, first MI-FB, second MI-FB, and third MI-FB sessions, respectively. This study demonstrated the AO + MI gave an easier way for subjects to conform their imagery actions, and the BCI performance was improved with the continual learning of the MI-FB training process.

The Full Informational Spectral Analysis for Auditory Steady-State Responses in Human Brain Using the Combination of Canonical Correlation Analysis and Holo-Hilbert Spectral Analysis.

Auditory steady-state response (ASSR) is a translational biomarker for several neurological and psychiatric disorders, such as hearing loss, schizophrenia, bipolar disorder, autism, etc. The ASSR is sinusoidal electroencephalography (EEG)/magnetoencephalography (MEG) responses induced by periodically presented auditory stimuli. Traditional frequency analysis assumes ASSR is a stationary response, which can be analyzed using linear analysis approaches, such as Fourier analysis or Wavelet. However, recent studies have reported that the human steady-state responses are dynamic and can be modulated by the subject's attention, wakefulness state, mental load, and mental fatigue. The amplitude modulations on the measured oscillatory responses can result in the spectral broadening or frequency splitting on the Fourier spectrum, owing to the trigonometric product-to-sum formula. Accordingly, in this study, we analyzed the human ASSR by the combination of canonical correlation analysis (CCA) and Holo-Hilbert spectral analysis (HHSA). The CCA was used to extract ASSR-related signal features, and the HHSA was used to decompose the extracted ASSR responses into amplitude modulation (AM) components and frequency modulation (FM) components, in which the FM frequency represents the fast-changing intra-mode frequency and the AM frequency represents the slow-changing inter-mode frequency. In this paper, we aimed to study the AM and FM spectra of ASSR responses in a 37 Hz steady-state auditory stimulation. Twenty-five healthy subjects were recruited for this study, and each subject was requested to participate in two auditory stimulation sessions, including one right-ear and one left-ear monaural steady-state auditory stimulation. With the HHSA, both the 37 Hz (fundamental frequency) and the 74 Hz (first harmonic frequency) auditory responses were successfully extracted. Examining the AM spectra, the 37 Hz and the 74 Hz auditory responses were modulated by distinct AM spectra, each with at least three composite frequencies. In contrast to the results of traditional Fourier spectra, frequency splitting was seen at 37 Hz, and a spectral peak was obscured at 74 Hz in Fourier spectra. The proposed method effectively corrects the frequency splitting problem resulting from time-varying amplitude changes. Our results have validated the HHSA as a useful tool for steady-state response (SSR) studies so that the misleading or wrong interpretation caused by amplitude modulation in the traditional Fourier spectrum can be avoided.

Phase-Approaching Stimulation Sequence for SSVEP-Based BCI: A Practical Use in VR/AR HMD.

Steady-state visual evoked potential (SSVEP) has been used to implement brain-computer interface (BCI) due to its advantages of high information transfer rate (ITR) and high accuracy. In recent years, owing to the developments of head-mounted device (HMD), the HMD has become a popular device to implement SSVEP-based BCI. However, an HMD with fixed frame rate only can flash at its subharmonic frequencies which limits the available number of stimulation frequencies for SSVEP-based BCI. In order to increase the number of available commands for SSVEP-based BCI, we proposed a phase-approaching (PA) method to generate visual stimulation sequences at user-specified frequency on an HMD. The flickering sequence generated by our PA method (PAS sequence) tries to approximate user-specified stimulation frequency by means of minimizing the difference of accumulated phases between our PAS sequence and the ideal wave of user-specified frequency. The generated sequence of PA method determines the brightness state for each frame to approach the accumulated phase of the ideal wave. The SSVEPs evoked from stimulators, driven by PAS sequences, were analyzed using canonical correlation analysis (CCA) to identify user's gazed target. In this study, a six-command SSVEP-based BCI was designed to operate a flying drone. The ITR and detection accuracy are 36.84 bits/min and 93.30%, respectively.

Analyses of EEG Oscillatory Activities during Slow and Fast Repetitive Movements using Holo-Hilbert Spectral Analysis.

Neural oscillatory activities existing in multiple fre-quency bands usually represent different levels of neurophysiolog-ical meanings, from micro-scale to macro-scale organizations. In this study, we adopted Holo-Hilbert spectral analysis (HHSA) to study the amplitude-modulated (AM) and frequency-modulated (FM) components in sensorimotor Mu rhythm, induced by slow- and fast-rate repetitive movements. The HHSA-based approach is a two-layer empirical mode decomposition (EMD) architecture, which firstly decomposes the EEG signal into a series of frequency-modulated intrinsic mode functions (IMF) and then decomposes each frequency-modulated IMF into a set of amplitude-modulated IMFs. With the HHSA, the FM and AM components were incor-porated with their instantaneous power to achieve full-informa-tional spectral analysis. We observed that the instantaneous power induced by slow-rate movements was significantly higher than that induced by fast-rate movements (p < 0.01, Wilcoxon signed rank test). The alpha-band AM frequencies induced by slow-rate movements were higher than those induced by fast-rate move-ments, while no statistical difference was found in beta-band AM frequencies. In addition, to study the functional coupling between the primary sensorimotor area and other brain regions, spectral coherence was applied and statistical difference was found in frontal area in slow-rate versus fast-rate movements. The discrep-ancy between slow- and fast-rate movements might be owing to the change of motor functional modes from default mode network (DMN) to automatic timing with the increase of movement rates. The use of HHSA for oscillatory activity analysis can be an effi-cient tool to provide informative interaction among different fre-quency bands.

Study of Repetitive Movements Induced Oscillatory Activities in Healthy Subjects and Chronic Stroke Patients.

Repetitive movements at a constant rate require the integration of internal time counting and motor neural networks. Previous studies have proved that humans can follow short durations automatically (automatic timing) but require more cognitive efforts to track or estimate long durations. In this study, we studied sensorimotor oscillatory activities in healthy subjects and chronic stroke patients when subjects were performing repetitive finger movements. We found the movement-modulated changes in alpha and beta oscillatory activities were decreased with the increase of movement rates in finger lifting of healthy subjects and the non-paretic hands in stroke patients, whereas no difference was found in the paretic-hand movements at different movement rates in stroke patients. The significant difference in oscillatory activities between movements of non-paretic hands and paretic hands could imply the requirement of higher cognitive efforts to perform fast repetitive movements in paretic hands. The sensorimotor oscillatory response in fast repetitive movements could be a possible indicator to probe the recovery of motor function in stroke patients.

Evaluate the Feasibility of Using Frontal SSVEP to Implement an SSVEP-Based BCI in Young, Elderly and ALS Groups.

This paper studies the amplitude-frequency characteristic of frontal steady-state visual evoked potential (SSVEP) and its feasibility as a control signal for brain computer interface (BCI). SSVEPs induced by different stimulation frequencies, from 13 ~ 31 Hz in 2 Hz steps, were measured in eight young subjects, eight elders and seven ALS patients. Each subject was requested to participate in a calibration study and an application study. The calibration study was designed to find the amplitude-frequency characteristics of SSVEPs recorded from Oz and Fpz positions, while the application study was designed to test the feasibility of using frontal SSVEP to control a two-command SSVEP-based BCI. The SSVEP amplitude was detected by an epoch-average process which enables artifact-contaminated epochs can be removed. The seven ALS patients were severely impaired, and four patients, who were incapable of completing our BCI task, were excluded from calculation of BCI performance. The averaged accuracies, command transfer intervals and information transfer rates in operating frontal SSVEP-based BCI were 96.1%, 3.43 s/command, and 14.42 bits/min in young subjects; 91.8%, 6.22 s/command, and 6.16 bits/min in elders; 81.2%, 12.14 s/command, and 1.51 bits/min in ALS patients, respectively. The frontal SSVEP could be an alternative choice to design SSVEP-based BCI.