A Bimodal Deep Learning Architecture for EEG-fNIRS Decoding of Overt and Imagined Speech.
IEEE Trans Biomed Eng
; 69(6): 1983-1994, 2022 06.
Article
in English
| MEDLINE | ID: covidwho-1997179
ABSTRACT
OBJECTIVE:
Brain-computer interfaces (BCI) studies are increasingly leveraging different attributes of multiple signal modalities simultaneously. Bimodal data acquisition protocols combining the temporal resolution of electroencephalography (EEG) with the spatial resolution of functional near-infrared spectroscopy (fNIRS) require novel approaches to decoding.METHODS:
We present an EEG-fNIRS Hybrid BCI that employs a new bimodal deep neural network architecture consisting of two convolutional sub-networks (subnets) to decode overt and imagined speech. Features from each subnet are fused before further feature extraction and classification. Nineteen participants performed overt and imagined speech in a novel cue-based paradigm enabling investigation of stimulus and linguistic effects on decoding.RESULTS:
Using the hybrid approach, classification accuracies (46.31% and 34.29% for overt and imagined speech, respectively (chance 25%)) indicated a significant improvement on EEG used independently for imagined speech (p = 0.020) while tending towards significance for overt speech (p = 0.098). In comparison with fNIRS, significant improvements for both speech-types were achieved with bimodal decoding (p<0.001). There was a mean difference of â¼12.02% between overt and imagined speech with accuracies as high as 87.18% and 53%. Deeper subnets enhanced performance while stimulus effected overt and imagined speech in significantly different ways.CONCLUSION:
The bimodal approach was a significant improvement on unimodal results for several tasks. Results indicate the potential of multi-modal deep learning for enhancing neural signal decoding.SIGNIFICANCE:
This novel architecture can be used to enhance speech decoding from bimodal neural signals.
Full text:
Available
Collection:
International databases
Database:
MEDLINE
Main subject:
Brain-Computer Interfaces
/
Deep Learning
Limits:
Humans
Language:
English
Journal:
IEEE Trans Biomed Eng
Year:
2022
Document Type:
Article
Similar
MEDLINE
...
LILACS
LIS