Imagined speech can be decoded from low- and cross-frequency intracranial EEG features.

Reconstructing intended speech from neural activity using brain-computer interfaces holds great promises for people with severe speech production deficits. While decoding overt speech has progressed, decoding imagined speech has met limited success, mainly because the associated neural signals are weak and variable compared to overt speech, hence difficult to decode by learning algorithms. We obtained three electrocorticography datasets from 13 patients, with electrodes implanted for epilepsy evaluation, who performed overt and imagined speech production tasks. Based on recent theories of speech neural processing, we extracted consistent and specific neural features usable for future brain computer interfaces, and assessed their performance to discriminate speech items in articulatory, phonetic, and vocalic representation spaces. While high-frequency activity provided the best signal for overt speech, both low- and higher-frequency power and local cross-frequency contributed to imagined speech decoding, in particular in phonetic and vocalic, i.e. perceptual, spaces. These findings show that low-frequency power and cross-frequency dynamics contain key information for imagined speech decoding.

Using Coherence-based spectro-spatial filters for stimulus features prediction from electro-corticographic recordings.

The traditional approach in neuroscience relies on encoding models where brain responses are related to different stimuli in order to establish dependencies. In decoding tasks, on the contrary, brain responses are used to predict the stimuli, and traditionally, the signals are assumed stationary within trials, which is rarely the case for natural stimuli. We hypothesize that a decoding model assuming each experimental trial as a realization of a random process more likely reflects the statistical properties of the undergoing process compared to the assumption of stationarity. Here, we propose a Coherence-based spectro-spatial filter that allows for reconstructing stimulus features from brain signal's features. The proposed method extracts common patterns between features of the brain signals and the stimuli that produced them. These patterns, originating from different recording electrodes are combined, forming a spatial filter that produces a unified prediction of the presented stimulus. This approach takes into account frequency, phase, and spatial distribution of brain features, hence avoiding the need to predefine specific frequency bands of interest or phase relationships between stimulus and brain responses manually. Furthermore, the model does not require the tuning of hyper-parameters, reducing significantly the computational load attached to it. Using three different cognitive tasks (motor movements, speech perception, and speech production), we show that the proposed method consistently improves stimulus feature predictions in terms of correlation (group averages of 0.74 for motor movements, 0.84 for speech perception, and 0.74 for speech production) in comparison with other methods based on regularized multivariate regression, probabilistic graphical models and artificial neural networks. Furthermore, the model parameters revealed those anatomical regions and spectral components that were discriminant in the different cognitive tasks. This novel method does not only provide a useful tool to address fundamental neuroscience questions, but could also be applied to neuroprosthetics.

The use of intracranial recordings to decode human language: Challenges and opportunities.

Decoding speech from intracranial recordings serves two main purposes: understanding the neural correlates of speech processing and decoding speech features for targeting speech neuroprosthetic devices. Intracranial recordings have high spatial and temporal resolution, and thus offer a unique opportunity to investigate and decode the electrophysiological dynamics underlying speech processing. In this review article, we describe current approaches to decoding different features of speech perception and production - such as spectrotemporal, phonetic, phonotactic, semantic, and articulatory components - using intracranial recordings. A specific section is devoted to the decoding of imagined speech, and potential applications to speech prosthetic devices. We outline the challenges in decoding human language, as well as the opportunities in scientific and neuroengineering applications.

Decoding Inner Speech Using Electrocorticography: Progress and Challenges Toward a Speech Prosthesis.

Certain brain disorders resulting from brainstem infarcts, traumatic brain injury, cerebral palsy, stroke, and amyotrophic lateral sclerosis, limit verbal communication despite the patient being fully aware. People that cannot communicate due to neurological disorders would benefit from a system that can infer internal speech directly from brain signals. In this review article, we describe the state of the art in decoding inner speech, ranging from early acoustic sound features, to higher order speech units. We focused on intracranial recordings, as this technique allows monitoring brain activity with high spatial, temporal, and spectral resolution, and therefore is a good candidate to investigate inner speech. Despite intense efforts, investigating how the human cortex encodes inner speech remains an elusive challenge, due to the lack of behavioral and observable measures. We emphasize various challenges commonly encountered when investigating inner speech decoding, and propose potential solutions in order to get closer to a natural speech assistive device.

Neural Encoding of Auditory Features during Music Perception and Imagery.

Despite many behavioral and neuroimaging investigations, it remains unclear how the human cortex represents spectrotemporal sound features during auditory imagery, and how this representation compares to auditory perception. To assess this, we recorded electrocorticographic signals from an epileptic patient with proficient music ability in 2 conditions. First, the participant played 2 piano pieces on an electronic piano with the sound volume of the digital keyboard on. Second, the participant replayed the same piano pieces, but without auditory feedback, and the participant was asked to imagine hearing the music in his mind. In both conditions, the sound output of the keyboard was recorded, thus allowing precise time-locking between the neural activity and the spectrotemporal content of the music imagery. This novel task design provided a unique opportunity to apply receptive field modeling techniques to quantitatively study neural encoding during auditory mental imagery. In both conditions, we built encoding models to predict high gamma neural activity (70-150 Hz) from the spectrogram representation of the recorded sound. We found robust spectrotemporal receptive fields during auditory imagery with substantial, but not complete overlap in frequency tuning and cortical location compared to receptive fields measured during auditory perception.

Rapid tuning shifts in human auditory cortex enhance speech intelligibility.

Experience shapes our perception of the world on a moment-to-moment basis. This robust perceptual effect of experience parallels a change in the neural representation of stimulus features, though the nature of this representation and its plasticity are not well-understood. Spectrotemporal receptive field (STRF) mapping describes the neural response to acoustic features, and has been used to study contextual effects on auditory receptive fields in animal models. We performed a STRF plasticity analysis on electrophysiological data from recordings obtained directly from the human auditory cortex. Here, we report rapid, automatic plasticity of the spectrotemporal response of recorded neural ensembles, driven by previous experience with acoustic and linguistic information, and with a neurophysiological effect in the sub-second range. This plasticity reflects increased sensitivity to spectrotemporal features, enhancing the extraction of more speech-like features from a degraded stimulus and providing the physiological basis for the observed 'perceptual enhancement' in understanding speech.

Word pair classification during imagined speech using direct brain recordings.

People that cannot communicate due to neurological disorders would benefit from an internal speech decoder. Here, we showed the ability to classify individual words during imagined speech from electrocorticographic signals. In a word imagery task, we used high gamma (70-150 Hz) time features with a support vector machine model to classify individual words from a pair of words. To account for temporal irregularities during speech production, we introduced a non-linear time alignment into the SVM kernel. Classification accuracy reached 88% in a two-class classification framework (50% chance level), and average classification accuracy across fifteen word-pairs was significant across five subjects (mean = 58%; p < 0.05). We also compared classification accuracy between imagined speech, overt speech and listening. As predicted, higher classification accuracy was obtained in the listening and overt speech conditions (mean = 89% and 86%, respectively; p < 0.0001), where speech stimuli were directly presented. The results provide evidence for a neural representation for imagined words in the temporal lobe, frontal lobe and sensorimotor cortex, consistent with previous findings in speech perception and production. These data represent a proof of concept study for basic decoding of speech imagery, and delineate a number of key challenges to usage of speech imagery neural representations for clinical applications.

Transcranial Magnetic Stimulation Changes Response Selectivity of Neurons in the Visual Cortex.

BACKGROUND: Transcranial magnetic stimulation (TMS) is used to selectively alter neuronal activity of specific regions in the cerebral cortex. TMS is reported to induce either transient disruption or enhancement of different neural functions. However, its effects on tuning properties of sensory neurons have not been studied quantitatively. OBJECTIVE/HYPOTHESIS: Here, we use specific TMS application parameters to determine how they may alter tuning characteristics (orientation, spatial frequency, and contrast sensitivity) of single neurons in the cat's visual cortex. METHODS: Single unit spikes were recorded with tungsten microelectrodes from the visual cortex of anesthetized and paralyzed cats (12 males). Repetitive TMS (4 Hz, 4 s) was delivered with a 70 mm figure-8 coil. We quantified basic tuning parameters of individual neurons for each pre- and post-TMS condition. The statistical significance of changes for each tuning parameter between the two conditions was evaluated with a Wilcoxon signed-rank test. RESULTS: We generally find long-lasting suppression which persists well beyond the stimulation period. Pre- and post-TMS orientation tuning curves show constant peak values. However, strong suppression at non-preferred orientations tends to narrow the widths of tuning curves. Spatial frequency tuning exhibits an asymmetric change in overall shape, which results in an emphasis on higher frequencies. Contrast tuning curves show nonlinear changes consistent with a gain control mechanism. CONCLUSIONS: These findings suggest that TMS causes extended interruption of the balance between sub-cortical and intra-cortical inputs.

Decoding spectrotemporal features of overt and covert speech from the human cortex.

Auditory perception and auditory imagery have been shown to activate overlapping brain regions. We hypothesized that these phenomena also share a common underlying neural representation. To assess this, we used electrocorticography intracranial recordings from epileptic patients performing an out loud or a silent reading task. In these tasks, short stories scrolled across a video screen in two conditions: subjects read the same stories both aloud (overt) and silently (covert). In a control condition the subject remained in a resting state. We first built a high gamma (70-150 Hz) neural decoding model to reconstruct spectrotemporal auditory features of self-generated overt speech. We then evaluated whether this same model could reconstruct auditory speech features in the covert speech condition. Two speech models were tested: a spectrogram and a modulation-based feature space. For the overt condition, reconstruction accuracy was evaluated as the correlation between original and predicted speech features, and was significant in each subject (p < 10(-5); paired two-sample t-test). For the covert speech condition, dynamic time warping was first used to realign the covert speech reconstruction with the corresponding original speech from the overt condition. Reconstruction accuracy was then evaluated as the correlation between original and reconstructed speech features. Covert reconstruction accuracy was compared to the accuracy obtained from reconstructions in the baseline control condition. Reconstruction accuracy for the covert condition was significantly better than for the control condition (p < 0.005; paired two-sample t-test). The superior temporal gyrus, pre- and post-central gyrus provided the highest reconstruction information. The relationship between overt and covert speech reconstruction depended on anatomy. These results provide evidence that auditory representations of covert speech can be reconstructed from models that are built from an overt speech data set, supporting a partially shared neural substrate.

Decoding speech for understanding and treating aphasia.

Aphasia is an acquired language disorder with a diverse set of symptoms that can affect virtually any linguistic modality across both the comprehension and production of spoken language. Partial recovery of language function after injury is common but typically incomplete. Rehabilitation strategies focus on behavioral training to induce plasticity in underlying neural circuits to maximize linguistic recovery. Understanding the different neural circuits underlying diverse language functions is a key to developing more effective treatment strategies. This chapter discusses a systems identification analytic approach to the study of linguistic neural representation. The focus of this framework is a quantitative, model-based characterization of speech and language neural representations that can be used to decode, or predict, speech representations from measured brain activity. Recent results of this approach are discussed in the context of applications to understanding the neural basis of aphasia symptoms and the potential to optimize plasticity during the rehabilitation process.

Reconstructing speech from human auditory cortex.

How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.

State-dependent variability of neuronal responses to transcranial magnetic stimulation of the visual cortex.

Electrical brain stimulation is a promising tool for both experimental and clinical applications. However, the effects of stimulation on neuronal activity are highly variable and poorly understood. To investigate the basis of this variability, we performed extracellular recordings in the visual cortex following application of transcranial magnetic stimulation (TMS). Our measurements of spiking and local field potential activity exhibit two types of response patterns which are characterized by the presence or absence of spontaneous discharge following stimulation. This variability can be partially explained by state-dependent effects, in which higher pre-TMS activity predicts larger post-TMS responses. These results reveal the possibility that variability in the neural response to TMS can be exploited to optimize the effects of stimulation. It is conceivable that this feature could be utilized in real time during the treatment of clinical disorders.

Transcranial magnetic stimulation elicits coupled neural and hemodynamic consequences.

Transcranial magnetic stimulation (TMS) is an increasingly common technique used to selectively modify neural processing. However, application of TMS is limited by uncertainty concerning its physiological effects. We applied TMS to the cat visual cortex and evaluated the neural and hemodynamic consequences. Short TMS pulse trains elicited initial activation (approximately 1 minute) and prolonged suppression (5 to 10 minutes) of neural responses. Furthermore, TMS disrupted the temporal structure of activity by altering phase relationships between neural signals. Despite the complexity of this response, neural changes were faithfully reflected in hemodynamic signals; quantitative coupling was present over a range of stimulation parameters. These results demonstrate long-lasting neural responses to TMS and support the use of hemodynamic-based neuroimaging to effectively monitor these changes over time.

Analysis of oxygen metabolism implies a neural origin for the negative BOLD response in human visual cortex.

The sustained negative blood oxygenation level-dependent (BOLD) response in functional MRI is observed universally, but its interpretation is controversial. The origin of the negative response is of fundamental importance because it could provide a measurement of neural deactivation. However, a substantial component of the negative response may be due to a non-neural hemodynamic artifact. To distinguish these possibilities, we have measured evoked BOLD, cerebral blood flow (CBF), and oxygen metabolism responses to a fixed visual stimulus from two different baseline conditions. One is a normal resting baseline, and the other is a lower baseline induced by a sustained negative response. For both baseline conditions, CBF and oxygen metabolism responses reach the same peak amplitude. Consequently, evoked responses from the negative baseline are larger than those from the resting baseline. The larger metabolic response from negative baseline presumably reflects a greater neural response that is required to reach the same peak amplitude as that from resting baseline. Furthermore, the ratio of CBF to oxygen metabolism remains approximately the same from both baseline states (approximately 2:1). This tight coupling between hemodynamic and metabolic components implies that the magnitude of any hemodynamic artifact is inconsequential. We conclude that the negative response is a functionally significant index of neural deactivation in early visual cortex.

Subcortical discrimination of unperceived objects during binocular rivalry.

Rapid identification of behaviorally relevant objects is important for survival. In humans, the neural computations for visually discriminating complex objects involve inferior temporal cortex (IT). However, less detailed but faster form processing may also occur in a phylogenetically older subcortical visual system that terminates in the amygdala. We used binocular rivalry to present stimuli without conscious awareness, thereby eliminating the IT object representation and isolating subcortical visual input to the amygdala. Functional magnetic resonance imaging revealed significant brain activation in the left amygdala but not in object-selective IT in response to unperceived fearful faces compared to unperceived nonface objects. These findings indicate that, for certain behaviorally relevant stimuli, a high-level cortical representation in IT is not required for object discrimination in the amygdala.