Sublexical cues affect degraded speech processing: insights from fMRI.

In natural listening situations, speech perception is often impaired by degraded speech sounds arriving at the ear. Contextual speech information can improve the perception of degraded speech and modify neuronal responses elicited by degraded speech. However, most studies on context effects on neural responses to degraded speech confounded lexico-semantic and sublexical cues. Here, we used fMRI to investigate how prior sublexical speech (e.g. pseudowords cues) affects neural responses to degraded sublexical speech and hence its processing and recognition. Each trial consisted of three consecutively presented pseudowords, of which the first and third were identical and degraded. The second pseudoword was always presented in clear form and either matched or did not match the degraded pseudowords. Improved speech processing through sublexical processing was associated with BOLD activation increases in frontal, temporal, and parietal regions, including the primary auditory cortex (PAC), posterior superior temporal cortex, angular gyrus, supramarginal gyrus, middle temporal cortex, and somato-motor cortex. These brain regions are part of a speech processing network and are involved in lexico-semantic processing. To further investigate the adaptive changes in PAC, we conducted a bilateral region of interest analysis on PAC subregions. PAC ROIs showed bilaterally increased activation in the match condition compared with the mismatch condition. Our results show that the perception of unintelligible degraded speech is improved and the neuronal population response is enhanced after exposure to intact sublexical cues. Furthermore, our findings indicate that the processing of clear meaningless sublexical speech preceding degraded speech could enhance the activity in the brain regions that belong to the cortical speech processing network previously reported in studies investigating lexico-semantic speech.

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex.

During natural speech perception, humans must parse temporally continuous auditory and visual speech signals into sequences of words. However, most studies of speech perception present only single words or syllables. We used electrocorticography (subdural electrodes implanted on the brains of epileptic patients) to investigate the neural mechanisms for processing continuous audiovisual speech signals consisting of individual sentences. Using partial correlation analysis, we found that posterior superior temporal gyrus (pSTG) and medial occipital cortex tracked both the auditory and the visual speech envelopes. These same regions, as well as inferior temporal cortex, responded more strongly to a dynamic video of a talking face compared to auditory speech paired with a static face. Occipital cortex and pSTG carry temporal information about both auditory and visual speech dynamics. Visual speech tracking in pSTG may be a mechanism for enhancing perception of degraded auditory speech.

A Double Dissociation between Anterior and Posterior Superior Temporal Gyrus for Processing Audiovisual Speech Demonstrated by Electrocorticography.

Human speech can be comprehended using only auditory information from the talker's voice. However, comprehension is improved if the talker's face is visible, especially if the auditory information is degraded as occurs in noisy environments or with hearing loss. We explored the neural substrates of audiovisual speech perception using electrocorticography, direct recording of neural activity using electrodes implanted on the cortical surface. We observed a double dissociation in the responses to audiovisual speech with clear and noisy auditory component within the superior temporal gyrus (STG), a region long known to be important for speech perception. Anterior STG showed greater neural activity to audiovisual speech with clear auditory component, whereas posterior STG showed similar or greater neural activity to audiovisual speech in which the speech was replaced with speech-like noise. A distinct border between the two response patterns was observed, demarcated by a landmark corresponding to the posterior margin of Heschl's gyrus. To further investigate the computational roles of both regions, we considered Bayesian models of multisensory integration, which predict that combining the independent sources of information available from different modalities should reduce variability in the neural responses. We tested this prediction by measuring the variability of the neural responses to single audiovisual words. Posterior STG showed smaller variability than anterior STG during presentation of audiovisual speech with noisy auditory component. Taken together, these results suggest that posterior STG but not anterior STG is important for multisensory integration of noisy auditory and visual speech.

Human centromedian-parafascicular complex signals sensory cues for goal-oriented behavior selection.

Experimental research has shown that the centromedian-parafascicular complex (CM-Pf) of the intralaminar thalamus is activated in attentional orienting and processing of behaviorally relevant stimuli. These observations resulted in the hypothesis that the CM-Pf plays a pivotal role in goal-oriented behavior selection. We here set out to test this hypothesis with electrophysiological recordings from patients with electrodes implanted in CM-Pf for deep brain stimulation (DBS) treatment of chronic neuropathic pain. Six patients participated in (1) an auditory three-class oddball experiment, which required a button press to target tones, but not to standard and deviant tones and in (2) a multi-speaker experiment with a target word that required attention selection and a target image that required response selection. Subjects showed transient neural responses (8-15Hz) to the target tone and the target word. Two subjects additionally showed transient neural responses (15-25Hz) to the target image. All sensory target stimuli were related to an internal goal and required a behavior selection (attention selection, response selection). In group analyses, neural responses were greater to target tones than deviant and standard tones and to target words than other task-relevant words that did not require attention selection. The transient neural responses occurred after the target stimuli but prior to the overt behavioral response. Our results demonstrate that in human subjects the CM-Pf is involved in signaling sensory inputs related to goal-oriented selection of behavior.

Electrocorticography Reveals Enhanced Visual Cortex Responses to Visual Speech.

Human speech contains both auditory and visual components, processed by their respective sensory cortices. We test a simple model in which task-relevant speech information is enhanced during cortical processing. Visual speech is most important when the auditory component is uninformative. Therefore, the model predicts that visual cortex responses should be enhanced to visual-only (V) speech compared with audiovisual (AV) speech. We recorded neuronal activity as patients perceived auditory-only (A), V, and AV speech. Visual cortex showed strong increases in high-gamma band power and strong decreases in alpha-band power to V and AV speech. Consistent with the model prediction, gamma-band increases and alpha-band decreases were stronger for V speech. The model predicts that the uninformative nature of the auditory component (not simply its absence) is the critical factor, a prediction we tested in a second experiment in which visual speech was paired with auditory white noise. As predicted, visual speech with auditory noise showed enhanced visual cortex responses relative to AV speech. An examination of the anatomical locus of the effects showed that all visual areas, including primary visual cortex, showed enhanced responses. Visual cortex responses to speech are enhanced under circumstances when visual information is most important for comprehension.

Altered intrinsic neuronal interactions in the visual cortex of the blind.

In congenital blindness, the brain develops under severe sensory deprivation and undergoes remarkable plastic changes in both structure and function. Visually deprived occipital cortical regions are histologically and morphologically altered and exhibit a strikingly remodeled functional state: absolute levels of neural activity are heightened and are modulated by nonvisual sensory stimulation as well as higher cognitive processes. However, the neuronal mechanisms that underlie this altered functional state remain largely unknown. Here, we show that the visual cortex of the congenitally blind exhibits a characteristic gain in frequency-specific intrinsic neuronal interactions. We studied oscillatory activity in 11 congenitally blind humans and matched sighted control subjects with magnetoencephalography at rest. We found increased spontaneous correlations of delta band (1-3 Hz) and gamma band (76-128 Hz) oscillations across the visual cortex of the blind that were functionally coupled. Local delta phase modulated gamma amplitude. Furthermore, classical resting rhythms (8-20 Hz) were reduced in amplitude but showed no altered correlation pattern. Our results suggest that both decreased inhibition and circuit mechanisms that support active processing are intrinsic features underlying the altered functional state of the visual cortex in congenitally blind individuals.

Noise alters beta-band activity in superior temporal cortex during audiovisual speech processing.

Speech recognition is improved when complementary visual information is available, especially under noisy acoustic conditions. Functional neuroimaging studies have suggested that the superior temporal sulcus (STS) plays an important role for this improvement. The spectrotemporal dynamics underlying audiovisual speech processing in the STS, and how these dynamics are affected by auditory noise, are not well understood. Using electroencephalography, we investigated how auditory noise affects audiovisual speech processing in event-related potentials (ERPs) and oscillatory activity. Spoken syllables were presented in audiovisual (AV) and auditory only (A) trials at three different auditory noise levels (no, low, and high). Responses to A stimuli were subtracted from responses to AV stimuli, separately for each noise level, and these responses were subjected to the statistical analysis. Central ERPs differed between the no noise and the two noise conditions from 130 to 150 ms and 170 to 210 ms after auditory stimulus onset. Source localization using the local autoregressive average procedure revealed an involvement of the lateral temporal lobe, encompassing the superior and middle temporal gyrus. Neuronal activity in the beta-band (16 to 32 Hz) was suppressed at central channels around 100 to 400 ms after auditory stimulus onset in the averaged AV minus A signal over the three noise levels. This suppression was smaller in the high noise compared to the no noise and low noise condition, possibly reflecting disturbed recognition or altered processing of multisensory speech stimuli. Source analysis of the beta-band effect using linear beamforming demonstrated an involvement of the STS. Our study shows that auditory noise alters audiovisual speech processing in ERPs localized to lateral temporal lobe and provides evidence that beta-band activity in the STS plays a role for audiovisual speech processing under regular and noisy acoustic conditions.

Functionally specific oscillatory activity correlates between visual and auditory cortex in the blind.

Many studies have shown that the visual cortex of blind humans is activated in non-visual tasks. However, the electrophysiological signals underlying this cross-modal plasticity are largely unknown. Here, we characterize the neuronal population activity in the visual and auditory cortex of congenitally blind humans and sighted controls in a complex cognitive task. We recorded magnetoencephalographic responses from participants performing semantic categorization of meaningful sounds that followed the presentation of a semantically related or unrelated haptic object. Source analysis of the spectrally resolved magnetoencephalography data revealed that: (i) neuronal responses to sounds were stronger and longer lasting in the auditory cortex of blind subjects; (ii) auditory stimulation elicited strong oscillatory responses in the visual cortex of blind subjects that closely resembled responses to visual stimulation in sighted humans; (iii) the signal in the gamma frequency range was modulated by semantic congruency between the sounds and the preceding haptic objects; and (iv) signal power in the gamma range was correlated on a trial-by-trial basis between auditory and visual cortex in blind subjects, and the strength of this correlation was modulated by semantic congruency. Our results suggest that specifically oscillatory activity in the gamma range reflects non-visual processing in the visual cortex of blind individuals. Moreover, our results provide evidence that the deprived visual cortex is functionally integrated into a larger network that serves non-visual functions.