Audio-visual synchrony modulates the ventriloquist illusion and its neural/spatial representation in the auditory cortex.

An essential task of our perceptual systems is to bind together the distinctive features of single objects and events into unitary percepts, even when those features are registered in different sensory modalities. In cases where auditory and visual inputs are spatially incongruent, they may still be perceived as belonging to a single event at the location of the visual stimulus - a phenomenon known as the 'ventriloquist illusion'. The present study examined how audio-visual temporal congruence influences the ventriloquist illusion and characterized its neural underpinnings with functional magnetic resonance imaging (fMRI). Behaviorally, the ventriloquist illusion was reduced for asynchronous versus synchronous audio-visual stimuli, in accordance with previous reports. Neural activity patterns associated with the ventriloquist effect were consistently observed in the planum temporale (PT), with a reduction in illusion-related fMRI-signals ipsilateral to visual stimulation for central sounds perceived peripherally and a contralateral increase in illusion-related fMRI-signals for peripheral sounds perceived centrally. Moreover, it was found that separate but adjacent regions within the PT were preferentially activated for ventriloquist illusions produced by synchronous and asynchronous audio-visual stimulation. We conclude that the left-right balance of neural activity in the PT represents the neural code that underlies the ventriloquist illusion, with greater activity in the cerebral hemisphere contralateral to the direction of the perceived shift of sound location.

Task-demands and audio-visual stimulus configurations modulate neural activity in the human thalamus.

Recent electrophysiological studies have reported short latency modulations in cortical regions for multisensory stimuli, thereby suggesting a subcortical, possibly thalamic origin of these modulations. Concurrently, there is an ongoing debate, whether multisensory interplay reflects automatic, bottom-up driven processes or relies on top-down influences. Here, we dissociated the effects of task set and stimulus configurations on BOLD-signals in the human thalamus with event-related functional magnetic resonance imaging (fMRI). We orthogonally manipulated temporal and spatial congruency of audio-visual stimulus configurations, while subjects judged either their temporal or spatial congruency. Voxel-based fMRI results revealed increased fMRI-signals for the temporal versus spatial task in posterior and central thalamus, respectively. A more sensitive region of interest (ROI)-analysis confirmed that the posterior thalamic nuclei showed a preference for the temporal task and central thalamic nuclei for the spatial task. Moreover, the ROI-analysis also revealed enhanced fMRI-signals for spatially incongruent stimuli in the central thalamus. Together, our results demonstrate that both audio-visual stimulus configurations and task-related processing of spatial or temporal stimulus features selectively modulate thalamic processing and thus are in a position to influence cortical processing at an early stage.

Neural basis of multisensory looming signals.

Approaching or looming signals are often related to extremely relevant environmental events (e.g. threats or collisions) making these signals critical for survival. However, the neural network underlying multisensory looming processing is not yet fully understood. Using functional magnetic resonance imaging (fMRI) we identified the neural correlates of audiovisual looming processing in humans: audiovisual looming (vs. receding) signals enhance fMRI-responses in low-level visual and auditory areas plus multisensory cortex (superior temporal sulcus; plus parietal and frontal structures). When characterizing the fMRI-response profiles for multisensory looming stimuli, we found significant enhancements relative to the mean and maximum of unisensory responses in looming-sensitive visual and auditory cortex plus STS. Superadditive enhancements were observed in visual cortex. Subject-specific region-of-interest analyses further revealed superadditive response profiles within all sensory-specific looming-sensitive structures plus bilateral STS for audiovisual looming vs. summed unisensory looming conditions. Finally, we observed enhanced connectivity of bilateral STS with low-level visual areas in the context of looming processing. This enhanced coupling of STS with unisensory regions might potentially serve to enhance the salience of unisensory stimulus features and is accompanied by superadditive fMRI-responses. We suggest that this preference in neural signaling for looming stimuli effectively informs animals to avoid potential threats or collisions.

Thalamic influences on multisensory integration.

In everyday life our brain often receives information about events and objects in the real world via several sensory modalities, because natural objects often stimulate more than one sense. These different types of information are processed in our brain along different sensory-specific pathways, but are finally integrated into a unified percept. During the last years, studies provided compelling evidence that the neural basis of multisensory integration is not restricted to higher association areas of the cortex, but can already occur at low-level stages of sensory cortical processing and even in subcortical structures. In this article we will review the potential role of several thalamic structures in multisensory interplay and discuss their extensive anatomical connections with sensory-specific and multisensory cortical structures. We conclude that sensory-specific thalamic structures may act as a crucial processing node of multisensory interplay in addition to their traditional role as sensory relaying structure.

Sound-induced enhancement of low-intensity vision: multisensory influences on human sensory-specific cortices and thalamic bodies relate to perceptual enhancement of visual detection sensitivity.

Combining information across modalities can affect sensory performance. We studied how co-occurring sounds modulate behavioral visual detection sensitivity (d'), and neural responses, for visual stimuli of higher or lower intensity. Co-occurrence of a sound enhanced human detection sensitivity for lower- but not higher-intensity visual targets. Functional magnetic resonance imaging (fMRI) linked this to boosts in activity-levels for sensory-specific visual and auditory cortex, plus multisensory superior temporal sulcus (STS), specifically for a lower-intensity visual event when paired with a sound. Thalamic structures in visual and auditory pathways, the lateral and medial geniculate bodies, respectively (LGB, MGB), showed a similar pattern. Subject-by-subject psychophysical benefits correlated with corresponding fMRI signals in visual, auditory, and multisensory regions. We also analyzed differential "coupling" patterns of LGB and MGB with other regions in the different experimental conditions. Effective-connectivity analyses showed enhanced coupling of sensory-specific thalamic bodies with the affected cortical sites during enhanced detection of lower-intensity visual events paired with sounds. Coupling strength between visual and auditory thalamus with cortical regions, including STS, covaried parametrically with the psychophysical benefit for this specific multisensory context. Our results indicate that multisensory enhancement of detection sensitivity for low-contrast visual stimuli by co-occurring sounds reflects a brain network involving not only established multisensory STS and sensory-specific cortex but also visual and auditory thalamus.

Closer in time when farther in space--spatial factors in audiovisual temporal integration.

We investigated the effect of visual eccentricity and spatial alignment on judgments of audiovisual synchrony. Sequences of flashes at 4, 6, and 8 Hz were presented centrally, or at horizontal eccentricities of 6 degrees or 18 degrees. Concurrent sequences of clicks were presented at the same rate as the flashes, or at higher or lower rates. Subjects judged whether the flash rate was the same as (synchronous with), faster than, or slower than the click rate. With the 4- and 6-Hz flash rates, subjects' judgments of audiovisual synchrony increased with increasing eccentricity, but only when the click rate was more rapid than the flash rate. This effect remained even when the size of the peripheral visual stimuli was adjusted to compensate for cortical magnification, and was not significantly influenced by the spatial proximity of the auditory and visual signals. However, it was absent when the auditory and visual stimuli were presented serially rather than concurrently. With the 8-Hz flash rate, synchrony judgments were prevalent irrespective of eccentricity. When two serially presented flash rates were compared, visual-visual matching judgments increased with eccentricity at flash rates of 6 Hz and higher, but decreased at flash rates below 6 Hz. Finally, when two concurrent flash rates were compared, visual-visual synchrony judgments increased with eccentricity at all flash-rate combinations. Together, these results suggest that while perceptual uncertainty can play a role in synchrony judgments at rates of 6 Hz and higher, below 6 Hz eccentricity produces a widening of the window of apparent audiovisual temporal synchrony which perceptual uncertainty cannot explain.