Violation of identity-specific action-effect prediction increases pupil size and attenuates auditory event-related potentials at P2 latencies when action-effects are behaviorally relevant.

Self-initiated sensory action effects are widely assumed to lead to less intense perception and reduced neural responses compared to externally triggered stimuli (sensory attenuation). However, it is unclear if sensory attenuation occurs in all cases of action-effect prediction. Specifically, when predicted action-effects are relevant to determine follow-up actions attenuation could be detrimental. We quantified auditory event-related potentials (ERP) in electroencephalography (EEG) when human participants created two-sound sequences by pressing two keys on a keyboard associated with different pitch, giving rise to identity-specific action-effect prediction after the first keypress. The first sound corresponded to (congruent) or violated (incongruent) the predicted pitch and was either relevant for the selection of the second keypress to correctly complete the sequence (Relevance) or irrelevant (Control Movement), or there was only one keypress and sound (Baseline). We found a diminished P2-timed ERP component in incongruent compared to congruent trials when the sound was relevant for the subsequent action. This effect of action-effect prediction was due to an ERP reduction for incongruent relevant sounds compared to incongruent irrelevant sounds at P2 latencies and correlated negatively with modulations of pupil dilation. Contrary to our expectation, we did not observe an N1 modulation by congruency in any condition. Attenuation of the N1 component seems absent for predicted identity-specific auditory action effects, while P2-timed ERPs as well as pupil size are sensitive to predictability, at least when action effects are relevant for the selection of the next action. Incongruent relevant stimuli thereby take a special place and seem to be subject to attentional modulations and error processing.

Differential effects of intra-modal and cross-modal reward value on perception: ERP evidence.

In natural environments objects comprise multiple features from the same or different sensory modalities but it is not known how perception of an object is affected by the value associations of its constituent parts. The present study compares intra- and cross-modal value-driven effects on behavioral and electrophysiological correlates of perception. Human participants first learned the reward associations of visual and auditory cues. Subsequently, they performed a visual discrimination task in the presence of previously rewarded, task-irrelevant visual or auditory cues (intra- and cross-modal cues, respectively). During the conditioning phase, when reward associations were learned and reward cues were the target of the task, high value stimuli of both modalities enhanced the electrophysiological correlates of sensory processing in posterior electrodes. During the post-conditioning phase, when reward delivery was halted and previously rewarded stimuli were task-irrelevant, cross-modal value significantly enhanced the behavioral measures of visual sensitivity, whereas intra-modal value produced only an insignificant decrement. Analysis of the simultaneously recorded event-related potentials (ERPs) of posterior electrodes revealed similar findings. We found an early (90-120 ms) suppression of ERPs evoked by high-value, intra-modal stimuli. Cross-modal stimuli led to a later value-driven modulation, with an enhancement of response positivity for high- compared to low-value stimuli starting at the N1 window (180-250 ms) and extending to the P3 (300-600 ms) responses. These results indicate that sensory processing of a compound stimulus comprising a visual target and task-irrelevant visual or auditory cues is modulated by the reward value of both sensory modalities, but such modulations rely on distinct underlying mechanisms.

Cross-Modal Integration of Reward Value during Oculomotor Planning.

Reward value guides goal-directed behavior and modulates early sensory processing. Rewarding stimuli are often multisensory, but it is not known how reward value is combined across sensory modalities. Here we show that the integration of reward value critically depends on whether the distinct sensory inputs are perceived to emanate from the same multisensory object. We systematically manipulated the congruency in monetary reward values and the relative spatial positions of co-occurring auditory and visual stimuli that served as bimodal distractors during an oculomotor task performed by healthy human participants (male and female). The amount of interference induced by the distractors was used as an indicator of their perceptual salience. Our results across two experiments show that when reward value is linked to each modality separately, the value congruence between vision and audition determines the combined salience of the bimodal distractors. However, the reward value of vision wins over the value of audition if the two modalities are perceived to convey conflicting information regarding the spatial position of the bimodal distractors. These results show that in a task that highly relies on the processing of visual spatial information, the reward values from multiple sensory modalities are integrated with each other, each with their respective weights. This weighting depends on the strength of prior beliefs regarding a common source for incoming unisensory signals based on their congruency in reward value and perceived spatial alignment.