Synchronous motor imagery and visual feedback of finger movement elicit the moving rubber hand illusion, at least in illusion-susceptible individuals.

Recent evidence suggests that imagined auditory and visual sensory stimuli can be integrated with real sensory information from a different sensory modality to change the perception of external events via cross-modal multisensory integration mechanisms. Here, we explored whether imagined voluntary movements can integrate visual and proprioceptive cues to change how we perceive our own limbs in space. Participants viewed a robotic hand wearing a glove repetitively moving its right index finger up and down at a frequency of 1 Hz, while they imagined executing the corresponding movements synchronously or asynchronously (kinesthetic-motor imagery); electromyography (EMG) from the participants' right index flexor muscle confirmed that the participants kept their hand relaxed while imagining the movements. The questionnaire results revealed that the synchronously imagined movements elicited illusory ownership and a sense of agency over the moving robotic hand-the moving rubber hand illusion-compared with asynchronously imagined movements; individuals who affirmed experiencing the illusion with real synchronous movement also did so with synchronous imagined movements. The results from a proprioceptive drift task further demonstrated a shift in the perceived location of the participants' real hand toward the robotic hand in the synchronous versus the asynchronous motor imagery condition. These results suggest that kinesthetic motor imagery can be used to replace veridical congruent somatosensory feedback from a moving finger in the moving rubber hand illusion to trigger illusory body ownership and agency, but only if the temporal congruence rule of the illusion is obeyed. This observation extends previous studies on the integration of mental imagery and sensory perception to the case of multisensory bodily awareness, which has potentially important implications for research into embodiment of brain-computer interface controlled robotic prostheses and computer-generated limbs in virtual reality.

Avatar Embodiment Enhances Haptic Confidence on the Out-of-Body Touch Illusion.

In the real world, our bodies influence how we perceive ourselves and how others perceive us. Our body can also affect estimations of object sizes and distances. But how does our body affect our haptic experience? Here, we examined the modulation of a visuo-haptic illusion of touch on a virtual stick in virtual reality, when participants were embodied in an avatar and when they were not. During the experiments participants (n = 49) received successions of three taps delivered from two independent controllers while they saw visual stimuli presented sequentially along the virtual stick. The stimulation pattern resulted in a robust illusion of tapping directly on the virtual stick. After each trial, participants were asked to report where they perceived the taps. We found that participants in both the body and no-body conditions displaced the second tap toward the center of the stick, and reported similar levels of certainty about their reported location. However, the illusion of touch on the stick, as measured by the reported location of the tap, was significantly stronger for those who had a virtual body than those who did not. Therefore, our study shows that avatar embodiment can change haptic perception.

Mental Imagery Induces Cross-Modal Sensory Plasticity and Changes Future Auditory Perception.

Can what we imagine in our minds change how we perceive the world in the future? A continuous process of multisensory integration and recalibration is responsible for maintaining a correspondence between the senses (e.g., vision, touch, audition) and, ultimately, a stable and coherent perception of our environment. This process depends on the plasticity of our sensory systems. The so-called ventriloquism aftereffect-a shift in the perceived localization of sounds presented alone after repeated exposure to spatially mismatched auditory and visual stimuli-is a clear example of this type of plasticity in the audiovisual domain. In a series of six studies with 24 participants each, we investigated an imagery-induced ventriloquism aftereffect in which imagining a visual stimulus elicits the same frequency-specific auditory aftereffect as actually seeing one. These results demonstrate that mental imagery can recalibrate the senses and induce the same cross-modal sensory plasticity as real sensory stimuli.

Generic HRTFs May be Good Enough in Virtual Reality. Improving Source Localization through Cross-Modal Plasticity.

Auditory spatial localization in humans is performed using a combination of interaural time differences, interaural level differences, as well as spectral cues provided by the geometry of the ear. To render spatialized sounds within a virtual reality (VR) headset, either individualized or generic Head Related Transfer Functions (HRTFs) are usually employed. The former require arduous calibrations, but enable accurate auditory source localization, which may lead to a heightened sense of presence within VR. The latter obviate the need for individualized calibrations, but result in less accurate auditory source localization. Previous research on auditory source localization in the real world suggests that our representation of acoustic space is highly plastic. In light of these findings, we investigated whether auditory source localization could be improved for users of generic HRTFs via cross-modal learning. The results show that pairing a dynamic auditory stimulus, with a spatio-temporally aligned visual counterpart, enabled users of generic HRTFs to improve subsequent auditory source localization. Exposure to the auditory stimulus alone or to asynchronous audiovisual stimuli did not improve auditory source localization. These findings have important implications for human perception as well as the development of VR systems as they indicate that generic HRTFs may be enough to enable good auditory source localization in VR.

The uncanny valley of haptics.

During teleoperation and virtual reality experiences, enhanced haptic feedback incongruent with other sensory cues can reduce subjective realism, producing an uncanny valley of haptics.

The Content of Imagined Sounds Changes Visual Motion Perception in the Cross-Bounce Illusion.

Can what we imagine hearing change what we see? Whether imagined sensory stimuli are integrated with external sensory stimuli to shape our perception of the world has only recently begun to come under scrutiny. Here, we made use of the cross-bounce illusion in which an auditory stimulus presented at the moment two passing objects meet promotes the perception that the objects bounce off rather than cross by one another to examine whether the content of imagined sound changes visual motion perception in a manner that is consistent with multisensory integration. The results from this study revealed that auditory imagery of a sound with acoustic properties typical of a collision (i.e., damped sound) promoted the bounce-percept, but auditory imagery of the same sound played backwards (i.e., ramped sound) did not. Moreover, the vividness of the participants' auditory imagery predicted the strength of this imagery-induced illusion. In a separate experiment, we ruled out the possibility that changes in attention (i.e., sensitivity index d') or response bias (response bias index c) were sufficient to explain this effect. Together, these findings suggest that this imagery-induced multisensory illusion reflects the successful integration of real and imagined cross-modal sensory stimuli, and more generally, that what we imagine hearing can change what we see.

Dissociating the Neural Basis of Conceptual Self-Awareness from Perceptual Awareness and Unaware Self-Processing.

Conceptual self-awareness is a mental state in which the content of one's consciousness refers to a particular aspect of semantic knowledge about oneself. This form of consciousness plays a crucial role in shaping human behavior; however, little is known about its neural basis. Here, we use functional magnetic resonance imaging (fMRI) and a visual masked priming paradigm to dissociate the neural responses related to the awareness of semantic autobiographical information (one's own name, surname, etc.) from the awareness of information related to any visual stimulus (perceptual awareness), as well as from the unaware processing of self-relevant stimuli. To detect brain activity that is highly selective for self-relevant information, we used the blood-oxygen-level-dependent (BOLD) adaptation approach, which goes beyond the spatial limitations of conventional fMRI. We found that self-awareness was associated with BOLD adaptation in the medial frontopolar-retrosplenial areas, whereas perceptual awareness and unaware self-processing were associated with BOLD adaptation in the lateral fronto-parietal areas and the inferior temporal cortex, respectively. Thus, using a direct manipulation of conscious awareness we demonstrate for the first time that the neural basis of conceptual self-awareness is neuroanatomically distinct from the network mediating perceptual awareness of the sensory environment or unaware processing of self-related stimuli.

Auditory Motion Elicits a Visual Motion Aftereffect.

The visual motion aftereffect is a visual illusion in which exposure to continuous motion in one direction leads to a subsequent illusion of visual motion in the opposite direction. Previous findings have been mixed with regard to whether this visual illusion can be induced cross-modally by auditory stimuli. Based on research on multisensory perception demonstrating the profound influence auditory perception can have on the interpretation and perceived motion of visual stimuli, we hypothesized that exposure to auditory stimuli with strong directional motion cues should induce a visual motion aftereffect. Here, we demonstrate that horizontally moving auditory stimuli induced a significant visual motion aftereffect-an effect that was driven primarily by a change in visual motion perception following exposure to leftward moving auditory stimuli. This finding is consistent with the notion that visual and auditory motion perception rely on at least partially overlapping neural substrates.

The fusion of mental imagery and sensation in the temporal association cortex.

It is well understood that the brain integrates information that is provided to our different senses to generate a coherent multisensory percept of the world around us (Stein and Stanford, 2008), but how does the brain handle concurrent sensory information from our mind and the external world? Recent behavioral experiments have found that mental imagery--the internal representation of sensory stimuli in one's mind--can also lead to integrated multisensory perception (Berger and Ehrsson, 2013); however, the neural mechanisms of this process have not yet been explored. Here, using functional magnetic resonance imaging and an adapted version of a well known multisensory illusion (i.e., the ventriloquist illusion; Howard and Templeton, 1966), we investigated the neural basis of mental imagery-induced multisensory perception in humans. We found that simultaneous visual mental imagery and auditory stimulation led to an illusory translocation of auditory stimuli and was associated with increased activity in the left superior temporal sulcus (L. STS), a key site for the integration of real audiovisual stimuli (Beauchamp et al., 2004a, 2010; Driver and Noesselt, 2008; Ghazanfar et al., 2008; Dahl et al., 2009). This imagery-induced ventriloquist illusion was also associated with increased effective connectivity between the L. STS and the auditory cortex. These findings suggest an important role of the temporal association cortex in integrating imagined visual stimuli with real auditory stimuli, and further suggest that connectivity between the STS and auditory cortex plays a modulatory role in spatially localizing auditory stimuli in the presence of imagined visual stimuli.

Mental imagery changes multisensory perception.

Multisensory interactions are the norm in perception, and an abundance of research on the interaction and integration of the senses has demonstrated the importance of combining sensory information from different modalities on our perception of the external world. However, although research on mental imagery has revealed a great deal of functional and neuroanatomical overlap between imagery and perception, this line of research has primarily focused on similarities within a particular modality and has yet to address whether imagery is capable of leading to multisensory integration. Here, we devised novel versions of classic multisensory paradigms to systematically examine whether imagery is capable of integrating with perceptual stimuli to induce multisensory illusions. We found that imagining an auditory stimulus at the moment two moving objects met promoted an illusory bounce percept, as in the classic cross-bounce illusion; an imagined visual stimulus led to the translocation of sound toward the imagined stimulus, as in the classic ventriloquist illusion; and auditory imagery of speech stimuli led to a promotion of an illusory speech percept in a modified version of the McGurk illusion. Our findings provide support for perceptually based theories of imagery and suggest that neuronal signals produced by imagined stimuli can integrate with signals generated by real stimuli of a different sensory modality to create robust multisensory percepts. These findings advance our understanding of the relationship between imagery and perception and provide new opportunities for investigating how the brain distinguishes between endogenous and exogenous sensory events.

Did I read or did I name? Diminished awareness of processes yielding identical 'outputs'.

It has been proposed that incompatible intentions (e.g., to inhale and not inhale while holding one's breath while underwater) are an essential ingredient of conscious conflict. Laboratory tasks such as the Stroop color naming task can instantiate conscious conflict innocuously. However, little research has explored what occurs subjectively at the other end of conflict, when intentions are harmonious. The hypothesis of synchrony blindness proposes that, during harmonious processing, not only may one not experience any conflict, but one may also be unaware that more than one process yielded the same intention/action plan. Accordingly, in the Stroop task, participants reported less of an urge to err (by reading) when words were presented in the congruent condition (e.g., RED presented in red) than when the very same words were presented in standard font color, suggesting that awareness of word-reading was diminished experimentally. The implications of this finding for theories about consciousness are discussed.

Cognitive and neural components of the phenomenology of agency.

A primary aspect of the self is the sense of agency ­ the sense that one is causing an action. In the spirit of recent reductionistic approaches to other complex, multifaceted phenomena (e.g., working memory; cf. Johnson &Johnson, 2009), we attempt to unravel the sense of agency by investigating its most basic components, without invoking high-level conceptual or 'central executive' processes. After considering the high-level components of agency, we examine the cognitive and neural underpinnings of its low-level components, which include basic consciousness and subjective urges (e.g., the urge to breathe when holding one's breath). Regarding urges, a quantitative review revealed that certain inter-representational dynamics (conflicts between action plans, as when holding one's breath) reliably engender fundamental aspects both of the phenomenology of agency and of 'something countering the will of the self'. The neural correlates of such dynamics, for both primordial urges (e.g., air hunger) and urges elicited in laboratory interference tasks, are entertained. In addition, we discuss the implications of this unique perspective for the study of disorders involving agency.

Mind control? Creating illusory intentions through a phony brain-computer interface.

Can one be fooled into believing that one intended an action that one in fact did not intend? Past experimental paradigms have demonstrated that participants, when provided with false perceptual feedback about their actions, can be fooled into misperceiving the nature of their intended motor act. However, because veridical proprioceptive/perceptual feedback limits the extent to which participants can be fooled, few studies have been able to answer our question and induce the illusion to intend. In a novel paradigm addressing this question, participants were instructed to move a line on the computer screen by use of a phony brain-computer interface. Line movements were actually controlled by computer program. Demonstrating the illusion to intend, participants reported more intentions to move the line when it moved frequently than when it moved infrequently. Consistent with ideomotor theory, the finding illuminates the intimate liaisons among ideomotor processing, the sense of agency, and action production.

Subjective aspects of cognitive control at different stages of processing.

Although research on cognitive control has addressed the effects that different forms of cognitive interference have on behavior and the activities of certain brain regions, until recently, the effects of interference on subjective experience have not been addressed. We demonstrate that, at the level of the individual trial, participants can reliably introspect the subjective aspects (e.g., perceptions of difficulty, competition, and control) of responding in interference paradigms. Similar subjective effects were obtained for both expressed and unexpressed (subvocalized) actions. Few participants discerned the source of these effects. These basic findings illuminate aspects of cognitive control and cognitive effort. In addition, these data have implications for the study of response interference in affect and self-control, and they begin to address theories regarding the function of consciousness.