Threat Detection in Nearby Space Mobilizes Human Ventral Premotor Cortex, Intraparietal Sulcus, and Amygdala.

In the monkey brain, the precentral gyrus and ventral intraparietal area are two interconnected brain regions that form a system for detecting and responding to events in nearby "peripersonal" space (PPS), with threat detection as one of its major functions. Behavioral studies point toward a similar defensive function of PPS in humans. Here, our aim was to find support for this hypothesis by investigating if homolog regions in the human brain respond more strongly to approaching threatening stimuli. During fMRI scanning, naturalistic social stimuli were presented in a 3D virtual environment. Our results showed that the ventral premotor cortex and intraparietal sulcus responded more strongly to threatening stimuli entering PPS. Moreover, we found evidence for the involvement of the amygdala and anterior insula in processing threats. We propose that the defensive function of PPS may be supported by a subcortical circuit that sends information about the relevance of the stimulus to the premotor cortex and intraparietal sulcus, where action preparation is facilitated when necessary.

First-Person Virtual Embodiment Modulates the Cortical Network that Encodes the Bodily Self and Its Surrounding Space during the Experience of Domestic Violence.

Social aggression, such as domestic violence, has been associated with a reduced ability to take on others' perspectives. In this naturalistic imaging study, we investigated whether training human participants to take on a first-person embodied perspective during the experience of domestic violence enhances the identification with the victim and elicits brain activity associated with the monitoring of the body and surrounding space and the experience of threat. We combined fMRI measurements with preceding virtual reality exposure from either first-person perspective (1PP) or third-person perspective (3PP) to manipulate whether the domestic abuse stimulus was perceived as directed to oneself or another. We found that 1PP exposure increased body ownership and identification with the virtual victim. Furthermore, when the stimulus was perceived as directed toward oneself, the brain network that encodes the bodily self and its surrounding space was more strongly synchronized across participants and connectivity increased from premotor cortex (PM) and intraparietal sulcus towards superior parietal lobe. Additionally, when the stimulus came near the body, brain activity in the amygdala (AMG) strongly synchronized across participants. Exposure to 3PP reduced synchronization of brain activity in the personal space network, increased modulation of visual areas and strengthened functional connectivity between PM, supramarginal gyrus and primary visual cortex. In conclusion, our results suggest that 1PP embodiment training enhances experience from the viewpoint of the virtual victim, which is accompanied by synchronization in the fronto-parietal network to predict actions toward the body and in the AMG to signal the proximity of the stimulus.

Amygdala responds to direct gaze in real but not in computer-generated faces.

Computer-generated (CG) faces are an important visual interface for human-computer interaction in social contexts. Here we investigated whether the human brain processes emotion and gaze similarly in real and carefully matched CG faces. Real faces evoked greater responses in the fusiform face area than CG faces, particularly for fearful expressions. Emotional (angry and fearful) facial expressions evoked similar activations in the amygdala in real and CG faces. Direct as compared with averted gaze elicited greater fMRI responses in the amygdala regardless of facial expression but only for real and not for CG faces. We observed an interaction effect between gaze and emotion (i.e., the shared signal effect) in the right posterior temporal sulcus and other regions, but not in the amygdala, and we found no evidence for different shared signal effects in real and CG faces. Taken together, the present findings highlight similarities (emotional processing in the amygdala) and differences (overall processing in the fusiform face area, gaze processing in the amygdala) in the neural processing of real and CG faces.

Virtual reality and the new psychophysics.

Virtual reality (VR) promises methodological rigour with the extra benefit of allowing us to study the context-dependent behaviour of individuals in their natural environment. Pan and Hamilton (2018, Br. J. Psychol.) provide a useful overview of methodological recommendations for using VR. Here, we highlight some other aspects of the use of VR. Our first argument is that VR can be useful by virtue of its differences from the normal perceptual environment. That is, by virtue of its relative non-realism and poverty of its perceptual elements, it can actually offer increased clarity with respect to the features of interest for the researcher. Our second argument is that VR exerts its measurable influence more by eliciting an acceptance of the virtual world (i.e., 'suspension of disbelief') rather than by eliciting a true belief of the realism of the VR environment. We conclude by providing a novel suggestion for combining neuroimaging methods with embodied VR that relies on the suspension of disbelief.

fMRI-based Multivariate Pattern Analyses Reveal Imagery Modality and Imagery Content Specific Representations in Primary Somatosensory, Motor and Auditory Cortices.

Previous studies have shown that the early visual cortex contains content-specific representations of stimuli during visual imagery, and that these representational patterns of imagery content have a perceptual basis. To date, there is little evidence for the presence of a similar organization in the auditory and tactile domains. Using fMRI-based multivariate pattern analyses we showed that primary somatosensory, auditory, motor, and visual cortices are discriminative for imagery of touch versus sound. In the somatosensory, motor and visual cortices the imagery modality discriminative patterns were similar to perception modality discriminative patterns, suggesting that top-down modulations in these regions rely on similar neural representations as bottom-up perceptual processes. Moreover, we found evidence for content-specific representations of the stimuli during auditory imagery in the primary somatosensory and primary motor cortices. Both the imagined emotions and the imagined identities of the auditory stimuli could be successfully classified in these regions.

Brain-based decoding of mentally imagined film clips and sounds reveals experience-based information patterns in film professionals.

In the perceptual domain, it has been shown that the human brain is strongly shaped through experience, leading to expertise in highly-skilled professionals. What has remained unclear is whether specialization also shapes brain networks underlying mental imagery. In our fMRI study, we aimed to uncover modality-specific mental imagery specialization of film experts. Using multi-voxel pattern analysis we decoded from brain activity of professional cinematographers and sound designers whether they were imagining sounds or images of particular film clips. In each expert group distinct multi-voxel patterns, specific for the modality of their expertise, were found during classification of imagery modality. These patterns were mainly localized in the occipito-temporal and parietal cortex for cinematographers and in the auditory cortex for sound designers. We also found generalized patterns across perception and imagery that were distinct for the two expert groups: they involved frontal cortex for the cinematographers and temporal cortex for the sound designers. Notably, the mental representations of film clips and sounds of cinematographers contained information that went beyond modality-specificity. We were able to successfully decode the implicit presence of film genre from brain activity during mental imagery in cinematographers. The results extend existing neuroimaging literature on expertise into the domain of mental imagery and show that experience in visual versus auditory imagery can alter the representation of information in modality-specific association cortices.

Is it the real deal? Perception of virtual characters versus humans: an affective cognitive neuroscience perspective.

Recent developments in neuroimaging research support the increased use of naturalistic stimulus material such as film, avatars, or androids. These stimuli allow for a better understanding of how the brain processes information in complex situations while maintaining experimental control. While avatars and androids are well suited to study human cognition, they should not be equated to human stimuli. For example, the uncanny valley hypothesis theorizes that artificial agents with high human-likeness may evoke feelings of eeriness in the human observer. Here we review if, when, and how the perception of human-like avatars and androids differs from the perception of humans and consider how this influences their utilization as stimulus material in social and affective neuroimaging studies. First, we discuss how the appearance of virtual characters affects perception. When stimuli are morphed across categories from non-human to human, the most ambiguous stimuli, rather than the most human-like stimuli, show prolonged classification times and increased eeriness. Human-like to human stimuli show a positive linear relationship with familiarity. Secondly, we show that expressions of emotions in human-like avatars can be perceived similarly to human emotions, with corresponding behavioral, physiological and neuronal activations, with exception of physical dissimilarities. Subsequently, we consider if and when one perceives differences in action representation by artificial agents versus humans. Motor resonance and predictive coding models may account for empirical findings, such as an interference effect on action for observed human-like, natural moving characters. However, the expansion of these models to explain more complex behavior, such as empathy, still needs to be investigated in more detail. Finally, we broaden our outlook to social interaction, where virtual reality stimuli can be utilized to imitate complex social situations.

Enactive cinema paves way for understanding complex real-time social interaction in neuroimaging experiments.

We outline general theoretical and practical implications of what we promote as enactive cinema for the neuroscientific study of online socio-emotional interaction. In a real-time functional magnetic resonance imaging (rt-fMRI) setting, participants are immersed in cinematic experiences that simulate social situations. While viewing, their physiological reactions-including brain responses-are tracked, representing implicit and unconscious experiences of the on-going social situations. These reactions, in turn, are analyzed in real-time and fed back to modify the cinematic sequences they are viewing while being scanned. Due to the engaging cinematic content, the proposed setting focuses on living-by in terms of shared psycho-physiological epiphenomena of experience rather than active coping in terms of goal-oriented motor actions. It constitutes a means to parametrically modify stimuli that depict social situations and their broader environmental contexts. As an alternative to studying the variation of brain responses as a function of a priori fixed stimuli, this method can be applied to survey the range of stimuli that evoke similar responses across participants at particular brain regions of interest.

Integration of "what" and "where" in frontal cortex during visual imagery of scenes.

Imagination is a key function for many human activities, such as reminiscing, learning, or planning. Unravelling its neuro-biological basis is paramount to grasp the essence of our thoughts. Previous neuroimaging studies have identified brain regions subserving the visualisation of "what?" (e.g. faces or objects) and "where?" (e.g. spatial layout) content of mental images. However, the functional role of a common set of involved regions - the frontal regions - and their interplay with the "what" and "where" regions, has remained largely unspecified. This study combines functional MRI and electroencephalography to examine the full-brain network that underlies the visual imagery of complex scenes and to investigate the spectro-temporal properties of its nodes, especially of the frontal cortex. Our results indicate that frontal regions integrate the "what" and "where" content of our thoughts into one visually imagined scene. We link early synchronisation of anterior theta and beta oscillations to regional activation of right and central frontal cortices, reflecting retrieval and integration of information. These frontal regions orchestrate remote occipital-temporal regions (including calcarine sulcus and parahippocampal gyrus) that encode the detailed representations of the objects, and parietal "where" regions that encode the spatial layout into forming one coherent mental picture. Specifically the mesial superior frontal gyrus appears to have a principal integrative role, as its activity during the visualisation of the scene predicts subsequent performance on the imagery task.

Predicting EEG single trial responses with simultaneous fMRI and relevance vector machine regression.

The combination of electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) has been proposed as a tool to study brain dynamics with both high temporal and high spatial resolution. Integration through EEG-fMRI trial-by-trial coupling has been proposed as a method to combine the different data sets and achieve temporal expansion of the fMRI data (Eichele et al., 2005). To fully benefit of this type of analysis simultaneous EEG-fMRI acquisitions are necessary (Debener et al., 2006). Here we address the issue of predicting the signal in one modality using information from the other modality. We use multivariate Relevance Vector Machine (RVM) regression to "learn" the relation between fMRI activation patterns and simultaneously acquired EEG responses in the context of a complex cognitive task entailing an auditory cue, visual mental imagery and a control visual target. We show that multivariate regression is a valuable approach for predicting evoked and induced oscillatory EEG responses from fMRI time series. Prediction of EEG from fMRI is largely influenced by the overall filtering effects of the hemodynamic response function. However, a detailed analysis of the auditory evoked responses shows that there is a small but significant contribution of single trial modulations that can be exploited for linking spatially-distributed patterns of fMRI activation to specific components of the simultaneously-recorded EEG signal.

Multimodal imaging: an evaluation of univariate and multivariate methods for simultaneous EEG/fMRI.

The combination of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) has been proposed as a tool to study brain dynamics with both high temporal and high spatial resolution. Multimodal imaging techniques rely on the assumption of a common neuronal source for the different recorded signals. In order to maximally exploit the combination of these techniques, one needs to understand the coupling (i.e., the relation) between electroencephalographic (EEG) and fMRI blood oxygen level-dependent (BOLD) signals. Recently, simultaneous EEG-fMRI measurements have been used to investigate the relation between the two signals. Previous attempts at the analysis of simultaneous EEG-fMRI data reported significant correlations between regional BOLD activations and modulation of both event-related potential (ERP) and oscillatory EEG power, mostly in the alpha but also in other frequency bands. Beyond the correlation of the two measured brain signals, the relevant issue we address here is the ability of predicting the signal in one modality using information from the other modality. Using multivariate machine learning-based regression, we show how it is possible to predict EEG power oscillations from simultaneously acquired fMRI data during an eyes-open/eyes-closed task using either the original channels or the underlying cortically distributed sources as the relevant EEG signal for the analysis of multimodal data.