Multimodal deep neural decoding reveals highly resolved spatiotemporal profile of visual object representation in humans.

Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. Recently, neural decoding schemes, such as machine learning in functional magnetic resonance imaging (fMRI), has been employed to elucidate the underlying neural mechanisms. However, it remains unclear as to how spatially distributed brain regions temporally represent visual object categories and sub-categories. One promising strategy to address this issue is neural decoding with concurrently obtained neural response data of high spatial and temporal resolution. In this study, we explored the spatial and temporal organization of visual object representations using concurrent fMRI and electroencephalography (EEG), combined with neural decoding using deep neural networks (DNNs). We hypothesized that neural decoding by multimodal neural data with DNN would show high classification performance in visual object categorization (faces or non-face objects) and sub-categorization within faces and objects. Visualization of the fMRI DNN was more sensitive than that in the univariate approach and revealed that visual categorization occurred in brain-wide regions. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Combination of the two DNNs improved the classification performance for both categorization and sub-categorization compared with fMRI DNN or EEG DNN alone. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner, and provide strong evidence of the ability of multimodal deep learning to uncover spatiotemporal neural machinery in sensory processing.

Neurophysiological dynamics for psychological resilience: A view from the temporal axis.

When an individual is faced with adversity, the brain and body work cooperatively to adapt to it. This adaptive process is termed psychological resilience, and recent studies have identified several neurophysiological factors ("neurophysiological resilience"), such as monoamines, oscillatory brain activity, hemodynamics, autonomic activity, stress hormones, and immune systems. Each factor is activated in an interactive manner during specific time windows after exposure to stress. Thus, the differences in psychological resilience levels among individuals can be characterized by differences in the temporal dynamics of neurophysiological resilience. In this review, after briefly introducing the frequently used approaches in this research field and the well-known factors of neurophysiological resilience, we summarize the temporal dynamics of neurophysiological resilience. This viewpoint clarifies an important time window, the more-than-one-hour scale, but the neurophysiological dynamics during this window remain elusive. To address this issue, we propose exploring brain-wide oscillatory activities using concurrent functional magnetic resonance imaging (fMRI) and electroencephalogram (EEG) techniques.

Ventromedial prefrontal cortex contributes to performance success by controlling reward-driven arousal representation in amygdala.

When preparing for a challenging task, potential rewards can cause physiological arousal that may impair performance. In this case, it is important to control reward-driven arousal while preparing for task execution. We recently examined neural representations of physiological arousal and potential reward magnitude during preparation, and found that performance failure was explained by relatively increased reward representation in the left caudate nucleus and arousal representation in the right amygdala (Watanabe, et al., 2019). Here we examine how prefrontal cortex influences the amygdala and caudate to control reward-driven arousal. Ventromedial prefrontal cortex (VMPFC) exhibited activity that was negatively correlated with trial-wise physiological arousal change, which identified this region as a potential modulator of amygdala and caudate. Next we tested the VMPFC - amygdala - caudate effective network using dynamic causal modeling (Friston et al., 2003). Post-hoc Bayesian model selection (Friston and Penny, 2011) identified a model that best fit data, in which amygdala activation was suppressively controlled by the VMPFC only in success trials. Furthermore, fixed connectivity strength from VMPFC to amygdala explained individual task performance. These findings highlight the role of effective connectivity from VMPFC to amygdala in order to control arousal during preparation for successful performance.

Reward-Driven Arousal Impacts Preparation to Perform a Task via Amygdala-Caudate Mechanisms.

Preparing for a challenging task can increase physiological arousal, in particular when potential incentives are large (e.g., a solo musical performance in front of an audience). Here, we examine how potential reward and its influence on arousal, measured by pupil dynamics, are represented in the brain while preparing for a challenging task. We further ask how neural representations during preparation relate to actual performance. Trials resulting in performance failure were characterized by increased pupil dilation as a function of increasing reward magnitude during preparation. Such failure trials were also associated with activation of the right amygdala representing pupil dilation, and the left caudate representing reward magnitude. Notably, increases in functional connectivity between amygdala and caudate preceded performance failure. These findings highlight increased connectivity between neural regions representing reward and arousal in circumstances where reward-driven arousal impairs performance.

Neural mechanisms of social dominance.

In a group setting, individuals' perceptions of their own level of dominance or of the dominance level of others, and the ability to adequately control their behavior based on these perceptions are crucial for living within a social environment. Recent advances in neural imaging and molecular technology have enabled researchers to investigate the neural substrates that support the perception of social dominance and the formation of a social hierarchy in humans. At the systems' level, recent studies showed that dominance perception is represented in broad brain regions which include the amygdala, hippocampus, striatum, and various cortical networks such as the prefrontal, and parietal cortices. Additionally, neurotransmitter systems such as the dopaminergic and serotonergic systems, modulate and are modulated by the formation of the social hierarchy in a group. While these monoamine systems have a wide distribution and multiple functions, it was recently found that the Neuropeptide B/W contributes to the perception of dominance and is present in neurons that have a limited projection primarily to the amygdala. The present review discusses the specific roles of these neural regions and neurotransmitter systems in the perception of dominance and in hierarchy formation.

Effects of subconscious and conscious emotions on human cue-reward association learning.

Life demands that we adapt our behaviour continuously in situations in which much of our incoming information is emotional and unrelated to our immediate behavioural goals. Such information is often processed without our consciousness. This poses an intriguing question of whether subconscious exposure to irrelevant emotional information (e.g. the surrounding social atmosphere) affects the way we learn. Here, we addressed this issue by examining whether the learning of cue-reward associations changes when an emotional facial expression is shown subconsciously or consciously prior to the presentation of a reward-predicting cue. We found that both subconscious (0.027 s and 0.033 s) and conscious (0.047 s) emotional signals increased the rate of learning, and this increase was smallest at the border of conscious duration (0.040 s). These data suggest not only that the subconscious and conscious processing of emotional signals enhances value-updating in cue-reward association learning, but also that the computational processes underlying the subconscious enhancement is at least partially dissociable from its conscious counterpart.

Reward prediction error signal enhanced by striatum-amygdala interaction explains the acceleration of probabilistic reward learning by emotion.

Learning does not only depend on rationality, because real-life learning cannot be isolated from emotion or social factors. Therefore, it is intriguing to determine how emotion changes learning, and to identify which neural substrates underlie this interaction. Here, we show that the task-independent presentation of an emotional face before a reward-predicting cue increases the speed of cue-reward association learning in human subjects compared with trials in which a neutral face is presented. This phenomenon was attributable to an increase in the learning rate, which regulates reward prediction errors. Parallel to these behavioral findings, functional magnetic resonance imaging demonstrated that presentation of an emotional face enhanced reward prediction error (RPE) signal in the ventral striatum. In addition, we also found a functional link between this enhanced RPE signal and increased activity in the amygdala following presentation of an emotional face. Thus, this study revealed an acceleration of cue-reward association learning by emotion, and underscored a role of striatum-amygdala interactions in the modulation of the reward prediction errors by emotion.

A single nucleotide polymorphism of the neuropeptide B/W receptor-1 gene influences the evaluation of facial expressions.

Neuropeptide B/W receptor-1 (NPBWR1) is expressed in discrete brain regions in rodents and humans, with particularly strong expression in the limbic system, including the central nucleus of the amygdala. Recently, Nagata-Kuroiwa et al. reported that Npbwr1(-/-) mice showed changes in social behavior, suggesting that NPBWR1 plays important roles in the emotional responses of social interactions.The human NPBWR1 gene has a single nucleotide polymorphism at nucleotide 404 (404A>T; SNP rs33977775). This polymorphism results in an amino acid change, Y135F. The results of an in vitro experiment demonstrated that this change alters receptor function. We investigated the effect of this variation on emotional responses to stimuli of showing human faces with four categories of emotional expressions (anger, fear, happiness, and neutral). Subjects' emotional levels on seeing these faces were rated on scales of hedonic valence, emotional arousal, and dominance (V-A-D). A significant genotype difference was observed in valence evaluation; the 404AT group perceived facial expressions more pleasantly than did the 404AA group, regardless of the category of facial expression. Statistical analysis of each combination of [V-A-D and facial expression] also showed that the 404AT group tended to feel less submissive to an angry face than did the 404AA group. Thus, a single nucleotide polymorphism of NPBWR1 seems to affect human behavior in a social context.

Comparison of the choice effect and the distance effect in a number-comparison task by FMRI.

Behavioral and neurophysiological studies of numerical comparisons have shown a "distance effect," whereby smaller numerical distances between two digits are associated with longer response times and higher activity in the parietal region. In this experiment, we introduced a two-choice condition (between either the smaller/lower or the larger/higher of two digits) and examined its effect on brain activity by fMRI. We observed longer response times and greater activity with the choice of smaller numbers ("choice effect") in several brain regions including the right temporo-parietal region, (pre)cuneus, superior temporal sulcus, precentral gyrus, superior frontal gyrus, bilateral insula, and anterior cingulate cortex. These regions correspond to areas that have been suggested to play a role in attentional shift and response conflict. However, brain activity associated with the distance effect disappeared even though the behavioral distance effect remained. Despite the absence of the distance effect on brain activity, several areas changed activity in relation to response time, including regions that were reported to change activity in both a distance effect and a reaction-time-related manner. The result suggested that the level of task load may change the activity of regions that are responsible for magnitude detection.

Behavioral and near-infrared spectroscopy study of the effects of distance and choice in a number comparison task.

Extensive behavioral and neurophysiological numerical comparison studies have shown that response times are longer and parietal activities are stronger when the numerical distance between two digits is smaller (the distance effect). However, only a few behavioral studies have considered the effect of the choice of larger or smaller numerals in numerical comparisons. Using near-infrared spectroscopy (NIRS), we investigated the neural basis of choosing larger/smaller numerals in number comparison tasks in which subjects were required to choose a larger or smaller digit. Our results showed that choosing a smaller digit induced significantly longer response times (the choice effect) and stronger parietal activities. We also obtained significantly longer response times as the distance effect in accordance with previous works. However, NIRS data did not show any significant difference corresponding to distance effect. Our results and previous studies suggest that the parietal cortex is involved not only in measuring numerical quantities, but also in choosing a numerically larger/smaller quantity among the categories of choice. Potentials and limitations of NIRS were discussed.