Study of the neural dynamics for understanding communication in terms of complex hetero systems.

The purpose of the research project was to establish a new research area named "neural information science for communication" by elucidating its neural mechanism. The research was performed in collaboration with applied mathematicians in complex-systems science and experimental researchers in neuroscience. The project included measurements of brain activity during communication with or without languages and analyses performed with the help of extended theories for dynamical systems and stochastic systems. The communication paradigm was extended to the interactions between human and human, human and animal, human and robot, human and materials, and even animal and animal.

How sound symbolism is processed in the brain: a study on Japanese mimetic words.

Sound symbolism is the systematic and non-arbitrary link between word and meaning. Although a number of behavioral studies demonstrate that both children and adults are universally sensitive to sound symbolism in mimetic words, the neural mechanisms underlying this phenomenon have not yet been extensively investigated. The present study used functional magnetic resonance imaging to investigate how Japanese mimetic words are processed in the brain. In Experiment 1, we compared processing for motion mimetic words with that for non-sound symbolic motion verbs and adverbs. Mimetic words uniquely activated the right posterior superior temporal sulcus (STS). In Experiment 2, we further examined the generalizability of the findings from Experiment 1 by testing another domain: shape mimetics. Our results show that the right posterior STS was active when subjects processed both motion and shape mimetic words, thus suggesting that this area may be the primary structure for processing sound symbolism. Increased activity in the right posterior STS may also reflect how sound symbolic words function as both linguistic and non-linguistic iconic symbols.

Stimulus-dependent adjustment of reward prediction error in the midbrain.

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus-reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.

Looking to the future: automatic regulation of attention between current performance and future plans.

We investigated neuro-cognitive mechanisms involved with coordination of attention between current task performance and future action plans in prospective memory. We developed a novel task paradigm with continuous performance of a prospective memory task, where trial intervals of prospective memory targets were systematically manipulated in a periodic cycle of expanding and contracting target intervals. We found that subjects' behaviour was significantly modulated without awareness of this temporal sequence of the targets: remembering to perform a prospective memory response to target events was more successful and faster in the expanding target interval phase, at the cost of lower and slower performance of ongoing tasks, while an opposite direction of this trade-off effect was observed in the contracting target interval phase. By using functional magnetic resonance imaging (fMRI), we identified the similar trade-off effect in activations in the anterior medial prefrontal cortices (activation elevation at the target responses as well as deactivation at the ongoing responses in the expanding phase as compared with the contracting phase). The opposite direction of the trade-off was observed in the anterior cingulate cortex. These results show a clear case in which attention between current task performance and future action plans in prospective memory tasks is automatically regulated without particular task instructions or strategic control processes initiated by subjects. We suggest that medial areas of the frontal cortex specifically mediate the automatic coordination of attentional resources between current task performance and future action plans.

Reactive mechanism of cognitive control system.

The prefrontal cortex (PFC) is thought to modulate the neural network state in favor of the processing of task-relevant sensory information prior to the presentation of sensory stimuli. However, this proactive control mechanism cannot always optimize the network state because of intrinsic fluctuation of neural activity upon arrival of sensory information. In the present study, we have investigated an additional control mechanism, in which the control process to regulate the behavior is adjusted to the trial-by-trial fluctuation in neural representations of sensory information. We asked normal human subjects to perform a variant of the Stroop task. Using functional magnetic resonance imaging, we isolated cognitive conflict at a sensory processing stage on a single-trial basis by calculating the difference in activation between task-relevant and task-irrelevant sensory areas. Activation in the dorsolateral PFC (DLPFC) covaried with the neural estimate of sensory conflict only on incongruent trials. Also, the coupling between the DLPFC and anterior cingulate cortex (ACC) was tighter on high-sensory conflict trials with fast response. The results suggest that although detection of sensory conflict is achieved by the DLPFC, online behavioral adjustment is achieved by interactive mechanisms between the DLPFC and ACC.

Task-specific signal transmission from prefrontal cortex in visual selective attention.

Our voluntary behaviors are thought to be controlled by top-down signals from the prefrontal cortex that modulate neural processing in the posterior cortices according to the behavioral goal. However, we have insufficient evidence for the causal effect of the top-down signals. We applied a single-pulse transcranial magnetic stimulation over the human prefrontal cortex and measured the strength of the top-down signals as an increase in the efficiency of neural impulse transmission. The impulse induced by the stimulation transmitted to different posterior visual areas depending on the domain of visual features to which subjects attended. We also found that the amount of impulse transmission was associated with the level of attentional preparation and the performance of visual selective-attention tasks, consistent with the causal role of prefrontal top-down signals.

Human posterior parietal cortex maintains color, shape and motion in visual short-term memory.

We used functional magnetic resonance imaging (fMRI) to investigate the neural substrate of visual short-term memory for objects defined by features processed in the dorsal and the ventral visual streams. Here we adopted the conventional delayed recognition task, whereas in addition to the more commonly used visual features of color and shape, motion direction was applied to define an item. Our behavioral results indicated that the capacity limit of visual short-term memory of motion direction was approximately two, which was significantly lower than those of color and shape, about three or four. We also found that storage capacity was significantly reduced when subjects were required to retain all three features superimposed in space. Meanwhile, fMRI results revealed that activity in the posterior part of the superior parietal lobe was memory-load dependent for all three features indicating that it collects and stores visual information from both the two visual processing streams, whereas the anterior part was load dependent only for motion.

Neural correlates of true memory, false memory, and deception.

We used functional magnetic resonance imaging (fMRI) to determine whether neural activity can differentiate between true memory, false memory, and deception. Subjects heard a series of semantically related words and were later asked to make a recognition judgment of old words, semantically related nonstudied words (lures for false recognition), and unrelated new words. They were also asked to make a deceptive response to half of the old and unrelated new words. There were 3 main findings. First, consistent with the notion that executive function supports deception, 2 types of deception (pretending to know and pretending not to know) recruited prefrontal activity. Second, consistent with the sensory reactivation hypothesis, the difference between true recognition and false recognition was found in the left temporoparietal regions probably engaged in the encoding of auditorily presented words. Third, the left prefrontal cortex was activated during pretending to know relative to correct rejection and false recognition, whereas the right anterior hippocampus was activated during false recognition relative to correct rejection and pretending to know. These findings indicate that fMRI can detect the difference in brain activity between deception and false memory despite the fact that subjects respond with "I know" to novel events in both processes.

Differential involvement of regions of rostral prefrontal cortex (Brodmann area 10) in time- and event-based prospective memory.

Rostral prefrontal cortex (approximating Brodmann area 10) has been shown repeatedly to have a role in the maintenance and realization of delayed intentions that are triggered by event cues (i.e., event-based prospective memory). The cerebral organization of the processes associated with the use of time cues (time-based prospective memory) has however received less attention. In two positron emission tomography (PET) studies we therefore examined brain activity associated with time- and event-based prospective memory tasks. In the time-based condition of the first study, young healthy volunteers were asked to make a prospective response based on their self-estimation of the passage of time while engaged in an attention-demanding ongoing activity. In the time-based condition of the second study, participants had a clock available in the ongoing task display and did not need to estimate the time for the prospective response. In the event-based condition of both studies, participants were asked to make a prospective response when prospective cues were presented in ongoing trials. Both studies showed activation differences in rostral prefrontal cortex according to whether the task was time- or event-based. In study one, an area of left superior frontal gyrus was more active in the time-based condition. In study two, three rostral prefrontal regions were more active in the time-based condition: right superior frontal gyrus, anterior medial frontal lobe and anterior cingulate gyrus. A region in left superior frontal gyrus, different from the area found in the first study, was more active in the event-based condition. These results indicate involvement of multiple brain regions of rostral prefrontal cortex in time- and event-based prospective memory. The results are interpreted as reflecting the differing processing demands made by event- or time-based prospective memory tasks, and the differing demands of time-based tasks according to whether a clock is present as an aid.

Neural correlates of context memory with real-world events.

There has been little evidence for the difference in the retrieval processes of when and where something happened, one of the important factors in understanding episodic memory. We used positron emission tomography (PET) to identify the neural networks associated with temporal and spatial context memory of events experienced under experimental conditions similar to those of everyday life. Before PET, subjects experienced 36 events. The events were divided into four groups of nine each. The subjects experienced the first two groups of events before a 15-min recess and the other two after the recess; they experienced the first and last groups of events in one room, took a recess in another room, and experienced the second and third groups in a different room. During PET, the subjects were scanned under three different retrieval tasks: a time-retrieval task, a place-retrieval task, and a simple recognition task. The results showed that the retrieval of time and space, compared with the simple recognition, was associated with activity in substantially different regions as well as a common region: time retrieval with the posterior part of the right orbitofrontal cortex and left inferior parietal lobule, place retrieval with two regions in right parietal association cortex, right posterior cingulate gyrus, left precentral gyrus, and right cerebellum, and both with the right inferior frontal gyrus. These findings indicate that there are unique areas, in addition to a common area, for retrieving temporal and spatial context.

Changes in brain activation patterns associated with learning of Korean words by Japanese: an fMRI study.

We used functional magnetic resonance imaging (fMRI) to explore the change in brain activation associated with the learning of Korean words written in Han-gul characters (K-words) by young Japanese at two stages. Subjects were 12 right-handed native Japanese without previous knowledge of Korean words and characters. On the first day they were taught the pronunciation and meaning of 20 K-words. Then, after the first fMRI session (on day 2), they were given a set of 20 cards with the words and corresponding photographs. They also received a tape and were instructed to memorize the 20 K-words by studying them every day until the day of the second fMRI session (day 16). During the fMRI sessions, 20 Japanese words written in kana syllabograms (J-words) and the 20 previously presented K-words, as well as 20 new K-words (Kn-words) were presented visually for silent reading. The first J-word reading, relative to the first K-word reading, showed activation in the left angular gyrus. K-word reading relative to J-word reading during both sessions showed activation in occipital regions. Within these activated areas, session by condition interaction was found only in the left angular gyrus. The interaction between session and condition resulted from the fact that the differences in blood oxygenation-level-dependent signals between K-words and J-words and between Kn-words and J-words were significantly greater in the first session than in the second session. From the results, we concluded that patterns of brain activation changed as the memory of the 20 K-words became fixed through daily practice and that reading of both Korean words and Japanese syllabograms engaged the left angular gyrus.

Thinking of the future and past: the roles of the frontal pole and the medial temporal lobes.

Human lesion data have indicated that the frontal polar area might be critically involved in having an insight into one's future. Retrospective memory mediated by medial temporal lobes and related structures, on the other hand, could be used to extract one's future prospects efficiently. In the present study, we investigated the roles of these two brain structures in thinking of the future and past by using positron emission tomography (PET) and a naturalistic task setting. We measured regional cerebral blood flow (rCBF) in healthy subjects while they were talking about their future prospects or past experiences, with regard to two different temporal windows (in years or days). Many areas in the frontal and the medial temporal lobes were activated during the future and past tasks compared with a control task requiring semantic retrieval. Among these, areas in anteromedial frontal pole showed greater activation during the future tasks than during the past tasks, showing significant effect of temporal distance from the present. Most areas in the medial temporal lobes showed greater or equivalent level of activations during the future tasks compared with the past tasks. The present results suggest that thinking of the future is closely related to retrospective memory, but that specific areas in the frontal pole and the medial temporal lobes are more involved with thinking of the future than that of the past.

Time-dependent contribution of the hippocampal complex when remembering the past: a PET study.

Previous studies of brain-damaged patients and functional neuro-imaging have consistently shown the importance of the hippocampal complex, i.e. the hippocampus and parahippocampal gyrus, in episodic memory retrieval. We wished to determine whether patterns of brain activation during memory retrieval as measured by PET are same or different when the oldness of a to-be-retrieved episode is manipulated. Using cue words, subjects remembered related episodes from three periods of their life, childhood, adolescence and recent period. The results showed an increase of parahippocampal activities during recall of episodes from childhood and recent period, but not from adolescence. These data suggest a possibility of time-dependent hippocampal contribution in episodic recall, which cannot be understood in simple terms of recent remote memory dichotomy.

Neural basis of temporal context memory: a functional MRI study.

Temporal context information is crucial to understanding human episodic memory. Human lesion and neuroimaging data indicate that prefrontal regions are important for retrieving temporal context memory, although the exact nature of their involvement is still unclear. We employed functional magnetic resonance imaging (fMRI) to elucidate the neural basis of two kinds of temporal context memory: the temporal order of items between lists and within a list. On the day of the fMRI experiment, subjects memorized a list of 30 pictures in the morning and another list of 30 pictures in the afternoon. During the scanning session, the subjects performed three tasks. In a between-lists task, they were asked to judge the temporal order between two items that had been presented in different lists. In a within-list task, they were asked to judge the temporal order between two items that had been presented in a single list. We found bilateral prefrontal activities during these two temporal context memory tasks compared with a simple item-recognition task. Furthermore, in direct comparison between these two tasks, we found differential prefrontal activities. Thus, right prefrontal activity was associated with temporal order judgment of items between lists, whereas left prefrontal activity was related to temporal order judgment of items within a list. These results indicate that retrieval processes of two kinds of temporal context memory are supported by different, but overlapping, sets of cerebral regions. We speculate that this reflects different cognitive processes for retrieving temporal context memory between separate episodes and within a single episode.

Encoding-related brain activity during deep processing of verbal materials: a PET study.

The recent advent of neuroimaging techniques provides an opportunity to examine brain regions related to a specific memory process such as episodic memory encoding. There is, however, a possibility that areas active during an assumed episodic memory encoding task, compared with a control task, involve not only areas directly relevant to episodic memory encoding processes but also areas associated with other cognitive processes for on-line information. We used positron emission tomography (PET) to differentiate these two kinds of regions. Normal volunteers were engaged in deep (semantic) or shallow (phonological) processing of new or repeated words during PET. Results showed that deep processing, compared with shallow processing, resulted in significantly better recognition performance and that this effect was associated with activation of various brain areas. Further analyses revealed that there were regions directly relevant to episodic memory encoding in the anterior part of the parahippocampal gyrus, inferior frontal gyrus, supramarginal gyrus, anterior cingulate gyrus, and medial frontal lobe in the left hemisphere. Our results demonstrated that several regions, including the medial temporal lobe, play a role in episodic memory encoding.

Medial temporal lobe activation during context-dependent relational processes in episodic retrieval: an fMRI study. Functional magnetic resonance imaging.

Previous studies have reported that the medial temporal lobe (MTL) structures contribute to the processing of relations among multiple stimuli in episodic encoding. There have been few studies, however, on the episodic retrieval requiring processing of relations among multiple components that was involved in our events. We used functional magnetic resonance imaging (fMRI) to investigate neural activities during the retrieval of relations within an organized episode and the recognition of an episodic component. Healthy, normal participants memorized 50 four-scene comic strips before fMRI scanning. In the retrieval phase with fMRI scanning, participants were engaged in three tasks: a visual identification (VI) task, a story recall (SR) task, and a picture recognition (PRe) task. In the VI task, participants were asked to judge whether they could identify at least one female character in the two scenes presented vertically. In the SR task, participants were shown the first and last scenes from strips memorized previously and asked to judge whether or not the two scenes were from the same strip. In the PRe task, participants were shown two scenes and asked to judge whether they both belonged to the memorized scenes. The two contrasts of SR with VI and PRe with VI demonstrated some commonly activated areas, such as the bilateral middle frontal gyrus and cerebellum. More importantly, the SR task differentially activated the bilateral parahippocampal gyrus, whereas the PRe task differentially activated right prefrontal areas, including the inferior frontal and precentral gyri. The results suggest that the activity of the MTL structures may be strongly associated with episodic memory retrieval requiring context-dependent relational processing.

Neural basis of the retrieval of people's names: evidence from brain-damaged patients and fMRI.

The aim of this study was to identify the neuroanatomical basis of the retrieval of people's names. Lesion data showed that patients with language-dominant temporal lobectomy had impairments in their ability to retrieve familiar and newly learned people's names, whereas patients with language-nondominant temporal lobectomy had difficulty retrieving newly learned people's names. Functional magnetic resonance imaging experiments revealed activations in the left temporal polar region during the retrieval of familiar and newly learned people's names, and in the right superior temporal and bilateral prefrontal cortices during the retrieval of newly learned information from face cues. These data provide new evidence that the left anterior temporal region is crucial for the retrieval of people's names irrespective of their familiarity and that the right superior temporal and bilateral prefrontal areas are crucial for the process of associating newly learned people's faces and names.

The role of the basal forebrain in episodic memory retrieval: a positron emission tomography study.

Human lesion data indicate that the basal forebrain or orbitofrontal cortex, or both, as well as medial temporal and diencephalic structures, is important for normal memory and that its disruption causes the pure amnesic syndrome, in which episodic memory is grossly impaired while other kinds of memory remain preserved. Among these critical areas, functional imaging studies have so far failed to detect activation of the basal forebrain, although activation in the nearby orbitofrontal cortex has been reported during episodic memory retrieval. We employed positron emission tomography to elucidate the neural basis of episodic memory recall utilizing two types of time cues and successfully detected activity in the basal forebrain for the first time. Specifically, recall of previously memorized words from temporal cues was associated with activity in the basal forebrain, right middle frontal gyrus, right superior temporal gyrus, and posterior cingulate gyrus, whereas their recall from person cues was associated with activity in the left insula, right middle frontal gyrus, and posterior cingulate gyrus. Furthermore, percentage increases of regional blood flow in the basal forebrain were correlated with behavioral data of successful recall. Our results provide clear evidence that the human basal forebrain has a specific role in episodic memory recall, especially that from time-contextual information.