Reinforcement Learning Model With Dynamic State Space Tested on Target Search Tasks for Monkeys: Extension to Learning Task Events.

Learning is a crucial basis for biological systems to adapt to environments. Environments include various states or episodes, and episode-dependent learning is essential in adaptation to such complex situations. Here, we developed a model for learning a two-target search task used in primate physiological experiments. In the task, the agent is required to gaze one of the four presented light spots. Two neighboring spots are served as the correct target alternately, and the correct target pair is switched after a certain number of consecutive successes. In order for the agent to obtain rewards with a high probability, it is necessary to make decisions based on the actions and results of the previous two trials. Our previous work achieved this by using a dynamic state space. However, to learn a task that includes events such as fixation to the initial central spot, the model framework should be extended. For this purpose, here we propose a "history-in-episode architecture." Specifically, we divide states into episodes and histories, and actions are selected based on the histories within each episode. When we compared the proposed model including the dynamic state space with the conventional SARSA method in the two-target search task, the former performed close to the theoretical optimum, while the latter never achieved target-pair switch because it had to re-learn each correct target each time. The reinforcement learning model including the proposed history-in-episode architecture and dynamic state scape enables episode-dependent learning and provides a basis for highly adaptable learning systems to complex environments.

Dynamic Axis-Tuned Cells in the Monkey Lateral Prefrontal Cortex during a Path-Planning Task.

The lateral prefrontal cortex (lPFC) plays a crucial role in the cognitive processes known as executive functions, which are necessary for the planning of goal-directed behavior in complex and constantly changing environments. To adapt to such environments, the lPFC must use its neuronal resources in a flexible manner. To investigate the mechanism by which lPFC neurons code directional information flexibly, the present study explored the tuning properties and time development of lPFC neurons in male Japanese monkeys during a path-planning task, which required them to move a cursor to a final goal in a stepwise manner within a checkerboard-like maze. We identified "axis-tuned" cells that preferred two opposing directions of immediate goals (i.e., vertical and horizontal directions). Among them, a considerable number of these axis-tuned cells dynamically transformed from vector tuning to a single final-goal direction. We also found that the activities of axis-tuned cells, especially pyramidal neurons, were also modulated by the abstract sequence patterns that the animals were to execute. These findings suggest that the axis-tuned cells change what they code (the type of behavioral goal) as well as how they code (their tuning shapes) so that the lPFC can represent a large number of possible actions or sequences with limited neuronal resources. The dynamic axis-tuned cells must reflect the flexible coding of behaviorally relevant information at the single neuron level by the lPFC to adapt to uncertain environments.SIGNIFICANCE STATEMENT The lateral PFC (lPFC) plays a crucial role in the planning of goal-directed behavior in uncertain environments. To adapt to such situations, the lPFC must flexibly encode behaviorally relevant information. Here, we investigated the goal-tuning properties of neuronal firing in the monkey lPFC during a path-planning task. We identified axis-tuned cells that preferred "up-down" or "left-right" immediate goals, and found that many were dynamically transformed from vector tuning to a final-goal direction. The activities of neurons, especially pyramidal neurons, were also modulated by the abstract sequence patterns. Our findings suggest that PFC neurons can alter not only what they code (behavioral goal) but also how they code (tuning shape) when coping with unpredictable environments with limited neuronal resources.

Surprise signals in the supplementary eye field: rectified prediction errors drive exploration-exploitation transitions.

Visual search is coordinated adaptively by monitoring and predicting the environment. The supplementary eye field (SEF) plays a role in oculomotor control and outcome evaluation. However, it is not clear whether the SEF is involved in adjusting behavioral modes based on preceding feedback. We hypothesized that the SEF drives exploration-exploitation transitions by generating "surprise signals" or rectified prediction errors, which reflect differences between predicted and actual outcomes. To test this hypothesis, we introduced an oculomotor two-target search task in which monkeys were required to find two valid targets among four identical stimuli. After they detected the valid targets, they exploited their knowledge of target locations to obtain a reward by choosing the two valid targets alternately. Behavioral analysis revealed two distinct types of oculomotor search patterns: exploration and exploitation. We found that two types of SEF neurons represented the surprise signals. The error-surprise neurons showed enhanced activity when the monkey received the first error feedback after the target pair change, and this activity was followed by an exploratory oculomotor search pattern. The correct-surprise neurons showed enhanced activity when the monkey received the first correct feedback after an error trial, and this increased activity was followed by an exploitative, fixed-type search pattern. Our findings suggest that error-surprise neurons are involved in the transition from exploitation to exploration and that correct-surprise neurons are involved in the transition from exploration to exploitation.

Increased firing irregularity as an emergent property of neural-state transition in monkey prefrontal cortex.

Flexible behaviors are organized by complex neural networks in the prefrontal cortex. Recent studies have suggested that such networks exhibit multiple dynamical states, and can switch rapidly from one state to another. In many complex systems such as the brain, the early-warning signals that may predict whether a critical threshold for state transitions is approaching are extremely difficult to detect. We hypothesized that increases in firing irregularity are a crucial measure for predicting state transitions in the underlying neuronal circuits of the prefrontal cortex. We used both experimental and theoretical approaches to test this hypothesis. Experimentally, we analyzed activities of neurons in the prefrontal cortex while monkeys performed a maze task that required them to perform actions to reach a goal. We observed increased firing irregularity before the activity changed to encode goal-to-action information. Theoretically, we constructed theoretical generic neural networks and demonstrated that changes in neuronal gain on functional connectivity resulted in a loss of stability and an altered state of the networks, accompanied by increased firing irregularity. These results suggest that assessing the temporal pattern of neuronal fluctuations provides important clues regarding the state stability of the prefrontal network. We also introduce a novel scheme that the prefrontal cortex functions in a metastable state near the critical point of bifurcation. According to this scheme, firing irregularity in the prefrontal cortex indicates that the system is about to change its state and the flow of information in a flexible manner, which is essential for executive functions. This metastable and/or critical dynamical state of the prefrontal cortex may account for distractibility and loss of flexibility in the prefrontal cortex in major mental illnesses such as schizophrenia.

Neural basis of impaired cognitive flexibility in patients with anorexia nervosa.

BACKGROUND: Impaired cognitive flexibility in anorexia nervosa (AN) causes clinical problems and makes the disease hard to treat, but its neural basis has yet to be fully elucidated. The purpose of this study was to evaluate the brain activity of individuals with AN while performing a task requiring cognitive flexibility on the Wisconsin Card Sorting Test (WCST), which is one of the most frequently used neurocognitive measures of cognitive flexibility and problem-solving ability. METHODS: Participants were 15 female AN patients and 15 age- and intelligence quotient-matched healthy control women. Participants completed the WCST while their brain activity was measured by functional magnetic resonance imaging during the task. Brain activation in response to set shifting error feedback and the correlation between such brain activity and set shifting performance were analyzed. RESULTS: The correct rate on the WCST was significantly poorer for AN patients than for controls. Patients showed poorer activity in the right ventrolateral prefrontal cortex and bilateral parahippocampal cortex on set shifting than controls. Controls showed a positive correlation between correct rate and ventrolateral prefrontal activity in response to set shifting whereas patients did not. CONCLUSION: These findings suggest dysfunction of the ventrolateral prefrontal cortex and parahippocampal cortex as a cause of impaired cognitive flexibility in AN patients.

Comparison of caregiver strain in Parkinson's disease between Yamagata, Japan, and Maryland, The United States.

BACKGROUND: Japan and the United States (US) have different cultures of caregiving including differences in family structure and social programs that may influence caregiver strain. Differences in caregiver strain between regions in Japan and in the US have not been investigated in patient-spouse dyads in PD. OBJECTIVES: To compare caregiver strain in spouses of PD patients between Yamagata, Japan and Maryland, US. Correlations between caregiver strain and patient/spousal variables are also examined. METHODS: In Yamagata and Maryland, spouses of patients with PD completed questionnaires assessing caregiver strain. Patients and spouses completed scales assessing mental health, and medical co-morbidity. PD severity and disability were assessed with the Unified Parkinson's Disease Rating Scale and the Schwab and England Activities of Daily Living Scale. Results in the two regions were compared with Chi-square and Student's t-tests. Relationships between caregiver strain and patient/spousal variables were analyzed with univariate correlations and multivariate regression. RESULTS: 178 Spouse-patient pairs were assessed. The level of caregiver strain in PD did not differ between Yamagata, Japan and Maryland, US despite differences in demographics and social support programs in the two regions. Yamagata spouses reported physical, time and financial constraints, while Maryland spouses reported more emotional distress. In both regions, spousal depression was a significant contributor to caregiver strain. CONCLUSION: Different approaches to reduce caregiver strain will likely be necessary in Yamagata and Maryland since the contributing factors to caregiver strain are influenced by differences in culture and social supports in each country.

Altered cognitive function of prefrontal cortex during error feedback in patients with irritable bowel syndrome, based on FMRI and dynamic causal modeling.

BACKGROUND & AIMS: Patients with irritable bowel syndrome (IBS) have increased activity in the insula and reduced activation of the dorsolateral prefrontal cortex (DLPFC) in response to visceral stimulation. We investigated whether they have latent impairments in cognitive flexibility because of dysfunction in the DLPFC and insula and altered connectivity between brain regions. METHODS: We analyzed data from 30 individuals with IBS (15 men; age, 21.7 ± 3.0 y) diagnosed based on Rome III criteria, along with 30 individuals matched for age, sex, and education level (controls). Event-related functional magnetic resonance imaging of the brain was performed to evaluate cognitive flexibility and was assessed by the Wisconsin Card Sorting Test, in which subjects are allowed to change choice criteria, defined as set-shifting in response to error feedback. Brain images were analyzed with statistical parametric mapping 5 and 8 software and dynamic causal modeling. RESULTS: Subjects with IBS had significantly more Nelson perseverative errors (P < .05) and set-maintenance difficulties (P < .05) than controls. They also showed significantly decreased activity of the right DLPFC (Brodmann's area 9; P < .001) and right hippocampus (P < .001), and significantly increased activity of the left posterior insula (P < .001) at error feedback during set-shifting. Dynamic causal modeling analysis during set-shifting revealed significantly less connectivity from the DLPFC to pre-supplementary motor area in subjects with IBS, compared with controls (P = .012). CONCLUSIONS: Individuals with IBS have latent impairments in cognitive flexibility as a result of altered activity of the DLPFC, insula, and hippocampus, and impaired connectivity between the DLPFC and pre-supplementary motor area.

Representational switching by dynamical reorganization of attractor structure in a network model of the prefrontal cortex.

The prefrontal cortex (PFC) plays a crucial role in flexible cognitive behavior by representing task relevant information with its working memory. The working memory with sustained neural activity is described as a neural dynamical system composed of multiple attractors, each attractor of which corresponds to an active state of a cell assembly, representing a fragment of information. Recent studies have revealed that the PFC not only represents multiple sets of information but also switches multiple representations and transforms a set of information to another set depending on a given task context. This representational switching between different sets of information is possibly generated endogenously by flexible network dynamics but details of underlying mechanisms are unclear. Here we propose a dynamically reorganizable attractor network model based on certain internal changes in synaptic connectivity, or short-term plasticity. We construct a network model based on a spiking neuron model with dynamical synapses, which can qualitatively reproduce experimentally demonstrated representational switching in the PFC when a monkey was performing a goal-oriented action-planning task. The model holds multiple sets of information that are required for action planning before and after representational switching by reconfiguration of functional cell assemblies. Furthermore, we analyzed population dynamics of this model with a mean field model and show that the changes in cell assemblies' configuration correspond to those in attractor structure that can be viewed as a bifurcation process of the dynamical system. This dynamical reorganization of a neural network could be a key to uncovering the mechanism of flexible information processing in the PFC.

Functional imaging studies of hyposmia in Parkinson's disease.

Hyposmia in Parkinson's disease (PD) was evaluated by using neuroimaging techniques. It is well known that olfactory impairments are one of the cardinal non-motor symptoms in PD. However, all smell tests used in previous studies depend on subjective answers by examinees and on sniffing of odorants, the latter of which may be impaired in PD as a consequence of motor impairments. We developed an fMRI system, which can visualize brain activation by olfactory stimuli during natural breathing. Although 7 age-matched controls demonstrated significant activations in various brain areas including precentral gyrus (BA6/6) and middle temporal gyrus (BA19/39) by the odorant stimuli, 9 patients with PD showed little activations by the same stimuli. These data suggest that the olfactory dysfunction in PD is not a simple reflection of impaired sniffing. Recent epidemiological studies demonstrate that the olfactory impairments may precede the onset of motor symptoms. Moreover, several pathological studies suggest that amygdala is one of the most frequently affected regions and is closely related to hyposmia in PD. Further brain imaging studies of hyposmia will shed light on the early pathological changes in PD.

Involvement of the prefrontal cortex in problem solving.

To achieve a behavioral goal in a complex environment, such as problem-solving situations, we must plan multiple steps of action. On planning a series of actions, we anticipate future events that will occur as a result of each action, and mentally organize the temporal sequence of events. To investigate the involvement of the lateral prefrontal cortex (PFC) in such multistep planning, we examined neuronal activity in the PFC while monkeys performed a maze path-finding task. In this task, we set monkeys the job of capturing a goal in the maze by moving a cursor on the screen. Cursor movement was linked to movements of each wrist. To dissociate the outcomes of the intended action from the motor commands, we trained the monkeys to use three different hand-cursor assignments. We found that monkeys were able to perform this task in a flexible manner. This report first introduces a problem-solving framework for studying the function of the PFC, from the view point of cognitive science. Then, this chapter will cover the neuronal representation of a series of actions, goal subgoal transformation, and synchrony of PFC neurons. We reported PFC neurons reflected final goals and immediate goals during the preparatory period. We also found some PFC neurons reflected each of all forthcoming steps of actions during the preparatory period and increased their activity step by step during the execution period. Recently, we found that the transient increase in synchronous activity of PFC neurons was involved in goal subgoal transformations. Our data suggest that the PFC is involved primarily in the dynamic representation of multiple future events that occur as a consequence of behavioral actions in problem-solving situations.

Microwave effect for glycosylation promoted by solid super acid in supercritical carbon dioxide.

The effects of microwave irradiation (2.45 GHz, 200 W) on glycosylation promoted by a solid super acid in supercritical carbon dioxide was investigated with particular attention paid to the structure of the acceptor substrate. Because of the symmetrical structure and high diffusive property of supercritical carbon dioxide, microwave irradiation did not alter the temperature of the reaction solution, but enhanced reaction yield when aliphatic acceptors are employed. Interestingly, the use of a phenolic acceptor under the same reaction conditions did not show these promoting effects due to microwave irradiation. In the case of aliphatic diol acceptors, the yield seemed to be dependent on the symmetrical properties of the acceptors. The results suggest that microwave irradiation do not affect the reactivity of the donor nor promoter independently. We conclude that the effect of acceptor structure on glycosylation yield is due to electric delocalization of hydroxyl group and dielectrically symmetric structure of whole molecule.

Discharge synchrony during the transition of behavioral goal representations encoded by discharge rates of prefrontal neurons.

To investigate the temporal relationship between synchrony in the discharge of neuron pairs and modulation of the discharge rate, we recorded the neuronal activity of the lateral prefrontal cortex of monkeys performing a behavioral task that required them to plan an immediate goal of action to attain a final goal. Information about the final goal was retrieved via visual instruction signals, whereas information about the immediate goal was generated internally. The synchrony of neuron pair discharges was analyzed separately from changes in the firing rate of individual neurons during a preparatory period. We focused on neuron pairs that exhibited a representation of the final goal followed by a representation of the immediate goal at a later stage. We found that changes in synchrony and discharge rates appeared to be complementary at different phases of the behavioral task. Synchrony was maximized during a specific phase in the preparatory period corresponding to a transitional stage when the neuronal activity representing the final goal was replaced with that representing the immediate goal. We hypothesize that the transient increase in discharge synchrony is an indication of a process that facilitates dynamic changes in the prefrontal neural circuits in order to undergo profound state changes.

Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans.

To achieve a behavioral goal in a complex environment, we must plan multiple steps of motor behavior. On planning a series of actions, we anticipate future events that will occur as a result of each action and mentally organize the temporal sequence of events. To investigate the involvement of the lateral prefrontal cortex (PFC) in such multistep planning, we examined neuronal activity in the PFC of monkeys performing a maze task that required the planning of stepwise cursor movements to reach a goal. During the preparatory period, PFC neurons reflected each of all forthcoming cursor movements, rather than arm movements. In contrast, in the primary motor cortex, most neuronal activity reflected arm movements but little of cursor movements during the preparatory period, as well as during movement execution. Our data suggest that the PFC is involved primarily in planning multiple future events that occur as a consequence of behavioral actions.

Colonic hyperalgesia triggered by proteinase-activated receptor-2 in mice: involvement of endogenous bradykinin.

Intracolonic (i.col.) administration of the PAR2-activating peptide (PAR2AP) SLIGRL-NH2 slowly develops visceral hypersensitivity to i.col. capsaicin in ddY mice. Thus, we further analyzed roles of PAR2 in colonic hypersensitivity, using the novel potent PAR2AP, 2-furoyl-LIGRL-NH2 and PAR2-knockout (KO) mice. In ddY mice, i.col. 2-furoyl-LIGRL-NH2 produced delayed (6 h later) facilitation of capsaicin-evoked visceral nociception, an effect being much more potent than SLIGRL-NH2. Such effects were mimicked by i.col. trypsin. In wild-type (WT), but not PAR2-KO, mice of C57BL/6 background, i.col. PAR2 agonists caused delayed facilitation of sensitivity to capsaicin. The PAR2-triggered visceral hypersensitivity was abolished by a bradykinin B2 receptor antagonist, HOE-140. Our data thus provide ultimate evidence for role of PAR2 in colonic hypersensitivity, and suggest involvement of the bradykinin-B2 pathway.

Suppression of pancreatitis-related allodynia/hyperalgesia by proteinase-activated receptor-2 in mice.

1 Proteinase-activated receptor-2 (PAR2), a receptor activated by trypsin and tryptase, is abundantly expressed in the gastrointestinal tract including the C-fiber terminal, and might play a role in processing of visceral pain. In the present study, we examined and characterized the roles of PAR2 in pancreatitis-related abdominal hyperalgesia/allodynia in mice. 2 Caerulein, administered i.p. once, caused a small increase in abdominal sensitivity to stimulation with von Frey hairs, without causing pancreatitis, in PAR2-knockout (KO) mice, but not wild-type (WT) mice. 3 Caerulein, given hourly six times in total, caused more profound abdominal hyperalgesia/allodynia in PAR2-KO mice, as compared with WT mice, although no significant differences were detected in the severity of pancreatitis between the KO and WT animals. 4 The PAR2-activating peptide, 2-furoyl-LIGRL-NH(2), coadministered repeatedly with caerulein six times in total, abolished the caerulein-evoked abdominal hyperalgesia/allodynia in WT, but not PAR2-KO, mice. Repeated doses of 2-furoyl-LIGRL-NH(2) moderately attenuated the severity of caerulein-induced pancreatitis in WT animals. 5 Our data from experiments using PAR2-KO mice provide evidence that PAR2 functions to attenuate pancreatitis-related abdominal hyperalgesia/allodynia without affecting pancreatitis itself, although the PAR2AP applied exogenously is not only antinociceptive but also anti-inflammatory.

Mechanisms for modulation of mouse gastrointestinal motility by proteinase-activated receptor (PAR)-1 and -2 in vitro.

Proteinase-activated receptor (PAR)-1 or -2 modulates gastrointestinal transit in vivo. To clarify the underlying mechanisms, we characterized contraction/relaxation caused by TFLLR-NH2 and SLIGRL-NH2, PAR-1- and -2-activating peptides, respectively, in gastric and small intestinal (duodenal, jejunal and ileal) smooth muscle isolated from wild-type and PAR-2-knockout mice. Either SLIGRL-NH2 or TFLLR-NH2 caused both relaxation and contraction in the gastrointestinal preparations from wild-type animals. Apamin, a K+ channel inhibitor, tended to enhance the peptide-evoked contraction in some of the gastrointestinal preparations, whereas it inhibited relaxation responses to either peptide completely in the stomach, but only partially in the small intestine. Indomethacin reduced the contraction caused by SLIGRL-NH2 or TFLLR-NH2 in both gastric and ileal preparations, but unaffected apamin-insensitive relaxant effect of either peptide in ileal preparations. Repeated treatment with capsaicin suppressed the contractile effect of either peptide in the stomach, but not clearly in the ileum, whereas it enhanced the apamin-insensitive relaxant effect in ileal preparations. In any gastrointestinal preparations from PAR-2-knockout mice, SLIGRL-NH2 produced no responses. Thus, the inhibitory component in tension modulation by PAR-1 and -2 involves both apamin-sensitive and -insensitive mechanisms in the small intestine, but is predominantly attributable to the former mechanism in the stomach. The excitatory component in the PAR-1 and -2 modulation may be mediated, in part, by activation of capsaicin-sensitive sensory nerves and/or endogenous prostaglandin formation. Our study thus clarifies the multiple mechanisms for gastrointestinal motility modulation by PAR-1 and -2, and also provides ultimate evidence for involvement of PAR-2.

Abrogation of bronchial eosinophilic inflammation and attenuated eotaxin content in protease-activated receptor 2-deficient mice.

Protease-activated receptor 2 (PAR2) belongs to the PAR family (PAR1 to PAR4), which is activated by serine proteases (trypsin, tryptase, etc.). In this study, we evaluated the role of PAR2 in allergic inflammation of airways using PAR2-deficient (PAR2(-/-)) mice. In wild- type mice, infiltration of eosinophils and high eotaxin content were found in bronchoalveolar lavage fluid (BALF) after ovalbumin (OA) sensitization and following challenge. In contrast, both OA-induced infiltration of eosinophils and increase of eotaxin content were abrogated in BALF from PAR2(-/-) mice. The activation of PAR2 might be essential in the production of eotaxin and consequential allergic inflammation in airways.

Representation of immediate and final behavioral goals in the monkey prefrontal cortex during an instructed delay period.

We examined neuronal activity in the lateral prefrontal cortex of monkeys performing a path-planning task in a maze that required the planning of actions in multiple steps. The animals received an instruction that prompted them to prepare to move a cursor in the maze stepwise from a starting position to a goal position by operating manipulanda with either arm. During a delay period in which the animal prepared to start the first of three cursor movements to approach the pre-instructed goal, we identified two types of neuronal activity: the first type reflected the position within the maze to which the animal intended to move the cursor as an initial step (an immediate goal) and the second type reflected the position within the maze that was to be captured as a final goal. Neither type reflected motor responses. We propose that these two types of neuronal activity are neuronal correlates that represent immediate and ultimate behavioral goals. This finding implicates the prefrontal cortex in governing goal-oriented sequential behavior rather than sensorimotor transformation.

Design of a head fixation device for experiments in behaving monkeys.

We have designed a new device for head fixation of behaving monkeys. The fixation device consists of a duralumin head ring mounted with four screw holders. It is firmly fixed to the animal's skull with four stainless steel screw pins. The head ring is then attached to a primate chair in any desirable position and angle using a set of adjustable plates. The device has been used for behavioral training that requires accurate gaze monitoring and for recording single-unit activity over a several-month period. The advantage of our device is that it is simple to use; it can be attached readily without major surgical procedures and it can be quickly removed when experiments are not running. This head fixation system is suitable for behavioral experiments and single-unit recording studies. It may also be applicable for studies on functional imaging of the macaque brain, by constructing it of non-magnetic materials.

[Aiming toward the supply of aseptic drugs--up to acceptance of the first patient].

As the ageing society has come, a categorized medical function has been advanced. Therefore, an improvement of medical care service for home care patients has been requested. One of the services provided by community pharmacists was to visit patient to give drug management advice based on the medical insurance and homecare insurance guidelines. In 1994, the Ministry of Health and Welfare expressed a degree of pharmacist's involvement to share patient's concern and was described as "the maturity of pharmacy". At present, the third step in the division of labor in medical services has taken its root. Then, an effort to arrange the fourth step (supply of aseptic drugs) will be needed in the future. Clinical exercises by pharmacists, acquisition of aseptic preparation skills and maintenance of facilities like a clean room will be required because the supply of aseptic drugs will be the first experience for a pharmacy. Neighboring medical facilities have to be prepared for accepting a supply of aseptic drugs along with the pharmacy's preparation. This time, we could have had an opportunity to accept the HPN patient. So we would like to report our progress and challenge.