Progression of phosphorylated α-synuclein in Macaca fuscata.

Prion-like spreading of abnormal proteins is proposed to occur in neurodegenerative diseases, and the progression of α-synuclein (α-syn) deposits has been reported in the brains of animal models injected with synthetic α-syn fibrils or pathological α-syn prepared from patients with Parkinson's disease (PD) and dementia with Lewy bodies (DLB). However, α-syn transmission in nonhuman primates, which are more similar to humans, has not been fully clarified. Here, we injected synthetic human α-syn fibrils into the left striatum of a macaque monkey (Macaca fuscata). At 3 months after the injection, we examined neurodegeneration and α-syn pathology in the brain using α-syn epitope-specific antibodies, antiphosphorylated α-syn antibodies (pSyn#64 and pSer129), anti-ubiquitin antibodies, and anti-p62 antibodies. Immunohistochemical examination with pSyn#64, pSer129, and α-syn epitope-specific antibodies revealed Lewy bodies, massive α-syn-positive neuronal intracytoplasmic inclusions (NCIs), and neurites in the left putamen. These inclusions were also positive for ubiquitin and p62. LB509, a human-specific α-syn antibody targeting amino acid residues 115-122, showed limited immunoreactivity around the injection site. The left substantia nigra (SN) and the bilateral frontal cortex also contained some NCIs and neurites. The left hemisphere, including parietal/temporal cortex presented sparse α-syn pathology, and no immunoreactivity was seen in olfactory nerves, amygdala, hippocampus, or right parietal/temporal cortex. Neuronal loss and gliosis in regions with α-syn pathology were mild, except for the left striatum and SN. Our results indicate that abnormal α-syn fibrils propagate throughout the brain of M. fuscata via projection, association, and commissural fibers, though the progression of α-syn pathology is limited.

Neuronal Representation of Object Choice in the Striatum of the Monkey.

According to a widely held view, the decision-making process can be conceptualized as a two-step process: "object choice," which does not include physical actions, followed by "movement choice," in which action is executed to obtain the object. Accumulating evidence in the field of decision neuroscience suggests that the cortico-basal ganglia circuits play a crucial role in decision-making. However, the underlying mechanisms of the object and movement choices remain poorly understood, mainly because the two processes occur simultaneously in most experiments. In this study, to uncover the neuronal basis of object choice in the striatum, the main input site of the basal ganglia, we designed a behavioral task in which the processes of object and movement choice were temporally separated, and recorded the single-unit activity of phasically active neurons (PANs) (n = 375) in the striatum of two monkeys. We focused our study mainly on neuronal representation during the object choice period, before movement choice, using a mutual information analysis. Population striatal activities significantly represented the information of the chosen object during the object choice period, which indicated that the monkeys actually made the object choice during the task. For the activity of each individual neuron during the object choice period, we identified offered object- and chosen object-type neurons, corresponding to pre- and post-decision signals, respectively. We also found the movement-type neurons during the movement period after the object choice. Most offered object- or chosen object-type neurons were not overlapped with movement-type neurons. The presence of object choice-related signals independent of movement signal in the striatum indicated that the striatum was part of the site where object choice was made within a cortico-basal ganglia circuit.

Monitoring and Updating of Action Selection for Goal-Directed Behavior through the Striatal Direct and Indirect Pathways.

The basal ganglia play key roles in adaptive behaviors guided by reward and punishment. However, despite accumulating knowledge, few studies have tested how heterogeneous signals in the basal ganglia are organized and coordinated for goal-directed behavior. In this study, we investigated neuronal signals of the direct and indirect pathways of the basal ganglia as rats performed a lever push/pull task for a probabilistic reward. In the dorsomedial striatum, we found that optogenetically and electrophysiologically identified direct pathway neurons encoded reward outcomes, whereas indirect pathway neurons encoded no-reward outcome and next-action selection. Outcome coding occurred in association with the chosen action. In support of pathway-specific neuronal coding, light activation induced a bias on repeat selection of the same action in the direct pathway, but on switch selection in the indirect pathway. Our data reveal the mechanisms underlying monitoring and updating of action selection for goal-directed behavior through basal ganglia circuits.

Distinct Functions of the Primate Putamen Direct and Indirect Pathways in Adaptive Outcome-Based Action Selection.

Cortico-basal ganglia circuits are critical regulators of reward-based decision making. Reinforcement learning models posit that action reward value is encoded by the firing activity of striatal medium spiny neurons (MSNs) and updated upon changing reinforcement contingencies by dopamine (DA) signaling to these neurons. However, it remains unclear how the anatomically distinct direct and indirect pathways through the basal ganglia are involved in updating action reward value under changing contingencies. MSNs of the direct pathway predominantly express DA D1 receptors and those of the indirect pathway predominantly D2 receptors, so we tested for distinct functions in behavioral adaptation by injecting D1 and D2 receptor antagonists into the putamen of two macaque monkeys performing a free choice task for probabilistic reward. In this task, monkeys turned a handle toward either a left or right target depending on an asymmetrically assigned probability of large reward. Reward probabilities of left and right targets changed after 30-150 trials, so the monkeys were required to learn the higher-value target choice based on action-outcome history. In the control condition, the monkeys showed stable selection of the higher-value target (that more likely to yield large reward) and kept choosing the higher-value target regardless of less frequent small reward outcomes. The monkeys also made flexible changes of selection away from the high-value target when two or three small reward outcomes occurred randomly in succession. DA D1 antagonist injection significantly increased the probability of the monkey switching to the alternate target in response to successive small reward outcomes. Conversely, D2 antagonist injection significantly decreased the switching probability. These results suggest distinct functions of D1 and D2 receptor-mediated signaling processes in action selection based on action-outcome history, with D1 receptor-mediated signaling promoting the stable choice of higher-value targets and D2 receptor-mediated signaling promoting a switch in action away from small reward outcomes. Therefore, direct and indirect pathways appear to have complementary functions in maintaining optimal goal-directed action selection and updating action value, which are dependent on D1 and D2 DA receptor signaling.

Characteristics of fast-spiking neurons in the striatum of behaving monkeys.

Inhibitory interneurons are the fundamental constituents of neural circuits that organize network outputs. The striatum as part of the basal ganglia is involved in reward-directed behaviors. However, the role of the inhibitory interneurons in this process remains unclear, especially in behaving monkeys. We recorded the striatal single neuron activity while monkeys performed reward-directed hand or eye movements. Presumed parvalbumin-containing GABAergic interneurons (fast-spiking neurons, FSNs) were identified based on narrow spike shapes in three independent experiments, though they were a small population (4.2%, 42/997). We found that FSNs are characterized by high-frequency and less-bursty discharges, which are distinct from the basic firing properties of the presumed projection neurons (phasically active neurons, PANs). Besides, the encoded information regarding actions and outcomes was similar between FSNs and PANs in terms of proportion of neurons, but the discharge selectivity was higher in PANs than that of FSNs. The coding of actions and outcomes in FSNs and PANs was consistently observed under various behavioral contexts in distinct parts of the striatum (caudate nucleus, putamen, and anterior striatum). Our results suggest that FSNs may enhance the discharge selectivity of postsynaptic output neurons (PANs) in encoding crucial variables for a reward-directed behavior.

Impact of stimulus uncanniness on speeded response.

In the uncanny valley phenomenon, the causes of the feeling of uncanniness as well as the impact of the uncanniness on behavioral performances still remain open. The present study investigated the behavioral effects of stimulus uncanniness, particularly with respect to speeded response. Pictures of fish were used as visual stimuli. Participants engaged in direction discrimination, spatial cueing, and dot-probe tasks. The results showed that pictures rated as strongly uncanny delayed speeded response in the discrimination of the direction of the fish. In the cueing experiment, where a fish served as a task-irrelevant and unpredictable cue for a peripheral target, we again observed that the detection of a target was slowed when the cue was an uncanny fish. Conversely, the dot-probe task suggested that uncanny fish, unlike threatening stimulus, did not capture visual spatial attention. These results suggested that stimulus uncanniness resulted in the delayed response, and importantly this modulation was not mediated by the feelings of threat.

Photosensitive-polyimide based method for fabricating various neural electrode architectures.

An extensive photosensitive-polyimide (PSPI)-based method for designing and fabricating various neural electrode architectures was developed. The method aims to broaden the design flexibility and expand the fabrication capability for neural electrodes to improve the quality of recorded signals and integrate other functions. After characterizing PSPI's properties for micromachining processes, we successfully designed and fabricated various neural electrodes even on a non-flat substrate using only one PSPI as an insulation material and without the time-consuming dry etching processes. The fabricated neural electrodes were an electrocorticogram (ECoG) electrode, a mesh intracortical electrode with a unique lattice-like mesh structure to fixate neural tissue, and a guide cannula electrode with recording microelectrodes placed on the curved surface of a guide cannula as a microdialysis probe. In vivo neural recordings using anesthetized rats demonstrated that these electrodes can be used to record neural activities repeatedly without any breakage and mechanical failures, which potentially promises stable recordings for long periods of time. These successes make us believe that this PSPI-based fabrication is a powerful method, permitting flexible design, and easy optimization of electrode architectures for a variety of electrophysiological experimental research with improved neural recording performance.

Effects of depression on reward-based decision making and variability of action in probabilistic learning.

BACKGROUND AND OBJECTIVES: Depression is characterized by low reward sensitivity in behavioral studies applying signal detection theory. We examined deficits in reward-based decision making in depressed participants during a probabilistic learning task, and used a reinforcement learning model to examine learning parameters during the task. METHODS: Thirty-six nonclinical undergraduates completed a probabilistic selection task. Participants were divided into depressed and non-depressed groups based on Center for Epidemiologic Studies-Depression (CES-D) cut scores. We then applied a reinforcement learning model to every participant's behavioral data. RESULTS: Depressed participants showed a reward-based decision making deficit and higher levels of the learning parameter τ, which modulates variability of action selection, as compared to non-depressed participants. Highly variable action selection is more random and characterized by difficulties with selecting a specific course of action. CONCLUSION: These results suggest that depression is characterized by deficits in reward-based decision making as well as high variability in terms of action selection.

Changing the structure of complex visuo-motor sequences selectively activates the fronto-parietal network.

Previous brain imaging studies investigating motor sequence complexity have mainly examined the effect of increasing the length of pre-learned sequences. The novel contribution of this research is that we varied the structure of complex visuo-motor sequences along two different dimensions using mxn paradigm. The complexity of sequences is increased from 12 movements (organized as a 2×6 task) to 24 movements (organized as 4×6 and 2×12 tasks). Behavioral results indicate that although the success rate attained was similar across the two complex tasks (2×12 and 4×6), a greater decrease in response times was observed for the 2×12 compared to the 4×6 condition at an intermediate learning stage. This decrease is possibly related to successful chunking across sets in the 2×12 task. In line with this, we observed a selective activation of the fronto-parietal network. Shifts of activation were observed from the ventral to dorsal prefrontal, lateral to medial premotor and inferior to superior parietal cortex from the early to intermediate learning stage concomitant with an increase in hyperset length. We suggest that these selective activations and shifts in activity during complex sequence learning are possibly related to chunking of motor sequences.

Improvement of neuronal cell adhesiveness on parylene with oxygen plasma treatment.

We improved adhesiveness of a neuron-like cell, PC12, on a Parylene-C surface by O(2) plasma treatment which changes the surface from hydrophobic to hydrophilic. Neural cell adhesiveness on the plasma-treated Parylene-C was more than twenty times better compared to non-treated Parylene-C and it was close to that on a conventional polystyrene tissue-culture dish.

Neurite outgrowth of PC12 cells on diX (parylene) family materials.

We investigated neuronal cell differentiation, particularly neurite outgrowth, on the surface of diX H and diX AM using an in vitro examination of a neuron-like rat pheochromocytoma cell line, PC12. diX H and diX AM are in the parylene family of diX C (or Parylene-C), which is widely used as a novel coating material to insulate neural electrodes, and they have been recently commercialized; diX H and diX AM offer different features of biocompatibility. Previously, we found that these new parylene materials have high cell adhesiveness to neuronal cells whereas the adhesiveness of diX C is extremely low. However, their cell differentiation remains unknown although neuronal cell differentiation plays a crucial role in their development and regeneration. This study showed that almost all PC12 cells adhering to the surface of diX AM and diX H were differentiated, but the neurite outgrowth was significantly larger on diX H than that on diX AM and a conventional polystyrene culture dish. The result suggests that diX H may be advantageous as a biocompatible coating material for a scaffold, which can be used on virtually any substrate to get various configurations in neural devices.

Stimulus-dependent adjustment of reward prediction error in the midbrain.

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus-reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.

Development of a BCI master switch based on single-trial detection of contingent negative variation related potentials.

To control the startup/shutdown of a conventional brain-computer interface (BCI) that is always running for daily use, we proposed and developed a new BCI system called a BCI master switch. We designed it with on/off switching functions by detecting the contingent negative variation (CNV)--related potentials. We chose CNV to improve the single-trial discrimination of user intentions to switch because CNV had a high signal-to-noise ratio and needed high concentration for its elicitation. We also applied a support vector machine (SVM) to improve the single-trial detection of CNV-related potentials. As the best parameters of SVM were estimated and applied, the offline evaluation's best performance achieved a CNV detection rate of 99.3% for the intention to switch and 2.1% for the intention not to switch. Remarkably, this performance was achieved from single-trial detection, imaginary response of user's intention without physical reaction, and the data from only one recording electrode. These results suggest that our proposed BCI system might work as a master switch by single-trial detection.

Neural correlates of cognitive dissonance and choice-induced preference change.

According to many modern economic theories, actions simply reflect an individual's preferences, whereas a psychological phenomenon called "cognitive dissonance" claims that actions can also create preference. Cognitive dissonance theory states that after making a difficult choice between two equally preferred items, the act of rejecting a favorite item induces an uncomfortable feeling (cognitive dissonance), which in turn motivates individuals to change their preferences to match their prior decision (i.e., reducing preference for rejected items). Recently, however, Chen and Risen [Chen K, Risen J (2010) J Pers Soc Psychol 99:573-594] pointed out a serious methodological problem, which casts a doubt on the very existence of this choice-induced preference change as studied over the past 50 y. Here, using a proper control condition and two measures of preferences (self-report and brain activity), we found that the mere act of making a choice can change self-report preference as well as its neural representation (i.e., striatum activity), thus providing strong evidence for choice-induced preference change. Furthermore, our data indicate that the anterior cingulate cortex and dorsolateral prefrontal cortex tracked the degree of cognitive dissonance on a trial-by-trial basis. Our findings provide important insights into the neural basis of how actions can alter an individual's preferences.

Multiple representations of belief states and action values in corticobasal ganglia loops.

Reward-related neural activities have been found in a variety of cortical and subcortical areas by neurophysiological and neuroimaging experiments. Here we present a unified view on how three subloops of the corticobasal ganglia network are involved in reward prediction and action selection using different types of information. The motor/premotor-posterior striatum loop is specialized for action-based value representation and movement selection. The orbitofrontal-ventral striatum loop is specialized for object-based value representation and target selection. The lateral prefrontal-anterior striatum loop is specialized for context-based value representation and context estimation. Furthermore, the medial prefrontal cortex (MPFC) coordinates these multiple value representations and actions at different levels of hierarchy by monitoring the error in predictions.

Efficient reinforcement learning: computational theories, neuroscience and robotics.

Reinforcement learning algorithms have provided some of the most influential computational theories for behavioral learning that depends on reward and penalty. After briefly reviewing supporting experimental data, this paper tackles three difficult theoretical issues that remain to be explored. First, plain reinforcement learning is much too slow to be considered a plausible brain model. Second, although the temporal-difference error has an important role both in theory and in experiments, how to compute it remains an enigma. Third, function of all brain areas, including the cerebral cortex, cerebellum, brainstem and basal ganglia, seems to necessitate a new computational framework. Computational studies that emphasize meta-parameters, hierarchy, modularity and supervised learning to resolve these issues are reviewed here, together with the related experimental data.

Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics.

In learning goal-directed behaviors, an agent has to consider not only the reward given at each state but also the consequences of dynamic state transitions associated with action selection. To understand brain mechanisms for action learning under predictable and unpredictable environmental dynamics, we measured brain activities by functional magnetic resonance imaging (fMRI) during a Markov decision task with predictable and unpredictable state transitions. Whereas the striatum and orbitofrontal cortex (OFC) were significantly activated both under predictable and unpredictable state transition rules, the dorsolateral prefrontal cortex (DLPFC) was more strongly activated under predictable than under unpredictable state transition rules. We then modelled subjects' choice behaviours using a reinforcement learning model and a Bayesian estimation framework and found that the subjects took larger temporal discount factors under predictable state transition rules. Model-based analysis of fMRI data revealed different engagement of striatum in reward prediction under different state transition dynamics. The ventral striatum was involved in reward prediction under both unpredictable and predictable state transition rules, although the dorsal striatum was dominantly involved in reward prediction under predictable rules. These results suggest different learning systems in the cortico-striatum loops depending on the dynamics of the environment: the OFC-ventral striatum loop is involved in action learning based on the present state, while the DLPFC-dorsal striatum loop is involved in action learning based on predictable future states.

Representation of action-specific reward values in the striatum.

The estimation of the reward an action will yield is critical in decision-making. To elucidate the role of the basal ganglia in this process, we recorded striatal neurons of monkeys who chose between left and right handle turns, based on the estimated reward probabilities of the actions. During a delay period before the choices, the activity of more than one-third of striatal projection neurons was selective to the values of one of the two actions. Fewer neurons were tuned to relative values or action choice. These results suggest representation of action values in the striatum, which can guide action selection in the basal ganglia circuit.

A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task.

Humans can acquire appropriate behaviors that maximize rewards on a trial-and-error basis. Recent electrophysiological and imaging studies have demonstrated that neural activity in the midbrain and ventral striatum encodes the error of reward prediction. However, it is yet to be examined whether the striatum is the main locus of reward-based behavioral learning. To address this, we conducted functional magnetic resonance imaging (fMRI) of a stochastic decision task involving monetary rewards, in which subjects had to learn behaviors involving different task difficulties that were controlled by probability. We performed a correlation analysis of fMRI data by using the explanatory variables derived from subject behaviors. We found that activity in the caudate nucleus was correlated with short-term reward and, furthermore, paralleled the magnitude of a subject's behavioral change during learning. In addition, we confirmed that this parallelism between learning and activity in the caudate nucleus is robustly maintained even when we vary task difficulty by controlling the probability. These findings suggest that the caudate nucleus is one of the main loci for reward-based behavioral learning.

Inter-module credit assignment in modular reinforcement learning.

Critical issues in modular or hierarchical reinforcement learning (RL) are (i) how to decompose a task into sub-tasks, (ii) how to achieve independence of learning of sub-tasks, and (iii) how to assure optimality of the composite policy for the entire task. The second and last requirements are often under trade-off. We propose a method for propagating the reward for the entire task achievement between modules. This is done in the form of a 'modular reward', which is calculated from the temporal difference of the module gating signal and the value of the succeeding module. We implement modular reward for a multiple model-based reinforcement learning (MMRL) architecture and show its effectiveness in simulations of a pursuit task with hidden states and a continuous-time non-linear control task.