Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans.

We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectation, voxels correlating with reward prediction errors in the ventral striatum and subgenual cingulate appear to be sensitive to devaluation. Moreover, regions encoding state prediction errors appear to be devaluation insensitive. We can also distinguish regions encoding predictions about outcome taste identity from predictions about expected spatial location. Regions encoding predictions about taste identity seem devaluation sensitive while those encoding predictions about an outcome's spatial location seem devaluation insensitive. These findings suggest the existence of multiple and distinct associative mechanisms in the brain and help identify putative neural correlates for the parallel expression of both devaluation sensitive and insensitive conditioned behaviors.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans.

Pavlovian learning depends on multiple and parallel associations leading to distinct classes of conditioned responses that vary in their flexibility following changes in the value of an associated outcome. Here, we aimed to differentiate brain areas involved in learning and encoding associations that are sensitive to changes in the value of an outcome from those that are not sensitive to such changes. To address this question, we combined a Pavlovian learning task with outcome devaluation, eye-tracking and functional magnetic resonance imaging. We used computational modeling to identify brain regions involved in learning stimulus-reward associations and stimulus-stimulus associations, by testing for brain areas correlating with reward-prediction errors and state-prediction errors, respectively. We found that, contrary to theoretical predictions about reward prediction errors being exclusively model-free, voxels correlating with reward prediction errors in the ventral striatum and subgenual anterior cingulate cortex were sensitive to devaluation. On the other hand, brain areas correlating with state prediction errors were found to be devaluation insensitive. In a supplementary analysis, we distinguished brain regions encoding predictions about outcome taste identity from those involved in encoding predictions about its expected spatial location. A subset of regions involved in taste identity predictions were devaluation sensitive while those involved in encoding predictions about spatial location were devaluation insensitive. These findings provide insights into the role of multiple associative mechanisms in the brain in mediating Pavlovian conditioned behavior - illustrating how distinct neural pathways can in parallel produce both devaluation sensitive and devaluation insensitive behaviors.

Mesolimbic Neurobehavioral Mechanisms of Reward Motivation in Anorexia Nervosa: A Multimodal Imaging Study.

Diminished motivation to pursue and obtain primary and secondary rewards has been demonstrated in anorexia nervosa (AN). However, the neurobehavioral mechanisms underlying the behavioral activation component of aberrant reward motivation remains incompletely understood. This work aims to explore this underexplored facet of reward motivation in AN. We recruited female adolescents with AN, restricting type (n = 32) and a healthy control group (n = 28). All participants underwent functional magnetic resonance imaging (fMRI) while performing a monetary reward task. Diffusion MRI data was also collected to examine the reward motivation circuit's structural connectivity. Behavioral results demonstrated slower speed of reward-seeking behavior in those with AN compared with controls. Accompanying this was lower functional connectivity and reduced white matter structural integrity of the connection between the ventral tegmental area/substantia nigra pars compacta and the nucleus accumbens within the mesolimbic circuit. Further, there was evidence of neurobehavioral decoupling in AN between reward-seeking behavior and mesolimbic regional activation and functional connectivity. Aberrant activity of the bed nucleus of the stria terminalis (BNST) and its connectivity with the mesolimbic system was also evident in AN during the reward motivation period. Our findings suggest functional and structural dysconnectivity within a mesolimbic reward circuit, neurofunctional decoupling from reward-seeking behavior, and abnormal BNST function and circuit interaction with the mesolimbic system. These results show behavioral indicators of aberrant reward motivation in AN, particularly in its activational component. This is mediated neuronally by mesolimbic reward circuit functional and structural dysconnectivity as well as neurobehavioral decoupling. Based on these findings, we suggest a novel circuit-based mechanism of impaired reward processing in AN, with the potential for translation to developing more targeted and effective treatments in this difficult-to-treat psychiatric condition.

White matter tracts characteristics in habitual decision-making circuit underlie ritual behaviors in anorexia nervosa.

Anorexia nervosa (AN) is a difficult to treat, pernicious psychiatric disorder that has been linked to decision-making abnormalities. We examined the structural characteristics of habitual and goal-directed decision-making circuits and their connecting white matter tracts in 32 AN and 43 healthy controls across two independent data sets of adults and adolescents as an explanatory sub-study. Total bilateral premotor/supplementary motor area-putamen tracts in the habit circuit had a significantly higher volume in adults with AN, relative to controls. Positive correlations were found between both the number of tracts and white matter volume (WMV) in the habit circuit, and the severity of ritualistic/compulsive behaviors in adults and adolescents with AN. Moreover, we found a significant influence of the habit circuit WMV on AN ritualistic/compulsive symptom severity, depending on the preoccupations symptom severity levels. These findings suggest that AN is associated with white matter plasticity alterations in the habit circuit. The association between characteristics of habit circuit white matter tracts and AN behavioral symptoms provides support for a circuit based neurobiological model of AN, and identifies the habit circuit as a focus for further investigation to aid in development of novel and more effective treatments based on brain-behavior relationships.

Restructuring of amygdala subregion apportion across adolescence.

Total amygdala volumes develop in association with sex and puberty, and postmortem studies find neuronal numbers increase in a nuclei specific fashion across development. Thus, amygdala subregions and composition may evolve with age. Our goal was to examine if amygdala subregion absolute volumes and/or relative proportion varies as a function of age, sex, or puberty in a large sample of typically developing adolescents (N = 408, 43 % female, 10-17 years). Utilizing the in vivo CIT168 atlas, we quantified 9 subregions and implemented Generalized Additive Mixed Models to capture potential non-linear associations with age and pubertal status between sexes. Only males showed significant age associations with the basolateral ventral and paralaminar subdivision (BLVPL), central nucleus (CEN), and amygdala transition area (ATA). Again, only males showed relative differences in the proportion of the BLVPL, CEN, ATA, along with lateral (LA) and amygdalostriatal transition area (ASTA), with age. Using a best-fit modeling approach, age, and not puberty, was found to drive these associations. The results suggest that amygdala subregions show unique variations with age in males across adolescence. Future research is warranted to determine if our findings may contribute to sex differences in mental health that emerge across adolescence.

Evidence for model-based encoding of Pavlovian contingencies in the human brain.

Prominent accounts of Pavlovian conditioning successfully approximate the frequency and intensity of conditioned responses under the assumption that learning is exclusively model-free; that animals do not develop a cognitive map of events. However, these model-free approximations fall short of comprehensively capturing learning and behavior in Pavlovian conditioning. We therefore performed multivoxel pattern analysis of high-resolution functional MRI data in human participants to test for the encoding of stimulus-stimulus associations that could support model-based computations during Pavlovian conditioning. We found that dissociable sub-regions of the striatum encode predictions of stimulus-stimulus associations and predictive value, in a manner that is directly related to learning performance. Activity patterns in the orbitofrontal cortex were also found to be related to stimulus-stimulus as well as value encoding. These results suggest that the brain encodes model-based representations during Pavlovian conditioning, and that these representations are utilized in the service of behavior.

Behavioural evidence for parallel outcome-sensitive and outcome-insensitive Pavlovian learning systems in humans.

There is a dichotomy in instrumental conditioning between goal-directed actions and habits that are distinguishable on the basis of their relative sensitivity to changes in outcome value. It is less clear whether a similar distinction applies in Pavlovian conditioning, where responses have been found to be predominantly outcome sensitive. To test for both devaluation insensitive and devaluation sensitive Pavlovian conditioning in humans, we conducted four experiments combining Pavlovian conditioning and outcome devaluation procedures while measuring multiple conditioned responses. Our results suggest that Pavlovian conditioning involves two distinct types of learning: one that learns the current value of the outcome which is sensitive to devaluation, and one that learns about the spatial localisation of the outcome which is insensitive to devaluation. Our findings have implications for the mechanistic understanding of Pavlovian conditioning and provide a more nuanced understanding of Pavlovian mechanisms that might contribute to a number of psychiatric disorders.

A high-resolution probabilistic in vivo atlas of human subcortical brain nuclei.

Recent advances in magnetic resonance imaging methods, including data acquisition, pre-processing and analysis, have benefited research on the contributions of subcortical brain nuclei to human cognition and behavior. At the same time, these developments have led to an increasing need for a high-resolution probabilistic in vivo anatomical atlas of subcortical nuclei. In order to address this need, we constructed high spatial resolution, three-dimensional templates, using high-accuracy diffeomorphic registration of T1- and T2- weighted structural images from 168 typical adults between 22 and 35 years old. In these templates, many tissue boundaries are clearly visible, which would otherwise be impossible to delineate in data from individual studies. The resulting delineations of subcortical nuclei complement current histology-based atlases. We further created a companion library of software tools for atlas development, to offer an open and evolving resource for the creation of a crowd-sourced in vivo probabilistic anatomical atlas of the human brain.

Neural computations underlying inverse reinforcement learning in the human brain.

In inverse reinforcement learning an observer infers the reward distribution available for actions in the environment solely through observing the actions implemented by another agent. To address whether this computational process is implemented in the human brain, participants underwent fMRI while learning about slot machines yielding hidden preferred and non-preferred food outcomes with varying probabilities, through observing the repeated slot choices of agents with similar and dissimilar food preferences. Using formal model comparison, we found that participants implemented inverse RL as opposed to a simple imitation strategy, in which the actions of the other agent are copied instead of inferring the underlying reward structure of the decision problem. Our computational fMRI analysis revealed that anterior dorsomedial prefrontal cortex encoded inferences about action-values within the value space of the agent as opposed to that of the observer, demonstrating that inverse RL is an abstract cognitive process divorceable from the values and concerns of the observer him/herself.

Distinct prediction errors in mesostriatal circuits of the human brain mediate learning about the values of both states and actions: evidence from high-resolution fMRI.

Prediction-error signals consistent with formal models of "reinforcement learning" (RL) have repeatedly been found within dopaminergic nuclei of the midbrain and dopaminoceptive areas of the striatum. However, the precise form of the RL algorithms implemented in the human brain is not yet well determined. Here, we created a novel paradigm optimized to dissociate the subtypes of reward-prediction errors that function as the key computational signatures of two distinct classes of RL models-namely, "actor/critic" models and action-value-learning models (e.g., the Q-learning model). The state-value-prediction error (SVPE), which is independent of actions, is a hallmark of the actor/critic architecture, whereas the action-value-prediction error (AVPE) is the distinguishing feature of action-value-learning algorithms. To test for the presence of these prediction-error signals in the brain, we scanned human participants with a high-resolution functional magnetic-resonance imaging (fMRI) protocol optimized to enable measurement of neural activity in the dopaminergic midbrain as well as the striatal areas to which it projects. In keeping with the actor/critic model, the SVPE signal was detected in the substantia nigra. The SVPE was also clearly present in both the ventral striatum and the dorsal striatum. However, alongside these purely state-value-based computations we also found evidence for AVPE signals throughout the striatum. These high-resolution fMRI findings suggest that model-free aspects of reward learning in humans can be explained algorithmically with RL in terms of an actor/critic mechanism operating in parallel with a system for more direct action-value learning.

Learning, Reward, and Decision Making.

In this review, we summarize findings supporting the existence of multiple behavioral strategies for controlling reward-related behavior, including a dichotomy between the goal-directed or model-based system and the habitual or model-free system in the domain of instrumental conditioning and a similar dichotomy in the realm of Pavlovian conditioning. We evaluate evidence from neuroscience supporting the existence of at least partly distinct neuronal substrates contributing to the key computations necessary for the function of these different control systems. We consider the nature of the interactions between these systems and show how these interactions can lead to either adaptive or maladaptive behavioral outcomes. We then review evidence that an additional system guides inference concerning the hidden states of other agents, such as their beliefs, preferences, and intentions, in a social context. We also describe emerging evidence for an arbitration mechanism between model-based and model-free reinforcement learning, placing such a mechanism within the broader context of the hierarchical control of behavior.

In vivo delineation of subdivisions of the human amygdaloid complex in a high-resolution group template.

The nuclei of the human amygdala remain difficult to distinguish in individual subject structural magnetic resonance images. However, interpretation of the amygdala's role in whole brain networks requires accurate localization of functional activity to a particular nucleus or subgroup of nuclei. To address this, high spatial resolution, three-dimensional templates, using joint high accuracy diffeomorphic registration of T1- and T2-weighted structural images from 168 typical adults between 22 and 35 years old released by the Human Connectome Project were constructed. Several internuclear boundaries are clearly visible in these templates, which would otherwise be impossible to delineate in individual subject data. A probabilistic atlas of major nuclei and nuclear groups was constructed in this template space and mapped back to individual spaces by inversion of the individual diffeomorphisms. Group level analyses revealed a slight (∼2%) bias toward larger total amygdala and nuclear volumes in the right hemisphere. No substantial sex or age differences were found in amygdala volumes normalized to total intracranial volume, or subdivision volumes normalized to amygdala volume. The current delineation provides a finer parcellation of the amygdala with more accurate external boundary definition than current histology-based atlases when used in conjunction with high accuracy registration methods, such as diffeomorphic warping. These templates and delineation are intended to be an open and evolving resource for future functional and structural imaging studies of the human amygdala. Hum Brain Mapp 37:3979-3998, 2016. © 2016 Wiley Periodicals, Inc.

Regional specialization within the human striatum for diverse psychological functions.

Decades of animal and human neuroimaging research have identified distinct, but overlapping, striatal zones, which are interconnected with separable corticostriatal circuits, and are crucial for the organization of functional systems. Despite continuous efforts to subdivide the human striatum based on anatomical and resting-state functional connectivity, characterizing the different psychological processes related to each zone remains a work in progress. Using an unbiased, data-driven approach, we analyzed large-scale coactivation data from 5,809 human imaging studies. We (i) identified five distinct striatal zones that exhibited discrete patterns of coactivation with cortical brain regions across distinct psychological processes and (ii) identified the different psychological processes associated with each zone. We found that the reported pattern of cortical activation reliably predicted which striatal zone was most strongly activated. Critically, activation in each functional zone could be associated with distinct psychological processes directly, rather than inferred indirectly from psychological functions attributed to associated cortices. Consistent with well-established findings, we found an association of the ventral striatum (VS) with reward processing. Confirming less well-established findings, the VS and adjacent anterior caudate were associated with evaluating the value of rewards and actions, respectively. Furthermore, our results confirmed a sometimes overlooked specialization of the posterior caudate nucleus for executive functions, often considered the exclusive domain of frontoparietal cortical circuits. Our findings provide a precise functional map of regional specialization within the human striatum, both in terms of the differential cortical regions and psychological functions associated with each striatal zone.

Distinct Contributions of Ventromedial and Dorsolateral Subregions of the Human Substantia Nigra to Appetitive and Aversive Learning.

The role of neurons in the substantia nigra (SN) and ventral tegmental area (VTA) of the midbrain in contributing to the elicitation of reward prediction errors during appetitive learning has been well established. Less is known about the differential contribution of these midbrain regions to appetitive versus aversive learning, especially in humans. Here we scanned human participants with high-resolution fMRI focused on the SN and VTA while they participated in a sequential Pavlovian conditioning paradigm involving an appetitive outcome (a pleasant juice), as well as an aversive outcome (an unpleasant bitter and salty flavor). We found a degree of regional specialization within the SN: Whereas a region of ventromedial SN correlated with a temporal difference reward prediction error during appetitive Pavlovian learning, a dorsolateral area correlated instead with an aversive expected value signal in response to the most distal cue, and to a reward prediction error in response to the most proximal cue to the aversive outcome. Furthermore, participants' affective reactions to both the appetitive and aversive conditioned stimuli more than 1 year after the fMRI experiment was conducted correlated with activation in the ventromedial and dorsolateral SN obtained during the experiment, respectively. These findings suggest that, whereas the human ventromedial SN contributes to long-term learning about rewards, the dorsolateral SN may be particularly important for long-term learning in aversive contexts. SIGNIFICANCE STATEMENT: The role of the substantia nigra (SN) and ventral tegmental area (VTA) in appetitive learning is well established, but less is known about their contribution to aversive compared with appetitive learning, especially in humans. We used high-resolution fMRI to measure activity in the SN and VTA while participants underwent higher-order Pavlovian learning. We found a regional specialization within the SN: a ventromedial area was selectively engaged during appetitive learning, and a dorsolateral area during aversive learning. Activity in these areas predicted affective reactions to appetitive and aversive conditioned stimuli over 1 year later. These findings suggest that, whereas the human ventromedial SN contributes to long-term learning about rewards, the dorsolateral SN may be particularly important for long-term learning in aversive contexts.

Inhibiting PKMζ reveals dorsal lateral and dorsal medial striatum store the different memories needed to support adaptive behavior.

Evidence suggests that two regions of the striatum contribute differential support to instrumental response selection. The dorsomedial striatum (DMS) is thought to support expectancy-mediated actions, and the dorsolateral striatum (DLS) is thought to support habits. Currently it is unclear whether these regions store task-relevant information or just coordinate the learning and retention of these solutions by other brain regions. To address this issue, we developed a two-lever concurrent variable-interval reinforcement operant conditioning task and used it to assess the trained rat's sensitivity to contingency shifts. Consistent with the view that these two regions make different contributions to actions and habits, injecting the NMDA antagonist DL-AP5 into the DMS just prior to the shift impaired the rat's performance but enhanced performance when injected into the DLS. To determine if these regions support memory content, we first trained rats on a biased concurrent schedule (Lever 1: VI 40" and Lever 2: VI 10"). With the intent of "erasing" the memory content stored in striatum, after this training we inhibited the putative memory-maintenance protein kinase C isozyme protein kinase Mζ (PKMζ). Infusing zeta inhibitory peptide (ZIP) into the DLS enhanced the rat's ability to adapt to the contingency shift 2 d later, whereas injecting it into the DMS had the opposite effect. Infusing GluR2(3Y) into the DMS 1 h before ZIP infusions prevented ZIP from impairing the rat's sensitivity to the contingency shift. These results support the hypothesis that the DMS stores information needed to support actions and the DLS stores information needed to support habits.

Expectancy, ambiguity, and behavioral flexibility: separable and complementary roles of the orbital frontal cortex and amygdala in processing reward expectancies.

Appetitive goal-directed behavior can be associated with a cue-triggered expectancy that it will lead to a particular reward, a process thought to depend on the OFC and basolateral amygdala complex. We developed a biologically informed neural network model of this system to investigate the separable and complementary roles of these areas as the main components of a flexible expectancy system. These areas of interest are part of a neural network with additional subcortical areas, including the central nucleus of amygdala, ventral (limbic) and dorsomedial (associative) striatum. Our simulations are consistent with the view that the amygdala maintains Pavlovian associations through incremental updating of synaptic strength and that the OFC supports flexibility by maintaining an activation-based working memory of the recent reward history. Our model provides a mechanistic explanation for electrophysiological evidence that cue-related firing in OFC neurons is nonselectively early after a contingency change and why this nonselective firing is critical for promoting plasticity in the amygdala. This ambiguous activation results from the simultaneous maintenance of recent outcomes and obsolete Pavlovian contingencies in working memory. Furthermore, at the beginning of reversal, the OFC is critical for supporting responses that are no longer inappropriate. This result is inconsistent with an exclusive inhibitory account of OFC function.

Computational models of cognitive control.

Cognitive control refers to the ability to perform task-relevant processing in the face of other distractions or other forms of interference, in the absence of strong environmental support. It depends on the integrity of the prefrontal cortex and associated biological structures (e.g., the basal ganglia). Computational models have played an influential role in developing our understanding of this system, and we review current developments in three major areas: dynamic gating of prefrontal representations, hierarchies in the prefrontal cortex, and reward, motivation, and goal-related processing in prefrontal cortex. Models in these and other areas are advancing the field further forward.

Emotional salience changes the focus of spatial attention.

The interaction of emotional stimulus processing and attentional mechanisms has become a topic of intensive investigation. The present study combined aversive conditioning with a spatial attention paradigm that by simultaneously recording event-related potentials (ERPs) allowed for accessing the distribution of auditory spatial attention. In an initial conditioning phase, seven spatial locations (conditioned stimuli; CS-) were associated with an emotionally neutral and one additional location (CS+) with an emotionally aversive sound (US). Four locations were in the left and the four remaining locations were in the right hemifield. During the testing phase, either frequent single bursts of noise or infrequent double noise bursts were presented in a random sequence from these eight locations. Task of the participants was to attend to the most leftward (in half of the blocks) or the most rightward location (other half of the blocks) in order to press a button to any deviant sound at that location. Spatial attention elicited the well-known increased fronto-centrally distributed negativity in the ERPs. The presence of an aversively conditioned location in the vicinity of the spatially attended location resulted in a broadening of the attentional focus at processing stages as early as 100 ms after stimulus onset. These results suggest that emotional and attentional processing interact at early stages of stimulus processing and change sensory processing of environmental input.

Attentional control of associative learning--a possible role of the central cholinergic system.

How does attention interact with learning? Kruschke [Kruschke, J.K. (2001). Toward a unified Model of Attention in Associative Learning. J. Math. Psychol. 45, 812-863.] proposed a model (EXIT) that captures Mackintosh's [Mackintosh, N.J. (1975). A theory of attention: Variations in the associability of stimuli with reinforcement. Psychological Review, 82(4), 276-298.] framework for attentional modulation of associative learning. We developed a computational model that showed analogous interactions between selective attention and associative learning, but is significantly simplified and, in contrast to EXIT, is motivated by neurophysiological findings. Competition among input representations in the internal representation layer, which increases the contrast between stimuli, is critical for simulating these interactions in human behavior. Furthermore, this competition is modulated in a way that might be consistent with the phasic activation of the central cholinergic system, which modulates activity in sensory cortices. Specifically, phasic increases in acetylcholine can cause increased excitability of both pyramidal excitatory neurons in cortical layers II/III and cortical GABAergic inhibitory interneurons targeting the same pyramidal neurons. These effects result in increased attentional contrast in our model. This model thus represents an initial attempt to link human attentional learning data with underlying neural substrates.