From Prediction to Action: Dissociable Roles of Ventral Tegmental Area and Substantia Nigra Dopamine Neurons in Instrumental Reinforcement.

Reward seeking requires the coordination of motor programs to achieve goals. Midbrain dopamine neurons are critical for reinforcement, and their activation is sufficient for learning about cues, actions, and outcomes. Here we examine in detail the mechanisms underlying the ability of ventral tegmental area (VTA) and substantia nigra (SNc) dopamine neurons to support instrumental learning. By exploiting numerous behavioral tasks in combination with time-limited optogenetic manipulations in male and female rats, we reveal that VTA and SNc dopamine neurons generate reinforcement through separable psychological processes. VTA dopamine neurons imbue actions and their associated cues with motivational value that allows flexible and persistent pursuit, whereas SNc dopamine neurons support time-limited, precise, action-specific learning that is nonscalable and inflexible. This architecture is reminiscent of actor-critic reinforcement learning models with VTA and SNc instructing the critic and actor, respectively. Our findings indicate that heterogeneous dopamine systems support unique forms of instrumental learning that ultimately result in disparate reward-seeking strategies.SIGNIFICANCE STATEMENT Dopamine neurons in the midbrain are essential for learning, motivation, and movement. Here we describe in detail the ability of VTA and SNc dopamine neurons to generate instrumental reinforcement, a process where an agent learns about actions they can emit to earn reward. While rats will avidly work and learn to respond for activation of VTA and SNc dopamine neurons, we find that only VTA dopamine neurons imbue actions and their associated cues with motivational value that spur continued pursuit of reward. Our data support a hypothesis that VTA and SNc dopamine neurons engage distinct psychological processes that have consequences for our understanding of these neurons in health and disease.

Ventral Tegmental Dopamine Neurons Participate in Reward Identity Predictions.

Dopamine (DA) neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) encode reward prediction errors (RPEs) and are proposed to mediate error-driven learning. However, the learning strategy engaged by DA-RPEs remains controversial. RPEs might imbue predictive cues with pure value, independently of representations of their associated outcome. Alternatively, RPEs might promote learning about the sensory features (the identity) of the rewarding outcome. Here, we show that, although both VTA and SNc DA neuron activation reinforces instrumental responding, only VTA DA neuron activation during consumption of expected sucrose reward restores error-driven learning and promotes formation of a new cue→sucrose association. Critically, expression of VTA DA-dependent Pavlovian associations is abolished following sucrose devaluation, a signature of identity-based learning. These findings reveal that activation of VTA- or SNc-DA neurons engages largely dissociable learning processes with VTA-DA neurons capable of participating in outcome-specific predictive learning, and the role of SNc-DA neurons appears limited to reinforcement of instrumental responses.

Dopamine Prediction Errors in Reward Learning and Addiction: From Theory to Neural Circuitry.

Midbrain dopamine (DA) neurons are proposed to signal reward prediction error (RPE), a fundamental parameter in associative learning models. This RPE hypothesis provides a compelling theoretical framework for understanding DA function in reward learning and addiction. New studies support a causal role for DA-mediated RPE activity in promoting learning about natural reward; however, this question has not been explicitly tested in the context of drug addiction. In this review, we integrate theoretical models with experimental findings on the activity of DA systems, and on the causal role of specific neuronal projections and cell types, to provide a circuit-based framework for probing DA-RPE function in addiction. By examining error-encoding DA neurons in the neural network in which they are embedded, hypotheses regarding circuit-level adaptations that possibly contribute to pathological error signaling and addiction can be formulated and tested.

A Transgenic Rat for Investigating the Anatomy and Function of Corticotrophin Releasing Factor Circuits.

Corticotrophin-releasing factor (CRF) is a 41 amino acid neuropeptide that coordinates adaptive responses to stress. CRF projections from neurons in the central nucleus of the amygdala (CeA) to the brainstem are of particular interest for their role in motivated behavior. To directly examine the anatomy and function of CRF neurons, we generated a BAC transgenic Crh-Cre rat in which bacterial Cre recombinase is expressed from the Crh promoter. Using Cre-dependent reporters, we found that Cre expressing neurons in these rats are immunoreactive for CRF and are clustered in the lateral CeA (CeL) and the oval nucleus of the BNST. We detected major projections from CeA CRF neurons to parabrachial nuclei and the locus coeruleus, dorsal and ventral BNST, and more minor projections to lateral portions of the substantia nigra, ventral tegmental area, and lateral hypothalamus. Optogenetic stimulation of CeA CRF neurons evoked GABA-ergic responses in 11% of non-CRF neurons in the medial CeA (CeM) and 44% of non-CRF neurons in the CeL. Chemogenetic stimulation of CeA CRF neurons induced Fos in a similar proportion of non-CRF CeM neurons but a smaller proportion of non-CRF CeL neurons. The CRF1 receptor antagonist R121919 reduced this Fos induction by two-thirds in these regions. These results indicate that CeL CRF neurons provide both local inhibitory GABA and excitatory CRF signals to other CeA neurons, and demonstrate the value of the Crh-Cre rat as a tool for studying circuit function and physiology of CRF neurons.

The orbitofrontal cortex as part of a hierarchical neural system mediating choice between two good options.

Animals rely on environmental cues to identify potential rewards and select the best reward available. The orbitofrontal cortex (OFC) is proposed to encode sensory-specific representations of expected outcome. However, its contribution to the selection of a preferred outcome among different reward options is still unclear. We investigated the effect of transient OFC inactivation (achieved by presession injection of muscimol and baclofen) in a novel two-reward choice task. In discrete trials, rats could choose between a solution of polycose and an equally caloric, but highly preferred, solution of sucrose by visiting one of two liquid dispensers after the presentation of a specific cue signaling the availability of one or both of the solutions. We found that OFC inactivation did not affect outcome preference: rats maintained high preference for sucrose and adapted their behavioral responding when the cue-outcome contingencies were reversed. However, when rats were tested drug-free 24 h after OFC inactivation and reversal learning, memory for the newly learned contingencies was poor. These results suggest a potential conflict between OFC (encoding pre-reversal contingencies) and other brain circuits (encoding the new contingencies). Remarkably, repeating the OFC inactivation before the reversal memory test restored normal behavior, confirming the hypothesis of a dominant impact of OFC on other decision-making circuits. These results indicate that the representations encoded in the OFC, while not essential to the expression of outcome preference, exert hierarchical control on downstream decision-making circuits.

A causal link between prediction errors, dopamine neurons and learning.

Situations in which rewards are unexpectedly obtained or withheld represent opportunities for new learning. Often, this learning includes identifying cues that predict reward availability. Unexpected rewards strongly activate midbrain dopamine neurons. This phasic signal is proposed to support learning about antecedent cues by signaling discrepancies between actual and expected outcomes, termed a reward prediction error. However, it is unknown whether dopamine neuron prediction error signaling and cue-reward learning are causally linked. To test this hypothesis, we manipulated dopamine neuron activity in rats in two behavioral procedures, associative blocking and extinction, that illustrate the essential function of prediction errors in learning. We observed that optogenetic activation of dopamine neurons concurrent with reward delivery, mimicking a prediction error, was sufficient to cause long-lasting increases in cue-elicited reward-seeking behavior. Our findings establish a causal role for temporally precise dopamine neuron signaling in cue-reward learning, bridging a critical gap between experimental evidence and influential theoretical frameworks.

Cocaine-induced reinstatement in rats: evidence for a critical role of cocaine stimulus properties.

RATIONALE: The exact behavioral nature of drug-induced reinstatement of drug seeking is still debated. As an incentive, the drug can have general facilitatory influences on appetitive behaviors. As an interoceptive stimulus, the drug can acquire discriminative properties and control behavior. OBJECTIVE: This study assessed the relative contribution of the incentive versus discriminative properties of cocaine in food-seeking reinstatement. METHODS: In Experiment 1, eight groups of rats were trained to press a lever for food pellets and experienced cocaine (0, 5, 10, or 15 mg/kg; i.p.), either during the operant conditioning sessions or 4 h after, in another environment without food access. In Experiment 2, to dissociate the role of the operant response per se from the consummatory response, two groups of rats experienced food consumption under cocaine (10 mg/kg; i.p.) either during operant conditioning sessions or during alternate sessions of free access to the food. Then, for both experiments, food pellets were withheld and cocaine injections ceased (extinction). The reinstating effects of noncontingent cocaine (10 mg/kg; i.p.) and food pellet delivery were assessed. Locomotor activity was recorded to probe expression of behavioral sensitization. RESULTS: Cocaine reinstated lever pressing only in rats having previously performed the operant responses under cocaine. In contrast, food pellet delivery reinstated lever pressing independently of rats' history with cocaine. Locomotor sensitization was evidenced for all cocaine-pre-exposed rats, dissociating sensitization from reinstatement. CONCLUSIONS: When present during operant conditioning, the stimulus "cocaine" acquires conditioned properties which can then promote reinstatement of the extinguished behavior.

Level of operant training rather than cocaine intake predicts level of reinstatement.

RATIONALE: Extended cocaine self-administration has been shown to potentiate reinstatement. This increased vulnerability to relapse could be attributed not only to extended cocaine exposure but also to extended operant training. OBJECTIVE: This study was aimed at determining the influence of different operant training histories on cocaine-induced reinstatement when cocaine intake is kept constant. MATERIALS AND METHODS: Cocaine intake and operant training were dissociated by using experimental procedures generating different histories of operant training but almost identical histories of cocaine intake. Rats were first trained to self-administer cocaine at a classical unit dose (250 microg/inf, FR1), then in independent groups, the level of operant response was changed for the next 20 sessions by changing either the unit dose available (83, 250, or 750 microg/inf, Experiment 1) or the fixed ratio required (FR-1, FR-3, or FR-10, Experiment 2). Then, all rats were tested for reinstatement with different priming doses of cocaine (0, 5, 10, and 15 mg/kg; i.p.) at an early and late stage of an extinction period. RESULTS: Level of responding during training predicts the level of reinstatement later on, independently of the amount of cocaine consumed. High FR requirement and low unit dose access led to higher level of reinstatement at early and late stage of the extinction period, respectively. CONCLUSIONS: This study shows that the level of operant responding required to maintain optimal cocaine intake directly influences later levels of reinstatement. This finding suggests that environmental constrains that make drug-taking demanding and effortful may increase the vulnerability to relapse.