Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning.

In many cases, learning is thought to be driven by differences between the value of rewards we expect and rewards we actually receive. Yet learning can also occur when the identity of the reward we receive is not as expected, even if its value remains unchanged. Learning from changes in reward identity implies access to an internal model of the environment, from which information about the identity of the expected reward can be derived. As a result, such learning is not easily accounted for by model-free reinforcement learning theories such as temporal difference reinforcement learning (TDRL), which predicate learning on changes in reward value, but not identity. Here, we used unblocking procedures to assess learning driven by value- versus identity-based prediction errors. Rats were trained to associate distinct visual cues with different food quantities and identities. These cues were subsequently presented in compound with novel auditory cues and the reward quantity or identity was selectively changed. Unblocking was assessed by presenting the auditory cues alone in a probe test. Consistent with neural implementations of TDRL models, we found that the ventral striatum was necessary for learning in response to changes in reward value. However, this area, along with orbitofrontal cortex, was also required for learning driven by changes in reward identity. This observation requires that existing models of TDRL in the ventral striatum be modified to include information about the specific features of expected outcomes derived from model-based representations, and that the role of orbitofrontal cortex in these models be clearly delineated.

The Orphan Drug Act and the development of stem cell-based products for rare diseases.

The Orphan Drug Act encourages the development of products for rare diseases and conditions. Many conditions that stand to benefit from stem cell-based products are rare diseases. We address the Orphan Drug Act in relation to the development of stem cell-based products.

Orbitofrontal inactivation impairs reversal of Pavlovian learning by interfering with 'disinhibition' of responding for previously unrewarded cues.

Orbitofrontal cortex (OFC) is critical for reversal learning. Reversal deficits are typically demonstrated in complex settings that combine Pavlovian and instrumental learning. Yet recent work has implicated the OFC specifically in behaviors guided by cues and the features of the specific outcomes they predict. To test whether the OFC is important for reversing such Pavlovian associations in the absence of confounding instrumental requirements, we trained rats on a simple Pavlovian task in which two auditory cues were presented, one paired with a food pellet reward and the other presented without reward. After learning, we reversed the cue-outcome associations. For half the rats, OFC was inactivated prior to each reversal session. Inactivation of OFC impaired the ability of the rats to reverse conditioned responding. This deficit reflected the inability of inactivated rats to develop normal responding for the previously unrewarded cue; inactivation of OFC had no impact on the ability of the rats to inhibit responding to the previously rewarded cue. These data show that OFC is critical to reversal of Pavlovian responding, and that the role of OFC in this behavior cannot be explained as a simple deficit in response inhibition. Furthermore, the contrast between the normal inhibition of responding, reported here, and impaired inhibition of responding during Pavlovian over-expectation, reported previously, suggests the novel hypothesis that OFC may be particularly critical for learning (or behavior) when it requires the subject to generate predictions about outcomes by bringing together or integrating disparate pieces of associative information.

The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes.

Humans and other animals change their behavior in response to unexpected outcomes. The orbitofrontal cortex (OFC) is implicated in such adaptive responding, based on evidence from reversal tasks. Yet these tasks confound using information about expected outcomes with learning when those expectations are violated. OFC is critical for the former function; here we show it is also critical for the latter. In a Pavlovian overexpectation task, inactivation of OFC prevented learning driven by unexpected outcomes, even when performance was assessed later. We propose this reflects a critical contribution of outcome signaling by OFC to encoding of reward prediction errors elsewhere. In accord with this proposal, we report that signaling of reward predictions by OFC neurons was related to signaling of prediction errors by dopamine neurons in ventral tegmental area (VTA). Furthermore, bilateral inactivation of VTA or contralateral inactivation of VTA and OFC disrupted learning driven by unexpected outcomes.

The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards.

Cues that reliably predict rewards trigger the thoughts and emotions normally evoked by those rewards. Humans and other animals will work, often quite hard, for these cues. This is termed conditioned reinforcement. The ability to use conditioned reinforcers to guide our behaviour is normally beneficial; however, it can go awry. For example, corporate icons, such as McDonald's Golden Arches, influence consumer behaviour in powerful and sometimes surprising ways, and drug-associated cues trigger relapse to drug seeking in addicts and animals exposed to addictive drugs, even after abstinence or extinction. Yet, despite their prevalence, it is not known how conditioned reinforcers control human or other animal behaviour. One possibility is that they act through the use of the specific rewards they predict; alternatively, they could control behaviour directly by activating emotions that are independent of any specific reward. In other words, the Golden Arches may drive business because they evoke thoughts of hamburgers and fries, or instead, may be effective because they also evoke feelings of hunger or happiness. Moreover, different brain circuits could support conditioned reinforcement mediated by thoughts of specific outcomes versus more general affective information. Here we have attempted to address these questions in rats. Rats were trained to learn that different cues predicted different rewards using specialized conditioning procedures that controlled whether the cues evoked thoughts of specific outcomes or general affective representations common to different outcomes. Subsequently, these rats were given the opportunity to press levers to obtain short and otherwise unrewarded presentations of these cues. We found that rats were willing to work for cues that evoked either outcome-specific or general affective representations. Furthermore the orbitofrontal cortex, a prefrontal region important for adaptive decision-making, was critical for the former but not for the latter form of conditioned reinforcement.

Neural correlates of inflexible behavior in the orbitofrontal-amygdalar circuit after cocaine exposure.

Addiction is characterized by compulsive or inflexible behavior, observed both in the context of drug-seeking and in contexts unrelated to drugs. One possible contributor to these inflexible behaviors may be drug-induced dysfunction within circuits that support behavioral flexibility, including the basolateral amygdala (ABL) and the orbitofrontal cortex (OFC). Here we describe data demonstrating that chronic cocaine exposure causes long-lasting changes in encoding properties in the ABL and the OFC during learning and reversal in an odor-guided task. In particular, these data suggest that inflexible encoding in ABL neurons may be the proximal cause of cocaine-induced behavioral inflexibility, and that a loss of outcome-expectant encoding in OFC neurons could be a more distal contributor to this impairment. A similar mechanism of drug-induced orbitofrontal-amygdalar dysfunction may cause inflexible behavior when animals and addicts are exposed to drug-associated cues and contexts.

Should I stay or should I go? Transformation of time-discounted rewards in orbitofrontal cortex and associated brain circuits.

Animals prefer a small, immediate reward over a larger delayed reward (time discounting). Lesions of the orbitofrontal cortex (OFC) can either increase or decrease the breakpoint at which animals abandon the large delayed reward for the more immediate reward as the delay becomes longer. Here we argue that the varied effects of OFC lesions on delayed discounting reflect two different patterns of activity in OFC; one that bridges the gap between a response and an outcome and another that discounts delayed reward. These signals appear to reflect the spatial location of the reward and/or the action taken to obtain it, and are encoded independently from representations of absolute value. We suggest a dual role for output from OFC in both discounting delayed reward, while at the same time supporting new learning for them.

Conditioned reinforcement can be mediated by either outcome-specific or general affective representations.

Conditioned reinforcers are Pavlovian cues that support the acquisition and maintenance of new instrumental responses. Responding on the basis of conditioned rather than primary reinforcers is a pervasive part of modern life, yet we have a remarkably limited understanding of what underlying associative information is triggered by these cues to guide responding. Specifically, it is not certain whether conditioned reinforcers are effective because they evoke representations of specific outcomes or because they trigger general affective states that are independent of any specific outcome. This question has important implications for how different brain circuits might be involved in conditioned reinforcement. Here, we use specialized Pavlovian training procedures, reinforcer devaluation and transreinforcer blocking, to create cues that were biased to preferentially evoke either devaluation-insensitive, general affect representations or, devaluation-sensitive, outcome-specific representations. Subsequently, these cues, along with normally conditioned control cues, were presented contingent on lever pressing. We found that intact rats learned to lever press for either the outcome or the affect cues to the same extent as for a normally conditioned cue. These results demonstrate that conditioned reinforcers can guide responding through either type of associative information. Interestingly, conditioned reinforcement was abolished in rats with basolateral amygdala lesions. Consistent with the extant literature, this result suggests a general role for basolateral amygdala in conditioned reinforcement. The implications of these data, combined with recent reports from our laboratory of a more specialized role of orbitofrontal cortex in conditioned reinforcement, will be discussed.

Abnormal associative encoding in orbitofrontal neurons in cocaine-experienced rats during decision-making.

Recent evidence has linked exposure to addictive drugs to an inability to employ information about adverse consequences, or outcomes, to control behavior. For instance, addicts and drug-experienced animals fail to adapt their behavior to avoid adverse outcomes in gambling and reversal tasks or after changes in the value of expected rewards. These deficits are similar to those caused by damage to the orbitofrontal cortex, suggesting that addictive drugs may cause long-lasting changes in the representation of outcome associations in a circuit that includes the orbitofrontal cortex. Here we test this hypothesis by recording from orbitofrontal neurons in a discrimination task in rats previously exposed to cocaine (30 mg/kg i.p. for 14 days). We found that orbitofrontal neurons recorded in cocaine-experienced rats failed to signal the adverse outcome at the time a decision was made in the task. The loss of this signal was associated with abnormal changes in response latencies on aversive trials. Furthermore, upon reversal of the cue-outcome associations, orbitofrontal neurons in cocaine-treated rats with enduring reversal impairments failed to reverse their cue-selectivity, while orbitofrontal neurons in cocaine-treated rats with normal performance showed an increase in the plasticity of cue-selective firing after reversal. These results provide direct neurophysiological evidence that exposure to cocaine can cause behaviorally relevant changes in the processing of associative information in a circuit that includes the orbitofrontal cortex.

Prior cocaine exposure disrupts extinction of fear conditioning.

Psychostimulant exposure has been shown to cause molecular and cellular changes in prefrontal cortex. It has been hypothesized that these drug-induced changes might affect the operation of prefrontal-limbic circuits, disrupting their normal role in controlling behavior and thereby leading to compulsive drug-seeking. To test this hypothesis, we tested cocaine-treated rats in a fear conditioning, inflation, and extinction task, known to depend on medial prefrontal cortex and amygdala. Cocaine-treated rats conditioned and inflated similar to saline controls but displayed slower extinction learning. These results support the hypothesis that control processes in the medial prefrontal cortex are impaired by cocaine exposure.

State-dependent gating of sensory inputs by zona incerta.

We have previously shown that the GABAergic nucleus zona incerta (ZI) suppresses vibrissae-evoked responses in the posterior medial (POm) thalamus of the rodent somatosensory system. We proposed that this inhibitory incerto-thalamic pathway regulates POm responses during different behavioral states. Here we tested the hypothesis that the cholinergic reticular activating system, implicated in regulating states of arousal, modulates ZI activity. We show that stimulation of brain stem cholinergic nuclei (laterodorsal tegmental and pedunculopontine tegmental) results in suppression of spontaneous firing of ZI neurons. Iontophoretic application of the cholinergic agonist carbachol to ZI neurons suppresses both their spontaneous firing and their vibrissae-evoked responses. We also found that carbachol application to an in vitro slice preparation suppresses spontaneous firing of neurons in the ventral sector of ZI (ZIv). Finally, we demonstrate that the majority of ZIv neurons contain parvalbumin and project to POm. Based on these results, we present the state-dependent gating hypothesis, which states that differing behavioral states-regulated by the brain stem cholinergic system-modulate ZI activity, thereby regulating the response properties of higher-order nuclei such as POm.