RESUMEN
The orbitofrontal cortex (OFC) is crucial for tracking various aspects of expected outcomes, thereby helping to guide choices and support learning. Our previous study showed that the effects of reward timing and size on the activity of single units in OFC were dissociable when these attributes were manipulated independently (Roesch et al., 2006). However, in real-life decision-making scenarios, outcome features often change simultaneously, so here we investigated how OFC neurons in male rats integrate information about the timing and identity (flavor) of reward and respond to changes in these features, according to whether they were changed simultaneously or separately. We found that a substantial number of OFC neurons fired differentially to immediate versus delayed reward and to the different reward flavors. However, contrary to the previous study, selectivity for timing was strongly correlated with selectivity for identity. Taken together with the previous research, these results suggest that when reward features are correlated, OFC tends to "pack" them into unitary constructs, whereas when they are independent, OFC tends to "crack" them into separate constructs. Furthermore, we found that when both reward timing and flavor were changed, reward-responsive OFC neurons showed unique activity patterns preceding and during the omission of an expected reward. Interestingly, this OFC activity is similar and slightly preceded the ventral tegmental area dopamine (VTA DA) activity that observed in a previous study (Takahashi et al., 2023), consistent with a role for OFC in providing predictive information to VTA DA neurons.Significant Statement Although multiple features of outcomes can change simultaneously in real-life decision-making scenarios, how OFC neurons integrate information about the reward timing and identity and respond to changes in these features remains unexplored. Here we found that OFC neurons integrate information about reward timing and identity when the two features changed simultaneously. Combining with prior research (Roesch et al., 2006), these findings suggest that OFC tends to integrate correlated features into unitary constructs, while segregating independent features into separate constructs. Additionally, we observed distinct activities in general reward-responsive neurons preceding the omission of an expected reward when both identity and timing changed. This implies OFC might convey predictions to VTA that track reward timing separately based on reward identity.
RESUMEN
The orbitofrontal cortex (OFC) and hippocampus (HC) are both implicated in forming the cognitive or task maps that support flexible behavior. Previously, we used the dopamine neurons as a sensor or tool to measure the functional effects of OFC lesions (Takahashi et al., 2011). We recorded midbrain dopamine neurons as rats performed an odor-based choice task, in which errors in the prediction of reward were induced by manipulating the number or timing of the expected rewards across blocks of trials. We found that OFC lesions ipsilateral to the recording electrodes caused prediction errors to be degraded consistent with a loss in the resolution of the task states, particularly under conditions where hidden information was critical to sharpening the predictions. Here we have repeated this experiment, along with computational modeling of the results, in rats with ipsilateral HC lesions. The results show HC also shapes the map of our task, however unlike OFC, which provides information local to the trial, the HC appears to be necessary for estimating the upper-level hidden states based on the information that is discontinuous or separated by longer timescales. The results contrast the respective roles of the OFC and HC in cognitive mapping and add to evidence that the dopamine neurons access a rich information set from distributed regions regarding the predictive structure of the environment, potentially enabling this powerful teaching signal to support complex learning and behavior.
RESUMEN
Dopamine neuron activity is tied to the prediction error in temporal difference reinforcement learning models. These models make significant simplifying assumptions, particularly with regard to the structure of the predictions fed into the dopamine neurons, which consist of a single chain of timepoint states. Although this predictive structure can explain error signals observed in many studies, it cannot cope with settings where subjects might infer multiple independent events and outcomes. In the present study, we recorded dopamine neurons in the ventral tegmental area in such a setting to test the validity of the single-stream assumption. Rats were trained in an odor-based choice task, in which the timing and identity of one of several rewards delivered in each trial changed across trial blocks. This design revealed an error signaling pattern that requires the dopamine neurons to access and update multiple independent predictive streams reflecting the subject's belief about timing and potentially unique identities of expected rewards.
Asunto(s)
Refuerzo en Psicología , Área Tegmental Ventral , Ratas , Animales , Área Tegmental Ventral/fisiología , Aprendizaje/fisiología , Recompensa , Neuronas Dopaminérgicas/fisiología , Dopamina/fisiologíaRESUMEN
There is no single way to represent a task. Indeed, despite experiencing the same task events and contingencies, different subjects may form distinct task representations. As experimenters, we often assume that subjects represent the task as we envision it. However, such a representation cannot be taken for granted, especially in animal experiments where we cannot deliver explicit instruction regarding the structure of the task. Here, we tested how rats represent an odor-guided choice task in which two odor cues indicated which of two responses would lead to reward, whereas a third odor indicated free choice among the two responses. A parsimonious task representation would allow animals to learn from the forced trials what is the better option to choose in the free-choice trials. However, animals may not necessarily generalize across odors in this way. We fit reinforcement-learning models that use different task representations to trial-by-trial choice behavior of individual rats performing this task, and quantified the degree to which each animal used the more parsimonious representation, generalizing across trial types. Model comparison revealed that most rats did not acquire this representation despite extensive experience. Our results demonstrate the importance of formally testing possible task representations that can afford the observed behavior, rather than assuming that animals' task representations abide by the generative task structure that governs the experimental design.
Asunto(s)
Odorantes , Recompensa , Animales , Señales (Psicología) , Generalización Psicológica , Humanos , Ratas , Refuerzo en PsicologíaRESUMEN
Dopamine neurons respond to errors in predicting value-neutral sensory information. These data, combined with causal evidence that dopamine transients support sensory-based associative learning, suggest that the dopamine system signals a multidimensional prediction error. Yet such complexity is not evident in the activity of individual neurons or population averages. How then do downstream areas know what to learn in response to these signals? One possibility is that information about content is contained in the pattern of firing across many dopamine neurons. Consistent with this, here we show that the pattern of firing across a small group of dopamine neurons recorded in rats signals the identity of a mis-predicted sensory event. Further, this same information is reflected in the BOLD response elicited by sensory prediction errors in human midbrain. These data provide evidence that ensembles of dopamine neurons provide highly specific teaching signals, opening new possibilities for how this system might contribute to learning.
Asunto(s)
Potenciales de Acción , Neuronas Dopaminérgicas/fisiología , Aprendizaje , Mesencéfalo/fisiología , Animales , Modelos Neurológicos , RatasRESUMEN
Addiction is a disorder of behavioral control and learning. While this may reflect pre-existing propensities, drug use also clearly contributes by causing changes in outcome processing in prefrontal and striatal regions. This altered processing is associated with behavioral deficits, including changes in learning. These areas provide critical input to midbrain dopamine neurons regarding expected outcomes, suggesting that effects on learning may result from changes in dopaminergic error signaling. Here, we show that dopamine neurons recorded in rats that had self-administered cocaine failed to suppress firing on omission of an expected reward and exhibited lower amplitude and imprecisely timed increases in firing to an unexpected reward. Learning also appeared to have less of an effect on reward-evoked and cue-evoked firing in the cocaine-experienced rats. Overall, the changes are consistent with reduced fidelity of input regarding the expected outcomes, such as their size, timing, and overall value, because of cocaine use.
Asunto(s)
Potenciales de Acción/efectos de los fármacos , Cocaína/administración & dosificación , Inhibidores de Captación de Dopamina/administración & dosificación , Neuronas Dopaminérgicas/efectos de los fármacos , Autoadministración , Análisis de Varianza , Animales , Conducta de Elección , Condicionamiento Operante/efectos de los fármacos , Señales (Psicología) , Ratas , Recompensa , Área Tegmental Ventral/citologíaRESUMEN
Neurons in the orbitofrontal cortex (OFC) fire in anticipation of and during rewards. Such firing has been suggested to encode reward predictions and to account in some way for the role of this area in adaptive behavior and learning. However, it has also been reported that neural activity in OFC reflects reward prediction errors, which might drive learning directly. Here we tested this question by analyzing the firing of OFC neurons recorded in an odor discrimination task in which rats were trained to sample odor cues and respond left or right on each trial for reward. Neurons were recorded across blocks of trials in which we switched either the number or the flavor of the reward delivered in each well. Previously we have described how neurons in this dataset fired to the predictive cues (Stalnaker et al., 2014); here we focused on the firing in anticipation of and just after delivery of each drop of reward, looking specifically for differences in firing based on whether the reward number or flavor was unexpected or expected. Unlike dopamine neurons recorded in this setting, which exhibited phasic error-like responses after surprising changes in either reward number or reward flavor (Takahashi et al., 2017), OFC neurons showed no such error correlates and instead fired in a way that reflected reward predictions.
Asunto(s)
Potenciales de Acción/fisiología , Aprendizaje/fisiología , Neuronas/fisiología , Corteza Prefrontal/fisiología , Recompensa , Animales , Neuronas Dopaminérgicas/fisiología , Masculino , Neuronas/citología , Corteza Prefrontal/citología , Ratas , Ratas Long-EvansRESUMEN
Midbrain dopamine neurons have been proposed to signal prediction errors as defined in model-free reinforcement learning algorithms. While these algorithms have been extremely powerful in interpreting dopamine activity, these models do not register any error unless there is a difference between the value of what is predicted and what is received. Yet learning often occurs in response to changes in the unique features that characterize what is received, sometimes with no change in its value at all. Here, we show that classic error-signaling dopamine neurons also respond to changes in value-neutral sensory features of an expected reward. This suggests that dopamine neurons have access to a wider variety of information than contemplated by the models currently used to interpret their activity and that, while their firing may conform to predictions of these models in some cases, they are not restricted to signaling errors in the prediction of value.
Asunto(s)
Condicionamiento Operante/fisiología , Neuronas Dopaminérgicas/fisiología , Recompensa , Sensación/fisiología , Animales , Animales Modificados Genéticamente , Masculino , Modelos Neurológicos , Ratas , Área Tegmental Ventral/fisiologíaRESUMEN
Dopaminergic reward prediction errors in monkeys reflect inferential reward predictions that well-trained animals can make when associative rules change. Here, in a new analysis of previously described data, we test whether dopaminergic error signals in rats are influenced by inferential predictions and whether such effects depend on the orbitofrontal cortex (OFC). Dopamine neurons were recorded from controls or rats with ipsilateral OFC lesions during performance of a choice task in which odor cues signaled the availability of sucrose reward in 2 wells. To induce prediction errors, we manipulated either the timing or number of rewards delivered in each well across blocks of trials. Of importance, a change in reward at 1 well predicted a change in reward at the other on later trials. We compared behavior and neural activity on trials when such inference was possible versus trials involving the same reward change when inference was not possible. Rats responded faster when they could infer an increase in reward compared to when the same reward was coming but they could not infer a change. This inferential prediction was reflected in the firing of dopamine neurons in controls, which changed less to unexpected delivery (or omission) of reward and more to the new high-value cue on inference versus noninference trials. These effects were absent in dopamine neurons recorded in rats with ipsilateral OFC lesions. Thus, dopaminergic error signals recorded in rats are influenced by both experiential and inferential reward predictions, and the effects of inferential predictions depend on OFC. (PsycINFO Database Record
Asunto(s)
Conducta de Elección/fisiología , Dopamina/fisiología , Neuronas Dopaminérgicas/fisiología , Corteza Prefrontal/fisiología , Recompensa , Área Tegmental Ventral/fisiología , Potenciales de Acción , Animales , Señales (Psicología) , Masculino , Ratas , Ratas Long-EvansRESUMEN
We recently showed that ventral striatal (VS) lesions abolished reward prediction errors that reflect changes in reward timing while leaving relatively unaffected errors that reflect changes in reward number. Here we extended those results by examining whether normal learning-related changes in firing to the reward-predictive cues in the same dopamine neurons were also disrupted by VS lesions. This analysis revealed that dopamine neurons recorded in VS-lesioned rats failed to show value-related changes in firing to the cues in timing but showed normal changes in number blocks. This effect suggests that the loss of reward-evoked error signals in the timing blocks impaired encoding of the cue value in downstream areas, which then supply these predictions to dopamine neurons at the time of cue presentation. (PsycINFO Database Record
Asunto(s)
Condicionamiento Operante/fisiología , Dopamina/fisiología , Neuronas Dopaminérgicas , Recompensa , Animales , Señales (Psicología) , Aprendizaje/fisiología , Masculino , Ratas , Ratas Long-Evans , Transducción de SeñalRESUMEN
Addiction involves an inability to control drug-seeking behavior. While this may be thought of as secondary to an overwhelming desire for drugs, it could equally well reflect a failure of the brain mechanisms that allow addicts to learn about and mentally simulate non-drug consequences. Importantly, this process of mental simulation draws upon, but is not normally bound by, our past experiences. Rather we have the ability to think outside the box of our past, integrating knowledge gained from a variety of similar and not-so-similar life experiences to derive estimates or imagine what might happen next. These estimates influence our current behavior directly and also affect future behavior by serving as the background against which outcomes are evaluated to support learning. Here we will review evidence, from our own work using a Pavlovian over-expectation task as well as from other sources, that the orbitofrontal cortex is a critical node in the neural circuit that generates these estimates. Further we will offer the specific hypothesis that degradation of this function secondary to drug-induced changes is a critical and likely addressable part of addiction.
Asunto(s)
Imaginación , Corteza Prefrontal/fisiología , Trastornos Relacionados con Sustancias , Animales , Humanos , Trastornos Relacionados con Sustancias/patología , Trastornos Relacionados con Sustancias/psicología , Trastornos Relacionados con Sustancias/rehabilitaciónRESUMEN
Dopamine neurons signal reward prediction errors. This requires accurate reward predictions. It has been suggested that the ventral striatum provides these predictions. Here we tested this hypothesis by recording from putative dopamine neurons in the VTA of rats performing a task in which prediction errors were induced by shifting reward timing or number. In controls, the neurons exhibited error signals in response to both manipulations. However, dopamine neurons in rats with ipsilateral ventral striatal lesions exhibited errors only to changes in number and failed to respond to changes in timing of reward. These results, supported by computational modeling, indicate that predictions about the temporal specificity and the number of expected reward are dissociable and that dopaminergic prediction-error signals rely on the ventral striatum for the former but not the latter.
Asunto(s)
Ganglios Basales/fisiología , Dopamina/metabolismo , Neuronas Dopaminérgicas/metabolismo , Recompensa , Estriado Ventral/fisiología , Área Tegmental Ventral/fisiología , Animales , Ratas Long-EvansRESUMEN
Reciprocal connections between the orbitofrontal cortex (OFC) and the basolateral nucleus of the amygdala (BLA) provide a critical circuit for guiding normal behavior when information about expected outcomes is required. Recently, we reported that outcome signaling by OFC neurons is also necessary for learning in the face of unexpected outcomes during a Pavlovian over-expectation task. Key to learning in this task is the ability to build on prior learning to infer or estimate an amount of reward never previously received. OFC was critical to this process. Notably, in parallel work, we found that BLA was not necessary for learning in this setting. This suggested a dissociation in which the BLA might be critical for acquiring information about the outcomes but not for subsequently using it to make novel predictions. Here we evaluated this hypothesis by recording single-unit activity from BLA in rats during the same Pavlovian over-expectation task used previously. We found that spiking activity recorded in BLA in control rats did reflect novel outcome estimates derived from the integration of prior learning, however consistent with a model in which this process occurs in the OFC, these correlates were entirely abolished by ipsilateral OFC lesions. These data indicate that this information about these novel predictions is represented in the BLA, supported via direct or indirect input from the OFC, even though it does not appear to be necessary for learning. SIGNIFICANCE STATEMENT: The basolateral nucleus of the amygdala (BLA) and the orbitofrontal cortex (OFC) are involved in behavior that depends on knowledge of impending outcomes. Recently, we found that only the OFC was necessary for using such information for learning in a Pavlovian over-expectation task. The current experiment was designed to search for neural correlates of this process in the BLA and, if present, to ask whether they would still be dependent on OFC input. We found that although spiking activity in BLA in control rats did reflect the novel outcome estimates underlying learning, these correlates were entirely abolished by OFC lesions.
Asunto(s)
Amígdala del Cerebelo/fisiología , Corteza Prefrontal/fisiología , Amígdala del Cerebelo/citología , Animales , Condicionamiento Clásico , Señales (Psicología) , Estimulación Eléctrica , Electrodos Implantados , Fenómenos Electrofisiológicos , Extinción Psicológica , Lateralidad Funcional/fisiología , Aprendizaje , Masculino , Modelos Neurológicos , Neuronas/fisiología , Técnicas de Placa-Clamp , Corteza Prefrontal/citología , Ratas , Ratas Long-EvansRESUMEN
Addiction is characterized by a lack of insight into the likely outcomes of one's behavior. Insight, or the ability to imagine outcomes, is evident when outcomes have not been directly experienced. Using this concept, work in both rats and humans has recently identified neural correlates of insight in the medial and orbital prefrontal cortices. We found that these correlates were selectively abolished in rats by cocaine self-administration. Their abolition was associated with behavioral deficits and reduced synaptic efficacy in orbitofrontal cortex, the reversal of which by optogenetic activation restored normal behavior. These results provide a link between cocaine use and problems with insight. Deficits in these functions are likely to be particularly important for problems such as drug relapse, in which behavior fails to account for likely adverse outcomes. As such, our data provide a neural target for therapeutic approaches to address these defining long-term effects of drug use.
Asunto(s)
Concienciación/efectos de los fármacos , Conducta Animal/efectos de los fármacos , Cocaína/farmacología , Inhibidores de Captación de Dopamina/farmacología , Aprendizaje/efectos de los fármacos , Corteza Prefrontal/efectos de los fármacos , Animales , Cocaína/administración & dosificación , Cocaína/efectos adversos , Trastornos Relacionados con Cocaína/metabolismo , Trastornos Relacionados con Cocaína/fisiopatología , Modelos Animales de Enfermedad , Inhibidores de Captación de Dopamina/administración & dosificación , Inhibidores de Captación de Dopamina/efectos adversos , Masculino , Optogenética , Corteza Prefrontal/citología , Corteza Prefrontal/fisiopatología , Ratas , Ratas Long-Evans , Autoadministración , Sinapsis/efectos de los fármacosRESUMEN
Orbitofrontal cortex (OFC) has long been known to play an important role in decision making. However, the exact nature of that role has remained elusive. Here, we propose a unifying theory of OFC function. We hypothesize that OFC provides an abstraction of currently available information in the form of a labeling of the current task state, which is used for reinforcement learning (RL) elsewhere in the brain. This function is especially critical when task states include unobservable information, for instance, from working memory. We use this framework to explain classic findings in reversal learning, delayed alternation, extinction, and devaluation as well as more recent findings showing the effect of OFC lesions on the firing of dopaminergic neurons in ventral tegmental area (VTA) in rodents performing an RL task. In addition, we generate a number of testable experimental predictions that can distinguish our theory from other accounts of OFC function.
Asunto(s)
Cognición , Aprendizaje/fisiología , Corteza Prefrontal/fisiología , Potenciales de Acción/fisiología , Animales , Toma de Decisiones , Neuronas Dopaminérgicas/fisiología , Humanos , Modelos Psicológicos , Corteza Prefrontal/lesiones , Refuerzo en Psicología , Área Tegmental Ventral/citologíaRESUMEN
Since it was demonstrated the orbitofrontal cortex (OFC) is critical to reversal learning, there has been considerable interest in specifying its role in flexible, outcome-guided behavior. Behavioral paradigms from the learning theory tradition, such as outcome devaluation, blocking, Pavlovian to instrumental transfer, and overexpectation have been a driving force in this research. The use of these procedures has revealed OFC's unique role in forming and integrating information about specific features of events and outcomes to drive behavior and learning. These studies highlight the power and importance of learning theory principles in guiding neuroscience research.
Asunto(s)
Aprendizaje/fisiología , Modelos Psicológicos , Corteza Prefrontal/fisiología , Refuerzo en Psicología , Animales , Humanos , RecompensaRESUMEN
Imagination, defined as the ability to interpret reality in ways that diverge from past experience, is fundamental to adaptive behavior. This can be seen at a simple level in our capacity to predict novel outcomes in new situations. The ability to anticipate outcomes never before received can also influence learning if those imagined outcomes are not received. The orbitofrontal cortex is a key candidate for where the process of imagining likely outcomes occurs; however, its precise role in generating these estimates and applying them to learning remain open questions. Here we address these questions by showing that single-unit activity in the orbitofrontal cortex reflects novel outcome estimates. The strength of these neural correlates predicted both behavior and learning, learning that was abolished by temporally specific inhibition of orbitofrontal neurons. These results are consistent with the proposal that the orbitofrontal cortex is critical for integrating information to imagine future outcomes.
Asunto(s)
Condicionamiento Clásico/fisiología , Imaginación/fisiología , Corteza Prefrontal/fisiología , Potenciales de Acción/fisiología , Animales , Señales (Psicología) , Extinción Psicológica/fisiología , Masculino , Inhibición Neural/fisiología , Neuronas/fisiología , RatasRESUMEN
Learning is proposed to occur when there is a discrepancy between reward prediction and reward receipt. At least two separate systems are thought to exist: one in which predictions are proposed to be based on model-free or cached values; and another in which predictions are model-based. A basic neural circuit for model-free reinforcement learning has already been described. In the model-free circuit the ventral striatum (VS) is thought to supply a common-currency reward prediction to midbrain dopamine neurons that compute prediction errors and drive learning. In a model-based system, predictions can include more information about an expected reward, such as its sensory attributes or current, unique value. This detailed prediction allows for both behavioral flexibility and learning driven by changes in sensory features of rewards alone. Recent evidence from animal learning and human imaging suggests that, in addition to model-free information, the VS also signals model-based information. Further, there is evidence that the orbitofrontal cortex (OFC) signals model-based information. Here we review these data and suggest that the OFC provides model-based information to this traditional model-free circuitry and offer possibilities as to how this interaction might occur.
Asunto(s)
Ganglios Basales/fisiología , Lóbulo Frontal/fisiología , Aprendizaje/fisiología , Modelos Neurológicos , Red Nerviosa/fisiología , Animales , HumanosRESUMEN
The orbitofrontal cortex has been hypothesized to carry information regarding the value of expected rewards. Such information is essential for associative learning, which relies on comparisons between expected and obtained reward for generating instructive error signals. These error signals are thought to be conveyed by dopamine neurons. To test whether orbitofrontal cortex contributes to these error signals, we recorded from dopamine neurons in orbitofrontal-lesioned rats performing a reward learning task. Lesions caused marked changes in dopaminergic error signaling. However, the effect of lesions was not consistent with a simple loss of information regarding expected value. Instead, without orbitofrontal input, dopaminergic error signals failed to reflect internal information about the impending response that distinguished externally similar states leading to differently valued future rewards. These results are consistent with current conceptualizations of orbitofrontal cortex as supporting model-based behavior and suggest an unexpected role for this information in dopaminergic error signaling.
Asunto(s)
Potenciales de Acción/fisiología , Dopamina/metabolismo , Neuronas/fisiología , Corteza Prefrontal/citología , Recompensa , Animales , Biotina/análogos & derivados , Biotina/metabolismo , Conducta de Elección , Simulación por Computador , Estimulación Eléctrica , Masculino , Modelos Neurológicos , Ratas , Ratas Long-Evans , Factores de Tiempo , Tirosina 3-Monooxigenasa/metabolismo , Área Tegmental Ventral/citología , Área Tegmental Ventral/fisiologíaRESUMEN
While knowing what to expect is important, it is equally important to know when to expect it and to respond accordingly. This is apparent even in simple Pavlovian training situations in which animals learn to respond more strongly closer to reward delivery. Here we report that the nucleus accumbens core, an area well-positioned to represent information about the timing of impending rewards, plays a critical role in this timing function.