Expectancy-related changes in firing of dopamine neurons depend on hippocampus.

The orbitofrontal cortex (OFC) and hippocampus (HC) are both implicated in forming the cognitive or task maps that support flexible behavior. Previously, we used the dopamine neurons as a sensor or tool to measure the functional effects of OFC lesions (Takahashi et al., 2011). We recorded midbrain dopamine neurons as rats performed an odor-based choice task, in which errors in the prediction of reward were induced by manipulating the number or timing of the expected rewards across blocks of trials. We found that OFC lesions ipsilateral to the recording electrodes caused prediction errors to be degraded consistent with a loss in the resolution of the task states, particularly under conditions where hidden information was critical to sharpening the predictions. Here we have repeated this experiment, along with computational modeling of the results, in rats with ipsilateral HC lesions. The results show HC also shapes the map of our task, however unlike OFC, which provides information local to the trial, the HC appears to be necessary for estimating the upper-level hidden states based on the information that is discontinuous or separated by longer timescales. The results contrast the respective roles of the OFC and HC in cognitive mapping and add to evidence that the dopamine neurons access a rich information set from distributed regions regarding the predictive structure of the environment, potentially enabling this powerful teaching signal to support complex learning and behavior.

Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model.

Dopamine neuron activity is tied to the prediction error in temporal difference reinforcement learning models. These models make significant simplifying assumptions, particularly with regard to the structure of the predictions fed into the dopamine neurons, which consist of a single chain of timepoint states. Although this predictive structure can explain error signals observed in many studies, it cannot cope with settings where subjects might infer multiple independent events and outcomes. In the present study, we recorded dopamine neurons in the ventral tegmental area in such a setting to test the validity of the single-stream assumption. Rats were trained in an odor-based choice task, in which the timing and identity of one of several rewards delivered in each trial changed across trial blocks. This design revealed an error signaling pattern that requires the dopamine neurons to access and update multiple independent predictive streams reflecting the subject's belief about timing and potentially unique identities of expected rewards.

Ending on a familiar note: Perceived endings motivate repeat consumption.

People fill their free time by choosing between hedonic activities that are new and exciting (e.g., exploring a buzzed-about restaurant) versus old and familiar (e.g., revisiting the same old spot). The dominant psychological assumption is that people will prefer novelty, holding constant factors like cost, availability, and convenience between acquiring such options ("variety is the spice of life"). Eight preregistered experiments (total N = 5,889) reveal that people's attraction to novelty depends, at least in part, on their temporal context-namely, on perceived endings. As participants faced a shrinking window of opportunity to enjoy a general category of experience (even merely temporarily; e.g., eating one's last dessert before starting a diet), their hedonic preferences shifted away from new and exciting options and toward old favorites. This relative shift emerged across many domains (e.g., food, travel, music), situations (e.g., impending New Year's resolutions, COVID-19 shutdowns), and consequential behaviors (e.g., choices with financial stakes). Using both moderation and mediation approaches, we found that perceived endings increase the preference for familiarity because they increase people's desire to ensure a personally meaningful experience on which to end, and returning to old favorites is typically more meaningful than exploring novelty. Endings increased participants' preference for familiarity even when it meant sacrificing other desirable attributes (e.g., exciting stimulation). Together, these findings advance and bridge research on hedonic preferences, time and timing, and the motivational effects of change. Variety may be the "spice of life," but familiarity may be the spice of life's endings. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Pivotal voting: The opportunity to tip group decisions skews juries and other voting outcomes.

Many important social and policy decisions are made by small groups of people (e.g., juries, college admissions officers, or corporate boards) with the hope that a collective process will yield better and fairer decisions. In many instances, it is possible for these groups to fail to reach a decision by not garnering a minimum number of votes (e.g., hung juries). Our research finds that pivotal voters vote to avoid such decision failure-voters who can "tip" their group into a punishment decision will be more likely to do so. This effect is distinct from well-known social pressures to simply conform with others or reach unanimity. Using observational data from Louisiana court cases, we find a sharp discontinuity in juries' voting decisions at the threshold between indecision and conviction (Study 1). In a third-party punishment paradigm, pivotal voters were more likely to vote to punish a target than nonpivotal voters, even when holding social information constant (Study 2), and adopted harsher views about the target's deservingness of punishment (Study 3). Using vignettes, we find that pivotal voters are judged to be differentially responsible for the outcomes of their votes-those who "block" the group from reaching a punishment decision are deemed more responsible for the outcome than those who "fall in line" (Study 4). These findings provide insight into how we might improve group decision-making environments to ensure that their outcomes accurately reflect group members' actual beliefs and not the influence of social pressures.

Minimal cross-trial generalization in learning the representation of an odor-guided choice task.

There is no single way to represent a task. Indeed, despite experiencing the same task events and contingencies, different subjects may form distinct task representations. As experimenters, we often assume that subjects represent the task as we envision it. However, such a representation cannot be taken for granted, especially in animal experiments where we cannot deliver explicit instruction regarding the structure of the task. Here, we tested how rats represent an odor-guided choice task in which two odor cues indicated which of two responses would lead to reward, whereas a third odor indicated free choice among the two responses. A parsimonious task representation would allow animals to learn from the forced trials what is the better option to choose in the free-choice trials. However, animals may not necessarily generalize across odors in this way. We fit reinforcement-learning models that use different task representations to trial-by-trial choice behavior of individual rats performing this task, and quantified the degree to which each animal used the more parsimonious representation, generalizing across trial types. Model comparison revealed that most rats did not acquire this representation despite extensive experience. Our results demonstrate the importance of formally testing possible task representations that can afford the observed behavior, rather than assuming that animals' task representations abide by the generative task structure that governs the experimental design.

Social exploration: When people deviate from options explored by others.

People often face choices between known options and unknown ones. Our research documents a social-exploration effect: People are more likely to explore unknown options when they learn about known options from other people's experiences. Across four studies (N = 2,333), we used an incentive-compatible paradigm where participants chose between known and unknown options (e.g., cash bonuses). We found higher exploration rates (i.e., choosing of unknown options) when information about known options came from other people, compared with an unidentified source (Study 1a) or a computer (Studies 1b-4). We theorize that the social-exploration effect results from people's tendency to intuitively adopt a group-level perspective with other people: a "we"-perspective. Thus, in social contexts, people explore more to diversify their experience as a group. Supporting this account, we find the effect attenuates in exploration of losses, where people do not wish to adopt a group-level perspective of others' losses (Study 2). Furthermore, the effect is obtained only if others have experienced the outcome; not when they only revealed its content (Study 3). Finally, the social-exploration effect generalizes to everyday choices, such as choosing a movie to watch (Study 4). Taken together, these findings highlight the social aspect of individual exploration decisions and offer practical implications for how to encourage exploration. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Dopamine neuron ensembles signal the content of sensory prediction errors.

Dopamine neurons respond to errors in predicting value-neutral sensory information. These data, combined with causal evidence that dopamine transients support sensory-based associative learning, suggest that the dopamine system signals a multidimensional prediction error. Yet such complexity is not evident in the activity of individual neurons or population averages. How then do downstream areas know what to learn in response to these signals? One possibility is that information about content is contained in the pattern of firing across many dopamine neurons. Consistent with this, here we show that the pattern of firing across a small group of dopamine neurons recorded in rats signals the identity of a mis-predicted sensory event. Further, this same information is reflected in the BOLD response elicited by sensory prediction errors in human midbrain. These data provide evidence that ensembles of dopamine neurons provide highly specific teaching signals, opening new possibilities for how this system might contribute to learning.

Expectancy-Related Changes in Dopaminergic Error Signals Are Impaired by Cocaine Self-Administration.

Addiction is a disorder of behavioral control and learning. While this may reflect pre-existing propensities, drug use also clearly contributes by causing changes in outcome processing in prefrontal and striatal regions. This altered processing is associated with behavioral deficits, including changes in learning. These areas provide critical input to midbrain dopamine neurons regarding expected outcomes, suggesting that effects on learning may result from changes in dopaminergic error signaling. Here, we show that dopamine neurons recorded in rats that had self-administered cocaine failed to suppress firing on omission of an expected reward and exhibited lower amplitude and imprecisely timed increases in firing to an unexpected reward. Learning also appeared to have less of an effect on reward-evoked and cue-evoked firing in the cocaine-experienced rats. Overall, the changes are consistent with reduced fidelity of input regarding the expected outcomes, such as their size, timing, and overall value, because of cocaine use.

Orbitofrontal neurons signal reward predictions, not reward prediction errors.

Neurons in the orbitofrontal cortex (OFC) fire in anticipation of and during rewards. Such firing has been suggested to encode reward predictions and to account in some way for the role of this area in adaptive behavior and learning. However, it has also been reported that neural activity in OFC reflects reward prediction errors, which might drive learning directly. Here we tested this question by analyzing the firing of OFC neurons recorded in an odor discrimination task in which rats were trained to sample odor cues and respond left or right on each trial for reward. Neurons were recorded across blocks of trials in which we switched either the number or the flavor of the reward delivered in each well. Previously we have described how neurons in this dataset fired to the predictive cues (Stalnaker et al., 2014); here we focused on the firing in anticipation of and just after delivery of each drop of reward, looking specifically for differences in firing based on whether the reward number or flavor was unexpected or expected. Unlike dopamine neurons recorded in this setting, which exhibited phasic error-like responses after surprising changes in either reward number or reward flavor (Takahashi et al., 2017), OFC neurons showed no such error correlates and instead fired in a way that reflected reward predictions.

Dopamine Neurons Respond to Errors in the Prediction of Sensory Features of Expected Rewards.

Midbrain dopamine neurons have been proposed to signal prediction errors as defined in model-free reinforcement learning algorithms. While these algorithms have been extremely powerful in interpreting dopamine activity, these models do not register any error unless there is a difference between the value of what is predicted and what is received. Yet learning often occurs in response to changes in the unique features that characterize what is received, sometimes with no change in its value at all. Here, we show that classic error-signaling dopamine neurons also respond to changes in value-neutral sensory features of an expected reward. This suggests that dopamine neurons have access to a wider variety of information than contemplated by the models currently used to interpret their activity and that, while their firing may conform to predictions of these models in some cases, they are not restricted to signaling errors in the prediction of value.

Effects of inference on dopaminergic prediction errors depend on orbitofrontal processing.

Dopaminergic reward prediction errors in monkeys reflect inferential reward predictions that well-trained animals can make when associative rules change. Here, in a new analysis of previously described data, we test whether dopaminergic error signals in rats are influenced by inferential predictions and whether such effects depend on the orbitofrontal cortex (OFC). Dopamine neurons were recorded from controls or rats with ipsilateral OFC lesions during performance of a choice task in which odor cues signaled the availability of sucrose reward in 2 wells. To induce prediction errors, we manipulated either the timing or number of rewards delivered in each well across blocks of trials. Of importance, a change in reward at 1 well predicted a change in reward at the other on later trials. We compared behavior and neural activity on trials when such inference was possible versus trials involving the same reward change when inference was not possible. Rats responded faster when they could infer an increase in reward compared to when the same reward was coming but they could not infer a change. This inferential prediction was reflected in the firing of dopamine neurons in controls, which changed less to unexpected delivery (or omission) of reward and more to the new high-value cue on inference versus noninference trials. These effects were absent in dopamine neurons recorded in rats with ipsilateral OFC lesions. Thus, dopaminergic error signals recorded in rats are influenced by both experiential and inferential reward predictions, and the effects of inferential predictions depend on OFC. (PsycINFO Database Record

Ventral striatal lesions disrupt dopamine neuron signaling of differences in cue value caused by changes in reward timing but not number.

We recently showed that ventral striatal (VS) lesions abolished reward prediction errors that reflect changes in reward timing while leaving relatively unaffected errors that reflect changes in reward number. Here we extended those results by examining whether normal learning-related changes in firing to the reward-predictive cues in the same dopamine neurons were also disrupted by VS lesions. This analysis revealed that dopamine neurons recorded in VS-lesioned rats failed to show value-related changes in firing to the cues in timing but showed normal changes in number blocks. This effect suggests that the loss of reward-evoked error signals in the timing blocks impaired encoding of the cue value in downstream areas, which then supply these predictions to dopamine neurons at the time of cue presentation. (PsycINFO Database Record

Thinking Outside the Box: Orbitofrontal Cortex, Imagination, and How We Can Treat Addiction.

Addiction involves an inability to control drug-seeking behavior. While this may be thought of as secondary to an overwhelming desire for drugs, it could equally well reflect a failure of the brain mechanisms that allow addicts to learn about and mentally simulate non-drug consequences. Importantly, this process of mental simulation draws upon, but is not normally bound by, our past experiences. Rather we have the ability to think outside the box of our past, integrating knowledge gained from a variety of similar and not-so-similar life experiences to derive estimates or imagine what might happen next. These estimates influence our current behavior directly and also affect future behavior by serving as the background against which outcomes are evaluated to support learning. Here we will review evidence, from our own work using a Pavlovian over-expectation task as well as from other sources, that the orbitofrontal cortex is a critical node in the neural circuit that generates these estimates. Further we will offer the specific hypothesis that degradation of this function secondary to drug-induced changes is a critical and likely addressable part of addiction.

Temporal Specificity of Reward Prediction Errors Signaled by Putative Dopamine Neurons in Rat VTA Depends on Ventral Striatum.

Dopamine neurons signal reward prediction errors. This requires accurate reward predictions. It has been suggested that the ventral striatum provides these predictions. Here we tested this hypothesis by recording from putative dopamine neurons in the VTA of rats performing a task in which prediction errors were induced by shifting reward timing or number. In controls, the neurons exhibited error signals in response to both manipulations. However, dopamine neurons in rats with ipsilateral ventral striatal lesions exhibited errors only to changes in number and failed to respond to changes in timing of reward. These results, supported by computational modeling, indicate that predictions about the temporal specificity and the number of expected reward are dissociable and that dopaminergic prediction-error signals rely on the ventral striatum for the former but not the latter.

Neural Estimates of Imagined Outcomes in Basolateral Amygdala Depend on Orbitofrontal Cortex.

Reciprocal connections between the orbitofrontal cortex (OFC) and the basolateral nucleus of the amygdala (BLA) provide a critical circuit for guiding normal behavior when information about expected outcomes is required. Recently, we reported that outcome signaling by OFC neurons is also necessary for learning in the face of unexpected outcomes during a Pavlovian over-expectation task. Key to learning in this task is the ability to build on prior learning to infer or estimate an amount of reward never previously received. OFC was critical to this process. Notably, in parallel work, we found that BLA was not necessary for learning in this setting. This suggested a dissociation in which the BLA might be critical for acquiring information about the outcomes but not for subsequently using it to make novel predictions. Here we evaluated this hypothesis by recording single-unit activity from BLA in rats during the same Pavlovian over-expectation task used previously. We found that spiking activity recorded in BLA in control rats did reflect novel outcome estimates derived from the integration of prior learning, however consistent with a model in which this process occurs in the OFC, these correlates were entirely abolished by ipsilateral OFC lesions. These data indicate that this information about these novel predictions is represented in the BLA, supported via direct or indirect input from the OFC, even though it does not appear to be necessary for learning. SIGNIFICANCE STATEMENT: The basolateral nucleus of the amygdala (BLA) and the orbitofrontal cortex (OFC) are involved in behavior that depends on knowledge of impending outcomes. Recently, we found that only the OFC was necessary for using such information for learning in a Pavlovian over-expectation task. The current experiment was designed to search for neural correlates of this process in the BLA and, if present, to ask whether they would still be dependent on OFC input. We found that although spiking activity in BLA in control rats did reflect the novel outcome estimates underlying learning, these correlates were entirely abolished by OFC lesions.

Orbitofrontal activation restores insight lost after cocaine use.

Addiction is characterized by a lack of insight into the likely outcomes of one's behavior. Insight, or the ability to imagine outcomes, is evident when outcomes have not been directly experienced. Using this concept, work in both rats and humans has recently identified neural correlates of insight in the medial and orbital prefrontal cortices. We found that these correlates were selectively abolished in rats by cocaine self-administration. Their abolition was associated with behavioral deficits and reduced synaptic efficacy in orbitofrontal cortex, the reversal of which by optogenetic activation restored normal behavior. These results provide a link between cocaine use and problems with insight. Deficits in these functions are likely to be particularly important for problems such as drug relapse, in which behavior fails to account for likely adverse outcomes. As such, our data provide a neural target for therapeutic approaches to address these defining long-term effects of drug use.

Impact of sex incompatibility on the outcome of single-unit cord blood transplantation for adult patients with hematological malignancies.

Donor-recipient sex incompatibility has been associated with transplant outcomes in allogeneic hematopoietic SCT. Such outcomes might be because mHA encoded by Y chromosome genes could be immunological targets for allogeneic T cells and B cells to induce GVHD, GVL effect and graft failure. However, its effect on the outcome of cord blood transplantation (CBT) is yet to be clarified. We retrospectively analyzed 191 adult patients who received single-unit CBT after myeloablative conditioning for malignant disease in our institute. In multivariate analysis, male recipients with female donors had a higher incidence of extensive chronic GVHD (hazard ratio (HR) 2.97, P=0.02), and female recipients with male donors had a lower incidence of platelet engraftment (HR 0.56, P=0.02) compared with female recipients with female donors as the reference. Nevertheless, there was no increase in mortality following sex-incompatible CBT. These data suggested that donor-recipient sex compatibility does not have a significant impact on survival after myeloablative CBT for hematological malignancies.

Orbitofrontal cortex as a cognitive map of task space.

Orbitofrontal cortex (OFC) has long been known to play an important role in decision making. However, the exact nature of that role has remained elusive. Here, we propose a unifying theory of OFC function. We hypothesize that OFC provides an abstraction of currently available information in the form of a labeling of the current task state, which is used for reinforcement learning (RL) elsewhere in the brain. This function is especially critical when task states include unobservable information, for instance, from working memory. We use this framework to explain classic findings in reversal learning, delayed alternation, extinction, and devaluation as well as more recent findings showing the effect of OFC lesions on the firing of dopaminergic neurons in ventral tegmental area (VTA) in rodents performing an RL task. In addition, we generate a number of testable experimental predictions that can distinguish our theory from other accounts of OFC function.