Regulation of reinforcement learning parameters captures long-term changes in rat behaviour.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related.

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.

Bayesian Mapping of the Striatal Microcircuit Reveals Robust Asymmetries in the Probabilities and Distances of Connections.

The striatum's complex microcircuit is made by connections within and between its D1- and D2-receptor expressing projection neurons and at least five species of interneuron. Precise knowledge of this circuit is likely essential to understanding striatum's functional roles and its dysfunction in a wide range of movement and cognitive disorders. We introduce here a Bayesian approach to mapping neuron connectivity using intracellular recording data, which lets us simultaneously evaluate the probability of connection between neuron types, the strength of evidence for it, and its dependence on distance. Using it to synthesize a complete map of the mouse striatum, we find strong evidence for two asymmetries: a selective asymmetry of projection neuron connections, with D2 neurons connecting twice as densely to other projection neurons than do D1 neurons, but neither subtype preferentially connecting to another; and a length-scale asymmetry, with interneuron connection probabilities remaining non-negligible at more than twice the distance of projection neuron connections. We further show that our Bayesian approach can evaluate evidence for wiring changes, using data from the developing striatum and a mouse model of Huntington's disease. By quantifying the uncertainty in our knowledge of the microcircuit, our approach reveals a wide range of potential striatal wiring diagrams consistent with current data.SIGNIFICANCE STATEMENT To properly understand a neuronal circuit's function, it is important to have an accurate picture of the rate of connection between individual neurons and how this rate changes with the distance separating pairs of neurons. We present a Bayesian method for extracting this information from experimental data and apply it to the mouse striatum, a subcortical structure involved in learning and decision-making, which is made up of a variety of different projection neurons and interneurons. Our resulting statistical map reveals not just the most robust estimates of the probability of connection between neuron types, but also the strength of evidence for them, and their dependence on distance.

Impacts of inter-trial interval duration on a computational model of sign-tracking vs. goal-tracking behaviour.

In the context of Pavlovian conditioning, two types of behaviour may emerge within the population (Flagel et al. Nature, 469(7328): 53-57, 2011). Animals may choose to engage either with the conditioned stimulus (CS), a behaviour known as sign-tracking (ST) which is sensitive to dopamine inhibition for its acquisition, or with the food cup in which the reward or unconditioned stimulus (US) will eventually be delivered, a behaviour known as goal-tracking (GT) which is dependent on dopamine for its expression only. Previous work by Lesaint et al. (PLoS Comput Biol, 10(2), 2014) offered a computational explanation for these phenomena and led to the prediction that varying the duration of the inter-trial interval (ITI) would change the relative ST-GT proportion in the population as well as phasic dopamine responses. A recent study verified this prediction, but also found a rich variance of ST and GT behaviours within the trial which goes beyond the original computational model. In this paper, we provide a computational perspective on these novel results.

Dopamine blockade impairs the exploration-exploitation trade-off in rats.

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.

What determines the relationship between color naming, unique hues, and sensory singularities: Illuminations, surfaces, or photoreceptors?

The relationship between the sensory signal of the photoreceptors on one hand and color appearance and language on the other hand is completely unclear. A recent finding established a surprisingly accurate correlation between focal colors, unique hues, and so-called singularities in the laws governing how sensory signals for different surfaces change across illuminations. This article examines how this correlation with singularities depends on reflectances, illuminants, and cone sensitivities. Results show that this correlation holds for a large range of illuminants and for a large range of sensors, including sensors that are fundamentally different from human photoreceptors. In contrast, the spectral characteristics of the reflectance spectra turned out to be the key factor that determines the correlation between focal colors, unique hues, and sensory singularities. These findings suggest that the origins of color appearance and color language may be found in particular characteristics of the reflectance spectra that correspond to focal colors and unique hues.