Acetylcholine demixes heterogeneous dopamine signals for learning and moving.

Midbrain dopamine neurons promote reinforcement learning and movement vigor. A major outstanding question is how dopamine-recipient neurons in the striatum parse these heterogeneous signals. Here we characterized dopamine and acetylcholine release in the dorsomedial striatum (DMS) of rats performing a decision-making task. We found that dopamine acted as a reward prediction error (RPE), modulating behavior and DMS spiking on subsequent trials when coincident with pauses in cholinergic release. In contrast, at task events that elicited coincident bursts of acetylcholine and dopamine, dopamine preceded contralateral movements and predicted movement vigor without inducing plastic changes in DMS firing rates. Our findings provide a circuit-level mechanism by which cholinergic modulation allows the same dopamine signals to be used for either movement or learning depending on instantaneous behavioral context.

Curriculum learning inspired by behavioral shaping trains neural networks to adopt animal-like decision making strategies.

Recurrent neural networks (RNN) are ubiquitously used in neuroscience to capture both neural dynamics and behaviors of living systems. However, when it comes to complex cognitive tasks, traditional methods for training RNNs can fall short in capturing crucial aspects of animal behavior. To address this challenge, we take inspiration from a commonly used (though rarely appreciated) approach from the experimental neuroscientist's toolkit: behavioral shaping. Our solution leverages task compositionality and models the animal's relevant learning experiences prior to the task. Taking as target a temporal wagering task previously studied in rats, we designed a pretraining curriculum of simpler cognitive tasks that are prerequisites for performing it well. These pretraining tasks are not just simplified versions of the temporal wagering task, but reflect relevant sub-computations. We show that this approach is required for RNNs to adopt similar strategies as rats, including long-timescale inference of latent states, which conventional pretraining approaches fail to capture. Mechanistically, our pretraining supports the development of key dynamical systems features needed for implementing both inference and value-based decision making. Overall, our approach addresses a gap in neural network model training by incorporating inductive biases of animals, which is important when modeling complex behaviors that rely on computational abilities acquired from past experiences.

Estrogenic control of reward prediction errors and reinforcement learning.

Gonadal hormones act throughout the brain 1 , and nearly all neuropsychiatric disorders vary in symptom severity with hormonal fluctuations over the reproductive cycle, gestation, and perimenopause 2-4 . Yet the mechanisms by which hormones influence mental and cognitive processes are unclear. Exogenous estrogenic hormones modulate dopamine signaling in the nucleus accumbens core (NAcc) 5,6 , which instantiates reward prediction errors (RPEs) for reinforcement learning 7-16 . Here we show that endogenous estrogenic hormones enhance RPEs and sensitivity to previous rewards by regulating expression of dopamine reuptake proteins in the NAcc. We trained rats to perform a temporal wagering task with different reward states; rats adjusted how quickly they initiated trials across states, balancing effort against expected rewards. Dopamine release in the NAcc reflected RPEs that predicted and causally in-fluenced subsequent initiation times. When fertile, females more quickly adjusted their initiation times to match reward states due to enhanced dopaminergic RPEs in the NAcc. Proteomics revealed reduced expression of dopamine transporters in fertile stages of the reproductive cycle. Finally, genetic suppression of midbrain estrogen receptors eliminated hormonal modulation of behavior. Estrogenic hormones therefore control the rate of reinforcement learning by regulating RPEs via dopamine reuptake, providing a mechanism by which hormones influence neural dynamics for motivation and learning.

Distinct value computations support rapid sequential decisions.

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or cache values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Distinct value computations support rapid sequential decisions.

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or "cache" values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Transformation of acoustic information to sensory decision variables in the parietal cortex.

The process by which sensory evidence contributes to perceptual choices requires an understanding of its transformation into decision variables. Here, we address this issue by evaluating the neural representation of acoustic information in the auditory cortex-recipient parietal cortex, while gerbils either performed a two-alternative forced-choice auditory discrimination task or while they passively listened to identical acoustic stimuli. During task engagement, stimulus identity decoding performance from simultaneously recorded parietal neurons significantly correlated with psychometric sensitivity. In contrast, decoding performance during passive listening was significantly reduced. Principal component and geometric analyses revealed the emergence of low-dimensional encoding of linearly separable manifolds with respect to stimulus identity and decision, but only during task engagement. These findings confirm that the parietal cortex mediates a transition of acoustic representations into decision-related variables. Finally, using a clustering analysis, we identified three functionally distinct subpopulations of neurons that each encoded task-relevant information during separate temporal segments of a trial. Taken together, our findings demonstrate how parietal cortex neurons integrate and transform encoded auditory information to guide sound-driven perceptual decisions.

Subpopulations of neurons in lOFC encode previous and current rewards at time of choice.

Studies of neural dynamics in lateral orbitofrontal cortex (lOFC) have shown that subsets of neurons that encode distinct aspects of behavior, such as value, may project to common downstream targets. However, it is unclear whether reward history, which may subserve lOFC's well-documented role in learning, is represented by functional subpopulations in lOFC. Previously, we analyzed neural recordings from rats performing a value-based decision-making task, and we documented trial-by-trial learning that required lOFC (Constantinople et al., 2019). Here, we characterize functional subpopulations of lOFC neurons during behavior, including their encoding of task variables. We found five distinct clusters of lOFC neurons, either based on clustering of their trial-averaged peristimulus time histograms (PSTHs), or a feature space defined by their average conditional firing rates aligned to different task variables. We observed weak encoding of reward attributes, but stronger encoding of reward history, the animal's left or right choice, and reward receipt across all clusters. Only one cluster, however, encoded the animal's reward history at the time shortly preceding the choice, suggesting a possible role in integrating previous and current trial outcomes at the time of choice. This cluster also exhibits qualitatively similar responses to identified corticostriatal projection neurons in a recent study (Hirokawa et al., 2019), and suggests a possible role for subpopulations of lOFC neurons in mediating trial-by-trial learning.

Tracking self-performance in the prefrontal cortex: It's layered.

In this issue of Cell, Spellman and colleagues record and manipulate the activity of neurons in the medial prefrontal cortex of mice performing a task in which they must pay attention to different stimuli. They show that this brain region is important for monitoring the animals' performance, and neurons that appear to contribute to behavior reside in deep cortical layers.

Parietal Cortex Is Required for the Integration of Acoustic Evidence.

Sensory-driven decisions are formed by accumulating information over time. Although parietal cortex activity is thought to represent accumulated evidence for sensory-based decisions, recent perturbation studies in rodents and non-human primates have challenged the hypothesis that these representations actually influence behavior. Here, we asked whether the parietal cortex integrates acoustic features from auditory cortical inputs during a perceptual decision-making task. If so, we predicted that selective inactivation of this projection should impair subjects' ability to accumulate sensory evidence. We trained gerbils to perform an auditory discrimination task and obtained measures of integration time as a readout of evidence accumulation capability. Minimum integration time was calculated behaviorally as the shortest stimulus duration for which subjects could discriminate the acoustic signals. Direct pharmacological inactivation of parietal cortex increased minimum integration times, suggesting its role in the behavior. To determine the specific impact of sensory evidence, we chemogenetically inactivated the excitatory projections from auditory cortex to parietal cortex and found this was sufficient to increase minimum behavioral integration times. Our signal-detection-theory-based model accurately replicated behavioral outcomes and indicated that the deficits in task performance were plausibly explained by elevated sensory noise. Together, our findings provide causal evidence that parietal cortex plays a role in the network that integrates auditory features for perceptual judgments.

Lateral orbitofrontal cortex promotes trial-by-trial learning of risky, but not spatial, biases.

Individual choices are not made in isolation but are embedded in a series of past experiences, decisions, and outcomes. The effects of past experiences on choices, often called sequential biases, are ubiquitous in perceptual and value-based decision-making, but their neural substrates are unclear. We trained rats to choose between cued guaranteed and probabilistic rewards in a task in which outcomes on each trial were independent. Behavioral variability often reflected sequential effects, including increased willingness to take risks following risky wins, and spatial 'win-stay/lose-shift' biases. Recordings from lateral orbitofrontal cortex (lOFC) revealed encoding of reward history and receipt, and optogenetic inhibition of lOFC eliminated rats' increased preference for risk following risky wins, but spared other sequential effects. Our data show that different sequential biases are neurally dissociable, and the lOFC's role in adaptive behavior promotes learning of more abstract biases (here, biases for the risky option), but not spatial ones.

An Analysis of Decision under Risk in Rats.

In 1979, Daniel Kahneman and Amos Tversky published a ground-breaking paper titled "Prospect Theory: An Analysis of Decision under Risk," which presented a behavioral economic theory that accounted for the ways in which humans deviate from economists' normative workhorse model, Expected Utility Theory [1, 2]. For example, people exhibit probability distortion (they overweight low probabilities), loss aversion (losses loom larger than gains), and reference dependence (outcomes are evaluated as gains or losses relative to an internal reference point). We found that rats exhibited many of these same biases, using a task in which rats chose between guaranteed and probabilistic rewards. However, prospect theory assumes stable preferences in the absence of learning, an assumption at odds with alternative frameworks such as animal learning theory and reinforcement learning [3-7]. Rats also exhibited trial history effects, consistent with ongoing learning. A reinforcement learning model in which state-action values were updated by the subjective value of outcomes according to prospect theory reproduced rats' nonlinear utility and probability weighting functions and also captured trial-by-trial learning dynamics.

Fronto-parietal Cortical Circuits Encode Accumulated Evidence with a Diversity of Timescales.

Decision-making in dynamic environments often involves accumulation of evidence, in which new information is used to update beliefs and select future actions. Using in vivo cellular resolution imaging in voluntarily head-restrained rats, we examined the responses of neurons in frontal and parietal cortices during a pulse-based accumulation of evidence task. Neurons exhibited activity that predicted the animal's upcoming choice, previous choice, and graded responses that reflected the strength of the accumulated evidence. The pulsatile nature of the stimuli enabled characterization of the responses of neurons to a single quantum (pulse) of evidence. Across the population, individual neurons displayed extensive heterogeneity in the dynamics of responses to pulses. The diversity of responses was sufficiently rich to form a temporal basis for accumulated evidence estimated from a latent variable model. These results suggest that heterogeneous, often transient sensory responses distributed across the fronto-parietal cortex may support working memory on behavioral timescales. VIDEO ABSTRACT.

Sources of noise during accumulation of evidence in unrestrained and voluntarily head-restrained rats.

Decision-making behavior is often characterized by substantial variability, but its source remains unclear. We developed a visual accumulation of evidence task designed to quantify sources of noise and to be performed during voluntary head restraint, enabling cellular resolution imaging in future studies. Rats accumulated discrete numbers of flashes presented to the left and right visual hemifields and indicated the side that had the greater number of flashes. Using a signal-detection theory-based model, we found that the standard deviation in their internal estimate of flash number scaled linearly with the number of flashes. This indicates a major source of noise that, surprisingly, is not consistent with the widely used 'drift-diffusion modeling' (DDM) approach but is instead closely related to proposed models of numerical cognition and counting. We speculate that this form of noise could be important in accumulation of evidence tasks generally.

Prefrontal cortex HCN1 channels enable intrinsic persistent neural firing and executive memory function.

In many cortical neurons, HCN1 channels are the major contributors to Ih, the hyperpolarization-activated current, which regulates the intrinsic properties of neurons and shapes their integration of synaptic inputs, paces rhythmic activity, and regulates synaptic plasticity. Here, we examine the physiological role of Ih in deep layer pyramidal neurons in mouse prefrontal cortex (PFC), focusing on persistent activity, a form of sustained firing thought to be important for the behavioral function of the PFC during working memory tasks. We find that HCN1 contributes to the intrinsic persistent firing that is induced by a brief depolarizing current stimulus in the presence of muscarinic agonists. Deletion of HCN1 or acute pharmacological blockade of Ih decreases the fraction of neurons capable of generating persistent firing. The reduction in persistent firing is caused by the membrane hyperpolarization that results from the deletion of HCN1 or Ih blockade, rather than a specific role of the hyperpolarization-activated current in generating persistent activity. In vivo recordings show that deletion of HCN1 has no effect on up states, periods of enhanced synaptic network activity. Parallel behavioral studies demonstrate that HCN1 contributes to the PFC-dependent resolution of proactive interference during working memory. These results thus provide genetic evidence demonstrating the importance of HCN1 to intrinsic persistent firing and the behavioral output of the PFC. The causal role of intrinsic persistent firing in PFC-mediated behavior remains an open question.

Deep cortical layers are activated directly by thalamus.

The thalamocortical (TC) projection to layer 4 (L4) is thought to be the main route by which sensory organs communicate with cortex. Sensory information is believed to then propagate through the cortical column along the L4→L2/3→L5/6 pathway. Here, we show that sensory-evoked responses of L5/6 neurons in rats derive instead from direct TC synapses. Many L5/6 neurons exhibited sensory-evoked postsynaptic potentials with the same latencies as L4. Paired in vivo recordings from L5/6 neurons and thalamic neurons revealed substantial convergence of direct TC synapses onto diverse types of infragranular neurons, particularly in L5B. Pharmacological inactivation of L4 had no effect on sensory-evoked synaptic input to L5/6 neurons. L4 is thus not an obligatory distribution hub for cortical activity, and thalamus activates two separate, independent "strata" of cortex in parallel.

Effects and mechanisms of wakefulness on local cortical networks.

Mammalian brains generate internal activity independent of environmental stimuli. Internally generated states may bring about distinct cortical processing modes. To investigate how brain state impacts cortical circuitry, we recorded intracellularly from the same neurons, under anesthesia and subsequent wakefulness, in rat barrel cortex. In every cell examined throughout layers 2-6, wakefulness produced a temporal pattern of synaptic inputs differing markedly from those under anesthesia. Recurring periods of synaptic quiescence, prominent under anesthesia, were abolished by wakefulness, which produced instead a persistently depolarized state. This switch in dynamics was unaffected by elimination of afferent synaptic input from thalamus, suggesting that arousal alters cortical dynamics by neuromodulators acting directly on cortex. Indeed, blockade of noradrenergic, but not cholinergic, pathways induced synaptic quiescence during wakefulness. We conclude that global brain states can switch local recurrent networks into different regimes via direct neuromodulation.

Quantitative analysis of neurons with Kv3 potassium channel subunits, Kv3.1b and Kv3.2, in macaque primary visual cortex.

Voltage-gated potassium channels that are composed of Kv3 subunits exhibit distinct electrophysiological properties: activation at more depolarized potentials than other voltage-gated K+ channels and fast kinetics. These channels have been shown to contribute to the high-frequency firing of fast-spiking (FS) GABAergic interneurons in the rat and mouse brain. In the rodent neocortex there are distinct patterns of expression for the Kv3.1b and Kv3.2 channel subunits and of coexpression of these subunits with neurochemical markers, such as the calcium-binding proteins parvalbumin (PV) and calbindin D-28K (CB). The distribution of Kv3 channels and interrelationship with calcium-binding protein expression has not been investigated in primate cortex. We used immunoperoxidase and immunofluorescent labeling and stereological counting techniques to characterize the laminar and cell-type distributions of Kv3-immunoreactive (ir) neurons in macaque V1. We found that across the cortical layers approximately 25% of both Kv3.1b- and Kv3.2-ir neurons are non-GABAergic. In contrast, all Kv3-ir neurons in rodent cortex are GABAergic (Chow et al. [1999] J Neurosci. 19:9332-9345). The putatively excitatory Kv3-ir neurons were mostly located in layers 2, 3, and 4b. Further, the proportion of Kv3-ir neurons that express PV or CB also differs between macaque V1 and rodent cortex. These data indicate that, within the population of cortical neurons, a broader population of neurons, encompassing cells of a wider range of morphological classes may be capable of sustaining high-frequency firing in macaque V1.