Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning.

Different brain systems have been hypothesized to subserve multiple "experts" that compete to generate behavior. In reinforcement learning, two general processes, one model-free (MF) and one model-based (MB), are often modeled as a mixture of agents (MoA) and hypothesized to capture differences between automaticity vs. deliberation. However, shifts in strategy cannot be captured by a static MoA. To investigate such dynamics, we present the mixture-of-agents hidden Markov model (MoA-HMM), which simultaneously learns inferred action values from a set of agents and the temporal dynamics of underlying "hidden" states that capture shifts in agent contributions over time. Applying this model to a multi-step,reward-guided task in rats reveals a progression of within-session strategies: a shift from initial MB exploration to MB exploitation, and finally to reduced engagement. The inferred states predict changes in both response time and OFC neural encoding during the task, suggesting that these states are capturing real shifts in dynamics.

Place cell assemblies remain intact, despite reduced phase precession, after cholinergic disruption.

The hippocampal theta rhythm is frequently viewed as a clocking mechanism that coordinates the spiking activity of neurons across the hippocampus to form coherent neural assemblies. Phase precession is a form of temporal coding evidencing this mechanism and is degraded following systemic pharmacological disruption of cholinergic signaling. However, whether neural assemblies are commensurately degraded, as would be predicted from a clocking mechanism hypothesis, remains unknown. To address this, we recorded the spiking activity of hippocampal place cells as rats completed laps on a circle track for chocolate drink before versus during the influence of a systemic muscarinic acetylcholine receptor antagonist. We compared the integrity of hippocampal ensembles using three approaches. The first approach used cross-correlogram (CCG) analyses to ask if the relative spike-timing between pairs of cells became less reliable. The second used a general linear model based analysis to ask whether the activity of simultaneously recorded neurons became any less predictive of the spiking activity of single neurons. Finally, the third approach used a reconstruction analysis to ask if the population activity was any less informative regarding the environmental position of the animal and whether theta sequences were impaired. The results of all three analyses paint a consistent picture: systemic cholinergic disruption did not degrade assembly integrity. These data demonstrate that place cell assemblies do not depend upon high quality phase precession.

Precise spike timing dynamics of hippocampal place cell activity sensitive to cholinergic disruption.

New memory formation depends on both the hippocampus and modulatory effects of acetylcholine. The mechanism by which acetylcholine levels in the hippocampus enable new encoding remains poorly understood. Here, we tested the hypothesis that cholinergic modulation supports memory formation by leading to structured spike timing in the hippocampus. Specifically, we tested if phase precession in dorsal CA1 was reduced under the influence of a systemic cholinergic antagonist. Unit and field potential were recorded from the dorsal CA1 of rats as they completed laps on a circular track for food rewards before and during the influence of the systemically administered acetylcholine muscarinic receptor antagonist scopolamine. We found that scopolamine significantly reduced phase precession of spiking relative to the field theta, and that this was due to a decrease in the frequency of the spiking rhythmicity. We also found that the correlation between position and theta phase was significantly reduced. This effect was not due to changes in spatial tuning as tuning remained stable for those cells analyzed. Similarly, it was not due to changes in lap-to-lap reliability of spiking onset or offset relative to either position or phase as the reliability did not decrease following scopolamine administration. These findings support the hypothesis that memory impairments that follow muscarinic blockade are the result of degraded spike timing in the hippocampus.