Regulation of reinforcement learning parameters captures long-term changes in rat behaviour.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Estimation of the transmission delays in the basal ganglia of the macaque monkey and subsequent predictions about oscillatory activity under dopamine depletion.

The timescales of the dynamics of a system depend on the combination of the timescales of its components and of its transmission delays between components. Here, we combine experimental stimulation data from 10 studies in macaque monkeys that reveal the timing of excitatory and inhibitory events in the basal ganglia circuit, to estimate its set of transmission delays. In doing so, we reveal possible inconsistencies in the existing data, calling for replications, and we propose two possible sets of transmission delays. We then integrate these delays in a model of the primate basal ganglia that does not rely on direct and indirect pathways' segregation and show that extrastriatal dopaminergic depletion in the external part of the globus pallidus and in the subthalamic nucleus is sufficient to generate ß-band oscillations (in the high part, 20-35 Hz, of the band). More specifically, we show that D2 and D5 dopamine receptors in these nuclei play opposing roles in the emergence of these oscillations, thereby explaining how completely deactivating D5 receptors in the subthalamic nucleus can, paradoxically, cancel oscillations.

A neural substrate for negative affect dictates female parental behavior.

Parental behaviors secure the well-being of newborns and concomitantly limit negative affective states in adults, which emerge when coping with neonatal distress becomes challenging. Whether negative-affect-related neuronal circuits orchestrate parental actions is unknown. Here, we identify parental signatures in lateral habenula neurons receiving bed nucleus of stria terminalis innervation (BNSTLHb). We find that LHb neurons of virgin female mice increase their activity following pup distress vocalization and are necessary for pup-call-driven aversive behaviors. LHb activity rises during pup retrieval, a behavior worsened by LHb inactivation. Intersectional cell identification and transcriptional profiling associate BNSTLHb cells to parenting and outline a gene expression in female virgins similar to that in mothers but different from that in non-parental virgin male mice. Finally, tracking and manipulating BNSTLHb cell activity demonstrates their specificity for encoding negative affect and pup retrieval. Thus, a negative affect neural circuit processes newborn distress signals and may limit them by guiding female parenting.

Modulation of Hippocampal Network Oscillation by PICK1-Dependent Cell Surface Expression of mGlu3 Receptors.

Metabotropic glutamate receptor Type 3 (mGlu3) controls the sleep/wake architecture, which plays a role in the glutamatergic pathophysiology of schizophrenia. Interestingly, mGlu3 receptor expression is decreased in the brain of schizophrenic patients. However, little is known about the molecular mechanisms regulating mGlu3 receptors at the cell membrane. Subcellular receptor localization is strongly dependent on protein-protein interactions. Here we show that mGlu3 interacts with PICK1 and that this scaffolding protein is important for mGlu3 surface expression and function in hippocampal primary cultures. Disruption of their interaction via an mGlu3 C-terminal mimicking peptide or an inhibitor of the PDZ domain of PICK1 altered the functional expression of mGlu3 receptors in neurons. We next investigated the impact of disrupting the mGlu3-PICK1 interaction on hippocampal theta oscillations in vitro and in vivo in WT male mice. We found a decreased frequency of theta oscillations in organotypic hippocampal slices, similar to what was previously observed in mGlu3 KO mice. In addition, hippocampal theta power was reduced during rapid eye movement sleep, non-rapid eye movement (NREM) sleep, and wake states after intraventricular administration of the mGlu3 C-terminal mimicking peptide. Targeting the mGlu3-PICK1 complex could thus be relevant to the pathophysiology of schizophrenia.SIGNIFICANCE STATEMENT Dysregulation of the glutamatergic system might play a role in the pathophysiology of schizophrenia. Metabotropic glutamate receptors Type 3 (mGlu3) have been proposed as potential targets for schizophrenia. Understanding the molecular mechanisms regulating mGlu3 receptor at the cell membrane is critical toward comprehending how their dysfunction contributes to the pathogenesis of schizophrenia. Here we describe that the binding of the signaling and scaffolding protein PICK1 to mGlu3 receptors is important for their localization and physiological functions. The identification of new proteins that associate specifically to mGlu3 receptors will advance our understanding of the regulatory mechanisms associated with their targeting and function and ultimately might provide new therapeutic strategies to counter these psychiatric conditions.

Model-Based and Model-Free Replay Mechanisms for Reinforcement Learning in Neurorobotics.

Experience replay is widely used in AI to bootstrap reinforcement learning (RL) by enabling an agent to remember and reuse past experiences. Classical techniques include shuffled-, reversed-ordered- and prioritized-memory buffers, which have different properties and advantages depending on the nature of the data and problem. Interestingly, recent computational neuroscience work has shown that these techniques are relevant to model hippocampal reactivations recorded during rodent navigation. Nevertheless, the brain mechanisms for orchestrating hippocampal replay are still unclear. In this paper, we present recent neurorobotics research aiming to endow a navigating robot with a neuro-inspired RL architecture (including different learning strategies, such as model-based (MB) and model-free (MF), and different replay techniques). We illustrate through a series of numerical simulations how the specificities of robotic experimentation (e.g., autonomous state decomposition by the robot, noisy perception, state transition uncertainty, non-stationarity) can shed new lights on which replay techniques turn out to be more efficient in different situations. Finally, we close the loop by raising new hypotheses for neuroscience from such robotic models of hippocampal replay.

VTA dopamine neuron activity encodes social interaction and promotes reinforcement learning through social prediction error.

Social interactions are motivated behaviors that, in many species, facilitate learning. However, how the brain encodes the reinforcing properties of social interactions remains unclear. In this study, using in vivo recording in freely moving mice, we show that dopamine (DA) neurons of the ventral tegmental area (VTA) increase their activity during interactions with an unfamiliar conspecific and display heterogeneous responses. Using a social instrumental task, we then show that VTA DA neuron activity encodes social prediction error and drives social reinforcement learning. Thus, our findings suggest that VTA DA neurons are a neural substrate for a social learning signal that drives motivated behavior.

Deconstructing the contribution of sensory cues in social approach.

Social interaction is a complex and highly conserved behavior that safeguards survival and reproductive success. Although considerable progress has been made regarding our understanding of same-sex conspecific and non-aggressive interactions, questions regarding the precise contribution of sensory cues in social approach and their specific neurobiological correlates remain open. Here, by designing a series of experiments with diverse social and object stimuli manipulations in custom-made enclosures, we first sought to deconstruct key elements of social preference as assessed by the three-chamber task. Our results highlight the importance of social olfactory cues in approach behavior. Subsequently, we interrogated whether a social odor would activate dopaminergic neurons of the Ventral Tegmental Area in the same way as a juvenile conspecific would. Employing in vivo recordings in freely behaving mice, we observed an increase of the firing only during the transition toward the juvenile mouse and not during the transition toward the object impregnated with social odor, suggesting that these two experiences are distinct and can be differentiated at the neuronal level. Moreover, using a four-choice task, we further showed that mice prefer to explore complex social stimuli compared to isolated sensory cues. Our findings offer insights toward understanding how different sensory modalities contribute to the neurobiological basis of social behavior which can be essential when studying social deficits observed in autism-, depression-, anxiety-, or schizophrenia-related mouse models.

A biologically constrained spiking neural network model of the primate basal ganglia with overlapping pathways exhibits action selection.

Action selection has been hypothesized to be a key function of the basal ganglia, yet the nuclei involved, their interactions and the importance of the direct/indirect pathway segregation in such process remain debated. Here, we design a spiking computational model of the monkey basal ganglia derived from a previously published population model, initially parameterized to reproduce electrophysiological activity at rest and to embody as much quantitative anatomical data as possible. As a particular feature, both models exhibit the strong overlap between the direct and indirect pathways that has been documented in non-human primates. Here, we first show how the translation from a population to an individual neuron model was achieved, with the addition of a minimal number of parameters. We then show that our model performs action selection, even though it was built without any assumption on the activity carried out during behaviour. We investigate the mechanisms of this selection through circuit disruptions and found an instrumental role of the off-centre/on-surround structure of the MSN-STN-GPi circuit, as well as of the MSN-MSN and FSI-MSN projections. This validates their potency in enabling selection. We finally study the pervasive centromedian and parafascicular thalamic inputs that reach all basal ganglia nuclei and whose influence is therefore difficult to anticipate. Our model predicts that these inputs modulate the responsiveness of action selection, making them a candidate for the regulation of the speed-accuracy trade-off during decision-making.

Gelatinase Biosensor Reports Cellular Remodeling During Epileptogenesis.

Epileptogenesis is the gradual process responsible for converting a healthy brain into an epileptic brain. This process can be triggered by a wide range of factors, including brain injury or tumors, infections, and status epilepticus. Epileptogenesis results in aberrant synaptic plasticity, neuroinflammation and seizure-induced cell death. As Matrix Metalloproteinases (MMPs) play a crucial role in cellular plasticity by remodeling the extracellular matrix (ECM), gelatinases (MMP-2 and MMP-9) were recently highlighted as key players in epileptogenesis. In this work, we engineered a biosensor to report in situ gelatinase activity in a model of epileptogenesis. This biosensor encompasses a gelatinase-sensitive activatable cell penetrating peptide (ACPP) coupled to a TAMRA fluorophore, allowing fluorescence uptake in cells displaying endogenous gelatinase activities. In a preclinical mouse model of temporal lobe epilepsy (TLE), the intrahippocampal kainate injection, ACPPs revealed a localized distribution of gelatinase activities, refining temporal cellular changes during epileptogenesis. The activity was found particularly but not only in the ipsilateral hippocampus, starting from the CA1 area and spreading to dentate gyrus from the early stages throughout chronic epilepsy, notably in neurons and microglial cells. Thus, our work shows that ACPPs are suitable molecular imaging probes for detecting the spatiotemporal pattern of gelatinase activity during epileptogenesis, suggesting their possible use as vectors to target cellular reactive changes with treatment for epileptogenesis.

Modeling awake hippocampal reactivations with model-based bidirectional search.

Hippocampal offline reactivations during reward-based learning, usually categorized as replay events, have been found to be important for performance improvement over time and for memory consolidation. Recent computational work has linked these phenomena to the need to transform reward information into state-action values for decision making and to propagate it to all relevant states of the environment. Nevertheless, it is still unclear whether an integrated reinforcement learning mechanism could account for the variety of awake hippocampal reactivations, including variety in order (forward and reverse reactivated trajectories) and variety in the location where they occur (reward site or decision-point). Here, we present a model-based bidirectional search model which accounts for a variety of hippocampal reactivations. The model combines forward trajectory sampling from current position and backward sampling through prioritized sweeping from states associated with large reward prediction errors until the two trajectories connect. This is repeated until stabilization of state-action values (convergence), which could explain why hippocampal reactivations drastically diminish when the animal's performance stabilizes. Simulations in a multiple T-maze task show that forward reactivations are prominently found at decision-points while backward reactivations are exclusively generated at reward sites. Finally, the model can generate imaginary trajectories that are not allowed to the agent during task performance. We raise some experimental predictions and implications for future studies of the role of the hippocampo-prefronto-striatal network in learning.

Author Correction: A hippocampo-cerebellar centred network for the learning and execution of sequence-based navigation.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Impacts of inter-trial interval duration on a computational model of sign-tracking vs. goal-tracking behaviour.

In the context of Pavlovian conditioning, two types of behaviour may emerge within the population (Flagel et al. Nature, 469(7328): 53-57, 2011). Animals may choose to engage either with the conditioned stimulus (CS), a behaviour known as sign-tracking (ST) which is sensitive to dopamine inhibition for its acquisition, or with the food cup in which the reward or unconditioned stimulus (US) will eventually be delivered, a behaviour known as goal-tracking (GT) which is dependent on dopamine for its expression only. Previous work by Lesaint et al. (PLoS Comput Biol, 10(2), 2014) offered a computational explanation for these phenomena and led to the prediction that varying the duration of the inter-trial interval (ITI) would change the relative ST-GT proportion in the population as well as phasic dopamine responses. A recent study verified this prediction, but also found a rich variance of ST and GT behaviours within the trial which goes beyond the original computational model. In this paper, we provide a computational perspective on these novel results.

Dopamine blockade impairs the exploration-exploitation trade-off in rats.

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.

The mGlu7 receptor provides protective effects against epileptogenesis and epileptic seizures.

Finding new targets to control or reduce seizure activity is essential to improve the management of epileptic patients. We hypothesized that activation of the pre-synaptic and inhibitory metabotropic glutamate receptor type 7 (mGlu7) reduces spontaneous seizures. We tested LSP2-9166, a recently developed mGlu7/4 agonist with unprecedented potency on mGlu7 receptors, in two paradigms of epileptogenesis. In a model of chemically induced epileptogenesis (pentylenetetrazole systemic injection), LSP2-9166 induces an anti-epileptogenic effect rarely observed in preclinical studies. In particular, we found a bidirectional modulation of seizure progression by mGlu4 and mGlu7 receptors, the latter preventing kindling. In the intra-hippocampal injection of kainic acid mouse model that mimics the human mesial temporal lobe epilepsy, we found that LSP2-9166 reduces seizure frequency and hippocampal sclerosis. LSP2-9166 also acts as an anti-seizure drug on established seizures in both models tested. Specific modulation of the mGlu7 receptor could represent a novel approach to reduce pathological network remodeling.

Hippocampal replays under the scrutiny of reinforcement learning models.

Multiple in vivo studies have shown that place cells from the hippocampus replay previously experienced trajectories. These replays are commonly considered to mainly reflect memory consolidation processes. Some data, however, have highlighted a functional link between replays and reinforcement learning (RL). This theory, extensively used in machine learning, has introduced efficient algorithms and can explain various behavioral and physiological measures from different brain regions. RL algorithms could constitute a mechanistic description of replays and explain how replays can reduce the number of iterations required to explore the environment during learning. We review the main findings concerning the different hippocampal replay types and the possible associated RL models (either model-based, model-free, or hybrid model types). We conclude by tying these frameworks together. We illustrate the link between data and RL through a series of model simulations. This review, at the frontier between informatics and biology, paves the way for future work on replays.

Open-Ended Learning: A Conceptual Framework Based on Representational Redescription.

Reinforcement learning (RL) aims at building a policy that maximizes a task-related reward within a given domain. When the domain is known, i.e., when its states, actions and reward are defined, Markov Decision Processes (MDPs) provide a convenient theoretical framework to formalize RL. But in an open-ended learning process, an agent or robot must solve an unbounded sequence of tasks that are not known in advance and the corresponding MDPs cannot be built at design time. This defines the main challenges of open-ended learning: how can the agent learn how to behave appropriately when the adequate states, actions and rewards representations are not given? In this paper, we propose a conceptual framework to address this question. We assume an agent endowed with low-level perception and action capabilities. This agent receives an external reward when it faces a task. It must discover the state and action representations that will let it cast the tasks as MDPs in order to solve them by RL. The relevance of the action or state representation is critical for the agent to learn efficiently. Considering that the agent starts with a low level, task-agnostic state and action spaces based on its low-level perception and action capabilities, we describe open-ended learning as the challenge of building the adequate representation of states and actions, i.e., of redescribing available representations. We suggest an iterative approach to this problem based on several successive Representational Redescription processes, and highlight the corresponding challenges in which intrinsic motivations play a key role.

Action-Centered Exposure Therapy (ACET): A New Approach to the Use of Virtual Reality to the Care of People with Post-Traumatic Stress Disorder.

Post-Traumatic Stress Disorder (PTSD) can be seen as the result of dysfunctional beliefs that associate stimuli with a danger or a threat leading to anxious reactions. Exposure therapy is so far considered to be the most effective treatment, and research suggests that it is mainly based on a habituation process. Based on learning theories, it appears that a passive systemic exposure to traumatic stimuli should not be the best option for the treatment of PTSD. We hypothesis that an active learning of safer and healthier coping strategies combined with systematic exposure should be more effective in reducing the psychological distress associated with PTSD. In this paper, we describe the theoretical foundations of this approach that focuses on the action and activity of the patient in his or her exposure environment. In this approach, we take advantage of Virtual Reality technologies and learning mechanics of serious games to allow the patient to learn new safe associations while promoting the empowerment. We named this action-centered exposure therapy (ACET). This approach exploits behaviorism, cognitivism, and constructivism learning theories. With the different benefits of virtual reality technologies, this approach would easily integrate with in-virtuo exposure therapy and would allow us to exploit as much as possible the enormous potential of these technologies. As a first step toward validation, we present a case study that supports the ACET approach.

Seizure progression and inflammatory mediators promote pericytosis and pericyte-microglia clustering at the cerebrovasculature.

BACKGROUND: Cerebrovascular dysfunction and inflammation occur in epilepsy. Here we asked whether pericytes, a pivotal cellular component of brain capillaries, undergo pathological modifications during experimental epileptogenesis and in human epilepsy. We evaluated whether pro-inflammatory cytokines, present in the brain during seizures, contribute to pericyte morphological modifications. METHODS: In vivo, unilateral intra-hippocampal kainic acid (KA) injections were performed in NG2DsRed/C57BL6 mice to induce status epilepticus (SE), epileptogenesis, and spontaneous recurrent seizures (SRS). NG2DsRed mice were used to visualize pericytes during seizure progression. The effect triggered by recombinant IL-1ß, TNFα, or IL-6 on pericytes was evaluated in NG2DsRed hippocampal slices and in human-derived cell culture. Human brain specimens obtained from temporal lobe epilepsy (TLE) with or without sclerosis (HS) and focal cortical dysplasia (FCD-IIb) were evaluated for pericyte-microglial cerebrovascular assembly. RESULTS: A disarray of NG2DsRed+ pericyte soma and ramifications was found 72 h post-SE and 1 week post-SE (epileptogenesis) in the hippocampus. Pericyte modifications topographically overlapped with IBA1+ microglia clustering around the capillaries with cases of pericytes lodged within the microglial cells. Microglial clustering around the NG2DsRed pericytes lingered at SRS. Pericyte proliferation (Ki67+) occurred 72 h post-SE and during epileptogenesis and returned towards control levels at SRS. Human epileptic brain tissues showed pericyte-microglia assemblies with IBA1/HLA microglial cells outlining the capillary wall in TLE-HS and FCD-IIb specimens. Inflammatory mediators contributed to pericyte modifications, in particular IL-1ß elicited pericyte morphological changes and pericyte-microglia clustering in NG2DsRed hippocampal slices. Modifications also occurred when pro-inflammatory cytokines were added to an in vitro culture of pericytes. CONCLUSIONS: These results indicate the occurrence of pericytosis during seizures and introduce a pericyte-microglial mediated mechanism of blood-brain barrier dysfunction in epilepsy.

Enabling a sustainable and prosperous future through science and innovation in the bioeconomy at Agriculture and Agri-Food Canada.

Science and innovation are important components underpinning the agricultural and agri-food system in Canada. Canada's vast geographical area presents diverse, regionally specific requirements in addition to the 21st century agricultural challenges facing the overall sector. As the broader needs of the agricultural landscape have evolved and will continue to do so in the next few decades, there is a trend in place to transition towards a sustainable bioeconomy, contributing to reducing greenhouse gas emission and our dependency on non-renewable resources. We highlight some of the key policy drivers on an overarching national scale and those specific to agricultural research and innovation that are critical to fostering a supportive environment for innovation and a sustainable bioeconomy. As well, we delineate some major challenges and opportunities facing agriculture in Canada, including climate change, sustainable agriculture, clean technologies, and agricultural productivity, and some scientific initiatives currently underway to tackle these challenges. The use of various technologies and scientific efforts, such as Next Generation Sequencing, metagenomics analysis, satellite image analysis and mapping of soil moisture, and value-added bioproduct development will accelerate scientific development and innovation and its contribution to a sustainable and prosperous bioeconomy.