Prediction error drives associative learning and conditioned behavior in a spiking model of Drosophila larva.

Predicting reinforcement from sensory cues is beneficial for goal-directed behavior. In insect brains, underlying associations between cues and reinforcement, encoded by dopaminergic neurons, are formed in the mushroom body. We propose a spiking model of the Drosophila larva mushroom body. It includes a feedback motif conveying learned reinforcement expectation to dopaminergic neurons, which can compute prediction error as the difference between expected and present reinforcement. We demonstrate that this can serve as a driving force in learning. When combined with synaptic homeostasis, our model accounts for theoretically derived features of acquisition and loss of associations that depend on the intensity of the reinforcement and its temporal proximity to the cue. From modeling olfactory learning over the time course of behavioral experiments and simulating the locomotion of individual larvae toward or away from odor sources in a virtual environment, we conclude that learning driven by prediction errors can explain larval behavior.

The mushroom body output encodes behavioral decision during sensory-motor transformation.

Animals form a behavioral decision by evaluating sensory evidence on the background of past experiences and the momentary motivational state. In insects, we still lack understanding of how and at which stage of the recurrent sensory-motor pathway behavioral decisions are formed. The mushroom body (MB), a central brain structure in insects1 and crustaceans,2,3 integrates sensory input of different modalities4,5,6 with the internal state, the behavioral state, and external sensory context7,8,9,10 through a large number of recurrent, mostly neuromodulatory inputs,11,12 implicating a functional role for MBs in state-dependent sensory-motor transformation.13,14 A number of classical conditioning studies in honeybees15,16 and fruit flies17,18,19 have provided accumulated evidence that at its output, the MB encodes the valence of a sensory stimulus with respect to its behavioral relevance. Recent work has extended this notion of valence encoding to the context of innate behaviors.8,20,21,22 Here, we co-analyzed a defined feeding behavior and simultaneous extracellular single-unit recordings from MB output neurons (MBONs) in the cockroach in response to timed sensory stimulation with odors. We show that clear neuronal responses occurred almost exclusively during behaviorally responded trials. Early MBON responses to the sensory stimulus preceded the feeding behavior and predicted its occurrence or non-occurrence from the single-trial population activity. Our results therefore suggest that at its output, the MB does not merely encode sensory stimulus valence. We hypothesize instead that the MB output represents an integrated signal of internal state, momentary environmental conditions, and experience-dependent memory to encode a behavioral decision.

Efficient parameter calibration and real-time simulation of large-scale spiking neural networks with GeNN and NEST.

Spiking neural networks (SNNs) represent the state-of-the-art approach to the biologically realistic modeling of nervous system function. The systematic calibration for multiple free model parameters is necessary to achieve robust network function and demands high computing power and large memory resources. Special requirements arise from closed-loop model simulation in virtual environments and from real-time simulation in robotic application. Here, we compare two complementary approaches to efficient large-scale and real-time SNN simulation. The widely used NEural Simulation Tool (NEST) parallelizes simulation across multiple CPU cores. The GPU-enhanced Neural Network (GeNN) simulator uses the highly parallel GPU-based architecture to gain simulation speed. We quantify fixed and variable simulation costs on single machines with different hardware configurations. As a benchmark model, we use a spiking cortical attractor network with a topology of densely connected excitatory and inhibitory neuron clusters with homogeneous or distributed synaptic time constants and in comparison to the random balanced network. We show that simulation time scales linearly with the simulated biological model time and, for large networks, approximately linearly with the model size as dominated by the number of synaptic connections. Additional fixed costs with GeNN are almost independent of model size, while fixed costs with NEST increase linearly with model size. We demonstrate how GeNN can be used for simulating networks with up to 3.5 · 106 neurons (> 3 · 1012synapses) on a high-end GPU, and up to 250, 000 neurons (25 · 109 synapses) on a low-cost GPU. Real-time simulation was achieved for networks with 100, 000 neurons. Network calibration and parameter grid search can be efficiently achieved using batch processing. We discuss the advantages and disadvantages of both approaches for different use cases.

Minimal circuit motifs for second-order conditioning in the insect mushroom body.

In well-established first-order conditioning experiments, the concurrence of a sensory cue with reinforcement forms an association, allowing the cue to predict future reinforcement. In the insect mushroom body, a brain region central to learning and memory, such associations are encoded in the synapses between its intrinsic and output neurons. This process is mediated by the activity of dopaminergic neurons that encode reinforcement signals. In second-order conditioning, a new sensory cue is paired with an already established one that presumably activates dopaminergic neurons due to its predictive power of the reinforcement. We explored minimal circuit motifs in the mushroom body for their ability to support second-order conditioning using mechanistic models. We found that dopaminergic neurons can either be activated directly by the mushroom body's intrinsic neurons or via feedback from the output neurons via several pathways. We demonstrated that the circuit motifs differ in their computational efficiency and robustness. Beyond previous research, we suggest an additional motif that relies on feedforward input of the mushroom body intrinsic neurons to dopaminergic neurons as a promising candidate for experimental evaluation. It differentiates well between trained and novel stimuli, demonstrating robust performance across a range of model parameters.

A Mechanistic Model for Reward Prediction and Extinction Learning in the Fruit Fly.

Extinction learning, the ability to update previously learned information by integrating novel contradictory information, is of high clinical relevance for therapeutic approaches to the modulation of maladaptive memories. Insect models have been instrumental in uncovering fundamental processes of memory formation and memory update. Recent experimental results in Drosophila melanogaster suggest that, after the behavioral extinction of a memory, two parallel but opposing memory traces coexist, residing at different sites within the mushroom body (MB). Here, we propose a minimalistic circuit model of the Drosophila MB that supports classical appetitive and aversive conditioning and memory extinction. The model is tailored to the existing anatomic data and involves two circuit motives of central functional importance. It employs plastic synaptic connections between Kenyon cells (KCs) and MB output neurons (MBONs) in separate and mutually inhibiting appetitive and aversive learning pathways. Recurrent modulation of plasticity through projections from MBONs to reinforcement-mediating dopaminergic neurons (DAN) implements a simple reward prediction mechanism. A distinct set of four MBONs encodes odor valence and predicts behavioral model output. Subjecting our model to learning and extinction protocols reproduced experimental results from recent behavioral and imaging studies. Simulating the experimental blocking of synaptic output of individual neurons or neuron groups in the model circuit confirmed experimental results and allowed formulation of testable predictions. In the temporal domain, our model achieves rapid learning with a step-like increase in the encoded odor value after a single pairing of the conditioned stimulus (CS) with a reward or punishment, facilitating single-trial learning.

A spiking neural program for sensorimotor control during foraging in flying insects.

Foraging is a vital behavioral task for living organisms. Behavioral strategies and abstract mathematical models thereof have been described in detail for various species. To explore the link between underlying neural circuits and computational principles, we present how a biologically detailed neural circuit model of the insect mushroom body implements sensory processing, learning, and motor control. We focus on cast and surge strategies employed by flying insects when foraging within turbulent odor plumes. Using a spike-based plasticity rule, the model rapidly learns to associate individual olfactory sensory cues paired with food in a classical conditioning paradigm. We show that, without retraining, the system dynamically recalls memories to detect relevant cues in complex sensory scenes. Accumulation of this sensory evidence on short time scales generates cast-and-surge motor commands. Our generic systems approach predicts that population sparseness facilitates learning, while temporal sparseness is required for dynamic memory recall and precise behavioral control. Our work successfully combines biological computational principles with spike-based machine learning. It shows how knowledge transfer from static to arbitrary complex dynamic conditions can be achieved by foraging insects and may serve as inspiration for agent-based machine learning.

Numerical Cognition Based on Precise Counting with a Single Spiking Neuron.

Insects are able to solve basic numerical cognition tasks. We show that estimation of numerosity can be realized and learned by a single spiking neuron with an appropriate synaptic plasticity rule. This model can be efficiently trained to detect arbitrary spatiotemporal spike patterns on a noisy and dynamic background with high precision and low variance. When put to test in a task that requires counting of visual concepts in a static image it required considerably less training epochs than a convolutional neural network to achieve equal performance. When mimicking a behavioral task in free-flying bees that requires numerical cognition, the model reaches a similar success rate in making correct decisions. We propose that using action potentials to represent basic numerical concepts with a single spiking neuron is beneficial for organisms with small brains and limited neuronal resources.

Cockroaches Show Individuality in Learning and Memory During Classical and Operant Conditioning.

Animal personality and individuality are intensively researched in vertebrates and both concepts are increasingly applied to behavioral science in insects. However, only few studies have looked into individuality with respect to performance in learning and memory tasks. In vertebrates, individual learning capabilities vary considerably with respect to learning speed and learning rate. Likewise, honeybees express individual learning abilities in a wide range of classical conditioning protocols. Here, we study individuality in the learning and memory performance of cockroaches, both in classical and operant conditioning tasks. We implemented a novel classical (olfactory) conditioning paradigm where the conditioned response is established in the maxilla-labia response (MLR). Operant spatial learning was investigated in a forced two-choice task using a T-maze. Our results confirm individual learning abilities in classical conditioning of cockroaches that was reported for honeybees and vertebrates but contrast long-standing reports on stochastic learning behavior in fruit flies. In our experiments, most learners expressed a correct behavior after only a single learning trial showing a consistent high performance during training and test. We can further show that individual learning differences in insects are not limited to classical conditioning but equally appear in operant conditioning of the cockroach.

Neural coding: sparse but on time.

To code information efficiently, sensory systems use sparse representations. In a sparse code, a specific stimulus activates only few spikes in a small number of neurons. A new study shows that the temporal pattern across sparsely activated neurons encodes information, suggesting that the sparse code extends into the time domain.

A neuromorphic network for generic multivariate data classification.

Computational neuroscience has uncovered a number of computational principles used by nervous systems. At the same time, neuromorphic hardware has matured to a state where fast silicon implementations of complex neural networks have become feasible. En route to future technical applications of neuromorphic computing the current challenge lies in the identification and implementation of functional brain algorithms. Taking inspiration from the olfactory system of insects, we constructed a spiking neural network for the classification of multivariate data, a common problem in signal and data analysis. In this model, real-valued multivariate data are converted into spike trains using "virtual receptors" (VRs). Their output is processed by lateral inhibition and drives a winner-take-all circuit that supports supervised learning. VRs are conveniently implemented in software, whereas the lateral inhibition and classification stages run on accelerated neuromorphic hardware. When trained and tested on real-world datasets, we find that the classification performance is on par with a naïve Bayes classifier. An analysis of the network dynamics shows that stable decisions in output neuron populations are reached within less than 100 ms of biological time, matching the time-to-decision reported for the insect nervous system. Through leveraging a population code, the network tolerates the variability of neuronal transfer functions and trial-to-trial variation that is inevitably present on the hardware system. Our work provides a proof of principle for the successful implementation of a functional spiking neural network on a configurable neuromorphic hardware system that can readily be applied to real-world computing problems.

Local interneurons and projection neurons in the antennal lobe from a spiking point of view.

Local computation in microcircuits is an essential feature of distributed information processing in vertebrate and invertebrate brains. The insect antennal lobe represents a spatially confined local network that processes high-dimensional and redundant peripheral input to compute an efficient odor code. Social insects can rely on a particularly rich olfactory receptor repertoire, and they exhibit complex odor-guided behaviors. This corresponds with a high anatomical complexity of their antennal lobe network. In the honeybee, a large number of glomeruli that receive sensory input are interconnected by a dense network of local interneurons (LNs). Uniglomerular projection neurons (PNs) integrate sensory and recurrent local network input into an efficient spatio-temporal odor code. To investigate the specific computational roles of LNs and PNs, we measured several features of sub- and suprathreshold single-cell responses to in vivo odor stimulation. Using a semisupervised cluster analysis, we identified a combination of five characteristic features as sufficient to separate LNs and PNs from each other, independent of the applied odor-stimuli. The two clusters differed significantly in all these five features. PNs showed a higher spontaneous subthreshold activation, assumed higher peak response rates and a more regular spiking pattern. LNs reacted considerably faster to the onset of a stimulus, and their responses were more reliable across stimulus repetitions. We discuss possible mechanisms that can explain our results, and we interpret cell-type-specific characteristics with respect to their functional relevance.

Average group behavior does not represent individual behavior in classical conditioning of the honeybee.

Conditioned behavior as observed during classical conditioning in a group of identically treated animals provides insights into the physiological process of learning and memory formation. However, several studies in vertebrates found a remarkable difference between the group-average behavioral performance and the behavioral characteristics of individual animals. Here, we analyzed a large number of data (1640 animals) on olfactory conditioning in the honeybee (Apis mellifera). The data acquired during absolute and differential classical conditioning differed with respect to the number of conditioning trials, the conditioned odors, the intertrial intervals, and the time of retention tests. We further investigated data in which animals were tested for spontaneous recovery from extinction. In all data sets we found that the gradually increasing group-average learning curve did not adequately represent the behavior of individual animals. Individual behavior was characterized by a rapid and stable acquisition of the conditioned response (CR), as well as by a rapid and stable cessation of the CR following unrewarded stimuli. In addition, we present and evaluate different model hypotheses on how honeybees form associations during classical conditioning by implementing a gradual learning process on the one hand and an all-or-none learning process on the other hand. In summary, our findings advise that individual behavior should be recognized as a meaningful predictor for the internal state of a honeybee--irrespective of the group-average behavioral performance.

Mushroom body output neurons encode odor-reward associations.

Neural correlates of learning and memory formation have been reported at different stages of the olfactory pathway in both vertebrates and invertebrates. However, the contribution of different neurons to the formation of a memory trace is little understood. Mushroom bodies (MBs) in the insect brain are higher-order structures involved in integration of olfactory, visual, and mechanosensory information and in memory formation. Here we focus on the ensemble spiking activity of single MB output neurons (ENs) when honeybees learned to associate an odor with reward. A large group of ENs (∼50%) changed their odor response spectra by losing or gaining sensitivity for specific odors. This response switching was dominated by the rewarded stimulus (CS+), which evoked exclusively recruitment. The remaining ENs did not change their qualitative odor spectrum but modulated their tuning strength, again dominated by increased responses to the CS+. While the bees showed a conditioned response (proboscis extension) after a few acquisition trials, no short-term effects were observed in the neuronal activity. In both EN types, associative plastic changes occurred only during retention 3 h after conditioning. Thus, long-term but not short-term memory was reflected by increased EN activity to the CS+. During retention, the EN ensemble separated the CS+ most differently from the CS- and control odors ∼140 ms after stimulus onset. The learned behavioral response appeared ∼330 ms later. It is concluded that after memory consolidation, the ensemble activity of the MB output neurons predicts the meaning of the stimulus (reward) and may provide the prerequisite for the expression of the learned behavior.

Parallel representation of stimulus identity and intensity in a dual pathway model inspired by the olfactory system of the honeybee.

The honeybee Apis mellifera has a remarkable ability to detect and locate food sources during foraging, and to associate odor cues with food rewards. In the honeybee's olfactory system, sensory input is first processed in the antennal lobe (AL) network. Uniglomerular projection neurons (PNs) convey the sensory code from the AL to higher brain regions via two parallel but anatomically distinct pathways, the lateral and the medial antenno-cerebral tract (l- and m-ACT). Neurons innervating either tract show characteristic differences in odor selectivity, concentration dependence, and representation of mixtures. It is still unknown how this differential stimulus representation is achieved within the AL network. In this contribution, we use a computational network model to demonstrate that the experimentally observed features of odor coding in PNs can be reproduced by varying lateral inhibition and gain control in an otherwise unchanged AL network. We show that odor coding in the l-ACT supports detection and accurate identification of weak odor traces at the expense of concentration sensitivity, while odor coding in the m-ACT provides the basis for the computation and following of concentration gradients but provides weaker discrimination power. Both coding strategies are mutually exclusive, which creates a tradeoff between detection accuracy and sensitivity. The development of two parallel systems may thus reflect an evolutionary solution to this problem that enables honeybees to achieve both tasks during bee foraging in their natural environment, and which could inspire the development of artificial chemosensory devices for odor-guided navigation in robots.

Precisely timed signal transmission in neocortical networks with reliable intermediate-range projections.

The mammalian neocortex has a remarkable ability to precisely reproduce behavioral sequences or to reliably retrieve stored information. In contrast, spiking activity in behaving animals shows a considerable trial-to-trial variability and temporal irregularity. The signal propagation and processing underlying these conflicting observations is based on fundamental neurophysiological processes like synaptic transmission, signal integration within single cells, and spike formation. Each of these steps in the neuronal signaling chain has been studied separately to a great extend, but it has been difficult to judge how they interact and sum up in active sub-networks of neocortical cells. In the present study, we experimentally assessed the precision and reliability of small neocortical networks consisting of trans-columnar, intermediate-range projections (200-1000 mum) on a millisecond time-scale. Employing photo-uncaging of glutamate in acute slices, we activated a number of distant presynaptic cells in a spatio-temporally precisely controlled manner, while monitoring the resulting membrane potential fluctuations of a postsynaptic cell. We found that signal integration in this part of the network is highly reliable and temporally precise. As numerical simulations showed, the residual membrane potential variability can be attributed to amplitude variability in synaptic transmission and may significantly contribute to trial-to-trial output variability of a rate signal. However, it does not impair the temporal accuracy of signal integration. We conclude that signals from intermediate-range projections onto neocortical neurons are propagated and integrated in a highly reliable and precise manner, and may serve as a substrate for temporally precise signal transmission in neocortical networks.

Comparing information about arm movement direction in single channels of local and epicortical field potentials from monkey and human motor cortex.

Cortical field potentials have been used for decades in neurophysiological studies to probe spatio-temporal activity patterns of local populations of neurons. Recently, however, interest in these signals was spurred as they were proposed as potential control signals for neuronal motor prostheses, i.e., for devices fit to record and decode brain activity to restore motor functions in paralyzed patients. Little is known, however, about the functional significance of these cortical field potentials. Here we compared information about arm movement direction in two types of movement related cortical field potentials, obtained during a four direction center-out arm reaching paradigm: local field potentials (LFPs) recorded with intracortical micro-electrodes from monkey motor cortex, and epicortical field potentials (EFPs) recorded with macro-electrode arrays subdurally implanted on the surface of the human cerebral cortex. While monkey LFPs showed a typical sequence of positive and negative potential peaks, an initial negative peak was the most salient feature of human EFPs. Individual contralateral LFPs from the monkey motor cortex carried approximately twice as much decoded information (DI) about arm movement direction (median 0.27 bit) as did individual EFPs from the contralateral hand/arm area of primary motor cortex in humans (median 0.12 bit). This relation was similar to the relation between median peak signal-to-noise ratios for directional modulation of movement related potentials (MRPs) of both types of signals. We discuss possible reasons for the observed differences, amongst them epi- vs. intracortical recording and the different electrode dimensions used to measure EFPs and LFPs.