Training a spiking neuronal network model of visual-motor cortex to play a virtual racket-ball game using reinforcement learning.

Recent models of spiking neuronal networks have been trained to perform behaviors in static environments using a variety of learning rules, with varying degrees of biological realism. Most of these models have not been tested in dynamic visual environments where models must make predictions on future states and adjust their behavior accordingly. The models using these learning rules are often treated as black boxes, with little analysis on circuit architectures and learning mechanisms supporting optimal performance. Here we developed visual/motor spiking neuronal network models and trained them to play a virtual racket-ball game using several reinforcement learning algorithms inspired by the dopaminergic reward system. We systematically investigated how different architectures and circuit-motifs (feed-forward, recurrent, feedback) contributed to learning and performance. We also developed a new biologically-inspired learning rule that significantly enhanced performance, while reducing training time. Our models included visual areas encoding game inputs and relaying the information to motor areas, which used this information to learn to move the racket to hit the ball. Neurons in the early visual area relayed information encoding object location and motion direction across the network. Neuronal association areas encoded spatial relationships between objects in the visual scene. Motor populations received inputs from visual and association areas representing the dorsal pathway. Two populations of motor neurons generated commands to move the racket up or down. Model-generated actions updated the environment and triggered reward or punishment signals that adjusted synaptic weights so that the models could learn which actions led to reward. Here we demonstrate that our biologically-plausible learning rules were effective in training spiking neuronal network models to solve problems in dynamic environments. We used our models to dissect the circuit architectures and learning rules most effective for learning. Our model shows that learning mechanisms involving different neural circuits produce similar performance in sensory-motor tasks. In biological networks, all learning mechanisms may complement one another, accelerating the learning capabilities of animals. Furthermore, this also highlights the resilience and redundancy in biological systems.

NetPyNE, a tool for data-driven multiscale modeling of brain circuits.

Biophysical modeling of neuronal networks helps to integrate and interpret rapidly growing and disparate experimental datasets at multiple scales. The NetPyNE tool (www.netpyne.org) provides both programmatic and graphical interfaces to develop data-driven multiscale network models in NEURON. NetPyNE clearly separates model parameters from implementation code. Users provide specifications at a high level via a standardized declarative language, for example connectivity rules, to create millions of cell-to-cell connections. NetPyNE then enables users to generate the NEURON network, run efficiently parallelized simulations, optimize and explore network parameters through automated batch runs, and use built-in functions for visualization and analysis - connectivity matrices, voltage traces, spike raster plots, local field potentials, and information theoretic measures. NetPyNE also facilitates model sharing by exporting and importing standardized formats (NeuroML and SONATA). NetPyNE is already being used to teach computational neuroscience students and by modelers to investigate brain regions and phenomena.

The Cascade Analysis Tool: software to analyze and optimize care cascades.

Introduction: Cascades, which track the progressive stages of engagement on the path towards a successful outcome, are increasingly being employed to quantitatively assess progress towards targets associated with health and development responses. Maximizing the proportion of people with successful outcomes within a budget-constrained context requires identifying and implementing interventions that are not only effective, but also cost-effective. Methods: We developed a software application called the Cascade Analysis Tool that implements advanced analysis and optimization methods for understanding cascades, combined with the flexibility to enable application across a wide range of areas in health and development. The tool allows users to design the cascade, collate and enter data, and then use the built-in analysis methods in order to answer key policy questions, such as: understanding where the biggest drop-offs along the cascade are; visualizing how the cascade varies by population; investigating the impact of introducing a new intervention or scaling up/down existing interventions; and estimating how available funding should be optimally allocated among available interventions in order to achieve a variety of different objectives selectable by the user (such as optimizing cascade outcomes in target years). The Cascade Analysis Tool is available via a user-friendly web-based application, and comes with a user guide, a library of pre-made examples, and training materials. Discussion: Whilst the Cascade Analysis Tool is still in the early stages of existence, it has already shown promise in preliminary applications, and we believe there is potential for it to help make sense of the increasing quantities of data on cascades.

Optimization by Adaptive Stochastic Descent.

When standard optimization methods fail to find a satisfactory solution for a parameter fitting problem, a tempting recourse is to adjust parameters manually. While tedious, this approach can be surprisingly powerful in terms of achieving optimal or near-optimal solutions. This paper outlines an optimization algorithm, Adaptive Stochastic Descent (ASD), that has been designed to replicate the essential aspects of manual parameter fitting in an automated way. Specifically, ASD uses simple principles to form probabilistic assumptions about (a) which parameters have the greatest effect on the objective function, and (b) optimal step sizes for each parameter. We show that for a certain class of optimization problems (namely, those with a moderate to large number of scalar parameter dimensions, especially if some dimensions are more important than others), ASD is capable of minimizing the objective function with far fewer function evaluations than classic optimization methods, such as the Nelder-Mead nonlinear simplex, Levenberg-Marquardt gradient descent, simulated annealing, and genetic algorithms. As a case study, we show that ASD outperforms standard algorithms when used to determine how resources should be allocated in order to minimize new HIV infections in Swaziland.

Development and Clinical Evaluation of an mHealth Application for Stress Management.

A large number of individuals experience mental health disorders, with cognitive behavioral therapy (CBT) emerging as a standard practice for reduction in psychiatric symptoms, including stress, anger, anxiety, and depression. However, CBT is associated with significant patient dropout and lacks the means to provide objective data regarding a patient's experience and symptoms between sessions. Emerging wearables and mobile health (mHealth) applications represent an approach that may provide objective data to the patient and provider between CBT sessions. Here, we describe the development of a classifier of real-time physiological stress in a healthy population (n = 35) and apply it in a controlled clinical evaluation for armed forces veterans undergoing CBT for stress and anger management (n = 16). Using cardiovascular and electrodermal inputs from a wearable device, the classifier was able to detect physiological stress in a non-clinical sample with accuracy greater than 90%. In a small clinical sample, patients who used the classifier and an associated mHealth application were less likely to discontinue therapy (p = 0.016, d = 1.34) and significantly improved on measures of stress (p = 0.032, d = 1.61), anxiety (p = 0.050, d = 1.26), and anger (p = 0.046, d = 1.41) compared to controls undergoing CBT alone. Given the large number of individuals that experience mental health disorders and the unmet need for treatment, especially in developing nations, such mHealth approaches have the potential to provide or augment treatment at low cost in the absence of in-person care.

Identification of resilient individuals and those at risk for performance deficits under stress.

Human task performance is affected by exposure to physiological and psychological stress. The ability to measure the physiological response to stressors and correlate that to task performance could be used to identify resilient individuals or those at risk for stress-related performance decrements. Accomplishing this prior to performance under severe stress or the development of clinical stress disorders could facilitate focused preparation such as tailoring training to individual needs. Here we measure the effects of stress on physiological response and performance through behavior, physiological sensors, and subjective ratings, and identify which individuals are at risk for stress-related performance decrements. Participants performed military-relevant training tasks under stress in a virtual environment, with autonomic and hypothalamic-pituitary-adrenal axis (HPA) reactivity analyzed. Self-reported stress, as well as physiological indices of stress, increased in the group pre-exposed to socioevaluative stress. Stress response was effectively captured via electrodermal and cardiovascular measures of heart rate and skin conductance level. A resilience classification algorithm was developed based upon physiological reactivity, which correlated with baseline unstressed physiological and self-reported stress values. Outliers were identified in the experimental group that had a significant mismatch between self-reported stress and salivary cortisol. Baseline stress measurements were predictive of individual resilience to stress, including the impact stress had on physiological reactivity and performance. Such an approach may have utility in identifying individuals at risk for problems performing under severe stress. Continuing work has focused on adapting this method for military personnel, and assessing the utility of various coping and decision-making strategies on performance and physiological stress.

Motor cortex microcircuit simulation based on brain activity mapping.

The deceptively simple laminar structure of neocortex belies the complexity of intra- and interlaminar connectivity. We developed a computational model based primarily on a unified set of brain activity mapping studies of mouse M1. The simulation consisted of 775 spiking neurons of 10 cell types with detailed population-to-population connectivity. Static analysis of connectivity with graph-theoretic tools revealed that the corticostriatal population showed strong centrality, suggesting that would provide a network hub. Subsequent dynamical analysis confirmed this observation, in addition to revealing network dynamics that cannot be readily predicted through analysis of the wiring diagram alone. Activation thresholds depended on the stimulated layer. Low stimulation produced transient activation, while stronger activation produced sustained oscillations where the threshold for sustained responses varied by layer: 13% in layer 2/3, 54% in layer 5A, 25% in layer 5B, and 17% in layer 6. The frequency and phase of the resulting oscillation also depended on stimulation layer. By demonstrating the effectiveness of combined static and dynamic analysis, our results show how static brain maps can be related to the results of brain activity mapping.

Towards a real-time interface between a biomimetic model of sensorimotor cortex and a robotic arm.

Brain-machine interfaces can greatly improve the performance of prosthetics. Utilizing biomimetic neuronal modeling in brain machine interfaces (BMI) offers the possibility of providing naturalistic motor-control algorithms for control of a robotic limb. This will allow finer control of a robot, while also giving us new tools to better understand the brain's use of electrical signals. However, the biomimetic approach presents challenges in integrating technologies across multiple hardware and software platforms, so that the different components can communicate in real-time. We present the first steps in an ongoing effort to integrate a biomimetic spiking neuronal model of motor learning with a robotic arm. The biomimetic model (BMM) was used to drive a simple kinematic two-joint virtual arm in a motor task requiring trial-and-error convergence on a single target. We utilized the output of this model in real time to drive mirroring motion of a Barrett Technology WAM robotic arm through a user datagram protocol (UDP) interface. The robotic arm sent back information on its joint positions, which was then used by a visualization tool on the remote computer to display a realistic 3D virtual model of the moving robotic arm in real time. This work paves the way towards a full closed-loop biomimetic brain-effector system that can be incorporated in a neural decoder for prosthetic control, to be used as a platform for developing biomimetic learning algorithms for controlling real-time devices.

Reinforcement learning of two-joint virtual arm reaching in a computer model of sensorimotor cortex.

Neocortical mechanisms of learning sensorimotor control involve a complex series of interactions at multiple levels, from synaptic mechanisms to cellular dynamics to network connectomics. We developed a model of sensory and motor neocortex consisting of 704 spiking model neurons. Sensory and motor populations included excitatory cells and two types of interneurons. Neurons were interconnected with AMPA/NMDA and GABAA synapses. We trained our model using spike-timing-dependent reinforcement learning to control a two-joint virtual arm to reach to a fixed target. For each of 125 trained networks, we used 200 training sessions, each involving 15 s reaches to the target from 16 starting positions. Learning altered network dynamics, with enhancements to neuronal synchrony and behaviorally relevant information flow between neurons. After learning, networks demonstrated retention of behaviorally relevant memories by using proprioceptive information to perform reach-to-target from multiple starting positions. Networks dynamically controlled which joint rotations to use to reach a target, depending on current arm position. Learning-dependent network reorganization was evident in both sensory and motor populations: learned synaptic weights showed target-specific patterning optimized for particular reach movements. Our model embodies an integrative hypothesis of sensorimotor cortical learning that could be used to interpret future electrophysiological data recorded in vivo from sensorimotor learning experiments. We used our model to make the following predictions: learning enhances synchrony in neuronal populations and behaviorally relevant information flow across neuronal populations, enhanced sensory processing aids task-relevant motor performance and the relative ease of a particular movement in vivo depends on the amount of sensory information required to complete the movement.

Cortical information flow in Parkinson's disease: a composite network/field model.

The basal ganglia play a crucial role in the execution of movements, as demonstrated by the severe motor deficits that accompany Parkinson's disease (PD). Since motor commands originate in the cortex, an important question is how the basal ganglia influence cortical information flow, and how this influence becomes pathological in PD. To explore this, we developed a composite neuronal network/neural field model. The network model consisted of 4950 spiking neurons, divided into 15 excitatory and inhibitory cell populations in the thalamus and cortex. The field model consisted of the cortex, thalamus, striatum, subthalamic nucleus, and globus pallidus. Both models have been separately validated in previous work. Three field models were used: one with basal ganglia parameters based on data from healthy individuals, one based on data from individuals with PD, and one purely thalamocortical model. Spikes generated by these field models were then used to drive the network model. Compared to the network driven by the healthy model, the PD-driven network had lower firing rates, a shift in spectral power toward lower frequencies, and higher probability of bursting; each of these findings is consistent with empirical data on PD. In the healthy model, we found strong Granger causality between cortical layers in the beta and low gamma frequency bands, but this causality was largely absent in the PD model. In particular, the reduction in Granger causality from the main "input" layer of the cortex (layer 4) to the main "output" layer (layer 5) was pronounced. This may account for symptoms of PD that seem to reflect deficits in information flow, such as bradykinesia. In general, these results demonstrate that the brain's large-scale oscillatory environment, represented here by the field model, strongly influences the information processing that occurs within its subnetworks. Hence, it may be preferable to drive spiking network models with physiologically realistic inputs rather than pure white noise.

Reinforcement learning of targeted movement in a spiking neuronal model of motor cortex.

Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint "forearm" to a target. Learning was based on a reinforcement mechanism analogous to that of the dopamine system. This provided a global reward or punishment signal in response to decreasing or increasing distance from hand to target, respectively. Output was partially driven by Poisson motor babbling, creating stochastic movements that could then be shaped by learning. The virtual forearm consisted of a single segment rotated around an elbow joint, controlled by flexor and extensor muscles. The model consisted of 144 excitatory and 64 inhibitory event-based neurons, each with AMPA, NMDA, and GABA synapses. Proprioceptive cell input to this model encoded the 2 muscle lengths. Plasticity was only enabled in feedforward connections between input and output excitatory units, using spike-timing-dependent eligibility traces for synaptic credit or blame assignment. Learning resulted from a global 3-valued signal: reward (+1), no learning (0), or punishment (-1), corresponding to phasic increases, lack of change, or phasic decreases of dopaminergic cell firing, respectively. Successful learning only occurred when both reward and punishment were enabled. In this case, 5 target angles were learned successfully within 180 s of simulation time, with a median error of 8 degrees. Motor babbling allowed exploratory learning, but decreased the stability of the learned behavior, since the hand continued moving after reaching the target. Our model demonstrated that a global reinforcement signal, coupled with eligibility traces for synaptic plasticity, can train a spiking sensorimotor network to perform goal-directed motor behavior.

Electrostimulation as a prosthesis for repair of information flow in a computer model of neocortex.

Damage to a cortical area reduces not only information transmitted to other cortical areas, but also activation of these areas. This phenomenon, whereby the dynamics of a follower area are dramatically altered, is typically manifested as a marked reduction in activity. Ideally, neuroprosthetic stimulation would replace both information and activation. However, replacement of activation alone may be valuable as a means of restoring dynamics and information processing of other signals in this multiplexing system. We used neuroprosthetic stimulation in a computer model of the cortex to repair activation dynamics, using a simple repetitive stimulation to replace the more complex, naturalistic stimulation that had been removed. We found that we were able to restore activity in terms of neuronal firing rates. Additionally, we were able to restore information processing, measured as a restoration of causality between an experimentally recorded signal fed into the in silico brain and a cortical output. These results indicate that even simple neuroprosthetics that do not restore lost information may nonetheless be effective in improving the functionality of surrounding areas of cortex.

A large-scale neurocomputational model of task-oriented behavior selection and working memory in prefrontal cortex.

The prefrontal cortex (PFC) is crucially involved in the executive component of working memory, representation of task state, and behavior selection. This article presents a large-scale computational model of the PFC and associated brain regions designed to investigate the mechanisms by which working memory and task state interact to select adaptive behaviors from a behavioral repertoire. The model consists of multiple brain regions containing neuronal populations with realistic physiological and anatomical properties, including extrastriate visual cortical regions, the inferotemporal cortex, the PFC, the striatum, and midbrain dopamine (DA) neurons. The onset of a delayed match-to-sample or delayed nonmatch-to-sample task triggers tonic DA release in the PFC causing a switch into a persistent, stimulus-insensitive dynamic state that promotes the maintenance of stimulus representations within prefrontal networks. Other modeled prefrontal and striatal units select cognitive acceptance or rejection behaviors according to which task is active and whether prefrontal working memory representations match the current stimulus. Working memory task performance and memory fields of prefrontal delay units are degraded by extreme elevation or depletion of tonic DA levels. Analyses of cellular and synaptic activity suggest that hyponormal DA levels result in increased prefrontal activation, whereas hypernormal DA levels lead to decreased activation. Our simulation results suggest a range of predictions for behavioral, single-cell, and neuroimaging response data under the proposed task set and under manipulations of DA concentration.