Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning.

Artificial neural networks (ANNs) have been successfully trained to perform a wide range of sensory-motor behaviors. In contrast, the performance of spiking neuronal network (SNN) models trained to perform similar behaviors remains relatively suboptimal. In this work, we aimed to push the field of SNNs forward by exploring the potential of different learning mechanisms to achieve optimal performance. We trained SNNs to solve the CartPole reinforcement learning (RL) control problem using two learning mechanisms operating at different timescales: (1) spike-timing-dependent reinforcement learning (STDP-RL) and (2) evolutionary strategy (EVOL). Though the role of STDP-RL in biological systems is well established, several other mechanisms, though not fully understood, work in concert during learning in vivo. Recreating accurate models that capture the interaction of STDP-RL with these diverse learning mechanisms is extremely difficult. EVOL is an alternative method and has been successfully used in many studies to fit model neural responsiveness to electrophysiological recordings and, in some cases, for classification problems. One advantage of EVOL is that it may not need to capture all interacting components of synaptic plasticity and thus provides a better alternative to STDP-RL. Here, we compared the performance of each algorithm after training, which revealed EVOL as a powerful method for training SNNs to perform sensory-motor behaviors. Our modeling opens up new capabilities for SNNs in RL and could serve as a testbed for neurobiologists aiming to understand multi-timescale learning mechanisms and dynamics in neuronal circuits.

Detecting Spontaneous Neural Oscillation Events in Primate Auditory Cortex.

Electrophysiological oscillations in the brain have been shown to occur as multicycle events, with onset and offset dependent on behavioral and cognitive state. To provide a baseline for state-related and task-related events, we quantified oscillation features in resting-state recordings. We developed an open-source wavelet-based tool to detect and characterize such oscillation events (OEvents) and exemplify the use of this tool in both simulations and two invasively-recorded electrophysiology datasets: one from human, and one from nonhuman primate (NHP) auditory system. After removing incidentally occurring event-related potentials (ERPs), we used OEvents to quantify oscillation features. We identified ∼2 million oscillation events, classified within traditional frequency bands: δ, θ, α, ß, low γ, γ, and high γ. Oscillation events of 1-44 cycles could be identified in at least one frequency band 90% of the time in human and NHP recordings. Individual oscillation events were characterized by nonconstant frequency and amplitude. This result necessarily contrasts with prior studies which assumed frequency constancy, but is consistent with evidence from event-associated oscillations. We measured oscillation event duration, frequency span, and waveform shape. Oscillations tended to exhibit multiple cycles per event, verifiable by comparing filtered to unfiltered waveforms. In addition to the clear intraevent rhythmicity, there was also evidence of interevent rhythmicity within bands, demonstrated by finding that coefficient of variation of interval distributions and Fano factor (FF) measures differed significantly from a Poisson distribution assumption. Overall, our study provides an easy-to-use tool to study oscillation events at the single-trial level or in ongoing recordings, and demonstrates that rhythmic, multicycle oscillation events dominate auditory cortical dynamics.

Inter-Animal Variability in Activity Phase Is Constrained by Synaptic Dynamics in an Oscillatory Network.

The levels of voltage-gated and synaptic currents in the same neuron type can vary substantially across individuals. Yet, the phase relationships between neurons in oscillatory circuits are often maintained, even in the face of varying oscillation frequencies. We examined whether synaptic and intrinsic currents are matched to maintain constant activity phases across preparations, using the lateral pyloric (LP) neuron of the stomatogastric ganglion (STG) of the crab, Cancer borealis LP produces stable oscillatory bursts on release from inhibition, with an onset phase that is independent of oscillation frequency. We quantified the parameters that define the shape of the synaptic current inputs across preparations and found no linear correlations with voltage-gated currents. However, several synaptic parameters were correlated with oscillation period and burst onset phase, suggesting they may play a role in phase maintenance. We used dynamic clamp to apply artificial synaptic inputs and found that those synaptic parameters correlated with phase and period were ineffective in influencing burst onset. Instead, parameters that showed the least variability across preparations had the greatest influence. Thus, parameters that influence circuit phasing are constrained across individuals, while those that have little effect simply co-vary with phase and frequency.

Training a spiking neuronal network model of visual-motor cortex to play a virtual racket-ball game using reinforcement learning.

Recent models of spiking neuronal networks have been trained to perform behaviors in static environments using a variety of learning rules, with varying degrees of biological realism. Most of these models have not been tested in dynamic visual environments where models must make predictions on future states and adjust their behavior accordingly. The models using these learning rules are often treated as black boxes, with little analysis on circuit architectures and learning mechanisms supporting optimal performance. Here we developed visual/motor spiking neuronal network models and trained them to play a virtual racket-ball game using several reinforcement learning algorithms inspired by the dopaminergic reward system. We systematically investigated how different architectures and circuit-motifs (feed-forward, recurrent, feedback) contributed to learning and performance. We also developed a new biologically-inspired learning rule that significantly enhanced performance, while reducing training time. Our models included visual areas encoding game inputs and relaying the information to motor areas, which used this information to learn to move the racket to hit the ball. Neurons in the early visual area relayed information encoding object location and motion direction across the network. Neuronal association areas encoded spatial relationships between objects in the visual scene. Motor populations received inputs from visual and association areas representing the dorsal pathway. Two populations of motor neurons generated commands to move the racket up or down. Model-generated actions updated the environment and triggered reward or punishment signals that adjusted synaptic weights so that the models could learn which actions led to reward. Here we demonstrate that our biologically-plausible learning rules were effective in training spiking neuronal network models to solve problems in dynamic environments. We used our models to dissect the circuit architectures and learning rules most effective for learning. Our model shows that learning mechanisms involving different neural circuits produce similar performance in sensory-motor tasks. In biological networks, all learning mechanisms may complement one another, accelerating the learning capabilities of animals. Furthermore, this also highlights the resilience and redundancy in biological systems.

Short-term synaptic dynamics control the activity phase of neurons in an oscillatory network.

In oscillatory systems, neuronal activity phase is often independent of network frequency. Such phase maintenance requires adjustment of synaptic input with network frequency, a relationship that we explored using the crab, Cancer borealis, pyloric network. The burst phase of pyloric neurons is relatively constant despite a > two fold variation in network frequency. We used noise input to characterize how input shape influences burst delay of a pyloric neuron, and then used dynamic clamp to examine how burst phase depends on the period, amplitude, duration, and shape of rhythmic synaptic input. Phase constancy across a range of periods required a proportional increase of synaptic duration with period. However, phase maintenance was also promoted by an increase of amplitude and peak phase of synaptic input with period. Mathematical analysis shows how short-term synaptic plasticity can coordinately change amplitude and peak phase to maximize the range of periods over which phase constancy is achieved.

Functional roles of short-term synaptic plasticity with an emphasis on inhibition.

Almost all synapses show activity-dependent dynamic changes in efficacy. Numerous studies have explored the mechanisms underlying different forms of short-term synaptic plasticity (STP), but the functional role of STP for circuit output and animal behavior is less understood. This is particularly true for inhibitory synapses that can play widely varied roles in circuit activity. We review recent findings on the role of synaptic STP in sensory, pattern generating, thalamocortical, and hippocampal networks, with a focus on synaptic inhibition. These studies show a variety of functions including sensory adaptation and gating, dynamic gain control and rhythm generation. Because experimental manipulations of STP are difficult and nonspecific, a clear demonstration of STP function often requires a combination of experimental and computational techniques.

Capturing intracellular Ca2+ dynamics in computational models of neurodegenerative diseases.

Many signaling pathways crucial for homeostatic regulation, synaptic plasticity, apoptosis and immune response depend on Ca2+. Ca2+ dysregulation disrupts normal function of neurons and neuronal networks. This causes severe motor and cognitive disabilities. Understanding how Ca2+ dysregulation triggers disease onset and progression, and affects downstream processes, can help identify targets for treatments. Because of intermingling of molecular pathways, dissecting the role of individual mechanisms and establishing causality is very challenging. Computational models provide a way to decipher these processes. I review some computational models with Ca2+ dynamics to illustrate their predictive power, and note where extending those models to capture multiscale interaction of Ca2+ dependent molecular pathways can be useful for therapeutic and drug discovery purposes.

Dendritic diameters affect the spatial variability of intracellular calcium dynamics in computer models.

There is growing interest in understanding calcium dynamics in dendrites, both experimentally and computationally. Many processes influence these dynamics, but in dendrites there is a strong contribution of morphology because the peak calcium levels are strongly determined by the surface to volume ratio (SVR) of each branch, which is inversely related to branch diameter. In this study we explore the predicted variance of dendritic calcium concentrations due to local changes in dendrite diameter and how this is affected by the modeling approach used. We investigate this in a model of dendritic calcium spiking in different reconstructions of cerebellar Purkinje cells and in morphological analysis of neocortical and hippocampal pyramidal neurons. We report that many published models neglect diameter-dependent effects on calcium concentration and show how to implement this correctly in the NEURON simulator, both for phenomenological pool based models and for implementations using radial 1D diffusion. More detailed modeling requires simulation of 3D diffusion and we demonstrate that this does not dissipate the local concentration variance due to changes of dendritic diameter. In many cases 1D diffusion of models of calcium buffering give a good approximation provided an increased morphological resolution is implemented.

Stochastic calcium mechanisms cause dendritic calcium spike variability.

Bursts of dendritic calcium spikes play an important role in excitability and synaptic plasticity in many types of neurons. In single Purkinje cells, spontaneous and synaptically evoked dendritic calcium bursts come in a variety of shapes with a variable number of spikes. The mechanisms causing this variability have never been investigated thoroughly. In this study, a detailed computational model using novel simulation routines is applied to identify the roles that stochastic ion channels, spatial arrangements of ion channels, and stochastic intracellular calcium have toward producing calcium burst variability. Consistent with experimental recordings from rats, strong variability in the burst shape is observed in simulations. This variability persists in large model sizes in contrast to models containing only voltage-gated channels, where variability reduces quickly with increase of system size. Phase plane analysis of Hodgkin-Huxley spikes and of calcium bursts identifies fluctuation in phase space around probabilistic phase boundaries as the mechanism determining the dependence of variability on model size. Stochastic calcium dynamics are the main cause of calcium burst fluctuations, specifically the calcium activation of mslo/BK-type and SK2 channels. Local variability of calcium concentration has a significant effect at larger model sizes. Simulations of both spontaneous and synaptically evoked calcium bursts in a reconstructed dendrite show, in addition, strong spatial and temporal variability of voltage and calcium, depending on morphological properties of the dendrite. Our findings suggest that stochastic intracellular calcium mechanisms play a crucial role in dendritic calcium spike generation and are therefore an essential consideration in studies of neuronal excitability and plasticity.

Controlling Ca2+-activated K+ channels with models of Ca2+ buffering in Purkinje cells.

Intracellular Ca(2+) concentrations play a crucial role in the physiological interaction between Ca(2+) channels and Ca(2+)-activated K(+) channels. The commonly used model, a Ca(2+) pool with a short relaxation time, fails to simulate interactions occurring at multiple time scales. On the other hand, detailed computational models including various Ca(2+) buffers and pumps can result in large computational cost due to radial diffusion in large compartments, which may be undesirable when simulating morphologically detailed Purkinje cell models. We present a method using a compensating mechanism to replace radial diffusion and compared the dynamics of different Ca(2+) buffering models during generation of a dendritic Ca(2+) spike in a single compartment model of a PC dendritic segment with Ca(2+) channels of P- and T-type and Ca(2+)-activated K(+) channels of BK- and SK-type. The Ca(2+) dynamics models used are (1) a single Ca(2+) pool; (2) two Ca(2+) pools, respectively, for the fast and slow transients; (3) detailed Ca(2+) dynamics with buffers, pump, and diffusion; and (4) detailed Ca(2+) dynamics with buffers, pump, and diffusion compensation. Our results show that detailed Ca(2+) dynamics models have significantly better control over Ca(2+)-activated K(+) channels and lead to physiologically more realistic simulations of Ca(2+) spikes and bursting. Furthermore, the compensating mechanism largely eliminates the effect of removing diffusion from the model on Ca(2+) dynamics over multiple time scales.

Spatial ability, experience, and skill in laparoscopic surgery.

BACKGROUND: Previous research showing correlations between spatial ability and surgical skills has used participants in relatively early stages of training. Research in skill acquisition has shown that the role of cognitive abilities can diminish as skills become increasingly automatic. In this study, we explored the role of spatial ability in laparoscopic surgical skills in two groups, one experienced and the other relatively inexperienced. METHODS: Subjects were recruited from two videoscopic courses: an advanced course for experienced surgeons and a laparoscopic urological surgery course attended by participants with relatively little laparoscopic experience. Three measures were obtained: spatial abilities, videoscopic experience, and operative skills. RESULTS: A significant correlation (r = 0.393) was found between spatial ability and skills in the lower experience group but not among the experienced surgeons (r = 0.020). CONCLUSIONS: The results are consistent with the prediction that the importance of spatial ability in performance of laparoscopic skills should diminish with experience.

Complete laparoscopic ileal cystoplasty.

INTRODUCTION: Laparoscopic enterocystoplasty provides a minimally invasive approach to bladder augmentation in the patient with a neurogenic bladder. In previously published reports, portions of the procedure were performed extracorporeally. We report our technique of complete intracorporeal laparoscopic enterocystoplasty. TECHNICAL CONSIDERATIONS: Important elements of the operation include (a) preoperative evaluation of patient compliance and videourodynamic studies; (b) cystoscopic placement of externalized ureteral stents; (c) transperitoneal placement of five radially dilating trocars; (d) identification of the cecum; (e) proximal mobilization of ileum sufficient for pelvic placement; (f) measurement of ileal length with segment of precut vessel loop; (g) vertical cystotomy after incising peritoneum and entering the space of Retzius; (h) ileal division and side-to-side anastomosis using endoscopic gastrointestinal anastomosis staplers; (i) detubularization and freehand intracorporeal suturing into a U-shaped configuration; (j) fixing ileal patch at the 6 and 11-o'clock positions; (k) completion of ileal-bladder anastomosis in quadrants with running sutures; (l) irrigation of bladder and placement of a closed suction drain in the pelvis; and (m) cystogram 4 weeks postoperatively. CONCLUSIONS: Pure laparoscopic enterocystoplasty is an advanced procedure that is technically feasible and yields excellent results, but has unproven benefits. We perform the entire operation intracorporeally with traditional instruments and do not rely on suturing devices or extracorporeal knots. Additional experience and technological developments may result in routine laparoscopic urinary augmentation and continent diversion.