Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Collective sensing in electric fish.

A number of organisms, including dolphins, bats and electric fish, possess sophisticated active sensory systems that use self-generated signals (for example, acoustic or electrical emissions) to probe the environment1,2. Studies of active sensing in social groups have typically focused on strategies for minimizing interference from conspecific emissions2-4. However, it is well known from engineering that multiple spatially distributed emitters and receivers can greatly enhance environmental sensing (for example, multistatic radar and sonar)5-8. Here we provide evidence from modelling, neural recordings and behavioural experiments that the African weakly electric fish Gnathonemus petersii utilizes the electrical pulses of conspecifics to extend its electrolocation range, discriminate objects and increase information transmission. These results provide evidence for a new, collective mode of active sensing in which individual perception is enhanced by the energy emissions of nearby group members.

Ultra-high density electrodes improve detection, yield, and cell type identification in neuronal recordings.

To understand the neural basis of behavior, it is essential to sensitively and accurately measure neural activity at single neuron and single spike resolution. Extracellular electrophysiology delivers this, but it has biases in the neurons it detects and it imperfectly resolves their action potentials. To minimize these limitations, we developed a silicon probe with much smaller and denser recording sites than previous designs, called Neuropixels Ultra (NP Ultra). This device samples neuronal activity at ultra-high spatial density (~10 times higher than previous probes) with low noise levels, while trading off recording span. NP Ultra is effectively an implantable voltage-sensing camera that captures a planar image of a neuron's electrical field. We use a spike sorting algorithm optimized for these probes to demonstrate that the yield of visually-responsive neurons in recordings from mouse visual cortex improves up to ~3-fold. We show that NP Ultra can record from small neuronal structures including axons and dendrites. Recordings across multiple brain regions and four species revealed a subset of extracellular action potentials with unexpectedly small spatial spread and axon-like features. We share a large-scale dataset of these brain-wide recordings in mice as a resource for studies of neuronal biophysics. Finally, using ground-truth identification of three major inhibitory cortical cell types, we found that these cell types were discriminable with approximately 75% success, a significant improvement over lower-resolution recordings. NP Ultra improves spike sorting performance, detection of subcellular compartments, and cell type classification to enable more powerful dissection of neural circuit activity during behavior.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Collective Sensing in Electric Fish.

A number of organisms, including dolphins, bats, and electric fish, possess sophisticated active sensory systems that use self-generated signals (e.g. acoustic or electrical emissions) to probe the environment1,2. Studies of active sensing in social groups have typically focused on strategies for minimizing interference from conspecific emissions2-4. However, it is well-known from engineering that multiple spatially distributed emitters and receivers can greatly enhance environmental sensing (e.g. multistatic radar and sonar)5-8. Here we provide evidence from modeling, neural recordings, and behavioral experiments that the African weakly electric fish Gnathonemus petersii utilizes the electrical pulses of conspecifics to extend electrolocation range, discriminate objects, and increase information transmission. These results suggest a novel, collective mode of active sensing in which individual perception is enhanced by the energy emissions of nearby group members.

A mechanism for differential control of axonal and dendritic spiking underlying learning in a cerebellum-like circuit.

In addition to the action potentials used for axonal signaling, many neurons generate dendritic "spikes" associated with synaptic plasticity. However, in order to control both plasticity and signaling, synaptic inputs must be able to differentially modulate the firing of these two spike types. Here, we investigate this issue in the electrosensory lobe (ELL) of weakly electric mormyrid fish, where separate control over axonal and dendritic spikes is essential for the transmission of learned predictive signals from inhibitory interneurons to the output stage of the circuit. Through a combination of experimental and modeling studies, we uncover a novel mechanism by which sensory input selectively modulates the rate of dendritic spiking by adjusting the amplitude of backpropagating axonal action potentials. Interestingly, this mechanism does not require spatially segregated synaptic inputs or dendritic compartmentalization but relies instead on an electrotonically distant spike initiation site in the axon-a common biophysical feature of neurons.

An internal model for canceling self-generated sensory input in freely behaving electric fish.

Internal models that predict the sensory consequences of motor actions are vital for sensory, motor, and cognitive functions. However, the relationship between motor action and sensory input is complex, often varying from one moment to another depending on the state of the animal and the environment. The neural mechanisms for generating predictions under such challenging, real-world conditions remain largely unknown. Using novel methods for underwater neural recording, a quantitative analysis of unconstrained behavior, and computational modeling, we provide evidence for an unexpectedly sophisticated internal model at the first stage of active electrosensory processing in mormyrid fish. Closed-loop manipulations reveal that electrosensory lobe neurons are capable of simultaneously learning and storing multiple predictions of the sensory consequences of motor commands specific to different sensory states. These results provide mechanistic insights into how internal motor signals and information about the sensory environment are combined within a cerebellum-like circuitry to predict the sensory consequences of natural behavior.

A Dense Conformal Electrode Array for High Spatial Resolution Stimulation of Electrosensory Systems.

Studies of electrosensory systems have led to insights into to a number of general issues in biology. However, investigations of these systems have been limited by the inability to precisely control spatial patterns of electrosensory input. In this paper, an electrode array and a system to selectively stimulate spatially restricted regions of an electroreceptor array is presented. The array has 96 channels consisting of chrome/gold electrodes patterned on a flexible parylene-C substrate and encapsulated with another parylene-C layer. The conformability of the electrode array allows for optimal current driving and surface interface conditions. Recordings of neural activity at the first central processing stage in weakly electric mormyrid fish support the potential of this system for high spatial resolution stimulation and mapping of electrosensory systems.

Neural readout of a latency code in the active electrosensory system.

The latency of spikes relative to a stimulus conveys sensory information across modalities. However, in most cases, it remains unclear whether and how such latency codes are utilized by postsynaptic neurons. In the active electrosensory system of mormyrid fish, a latency code for stimulus amplitude in electroreceptor afferent nerve fibers (EAs) is hypothesized to be read out by a central reference provided by motor corollary discharge (CD). Here, we demonstrate that CD enhances sensory responses in postsynaptic granular cells of the electrosensory lobe but is not required for reading out EA input. Instead, diverse latency and spike count tuning across the EA population give rise to graded information about stimulus amplitude that can be read out by standard integration of converging excitatory synaptic inputs. Inhibitory control over the temporal window of integration renders two granular cell subclasses differentially sensitive to information derived from relative spike latency versus spike count.

Neurobiology: The power of pauses in electrocommunication.

A new study of social communication behavior in weakly electric fish identifies neural mechanisms that may account for the significance of silent pauses in communication.

A rapid whisker-based decision underlying skilled locomotion in mice.

Skilled motor behavior requires rapidly integrating external sensory input with information about internal state to decide which movements to make next. Using machine learning approaches for high-resolution kinematic analysis, we uncover the logic of a rapid decision underlying sensory-guided locomotion in mice. After detecting obstacles with their whiskers mice select distinct kinematic strategies depending on a whisker-derived estimate of obstacle location together with the position and velocity of their body. Although mice rely on whiskers for obstacle avoidance, lesions of primary whisker sensory cortex had minimal impact. While motor cortex manipulations affected the execution of the chosen strategy, the decision-making process remained largely intact. These results highlight the potential of machine learning for reductionist analysis of naturalistic behaviors and provide a case in which subcortical brain structures appear sufficient for mediating a relatively sophisticated sensorimotor decision.

Continual Learning in a Multi-Layer Network of an Electric Fish.

Distributing learning across multiple layers has proven extremely powerful in artificial neural networks. However, little is known about how multi-layer learning is implemented in the brain. Here, we provide an account of learning across multiple processing layers in the electrosensory lobe (ELL) of mormyrid fish and report how it solves problems well known from machine learning. Because the ELL operates and learns continuously, it must reconcile learning and signaling functions without switching its mode of operation. We show that this is accomplished through a functional compartmentalization within intermediate layer neurons in which inputs driving learning differentially affect dendritic and axonal spikes. We also find that connectivity based on learning rather than sensory response selectivity assures that plasticity at synapses onto intermediate-layer neurons is matched to the requirements of output neurons. The mechanisms we uncover have relevance to learning in the cerebellum, hippocampus, and cerebral cortex, as well as in artificial systems.

Generalization of learned responses in the mormyrid electrosensory lobe.

Appropriate generalization of learned responses to new situations is vital for adaptive behavior. We provide a circuit-level account of generalization in the electrosensory lobe (ELL) of weakly electric mormyrid fish. Much is already known in this system about a form of learning in which motor corollary discharge signals cancel responses to the uninformative input evoked by the fish's own electric pulses. However, for this cancellation to be useful under natural circumstances, it must generalize accurately across behavioral regimes, specifically different electric pulse rates. We show that such generalization indeed occurs in ELL neurons, and develop a circuit-level model explaining how this may be achieved. The mechanism involves regularized synaptic plasticity and an approximate matching of the temporal dynamics of motor corollary discharge and electrosensory inputs. Recordings of motor corollary discharge signals in mossy fibers and granule cells provide direct evidence for such matching.

Internally Generated Predictions Enhance Neural and Behavioral Detection of Sensory Stimuli in an Electric Fish.

Studies of cerebellum-like circuits in fish have demonstrated that synaptic plasticity shapes the motor corollary discharge responses of granule cells into highly-specific predictions of self-generated sensory input. However, the functional significance of such predictions, known as negative images, has not been directly tested. Here we provide evidence for improvements in neural coding and behavioral detection of prey-like stimuli due to negative images. In addition, we find that manipulating synaptic plasticity leads to specific changes in circuit output that disrupt neural coding and detection of prey-like stimuli. These results link synaptic plasticity, neural coding, and behavior and also provide a circuit-level account of how combining external sensory input with internally generated predictions enhances sensory processing.

A cerebellum-like circuit in the auditory system cancels responses to self-generated sounds.

The dorsal cochlear nucleus (DCN) integrates auditory nerve input with a diverse array of sensory and motor signals processed in circuitry similar to that of the cerebellum. Yet how the DCN contributes to early auditory processing has been a longstanding puzzle. Using electrophysiological recordings in mice during licking behavior, we show that DCN neurons are largely unaffected by self-generated sounds while remaining sensitive to external acoustic stimuli. Recordings in deafened mice, together with neural activity manipulations, indicate that self-generated sounds are cancelled by non-auditory signals conveyed by mossy fibers. In addition, DCN neurons exhibit gradual reductions in their responses to acoustic stimuli that are temporally correlated with licking. Together, these findings suggest that DCN may act as an adaptive filter for cancelling self-generated sounds. Adaptive filtering has been established previously for cerebellum-like sensory structures in fish, suggesting a conserved function for such structures across vertebrates.

Neural Mechanisms for Predicting the Sensory Consequences of Behavior: Insights from Electrosensory Systems.

Perception of the environment requires differentiating between external sensory inputs and those that are self-generated. Some of the clearest insights into the neural mechanisms underlying this process have come from studies of the electrosensory systems of fish. Neurons at the first stage of electrosensory processing generate negative images of the electrosensory consequences of the animal's own behavior. By canceling out the effects of predictable, self-generated inputs, negative images allow for the selective encoding of unpredictable, externally generated stimuli. Combined experimental and theoretical studies of electrosensory systems have led to detailed accounts of how negative images are formed at the level of synaptic plasticity rules, cells, and circuits. Here, I review these accounts and discuss their implications for understanding how predictions of the sensory consequences of behavior may be generated in other sensory structures and the cerebellum.

The Timing Is Right for Cerebellar Learning.

In this issue of Neuron, Suvrathan et al. (2016) provide a striking demonstration of a plasticity rule with temporal properties precisely matched to the computational requirements of behavioral learning and suggest major revisions to the rules for synaptic plasticity in the cerebellum.

A comparative approach to cerebellar function: insights from electrosensory systems.

Despite its simple and highly-ordered circuitry the function of the cerebellum remains a topic of vigorous debate. This review explores connections between the cerebellum and sensory processing structures that closely resemble the cerebellum in terms of their evolution, development, patterns of gene expression, and circuitry. Recent studies of cerebellum-like structures involved in electrosensory processing in fish have provided insights into the functions of granule cells and unipolar brush cells-cell types shared with the cerebellum. We also discuss the possibility, supported by recent studies, that generating and subtracting predictions of the sensory consequences of motor commands may be core functions shared by both cerebellum-like structures and the cerebellum.

Responses of cerebellar Purkinje cells during fictive optomotor behavior in larval zebrafish.

Although most studies of the cerebellum have been conducted in mammals, cerebellar circuitry is highly conserved across vertebrates, suggesting that studies of simpler systems may be useful for understanding cerebellar function. The larval zebrafish is particularly promising in this regard because of its accessibility to optical monitoring and manipulations of neural activity. Although several studies suggest that the cerebellum plays a role in behavior at larval stages, little is known about the signals conveyed by particular classes of cerebellar neurons. Here we use electrophysiological recordings to characterize subthreshold, simple spike, and climbing fiber responses in larval zebrafish Purkinje cells in the context of the fictive optomotor response (OMR)-a paradigm in which fish adjust motor output to stabilize their virtual position relative to a visual stimulus. Although visual responses were prominent in Purkinje cells, they lacked the direction or velocity sensitivity that would be expected for controlling the OMR. On the other hand, Purkinje cells exhibited strong responses during fictive swim bouts. Temporal characteristics of these responses are suggestive of a general role for the larval zebrafish cerebellum in controlling swimming. Climbing fibers encoded both visual and motor signals but did not appear to encode signals that could be used to adjust OMR gain, such as retinal slip. Finally, the observation of diverse relationships between simple spikes and climbing fiber responses in individual Purkinje cells highlights the importance of distinguishing between these two types of activity in calcium imaging experiments.