Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Interactive data exploration websites for large-scale electrophysiology.

Methodological advances in neuroscience have enabled the collection of massive datasets which demand innovative approaches for scientific communication. Existing platforms for data storage lack intuitive tools for data exploration, limiting our ability to interact effectively with these brain-wide datasets. We introduce two public websites: Data and Atlas developed for the International Brain Laboratory which provide access to millions of behavioral trials and hundreds of thousands of individual neurons. These interfaces allow users to discover both the raw and processed brain-wide data released by the IBL at the scale of the whole brain, individual sessions, trials, and neurons. By hosting these data interfaces as websites they are available cross-platform with no installation. By releasing each site's code as a modular open-source framework, other researchers can easily develop their own web interfaces and explore their own data. As neuroscience datasets continue to expand, customizable web interfaces offer a glimpse into a future of streamlined data exploration and act as blueprints for future tools.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Predicting Bordeaux red wine origins and vintages from raw gas chromatograms.

Connecting chemical properties to various wine characteristics is of great interest to the science of olfaction as well as the wine industry. We explored whether Bordeaux wine chemical identities and vintages (harvest year) can be inferred from a common and affordable chemical analysis, namely, a combination of gas chromatography (GC) and electron ionization mass spectrometry. Using 12 vintages (within the 1990-2007 range) from 7 estates of the Bordeaux region, we report that, remarkably, nonlinear dimensionality reduction techniques applied to raw gas chromatograms recover the geography of the Bordeaux region. Using machine learning, we found that we can not only recover the estate perfectly from gas chromatograms, but also the vintage with up to 50% accuracy. Interestingly, we observed that the entire chromatogram is informative with respect to geographic location and age, thus suggesting that the chemical identity of a wine is not defined by just a few molecules but is distributed over a large chemical spectrum. This study demonstrates the remarkable potential of GC analysis to explore fundamental questions about the origin and age of wine.

A modular architecture for organizing, processing and sharing neurophysiology data.

We describe an architecture for organizing, integrating and sharing neurophysiology data within a single laboratory or across a group of collaborators. It comprises a database linking data files to metadata and electronic laboratory notes; a module collecting data from multiple laboratories into one location; a protocol for searching and sharing data and a module for automatic analyses that populates a website. These modules can be used together or individually, by single laboratories or worldwide collaborations.

Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders.

Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from video data. Here we introduce a new video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this tool by extracting interpretable behavioral features from videos of three different head-fixed mouse preparations, as well as a freely moving mouse in an open field arena, and show how these interpretable features can facilitate downstream behavioral and neural analyses. We also show how the behavioral features produced by our model improve the precision and interpretation of these downstream analyses compared to using the outputs of either fully supervised or fully unsupervised methods alone.

Neural network models for DMT-induced visual hallucinations.

The regulatory role of the serotonergic system on conscious perception can be investigated perturbatorily with psychedelic drugs such as N,N-Dimethyltryptamine. There is increasing evidence that the serotonergic system gates prior (endogenous) and sensory (exogenous) information in the construction of a conscious experience. Using two generative deep neural networks as examples, we discuss how such models have the potential to be, firstly, an important medium to illustrate phenomenological visual effects of psychedelics-besides paintings, verbal reports and psychometric testing-and, secondly, their utility to conceptualize biological mechanisms of gating the influence of exogenous and endogenous information on visual perception.

Projection-specific deficits in synaptic transmission in adult Sapap3-knockout mice.

Obsessive-compulsive disorder (OCD) is a circuit disorder involving corticostriatal projections, which play a role in motor control. The Sapap3-knockout (KO) mouse is a mouse model to study OCD and recapitulates OCD-like compulsion through excessive grooming behavior, with skin lesions appearing at advanced age. Deficits in corticostriatal control provide a link to the pathophysiology of OCD. However, there remain significant gaps in the characterization of the Sapap3-KO mouse, with respect to age, specificity of synaptic dysfunction, and locomotor phenotype. We therefore investigated the corticostriatal synaptic phenotype of Sapap3-KO mice using patch-clamp slice electrophysiology, in adult mice and with projection specificity. We also analyzed grooming across age and locomotor phenotype with a novel, unsupervised machine learning technique (MoSeq). Increased grooming in Sapap3-KO mice without skin lesions was age independent. Synaptic deficits persisted in adulthood and involved the projections from the motor cortices and cingulate cortex to the dorsolateral and dorsomedial striatum. Decreased synaptic strength was evident at the input from the primary motor cortex by reduction in AMPA receptor function. Hypolocomotion, i.e., slowness of movement, was consistently observed in Sapap3-KO mice. Our findings emphasize the utility of young adult Sapap3-KO mice to investigate corticostriatal synaptic dysfunction in motor control.

Neural correlates of the DMT experience assessed with multivariate EEG.

Studying transitions in and out of the altered state of consciousness caused by intravenous (IV) N,N-Dimethyltryptamine (DMT - a fast-acting tryptamine psychedelic) offers a safe and powerful means of advancing knowledge on the neurobiology of conscious states. Here we sought to investigate the effects of IV DMT on the power spectrum and signal diversity of human brain activity (6 female, 7 male) recorded via multivariate EEG, and plot relationships between subjective experience, brain activity and drug plasma concentrations across time. Compared with placebo, DMT markedly reduced oscillatory power in the alpha and beta bands and robustly increased spontaneous signal diversity. Time-referenced and neurophenomenological analyses revealed close relationships between changes in various aspects of subjective experience and changes in brain activity. Importantly, the emergence of oscillatory activity within the delta and theta frequency bands was found to correlate with the peak of the experience - particularly its eyes-closed visual component. These findings highlight marked changes in oscillatory activity and signal diversity with DMT that parallel broad and specific components of the subjective experience, thus advancing our understanding of the neurobiological underpinnings of immersive states of consciousness.

Informativeness of Auditory Stimuli Does Not Affect EEG Signal Diversity.

Brain signal diversity constitutes a robust neuronal marker of the global states of consciousness. It has been demonstrated that, in comparison to the resting wakefulness, signal diversity is lower during unconscious states, and higher during psychedelic states. A plausible interpretation of these findings is that the neuronal diversity corresponds to the diversity of subjective conscious experiences. Therefore, in the present study we varied an information rate processed by the subjects and hypothesized that greater information rate will be related to richer and more differentiated phenomenology and, consequently, to greater signal diversity. To test this hypothesis speech recordings (excerpts from an audio-book) were presented to subjects at five different speeds (65, 83, 100, 117, and 135% of the original speed). By increasing or decreasing speed of the recordings we were able to, respectively, increase or decrease the presented information rate. We also included a backward (unintelligible) speech presentation and a resting-state condition (no auditory stimulation). We tested 19 healthy subjects and analyzed the recorded EEG signal (64 channels) in terms of Lempel-Ziv diversity (LZs). We report the following findings. First, our main hypothesis was not confirmed, as Bayes Factor indicates evidence for no effect when comparing LZs among five presentation speeds. Second, we found that LZs during the resting-state was greater than during processing of both meaningful and unintelligible speech. Third, an additional analysis uncovered a gradual decrease of diversity over the time-course of the experiment, which might reflect a decrease in vigilance. We thus speculate that higher signal diversity during the unconstrained resting-state might be due to a greater variety of experiences, involving spontaneous attention switching and mind wandering.

Increased spontaneous MEG signal diversity for psychoactive doses of ketamine, LSD and psilocybin.

What is the level of consciousness of the psychedelic state? Empirically, measures of neural signal diversity such as entropy and Lempel-Ziv (LZ) complexity score higher for wakeful rest than for states with lower conscious level like propofol-induced anesthesia. Here we compute these measures for spontaneous magnetoencephalographic (MEG) signals from humans during altered states of consciousness induced by three psychedelic substances: psilocybin, ketamine and LSD. For all three, we find reliably higher spontaneous signal diversity, even when controlling for spectral changes. This increase is most pronounced for the single-channel LZ complexity measure, and hence for temporal, as opposed to spatial, signal diversity. We also uncover selective correlations between changes in signal diversity and phenomenological reports of the intensity of psychedelic experience. This is the first time that these measures have been applied to the psychedelic state and, crucially, that they have yielded values exceeding those of normal waking consciousness. These findings suggest that the sustained occurrence of psychedelic phenomenology constitutes an elevated level of consciousness - as measured by neural signal diversity.

Global and local complexity of intracranial EEG decreases during NREM sleep.

Key to understanding the neuronal basis of consciousness is the characterization of the neural signatures of changes in level of consciousness during sleep. Here we analysed three measures of dynamical complexity on spontaneous depth electrode recordings from 10 epilepsy patients during wakeful rest (WR) and different stages of sleep: (i) Lempel-Ziv complexity, which is derived from how compressible the data are; (ii) amplitude coalition entropy, which measures the variability over time of the set of channels active above a threshold; (iii) synchrony coalition entropy, which measures the variability over time of the set of synchronous channels. When computed across sets of channels that are broadly distributed across multiple brain regions, all three measures decreased substantially in all participants during early-night non-rapid eye movement (NREM) sleep. This decrease was partially reversed during late-night NREM sleep, while the measures scored similar to WR during rapid eye movement (REM) sleep. This global pattern was in almost all cases mirrored at the local level by groups of channels located in a single region. In testing for differences between regions, we found elevated signal complexity in the frontal lobe. These differences could not be attributed solely to changes in spectral power between conditions. Our results provide further evidence that the level of consciousness correlates with neural dynamical complexity.

Complexity of Multi-Dimensional Spontaneous EEG Decreases during Propofol Induced General Anaesthesia.

Emerging neural theories of consciousness suggest a correlation between a specific type of neural dynamical complexity and the level of consciousness: When awake and aware, causal interactions between brain regions are both integrated (all regions are to a certain extent connected) and differentiated (there is inhomogeneity and variety in the interactions). In support of this, recent work by Casali et al (2013) has shown that Lempel-Ziv complexity correlates strongly with conscious level, when computed on the EEG response to transcranial magnetic stimulation. Here we investigated complexity of spontaneous high-density EEG data during propofol-induced general anaesthesia. We consider three distinct measures: (i) Lempel-Ziv complexity, which is derived from how compressible the data are; (ii) amplitude coalition entropy, which measures the variability in the constitution of the set of active channels; and (iii) the novel synchrony coalition entropy (SCE), which measures the variability in the constitution of the set of synchronous channels. After some simulations on Kuramoto oscillator models which demonstrate that these measures capture distinct 'flavours' of complexity, we show that there is a robustly measurable decrease in the complexity of spontaneous EEG during general anaesthesia.