Probing the Structure and Functional Properties of the Dropout-Induced Correlated Variability in Convolutional Neural Networks.

Computational neuroscience studies have shown that the structure of neural variability to an unchanged stimulus affects the amount of information encoded. Some artificial deep neural networks, such as those with Monte Carlo dropout layers, also have variable responses when the input is fixed. However, the structure of the trial-by-trial neural covariance in neural networks with dropout has not been studied, and its role in decoding accuracy is unknown. We studied the above questions in a convolutional neural network model with dropout in both the training and testing phases. We found that trial-by-trial correlation between neurons (i.e., noise correlation) is positive and low dimensional. Neurons that are close in a feature map have larger noise correlation. These properties are surprisingly similar to the findings in the visual cortex. We further analyzed the alignment of the main axes of the covariance matrix. We found that different images share a common trial-by-trial noise covariance subspace, and they are aligned with the global signal covariance. This evidence that the noise covariance is aligned with signal covariance suggests that noise covariance in dropout neural networks reduces network accuracy, which we further verified directly with a trial-shuffling procedure commonly used in neuroscience. These findings highlight a previously overlooked aspect of dropout layers that can affect network performance. Such dropout networks could also potentially be a computational model of neural variability.

Generalizing biological surround suppression based on center surround similarity via deep neural network models.

Sensory perception is dramatically influenced by the context. Models of contextual neural surround effects in vision have mostly accounted for Primary Visual Cortex (V1) data, via nonlinear computations such as divisive normalization. However, surround effects are not well understood within a hierarchy, for neurons with more complex stimulus selectivity beyond V1. We utilized feedforward deep convolutional neural networks and developed a gradient-based technique to visualize the most suppressive and excitatory surround. We found that deep neural networks exhibited a key signature of surround effects in V1, highlighting center stimuli that visually stand out from the surround and suppressing responses when the surround stimulus is similar to the center. We found that in some neurons, especially in late layers, when the center stimulus was altered, the most suppressive surround surprisingly can follow the change. Through the visualization approach, we generalized previous understanding of surround effects to more complex stimuli, in ways that have not been revealed in visual cortices. In contrast, the suppression based on center surround similarity was not observed in an untrained network. We identified further successes and mismatches of the feedforward CNNs to the biology. Our results provide a testable hypothesis of surround effects in higher visual cortices, and the visualization approach could be adopted in future biological experimental designs.

SymbiQuant: A Machine Learning Object Detection Tool for Polyploid Independent Estimates of Endosymbiont Population Size.

Quantifying the size of endosymbiont populations is challenging because endosymbionts are typically difficult or impossible to culture and commonly polyploid. Current approaches to estimating endosymbiont population sizes include quantitative PCR (qPCR) targeting endosymbiont genomic DNA and flow-cytometry. While qPCR captures genome copy number data, it does not capture the number of bacterial cells in polyploid endosymbiont populations. In contrast, flow cytometry can capture accurate estimates of whole host-level endosymbiont population size, but it is not readily able to capture data at the level of endosymbiotic host cells. To complement these existing approaches for estimating endosymbiont population size, we designed and implemented an object detection/segmentation tool for counting the number of endosymbiont cells in micrographs of host tissues. The tool, called SymbiQuant, which makes use of recent advances in deep neural networks includes a graphic user interface that allows for human curation of tool output. We trained SymbiQuant for use in the model aphid/Buchnera endosymbiosis and studied Buchnera population dynamics and phenotype over aphid postembryonic development. We show that SymbiQuant returns accurate counts of endosymbionts, and readily captures Buchnera phenotype. By replacing our training data with data composed of annotated microscopy images from other models of endosymbiosis, SymbiQuant has the potential for broad application. Our tool, which is available on GitHub, adds to the repertoire of methods researchers can use to study endosymbiosis at the organismal, genome, and now endosymbiotic host tissue or cell levels.

Inference via sparse coding in a hierarchical vision model.

Sparse coding has been incorporated in models of the visual cortex for its computational advantages and connection to biology. But how the level of sparsity contributes to performance on visual tasks is not well understood. In this work, sparse coding has been integrated into an existing hierarchical V2 model (Hosoya & Hyvärinen, 2015), but replacing its independent component analysis (ICA) with an explicit sparse coding in which the degree of sparsity can be controlled. After training, the sparse coding basis functions with a higher degree of sparsity resembled qualitatively different structures, such as curves and corners. The contributions of the models were assessed with image classification tasks, specifically tasks associated with mid-level vision including figure-ground classification, texture classification, and angle prediction between two line stimuli. In addition, the models were assessed in comparison with a texture sensitivity measure that has been reported in V2 (Freeman et al., 2013) and a deleted-region inference task. The results from the experiments show that although sparse coding performed worse than ICA at classifying images, only sparse coding was able to better match the texture sensitivity level of V2 and infer deleted image regions, both by increasing the degree of sparsity in sparse coding. Greater degrees of sparsity allowed for inference over larger deleted image regions. The mechanism that allows for this inference capability in sparse coding is described in this article.

Chronic recordings from the marmoset motor cortex reveals modulation of neural firing and local field potentials overlap with macaques.

Objective.The common marmoset has been increasingly used in neural interfacing studies due to its smaller size, easier handling, and faster breeding compared to Old World non-human primate (NHP) species. While assessment of cortical anatomy in marmosets has shown strikingly similar layout to macaques, comprehensive assessment of electrophysiological properties underlying forelimb reaching movements in this bridge species does not exist. The objective of this study is to characterize electrophysiological properties of signals recorded from the marmoset primary motor cortex (M1) during a reach task and compare with larger NHP models such that this smaller NHP model can be used in behavioral neural interfacing studies.Approach and main results.Neuronal firing rates and local field potentials (LFPs) were chronically recorded from M1 in three adult, male marmosets. Firing rates, mu + beta and high gamma frequency bands of LFPs were evaluated for modulation with respect to movement. Firing rate and regularity of neurons of the marmoset M1 were similar to that reported in macaques with a subset of neurons showing selectivity to movement direction. Movement phases (rest vs move) was classified from both neural spiking and LFPs. Microelectrode arrays provide the ability to sample small regions of the motor cortex to drive brain-machine interfaces (BMIs). The results demonstrate that marmosets are a robust bridge species for behavioral neuroscience studies with motor cortical electrophysiological signals recorded from microelectrode arrays that are similar to Old World NHPs.Significance. As marmosets represent an interesting step between rodent and macaque models, successful demonstration that neuron modulation in marmoset motor cortex is analogous to reports in macaques illustrates the utility of marmosets as a viable species for BMI studies.

Use of Machine Learning to Re-Assess Patterns of Multivariate Functional Recovery after Fluid Percussion Injury: Operation Brain Trauma Therapy.

Traumatic brain injury (TBI) is a leading cause of death and disability. Yet, despite immense research efforts, treatment options remain elusive. Translational failures in TBI are often attributed to the heterogeneity of the TBI population and limited methods to capture these individual variabilities. Advances in machine learning (ML) have the potential to further personalized treatment strategies and better inform translational research. However, the use of ML has yet to be widely assessed in pre-clinical neurotrauma research, where data are strictly limited in subject number. To better establish ML's feasibility, we utilized the fluid percussion injury (FPI) portion of the rich, rat data set collected by Operation Brain Trauma Therapy (OBTT), which tested multiple pharmacological treatments. Previous work has provided confidence that both unsupervised and supervised ML techniques can uncover useful insights from this OBTT pre-clinical research data set. As a proof-of-concept, we aimed to better evaluate the multi-variate recovery profiles afforded by the administration of nine different experimental therapies. We assessed supervised pairwise classifiers trained on a pre-processed data set that incorporated metrics from four feature groups to determine their ability to correctly identify specific drug treatments. In all but one of the possible pairwise combinations of minocycline, levetiracetam, erythropoietin, nicotinamide, and amantadine, the baseline was outperformed by one or more supervised classifiers, the exception being nicotinamide versus amantadine. Further, when the same methods were employed to assess different doses of the same treatment, the ML classifiers had greater difficulty in understanding which treatment each sample received. Our data serve as a critical first step toward identifying optimal treatments for specific subgroups of samples that are dependent on factors such as types and severity of traumatic injuries, as well as informing the prediction of therapeutic combinations that may lead to greater treatment effects than individual therapies.

Development and Evaluation of Machine Learning Models for Recovery Prediction after Treatment for Traumatic Brain Injury.

Traumatic brain injury (TBI) is a leading cause of death and disability yet treatment strategies remain elusive. Advances in machine learning present exciting opportunities for developing personalized medicine and informing laboratory research. However, their feasibility has yet to be widely assessed in animal research where data are typically limited or in the TBI field where each patient presents with a unique injury. The Operation Brain Trauma Therapy (OBTT) has amassed an animal dataset that spans multiple types of injury, treatment strategies, behavioral assessments, histological measures, and biomarker screenings. This paper aims to analyze these data using supervised learning techniques for the first time by partitioning the dataset into acute input metrics (i.e. 7 days post-injury) and a defined recovery outcome (i.e. memory retention). Preprocessing is then applied to transform the raw OBTT dataset, e.g. developing a class attribute by histogram binning, eliminating borderline cases, and applying principal component analysis (PCA). We find that these steps are also useful in establishing a treatment ranking; Minocycline, a therapy with no significant findings in the OBTT analyses, yields the highest percentage recovery in our ranking. Furthermore, of the seven classifiers we have evaluated, Naïve Bayes achieves the best performance (67%) and yields significant improvement over our baseline model on the preprocessed dataset with borderline elimination. We also investigate the effect of testing on individual treatment groups to evaluate which groups are difficult to classify, and note the interpretive qualities of our model that can be clinically relevant.Clinical Relevance- These studies establish methods for better analyzing multivariate functional recovery and understanding which measures affect prognosis following traumatic brain injury.

Deep neural networks capture texture sensitivity in V2.

Deep convolutional neural networks (CNNs) trained on visual objects have shown intriguing ability to predict some response properties of visual cortical neurons. However, the factors (e.g., if the model is trained or not, receptive field size) and computations (e.g., convolution, rectification, pooling, normalization) that give rise to such ability, at what level, and the role of intermediate processing stages in explaining changes that develop across areas of the cortical hierarchy are poorly understood. We focused on the sensitivity to textures as a paradigmatic example, since recent neurophysiology experiments provide rich data pointing to texture sensitivity in secondary (but not primary) visual cortex (V2). We initially explored the CNN without any fitting to the neural data and found that the first two layers of the CNN showed qualitative correspondence to the first two cortical areas in terms of texture sensitivity. We therefore developed a quantitative approach to select a population of CNN model neurons that best fits the brain neural recordings. We found that the CNN could develop compatibility to secondary cortex in the second layer following rectification and that this was improved following pooling but only mildly influenced by the local normalization operation. Higher layers of the CNN could further, though modestly, improve the compatibility with the V2 data. The compatibility was reduced when incorporating random rather than learned weights. Our results show that the CNN class of model is effective for capturing changes that develop across early areas of cortex, and has the potential to help identify the computations that give rise to hierarchical processing in the brain (code is available in GitHub).

Integrating Flexible Normalization into Midlevel Representations of Deep Convolutional Neural Networks.

Deep convolutional neural networks (CNNs) are becoming increasingly popular models to predict neural responses in visual cortex. However, contextual effects, which are prevalent in neural processing and in perception, are not explicitly handled by current CNNs, including those used for neural prediction. In primary visual cortex, neural responses are modulated by stimuli spatially surrounding the classical receptive field in rich ways. These effects have been modeled with divisive normalization approaches, including flexible models, where spatial normalization is recruited only to the degree that responses from center and surround locations are deemed statistically dependent. We propose a flexible normalization model applied to midlevel representations of deep CNNs as a tractable way to study contextual normalization mechanisms in midlevel cortical areas. This approach captures nontrivial spatial dependencies among midlevel features in CNNs, such as those present in textures and other visual stimuli, that arise from tiling high-order features geometrically. We expect that the proposed approach can make predictions about when spatial normalization might be recruited in midlevel cortical areas. We also expect this approach to be useful as part of the CNN tool kit, therefore going beyond more restrictive fixed forms of normalization.

Normalization and pooling in hierarchical models of natural images.

Divisive normalization and subunit pooling are two canonical classes of computation that have become widely used in descriptive (what) models of visual cortical processing. Normative (why) models from natural image statistics can help constrain the form and parameters of such classes of models. We focus on recent advances in two particular directions, namely deriving richer forms of divisive normalization, and advances in learning pooling from image statistics. We discuss the incorporation of such components into hierarchical models. We consider both hierarchical unsupervised learning from image statistics, and discriminative supervised learning in deep convolutional neural networks (CNNs). We further discuss studies on the utility and extensions of the convolutional architecture, which has also been adopted by recent descriptive models. We review the recent literature and discuss the current promises and gaps of using such approaches to gain a better understanding of how cortical neurons represent and process complex visual stimuli.

Stimulus- and goal-oriented frameworks for understanding natural vision.

Our knowledge of sensory processing has advanced dramatically in the last few decades, but this understanding remains far from complete, especially for stimuli with the large dynamic range and strong temporal and spatial correlations characteristic of natural visual inputs. Here we describe some of the issues that make understanding the encoding of natural images a challenge. We highlight two broad strategies for approaching this problem: a stimulus-oriented framework and a goal-oriented one. Different contexts can call for one framework or the other. Looking forward, recent advances, particularly those based in machine learning, show promise in borrowing key strengths of both frameworks and by doing so illuminating a path to a more comprehensive understanding of the encoding of natural stimuli.

Decoding of finger trajectory from ECoG using deep learning.

OBJECTIVE: Conventional decoding pipeline for brain-machine interfaces (BMIs) consists of chained different stages of feature extraction, time-frequency analysis and statistical learning models. Each of these stages uses a different algorithm trained in a sequential manner, which makes it difficult to make the whole system adaptive. The goal was to create an adaptive online system with a single objective function and a single learning algorithm so that the whole system can be trained in parallel to increase the decoding performance. Here, we used deep neural networks consisting of convolutional neural networks (CNN) and a special kind of recurrent neural network (RNN) called long short term memory (LSTM) to address these needs. APPROACH: We used electrocorticography (ECoG) data collected by Kubanek et al. The task consisted of individual finger flexions upon a visual cue. Our model combined a hierarchical feature extractor CNN and a RNN that was able to process sequential data and recognize temporal dynamics in the neural data. CNN was used as the feature extractor and LSTM was used as the regression algorithm to capture the temporal dynamics of the signal. MAIN RESULTS: We predicted the finger trajectory using ECoG signals and compared results for the least angle regression (LARS), CNN-LSTM, random forest, LSTM model (LSTM_HC, for using hard-coded features) and a decoding pipeline consisting of band-pass filtering, energy extraction, feature selection and linear regression. The results showed that the deep learning models performed better than the commonly used linear model. The deep learning models not only gave smoother and more realistic trajectories but also learned the transition between movement and rest state. SIGNIFICANCE: This study demonstrated a decoding network for BMI that involved a convolutional and recurrent neural network model. It integrated the feature extraction pipeline into the convolution and pooling layer and used LSTM layer to capture the state transitions. The discussed network eliminated the need to separately train the model at each step in the decoding pipeline. The whole system can be jointly optimized using stochastic gradient descent and is capable of online learning.

Adaptation in the visual cortex: a case for probing neuronal populations with natural stimuli.

The perception of, and neural responses to, sensory stimuli in the present are influenced by what has been observed in the past-a phenomenon known as adaptation. We focus on adaptation in visual cortical neurons as a paradigmatic example. We review recent work that represents two shifts in the way we study adaptation, namely (i) going beyond single neurons to study adaptation in populations of neurons and (ii) going beyond simple stimuli to study adaptation to natural stimuli. We suggest that efforts in these two directions, through a closer integration of experimental and modeling approaches, will enable a more complete understanding of cortical processing in natural environments.

Statistical Learning of Melodic Patterns Influences the Brain's Response to Wrong Notes.

The theory of statistical learning has been influential in providing a framework for how humans learn to segment patterns of regularities from continuous sensory inputs, such as speech and music. This form of learning is based on statistical cues and is thought to underlie the ability to learn to segment patterns of regularities from continuous sensory inputs, such as the transition probabilities in speech and music. However, the connection between statistical learning and brain measurements is not well understood. Here we focus on ERPs in the context of tone sequences that contain statistically cohesive melodic patterns. We hypothesized that implicit learning of statistical regularities would influence what was held in auditory working memory. We predicted that a wrong note occurring within a cohesive pattern (within-pattern deviant) would lead to a significantly larger brain signal than a wrong note occurring between cohesive patterns (between-pattern deviant), even though both deviant types were equally likely to occur with respect to the global tone sequence. We discuss this prediction within a simple Markov model framework that learns the transition probability regularities within the tone sequence. Results show that signal strength was stronger when cohesive patterns were violated and demonstrate that the transitional probability of the sequence influences the memory basis for melodic patterns. Our results thus characterize how informational units are stored in auditory memory trace for deviance detection and provide new evidence about how the brain organizes sequential sound input that is useful for perception.

Behavioral and neural constraints on hierarchical representations.

Central to behavior and cognition is the way that sensory stimuli are represented in neural systems. The distributions over such stimuli enjoy rich structure; however, how the brain captures and exploits these regularities is unclear. Here, we consider different sources of perhaps the most prevalent form of structure, namely hierarchies, in one of its most prevalent cases, namely the representation of images. We review experimental approaches across a range of subfields, spanning inference, memory recall, and visual adaptation, to investigate how these constrain hierarchical representations. We also discuss progress in building hierarchical models of the representation of images-this has the potential to clarify how the structure of the world is reflected in biological systems. We suggest there is a need for a closer embedding of recent advances in machine learning and computer vision into the design and interpretation of experiments, notably by utilizing the understanding of the structure of natural scenes and through the creation of hierarchically structured synthetic stimuli.

Distinguishing Neural Adaptation and Predictive Coding Hypotheses in Auditory Change Detection.

The auditory mismatch negativity (MMN) component of event-related potentials (ERPs) has served as a neural index of auditory change detection. MMN is elicited by presentation of infrequent (deviant) sounds randomly interspersed among frequent (standard) sounds. Deviants elicit a larger negative deflection in the ERP waveform compared to the standard. There is considerable debate as to whether the neural mechanism of this change detection response is due to release from neural adaptation (neural adaptation hypothesis) or from a prediction error signal (predictive coding hypothesis). Previous studies have not been able to distinguish between these explanations because paradigms typically confound the two. The current study disambiguated effects of stimulus-specific adaptation from expectation violation using a unique stimulus design that compared expectation violation responses that did and did not involve stimulus change. The expectation violation response without the stimulus change differed in timing, scalp distribution, and attentional modulation from the more typical MMN response. There is insufficient evidence from the current study to suggest that the negative deflection elicited by the expectation violation alone includes the MMN. Thus, we offer a novel hypothesis that the expectation violation response reflects a fundamentally different neural substrate than that attributed to the canonical MMN.

Specificity and timescales of cortical adaptation as inferences about natural movie statistics.

Adaptation is a phenomenological umbrella term under which a variety of temporal contextual effects are grouped. Previous models have shown that some aspects of visual adaptation reflect optimal processing of dynamic visual inputs, suggesting that adaptation should be tuned to the properties of natural visual inputs. However, the link between natural dynamic inputs and adaptation is poorly understood. Here, we extend a previously developed Bayesian modeling framework for spatial contextual effects to the temporal domain. The model learns temporal statistical regularities of natural movies and links these statistics to adaptation in primary visual cortex via divisive normalization, a ubiquitous neural computation. In particular, the model divisively normalizes the present visual input by the past visual inputs only to the degree that these are inferred to be statistically dependent. We show that this flexible form of normalization reproduces classical findings on how brief adaptation affects neuronal selectivity. Furthermore, prior knowledge acquired by the Bayesian model from natural movies can be modified by prolonged exposure to novel visual stimuli. We show that this updating can explain classical results on contrast adaptation. We also simulate the recent finding that adaptation maintains population homeostasis, namely, a balanced level of activity across a population of neurons with different orientation preferences. Consistent with previous disparate observations, our work further clarifies the influence of stimulus-specific and neuronal-specific normalization signals in adaptation.

Integrative properties of retinal ganglion cell electrical responsiveness depend on neurotrophic support and genotype in the mouse.

Early stages of glaucoma and optic neuropathies are thought to show inner retina remodeling and functional changes of retinal ganglion cells (RGCs) before they die. To assess RGC functional plasticity, we investigated the contrast-gain control properties of the pattern electroretinogram (PERG), a sensitive measure of RGC function, as an index of spatio-temporal integration occurring in the inner retina circuitry subserving PERG generators. We studied the integrative properties of the PERG in mice exposed to different conditions of neurotrophic support. We also investigated the effect of genotypic differences among mouse strains with different susceptibility to glaucoma (C57BL/6J, DBA/2J, DBA/2.Gpnmb(+)). Results show that the integrative properties of the PERG recorded in the standard C57BL/6J inbred mouse strain are impaired after deficit of neurotrophic support and partially restored after exogenous neurotrophic administration. Changes in PERG amplitude, latency, and contrast-dependent responses differ between mouse strains with different susceptibility to glaucoma. Results represent a proof of concept that the PERG could be used as a tool for in-vivo monitoring of RGC functional plasticity before RGC death, the effect of neuroactive treatments, as well as for high-throughput tool for phenotypic screening of different mouse genotypes.

Flexible gating of contextual influences in natural vision.

Identical sensory inputs can be perceived as markedly different when embedded in distinct contexts. Neural responses to simple stimuli are also modulated by context, but the contribution of this modulation to the processing of natural sensory input is unclear. We measured surround suppression, a quintessential contextual influence, in macaque primary visual cortex with natural images. We found that suppression strength varied substantially for different images. This variability was not well explained by existing descriptions of surround suppression, but it was predicted by Bayesian inference about statistical dependencies in images. In this framework, surround suppression was flexible: it was recruited when the image was inferred to contain redundancies and substantially reduced in strength otherwise. Thus, our results reveal a gating of a basic, widespread cortical computation by inference about the statistics of natural input.