Predictive coding with spiking neurons and feedforward gist signaling.

Predictive coding (PC) is an influential theory in neuroscience, which suggests the existence of a cortical architecture that is constantly generating and updating predictive representations of sensory inputs. Owing to its hierarchical and generative nature, PC has inspired many computational models of perception in the literature. However, the biological plausibility of existing models has not been sufficiently explored due to their use of artificial neurons that approximate neural activity with firing rates in the continuous time domain and propagate signals synchronously. Therefore, we developed a spiking neural network for predictive coding (SNN-PC), in which neurons communicate using event-driven and asynchronous spikes. Adopting the hierarchical structure and Hebbian learning algorithms from previous PC neural network models, SNN-PC introduces two novel features: (1) a fast feedforward sweep from the input to higher areas, which generates a spatially reduced and abstract representation of input (i.e., a neural code for the gist of a scene) and provides a neurobiological alternative to an arbitrary choice of priors; and (2) a separation of positive and negative error-computing neurons, which counters the biological implausibility of a bi-directional error neuron with a very high baseline firing rate. After training with the MNIST handwritten digit dataset, SNN-PC developed hierarchical internal representations and was able to reconstruct samples it had not seen during training. SNN-PC suggests biologically plausible mechanisms by which the brain may perform perceptual inference and learning in an unsupervised manner. In addition, it may be used in neuromorphic applications that can utilize its energy-efficient, event-driven, local learning, and parallel information processing nature.

Theta Phase Entrainment of Single-Cell Spiking in Rat Somatosensory Barrel Cortex and Secondary Visual Cortex Is Enhanced during Multisensory Discrimination Behavior.

Phase entrainment of cells by theta oscillations is thought to globally coordinate the activity of cell assemblies across different structures, such as the hippocampus and neocortex. This coordination is likely required for optimal processing of sensory input during recognition and decision-making processes. In quadruple-area ensemble recordings from male rats engaged in a multisensory discrimination task, we investigated phase entrainment of cells by theta oscillations in areas along the corticohippocampal hierarchy: somatosensory barrel cortex (S1BF), secondary visual cortex (V2L), perirhinal cortex (PER), and dorsal hippocampus (dHC). Rats discriminated between two 3D objects presented in tactile-only, visual-only, or both tactile and visual modalities. During task engagement, S1BF, V2L, PER, and dHC LFP signals showed coherent theta-band activity. We found phase entrainment of single-cell spiking activity to locally recorded as well as hippocampal theta activity in S1BF, V2L, PER, and dHC. While phase entrainment of hippocampal spikes to local theta oscillations occurred during sustained epochs of task trials and was nonselective for behavior and modality, somatosensory and visual cortical cells were only phase entrained during stimulus presentation, mainly in their preferred modality (S1BF, tactile; V2L, visual), with subsets of cells selectively phase-entrained during cross-modal stimulus presentation (S1BF: visual; V2L: tactile). This effect could not be explained by modulations of firing rate or theta amplitude. Thus, hippocampal cells are phase entrained during prolonged epochs, while sensory and perirhinal neurons are selectively entrained during sensory stimulus presentation, providing a brief time window for coordination of activity.

An integrative, multiscale view on neural theories of consciousness.

How is conscious experience related to material brain processes? A variety of theories aiming to answer this age-old question have emerged from the recent surge in consciousness research, and some are now hotly debated. Although most researchers have so far focused on the development and validation of their preferred theory in relative isolation, this article, written by a group of scientists representing different theories, takes an alternative approach. Noting that various theories often try to explain different aspects or mechanistic levels of consciousness, we argue that the theories do not necessarily contradict each other. Instead, several of them may converge on fundamental neuronal mechanisms and be partly compatible and complementary, so that multiple theories can simultaneously contribute to our understanding. Here, we consider unifying, integration-oriented approaches that have so far been largely neglected, seeking to combine valuable elements from various theories.

Deep brain stimulation of the central thalamus restores arousal and motivation in a zolpidem-responsive patient with akinetic mutism after severe brain injury.

After severe brain injury, zolpidem is known to cause spectacular, often short-lived, restorations of brain functions in a small subgroup of patients. Previously, we showed that these zolpidem-induced neurological recoveries can be paralleled by significant changes in functional connectivity throughout the brain. Deep brain stimulation (DBS) is a neurosurgical intervention known to modulate functional connectivity in a wide variety of neurological disorders. In this study, we used DBS to restore arousal and motivation in a zolpidem-responsive patient with severe brain injury and a concomitant disorder of diminished motivation, more than 10 years after surviving hypoxic ischemia. We found that DBS of the central thalamus, targeted at the centromedian-parafascicular complex, immediately restored arousal and was able to transition the patient from a state of deep sleep to full wakefulness. Moreover, DBS was associated with temporary restoration of communication and ability to walk and eat in an otherwise wheelchair-bound and mute patient. With the use of magnetoencephalography (MEG), we revealed that DBS was generally associated with a marked decrease in aberrantly high levels of functional connectivity throughout the brain, mimicking the effects of zolpidem. These results imply that 'pathological hyperconnectivity' after severe brain injury can be associated with reduced arousal and behavioral performance and that DBS is able to modulate connectivity towards a 'healthier baseline' with lower synchronization, and, can restore functional brain networks long after severe brain injury. The presence of hyperconnectivity after brain injury may be a possible future marker for a patient's responsiveness for restorative interventions, such as DBS, and suggests that lower degrees of overall brain synchronization may be conducive to cognition and behavioral responsiveness.

Triple dissociation of visual, auditory and motor processing in mouse primary visual cortex.

Primary sensory cortices respond to crossmodal stimuli-for example, auditory responses are found in primary visual cortex (V1). However, it remains unclear whether these responses reflect sensory inputs or behavioral modulation through sound-evoked body movement. We address this controversy by showing that sound-evoked activity in V1 of awake mice can be dissociated into auditory and behavioral components with distinct spatiotemporal profiles. The auditory component began at approximately 27 ms, was found in superficial and deep layers and originated from auditory cortex. Sound-evoked orofacial movements correlated with V1 neural activity starting at approximately 80-100 ms and explained auditory frequency tuning. Visual, auditory and motor activity were expressed by different laminar profiles and largely segregated subsets of neuronal populations. During simultaneous audiovisual stimulation, visual representations remained dissociable from auditory-related and motor-related activity. This three-fold dissociability of auditory, motor and visual processing is central to understanding how distinct inputs to visual cortex interact to support vision.

Neural correlates of object identity and reward outcome in the sensory cortical-hippocampal hierarchy: coding of motivational information in perirhinal cortex.

Neural circuits support behavioral adaptations by integrating sensory and motor information with reward and error-driven learning signals, but it remains poorly understood how these signals are distributed across different levels of the corticohippocampal hierarchy. We trained rats on a multisensory object-recognition task and compared visual and tactile responses of simultaneously recorded neuronal ensembles in somatosensory cortex, secondary visual cortex, perirhinal cortex, and hippocampus. The sensory regions primarily represented unisensory information, whereas hippocampus was modulated by both vision and touch. Surprisingly, the sensory cortices and the hippocampus coded object-specific information, whereas the perirhinal cortex did not. Instead, perirhinal cortical neurons signaled trial outcome upon reward-based feedback. A majority of outcome-related perirhinal cells responded to a negative outcome (reward omission), whereas a minority of other cells coded positive outcome (reward delivery). Our results highlight a distributed neural coding of multisensory variables in the cortico-hippocampal hierarchy. Notably, the perirhinal cortex emerges as a crucial region for conveying motivational outcomes, whereas distinct functions related to object identity are observed in the sensory cortices and hippocampus.

How deep is the brain? The shallow brain hypothesis.

Deep learning and predictive coding architectures commonly assume that inference in neural networks is hierarchical. However, largely neglected in deep learning and predictive coding architectures is the neurobiological evidence that all hierarchical cortical areas, higher or lower, project to and receive signals directly from subcortical areas. Given these neuroanatomical facts, today's dominance of cortico-centric, hierarchical architectures in deep learning and predictive coding networks is highly questionable; such architectures are likely to be missing essential computational principles the brain uses. In this Perspective, we present the shallow brain hypothesis: hierarchical cortical processing is integrated with a massively parallel process to which subcortical areas substantially contribute. This shallow architecture exploits the computational capacity of cortical microcircuits and thalamo-cortical loops that are not included in typical hierarchical deep learning and predictive coding networks. We argue that the shallow brain architecture provides several critical benefits over deep hierarchical structures and a more complete depiction of how mammalian brains achieve fast and flexible computational capabilities.

Local minimization of prediction errors drives learning of invariant object representations in a generative network model of visual perception.

The ventral visual processing hierarchy of the cortex needs to fulfill at least two key functions: perceived objects must be mapped to high-level representations invariantly of the precise viewing conditions, and a generative model must be learned that allows, for instance, to fill in occluded information guided by visual experience. Here, we show how a multilayered predictive coding network can learn to recognize objects from the bottom up and to generate specific representations via a top-down pathway through a single learning rule: the local minimization of prediction errors. Trained on sequences of continuously transformed objects, neurons in the highest network area become tuned to object identity invariant of precise position, comparable to inferotemporal neurons in macaques. Drawing on this, the dynamic properties of invariant object representations reproduce experimentally observed hierarchies of timescales from low to high levels of the ventral processing stream. The predicted faster decorrelation of error-neuron activity compared to representation neurons is of relevance for the experimental search for neural correlates of prediction errors. Lastly, the generative capacity of the network is confirmed by reconstructing specific object images, robust to partial occlusion of the inputs. By learning invariance from temporal continuity within a generative model, the approach generalizes the predictive coding framework to dynamic inputs in a more biologically plausible way than self-supervised networks with non-local error-backpropagation. This was achieved simply by shifting the training paradigm to dynamic inputs, with little change in architecture and learning rule from static input-reconstructing Hebbian predictive coding networks.

How 'visual' is the visual cortex? The interactions between the visual cortex and other sensory, motivational and motor systems as enabling factors for visual perception.

The definition of the visual cortex is primarily based on the evidence that lesions of this area impair visual perception. However, this does not exclude that the visual cortex may process more information than of retinal origin alone, or that other brain structures contribute to vision. Indeed, research across the past decades has shown that non-visual information, such as neural activity related to reward expectation and value, locomotion, working memory and other sensory modalities, can modulate primary visual cortical responses to retinal inputs. Nevertheless, the function of this non-visual information is poorly understood. Here we review recent evidence, coming primarily from studies in rodents, arguing that non-visual and motor effects in visual cortex play a role in visual processing itself, for instance disentangling direct auditory effects on visual cortex from effects of sound-evoked orofacial movement. These findings are placed in a broader framework casting vision in terms of predictive processing under control of frontal, reward- and motor-related systems. In contrast to the prevalent notion that vision is exclusively constructed by the visual cortical system, we propose that visual percepts are generated by a larger network-the extended visual system-spanning other sensory cortices, supramodal areas and frontal systems. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Spike-based coupling between single neurons and populations across rat sensory cortices, perirhinal cortex, and hippocampus.

Cortical computations require coordination of neuronal activity within and across multiple areas. We characterized spiking relationships within and between areas by quantifying coupling of single neurons to population firing patterns. Single-neuron population coupling (SNPC) was investigated using ensemble recordings from hippocampal CA1 region and somatosensory, visual, and perirhinal cortices. Within-area coupling was heterogeneous across structures, with area CA1 showing higher levels than neocortical regions. In contrast to known anatomical connectivity, between-area coupling showed strong firing coherence of sensory neocortices with CA1, but less with perirhinal cortex. Cells in sensory neocortices and CA1 showed positive correlations between within- and between-area coupling; these were weaker for perirhinal cortex. All four areas harbored broadcasting cells, connecting to multiple external areas, which was uncorrelated to within-area coupling strength. When examining correlations between SNPC and spatial coding, we found that, if such correlations were significant, they were negative. This result was consistent with an overall preservation of SNPC across different brain states, suggesting a strong dependence on intrinsic network connectivity. Overall, SNPC offers an important window on cell-to-population synchronization in multi-area networks. Instead of pointing to specific information-coding functions, our results indicate a primary function of SNPC in dynamically organizing communication in systems composed of multiple, interconnected areas.

A novel task to investigate vibrotactile detection in mice.

Throughout the last decades, understanding the neural mechanisms of sensory processing has been a key objective for neuroscientists. Many studies focused on uncovering the microcircuit-level architecture of somatosensation using the rodent whisker system as a model. Although these studies have significantly advanced our understanding of tactile processing, the question remains to what extent the whisker system can provide results translatable to the human somatosensory system. To address this, we developed a restrained vibrotactile detection task involving the limb system in mice. A vibrotactile stimulus was delivered to the hindlimb of head-fixed mice, who were trained to perform a Go/No-go detection task. Mice were able to learn this task with satisfactory performance and with reasonably short training times. In addition, the task we developed is versatile, as it can be combined with diverse neuroscience methods. Thus, this study introduces a novel task to study the neuron-level mechanisms of tactile processing in a system other than the more commonly studied whisker system.

Spontaneous variations in arousal modulate subsequent visual processing and local field potential dynamics in the ferret during quiet wakefulness.

Behavioral states affect neuronal responses throughout the cortex and influence visual processing. Quiet wakefulness (QW) is a behavioral state during which subjects are quiescent but awake and connected to the environment. Here, we examined the effects of pre-stimulus arousal variability on post-stimulus neural activity in the primary visual cortex and posterior parietal cortex in awake ferrets, using pupil diameter as an indicator of arousal. We observed that the power of stimuli-induced alpha (8-12 Hz) decreases when the arousal level increases. The peak of alpha power shifts depending on arousal. High arousal increases inter- and intra-areal coherence. Using a simplified model of laminar circuits, we show that this connectivity pattern is compatible with feedback signals targeting infragranular layers in area posterior parietal cortex and supragranular layers in V1. During high arousal, neurons in V1 displayed higher firing rates at their preferred orientations. Broad-spiking cells in V1 are entrained to high-frequency oscillations (>80 Hz), whereas narrow-spiking neurons are phase-locked to low- (12-18 Hz) and high-frequency (>80 Hz) rhythms. These results indicate that the variability and sensitivity of post-stimulus cortical responses and coherence depend on the pre-stimulus behavioral state and account for the neuronal response variability observed during repeated stimulation.

Coherent mapping of position and head direction across auditory and visual cortex.

Neurons in primary visual cortex (V1) may not only signal current visual input but also relevant contextual information such as reward expectancy and the subject's spatial position. Such contextual representations need not be restricted to V1 but could participate in a coherent mapping throughout sensory cortices. Here, we show that spiking activity coherently represents a location-specific mapping across auditory cortex (AC) and lateral, secondary visual cortex (V2L) of freely moving rats engaged in a sensory detection task on a figure-8 maze. Single-unit activity of both areas showed extensive similarities in terms of spatial distribution, reliability, and position coding. Importantly, reconstructions of subject position based on spiking activity displayed decoding errors that were correlated between areas. Additionally, we found that head direction, but not locomotor speed or head angular velocity, was an important determinant of activity in AC and V2L. By contrast, variables related to the sensory task cues or to trial correctness and reward were not markedly encoded in AC and V2L. We conclude that sensory cortices participate in coherent, multimodal representations of the subject's sensory-specific location. These may provide a common reference frame for distributed cortical sensory and motor processes and may support crossmodal predictive processing.

What is neurorepresentationalism? From neural activity and predictive processing to multi-level representations and consciousness.

This review provides an update on Neurorepresentationalism, a theoretical framework that defines conscious experience as multimodal, situational survey and explains its neural basis from brain systems constructing best-guess representations of sensations originating in our environment and body (Pennartz, 2015). It posits that conscious experience is characterized by five essential hallmarks: (i) multimodal richness, (ii) situatedness and immersion, (iii) unity and integration, (iv) dynamics and stability, and (v) intentionality. Consciousness is furthermore proposed to have a biological function, framed by the contrast between reflexes and habits (not requiring consciousness) versus goal-directed, planned behavior (requiring multimodal, situational survey). Conscious experience is therefore understood as a sensorily rich, spatially encompassing representation of body and environment, while we nevertheless have the impression of experiencing external reality directly. Contributions to understanding neural mechanisms underlying consciousness are derived from models for predictive processing, which are trained in an unsupervised manner, do not necessarily require overt action, and have been extended to deep neural networks. Even with predictive processing in place, however, the question remains why this type of neural network activity would give rise to phenomenal experience. Here, I propose to tackle the Hard Problem with the concept of multi-level representations which emergently give rise to multimodal, spatially wide superinferences corresponding to phenomenal experiences. Finally, Neurorepresentationalism is compared to other neural theories of consciousness, and its implications for defining indicators of consciousness in animals, artificial intelligence devices and immobile or unresponsive patients with disorders of consciousness are discussed.

Multisensory task demands temporally extend the causal requirement for visual cortex in perception.

Primary sensory areas constitute crucial nodes during perceptual decision making. However, it remains unclear to what extent they mainly constitute a feedforward processing step, or rather are continuously involved in a recurrent network together with higher-order areas. We found that the temporal window in which primary visual cortex is required for the detection of identical visual stimuli was extended when task demands were increased via an additional sensory modality that had to be monitored. Late-onset optogenetic inactivation preserved bottom-up, early-onset responses which faithfully encoded stimulus features, and was effective in impairing detection only if it preceded a late, report-related phase of the cortical response. Increasing task demands were marked by longer reaction times and the effect of late optogenetic inactivation scaled with reaction time. Thus, independently of visual stimulus complexity, multisensory task demands determine the temporal requirement for ongoing sensory-related activity in V1, which overlaps with report-related activity.

What is exactly the problem with panpsychism?

Merker et al.'s critique calls for a deeper analysis of panpsychism. In principle, the concept of integrated information can be applied to photodiodes and subatomic particles, but I suggest the main obstacle is the lack of any evidence to confirm the presence of consciousness. Also MRW's perspectivalist theory illustrates the difficulties in synthesizing a full-fledged theory of consciousness.

Higher order visual areas enhance stimulus responsiveness in mouse primary visual cortex.

Over the past few years, the various areas that surround the primary visual cortex (V1) in the mouse have been associated with many functions, ranging from higher order visual processing to decision-making. Recently, some studies have shown that higher order visual areas influence the activity of the primary visual cortex, refining its processing capabilities. Here, we studied how in vivo optogenetic inactivation of two higher order visual areas with different functional properties affects responses evoked by moving bars in the primary visual cortex. In contrast with the prevailing view, our results demonstrate that distinct higher order visual areas similarly modulate early visual processing. In particular, these areas enhance stimulus responsiveness in the primary visual cortex, by more strongly amplifying weaker compared with stronger sensory-evoked responses (for instance specifically amplifying responses to stimuli not moving along the direction preferred by individual neurons) and by facilitating responses to stimuli entering the receptive field of single neurons. Such enhancement, however, comes at the expense of orientation and direction selectivity, which increased when the selected higher order visual areas were inactivated. Thus, feedback from higher order visual areas selectively amplifies weak sensory-evoked V1 responses, which may enable more robust processing of visual stimuli.

Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding.

Recognising familiar places is a competence required in many engineering applications that interact with the real world such as robot navigation. Combining information from different sensory sources promotes robustness and accuracy of place recognition. However, mismatch in data registration, dimensionality, and timing between modalities remain challenging problems in multisensory place recognition. Spurious data generated by sensor drop-out in multisensory environments is particularly problematic and often resolved through adhoc and brittle solutions. An effective approach to these problems is demonstrated by animals as they gracefully move through the world. Therefore, we take a neuro-ethological approach by adopting self-supervised representation learning based on a neuroscientific model of visual cortex known as predictive coding. We demonstrate how this parsimonious network algorithm which is trained using a local learning rule can be extended to combine visual and tactile sensory cues from a biomimetic robot as it naturally explores a visually aliased environment. The place recognition performance obtained using joint latent representations generated by the network is significantly better than contemporary representation learning techniques. Further, we see evidence of improved robustness at place recognition in face of unimodal sensor drop-out. The proposed multimodal deep predictive coding algorithm presented is also linearly extensible to accommodate more than two sensory modalities, thereby providing an intriguing example of the value of neuro-biologically plausible representation learning for multimodal navigation.

Deep Gated Hebbian Predictive Coding Accounts for Emergence of Complex Neural Response Properties Along the Visual Cortical Hierarchy.

Predictive coding provides a computational paradigm for modeling perceptual processing as the construction of representations accounting for causes of sensory inputs. Here, we developed a scalable, deep network architecture for predictive coding that is trained using a gated Hebbian learning rule and mimics the feedforward and feedback connectivity of the cortex. After training on image datasets, the models formed latent representations in higher areas that allowed reconstruction of the original images. We analyzed low- and high-level properties such as orientation selectivity, object selectivity and sparseness of neuronal populations in the model. As reported experimentally, image selectivity increased systematically across ascending areas in the model hierarchy. Depending on the strength of regularization factors, sparseness also increased from lower to higher areas. The results suggest a rationale as to why experimental results on sparseness across the cortical hierarchy have been inconsistent. Finally, representations for different object classes became more distinguishable from lower to higher areas. Thus, deep neural networks trained using a gated Hebbian formulation of predictive coding can reproduce several properties associated with neuronal responses along the visual cortical hierarchy.