Multimodal Representation Learning for Place Recognition Using Deep Hebbian Predictive Coding.

Recognising familiar places is a competence required in many engineering applications that interact with the real world such as robot navigation. Combining information from different sensory sources promotes robustness and accuracy of place recognition. However, mismatch in data registration, dimensionality, and timing between modalities remain challenging problems in multisensory place recognition. Spurious data generated by sensor drop-out in multisensory environments is particularly problematic and often resolved through adhoc and brittle solutions. An effective approach to these problems is demonstrated by animals as they gracefully move through the world. Therefore, we take a neuro-ethological approach by adopting self-supervised representation learning based on a neuroscientific model of visual cortex known as predictive coding. We demonstrate how this parsimonious network algorithm which is trained using a local learning rule can be extended to combine visual and tactile sensory cues from a biomimetic robot as it naturally explores a visually aliased environment. The place recognition performance obtained using joint latent representations generated by the network is significantly better than contemporary representation learning techniques. Further, we see evidence of improved robustness at place recognition in face of unimodal sensor drop-out. The proposed multimodal deep predictive coding algorithm presented is also linearly extensible to accommodate more than two sensory modalities, thereby providing an intriguing example of the value of neuro-biologically plausible representation learning for multimodal navigation.

Fast, Flexible Closed-Loop Feedback: Tracking Movement in "Real-Millisecond-Time".

One of the principal functions of the brain is to control movement and rapidly adapt behavior to a changing external environment. Over the last decades our ability to monitor activity in the brain, manipulate it while also manipulating the environment the animal moves through, has been tackled with increasing sophistication. However, our ability to track the movement of the animal in real time has not kept pace. Here, we use a dynamic vision sensor (DVS) based event-driven neuromorphic camera system to implement real-time, low-latency tracking of a single whisker that mice can move at ∼25 Hz. The customized DVS system described here converts whisker motion into a series of events that can be used to estimate the position of the whisker and to trigger a position-based output interactively within 2 ms. This neuromorphic chip-based closed-loop system provides feedback rapidly and flexibly. With this system, it becomes possible to use the movement of whiskers or in principal, movement of any part of the body to reward, punish, in a rapidly reconfigurable way. These methods can be used to manipulate behavior, and the neural circuits that help animals adapt to changing values of a sequence of motor actions.

IntelliTable: Inclusively-Designed Furniture with Robotic Capabilities.

IntelliTable is a new proof-of-principle assistive technology system with robotic capabilities in the form of an elegant universal cantilever table able to move around by itself, or under user control. We describe the design and current capabilities of the table and the human-centered design methodology used in its development and initial evaluation. The IntelliTable study has delivered robotic platform programmed by a smartphone that can navigate around a typical home or care environment, avoiding obstacles, and positioning itself at the user's command. It can also be configured to navigate itself to pre-ordained places positions within an environment using ceiling tracking, responsive optical guidance and object-based sonar navigation.

Strategy change in vibrissal active sensing during rat locomotion.

During exploration, rats and other small mammals make rhythmic back-and-forth sweeps of their long facial whiskers (macrovibrissae) [1-3]. These "whisking" movements are modulated by head movement [4] and by vibrissal sensory input [5, 6] and hence are often considered "active" in the Gibsonian sense of being purposive and information seeking [7, 8]. An important hallmark of active sensing is the modification of the control strategy according to context [9]. Using a task in which rats were trained to run circuits for food, we tested the hypothesis that whisker control, as measured by high-speed videography, changes with contextual variables such as environment familiarity, risk of collision, and availability of visual cues. In novel environments, functionally blind rats moved at slow speeds and performed broad whisker sweeps. With greater familiarity, however, they moved more rapidly, protracted their whiskers further, and showed decreased whisking amplitude. These findings indicate a strategy change from using the vibrissae to explore nearby surfaces to using them primarily for "look ahead." In environments with increased risk of collision, functionally blind animals moved more slowly but protracted their whiskers further. Sighted animals also showed changes in whisker control strategy with increased familiarity, but these changes were different to those of the functionally blind strain. Sighted animals also changed their vibrissal behavior when visual cues were subsequently removed (by being placed in darkness). These contextual influences provide strong evidence of active control and demonstrate that the vibrissal system provides an accessible model of purposive behavior in mammals.

Whisker movements reveal spatial attention: a unified computational model of active sensing control in the rat.

Spatial attention is most often investigated in the visual modality through measurement of eye movements, with primates, including humans, a widely-studied model. Its study in laboratory rodents, such as mice and rats, requires different techniques, owing to the lack of a visual fovea and the particular ethological relevance of orienting movements of the snout and the whiskers in these animals. In recent years, several reliable relationships have been observed between environmental and behavioural variables and movements of the whiskers, but the function of these responses, as well as how they integrate, remains unclear. Here, we propose a unifying abstract model of whisker movement control that has as its key variable the region of space that is the animal's current focus of attention, and demonstrate, using computer-simulated behavioral experiments, that the model is consistent with a broad range of experimental observations. A core hypothesis is that the rat explicitly decodes the location in space of whisker contacts and that this representation is used to regulate whisker drive signals. This proposition stands in contrast to earlier proposals that the modulation of whisker movement during exploration is mediated primarily by reflex loops. We go on to argue that the superior colliculus is a candidate neural substrate for the siting of a head-centred map guiding whisker movement, in analogy to current models of visual attention. The proposed model has the potential to offer a more complete understanding of whisker control as well as to highlight the potential of the rodent and its whiskers as a tool for the study of mammalian attention.

The development of whisker control in rats in relation to locomotion.

Adult rats sweep their large facial whiskers (macrovibrissae) back and forth in a rhythmic pattern known as "whisking". Here we examine how these whisker movements develop in relation to other aspects of exploratory behavior, particularly locomotion. We analyzed 963 high-speed video recordings of neonatal rats, from P1 (Post-natal day 1) to P21, and measured the emergence of whisker control and of head, body, and limb movements. Prior to P11, whisker movements were largely limited to unilateral retractions accompanying head turns. Between P11 and P13 bilateral whisking emerged alongside increased forward locomotion and improved control of the head. Contact-induced modulations of whisking symmetry, synchrony, and whisker spread emerge shortly thereafter but continue to develop until at least P18, coinciding with the emergence of adult-like locomotion patterns such as rearing. Overall, whisking develops alongside increasing locomotor competence indicating that active vibrissal sensing plays an important role in the exploratory behavior of the developing animal.

Neural computation via neural geometry: a place code for inter-whisker timing in the barrel cortex?

The place theory proposed by Jeffress (1948) is still the dominant model of how the brain represents the movement of sensory stimuli between sensory receptors. According to the place theory, delays in signalling between neurons, dependent on the distances between them, compensate for time differences in the stimulation of sensory receptors. Hence the location of neurons, activated by the coincident arrival of multiple signals, reports the stimulus movement velocity. Despite its generality, most evidence for the place theory has been provided by studies of the auditory system of auditory specialists like the barn owl, but in the study of mammalian auditory systems the evidence is inconclusive. We ask to what extent the somatosensory systems of tactile specialists like rats and mice use distance dependent delays between neurons to compute the motion of tactile stimuli between the facial whiskers (or 'vibrissae'). We present a model in which synaptic inputs evoked by whisker deflections arrive at neurons in layer 2/3 (L2/3) somatosensory 'barrel' cortex at different times. The timing of synaptic inputs to each neuron depends on its location relative to sources of input in layer 4 (L4) that represent stimulation of each whisker. Constrained by the geometry and timing of projections from L4 to L2/3, the model can account for a range of experimentally measured responses to two-whisker stimuli. Consistent with that data, responses of model neurons located between the barrels to paired stimulation of two whiskers are greater than the sum of the responses to either whisker input alone. The model predicts that for neurons located closer to either barrel these supralinear responses are tuned for longer inter-whisker stimulation intervals, yielding a topographic map for the inter-whisker deflection interval across the surface of L2/3. This map constitutes a neural place code for the relative timing of sensory stimuli.

Active vibrissal sensing in rodents and marsupials.

In rats, the long facial whiskers (mystacial macrovibrissae) are repetitively and rapidly swept back and forth during exploration in a behaviour known as 'whisking'. In this paper, we summarize previous evidence from rats, and present new data for rat, mouse and the marsupial grey short-tailed opossum (Monodelphis domestica) showing that whisking in all three species is actively controlled both with respect to movement of the animal's body and relative to environmental structure. Using automatic whisker tracking, and Fourier analysis, we first show that the whisking motion of the mystacial vibrissae, in the horizontal plane, can be approximated as a blend of two sinusoids at the fundamental frequency (mean 8.5, 11.3 and 7.3 Hz in rat, mouse and opossum, respectively) and its second harmonic. The oscillation at the second harmonic is particularly strong in mouse (around 22 Hz) consistent with previous reports of fast whisking in that species. In all three species, we found evidence of asymmetric whisking during head turning and following unilateral object contacts consistent with active control of whisker movement. We propose that the presence of active vibrissal touch in both rodents and marsupials suggests that this behavioural capacity emerged at an early stage in the evolution of therian mammals.

Biomimetic vibrissal sensing for robots.

Active vibrissal touch can be used to replace or to supplement sensory systems such as computer vision and, therefore, improve the sensory capacity of mobile robots. This paper describes how arrays of whisker-like touch sensors have been incorporated onto mobile robot platforms taking inspiration from biology for their morphology and control. There were two motivations for this work: first, to build a physical platform on which to model, and therefore test, recent neuroethological hypotheses about vibrissal touch; second, to exploit the control strategies and morphology observed in the biological analogue to maximize the quality and quantity of tactile sensory information derived from the artificial whisker array. We describe the design of a new whiskered robot, Shrewbot, endowed with a biomimetic array of individually controlled whiskers and a neuroethologically inspired whisking pattern generation mechanism. We then present results showing how the morphology of the whisker array shapes the sensory surface surrounding the robot's head, and demonstrate the impact of active touch control on the sensory information that can be acquired by the robot. We show that adopting bio-inspired, low latency motor control of the rhythmic motion of the whiskers in response to contact-induced stimuli usefully constrains the sensory range, while also maximizing the number of whisker contacts. The robot experiments also demonstrate that the sensory consequences of active touch control can be usefully investigated in biomimetic robots.

Modeling the emergence of whisker direction maps in rat barrel cortex.

Based on measuring responses to rat whiskers as they are mechanically stimulated, one recent study suggests that barrel-related areas in layer 2/3 rat primary somatosensory cortex (S1) contain a pinwheel map of whisker motion directions. Because this map is reminiscent of topographic organization for visual direction in primary visual cortex (V1) of higher mammals, we asked whether the S1 pinwheels could be explained by an input-driven developmental process as is often suggested for V1. We developed a computational model to capture how whisker stimuli are conveyed to supragranular S1, and simulate lateral cortical interactions using an established self-organizing algorithm. Inputs to the model each represent the deflection of a subset of 25 whiskers as they are contacted by a moving stimulus object. The subset of deflected whiskers corresponds with the shape of the stimulus, and the deflection direction corresponds with the movement direction of the stimulus. If these two features of the inputs are correlated during the training of the model, a somatotopically aligned map of direction emerges for each whisker in S1. Predictions of the model that are immediately testable include (1) that somatotopic pinwheel maps of whisker direction exist in adult layer 2/3 barrel cortex for every large whisker on the rat's face, even peripheral whiskers; and (2) in the adult, neurons with similar directional tuning are interconnected by a network of horizontal connections, spanning distances of many whisker representations. We also propose specific experiments for testing the predictions of the model by manipulating patterns of whisker inputs experienced during early development. The results suggest that similar intracortical mechanisms guide the development of primate V1 and rat S1.

Technical integration of hippocampus, Basal Ganglia and physical models for spatial navigation.

Computational neuroscience is increasingly moving beyond modeling individual neurons or neural systems to consider the integration of multiple models, often constructed by different research groups. We report on our preliminary technical integration of recent hippocampal formation, basal ganglia and physical environment models, together with visualisation tools, as a case study in the use of Python across the modelling tool-chain. We do not present new modeling results here. The architecture incorporates leaky-integrator and rate-coded neurons, a 3D environment with collision detection and tactile sensors, 3D graphics and 2D plots. We found Python to be a flexible platform, offering a significant reduction in development time, without a corresponding significant increase in execution time. We illustrate this by implementing a part of the model in various alternative languages and coding styles, and comparing their execution times. For very large-scale system integration, communication with other languages and parallel execution may be required, which we demonstrate using the BRAHMS framework's Python bindings.

Active touch sensing in the rat: anticipatory and regulatory control of whisker movements during surface exploration.

Animals actively regulate the position and movement of their sensory systems to boost the quality and quantity of the sensory information they obtain. The rat vibrissal system is recognized to be an important model system in which to investigate such "active sensing" capabilities. The current study used high-speed video analysis to investigate whisker movements in untrained, freely moving rats encountering unexpected, vertical surfaces. A prominent feature of rat vibrissal movement is the repeated posterior-anterior sweep of the whiskers in which the macrovibrissae are seen to move largely in synchrony. Here we show that a second significant component of whisking behavior is the size of the arc, or "spread," between the whiskers. Observed spread is shown to vary over the whisk cycle and to substantially decrease during exploration of an unexpected surface. We further show that the timing of whisker movements is affected by surface contact such that 1) the whiskers rapidly cease forward protraction following an initial, unexpected contact, and may do so even more rapidly following contact with the same surface in the subsequent whisk cycle, and 2) retraction velocity is reduced following this latter contact, leading to longer second-contact durations. This evidence is taken to support two hypotheses: 1) that the relative velocities of different whiskers may be actively controlled by the rat and 2) that control of whisker velocity and timing may serve to increase the number and duration of whisker-surface contacts while ensuring that such contacts are made with a light touch.

Spike-timing in primary sensory neurons: a model of somatosensory transduction in the rat.

In previous work, we constructed a simple electro-mechanical model of transduction in the rat mystacial follicle that was able to replicate primary afferent response profiles to a variety of whisker deflection stimuli. Here, we update that model to fit newly available spike-timing response data, and demonstrate that the new model produces appropriate responses to richer stimuli, including pseudo white noise and natural textures, at a spike-timing level of detail. Additionally, we demonstrate reliable distributed encoding of multi-component oscillatory signals. No modifications were necessary to the mechanical model of the physical components of the follicle-sinus complex, supporting its generality. We conclude that this model, and its continued development, will aid the understanding both of somatosensory systems in general, and of physiological results from higher (e.g. thalamocortical) systems by accurately characterising the signals on which they operate.

Feedback control in active sensing: rat exploratory whisking is modulated by environmental contact.

Rats sweep their facial whiskers back and forth to generate tactile sensory information through contact with environmental structure. The neural processes operating on the signals arising from these whisker contacts are widely studied as a model of sensing in general, even though detailed knowledge of the natural circumstances under which such signals are generated is lacking. We used digital video tracking and wireless recording of mystacial electromyogram signals to assess the effects of whisker-object contact on whisking in freely moving animals exploring simple environments. Our results show that contact leads to reduced protraction (forward whisker motion) on the side of the animal ipsilateral to an obstruction and increased protraction on the contralateral side. Reduced ipsilateral protraction occurs rapidly and in the same whisk cycle as the initial contact. We conclude that whisker movements are actively controlled so as to increase the likelihood of environmental contacts while constraining such interactions to involve a gentle touch. That whisking pattern generation is under strong feedback control has important implications for understanding the nature of the signals reaching upstream neural processes.

Empirically inspired simulated electro-mechanical model of the rat mystacial follicle-sinus complex.

In whiskered animals, activity is evoked in the primary sensory afferent cells (trigeminal nerve) by mechanical stimulation of the whiskers. In some cell populations this activity is correlated well with continuous stimulus parameters such as whisker deflection magnitude, but in others it is observed to represent events such as whisker-stimulator contact or detachment. The transduction process is mediated by the mechanics of the whisker shaft and follicle-sinus complex (FSC), and the mechanics and electro-chemistry of mechanoreceptors within the FSC. An understanding of this transduction process and the nature of the primary neural codes generated is crucial for understanding more central sensory processing in the thalamus and cortex. However, the details of the peripheral processing are currently poorly understood. To overcome this deficiency in our knowledge, we constructed a simulated electro-mechanical model of the whisker-FSC-mechanoreceptor system in the rat and tested it against a variety of data drawn from the literature. The agreement was good enough to suggest that the model captures many of the key features of the peripheral whisker system in the rat.