Spiking Elementary Motion Detector in Neuromorphic Systems.

Apparent motion of the surroundings on an agent's retina can be used to navigate through cluttered environments, avoid collisions with obstacles, or track targets of interest. The pattern of apparent motion of objects, (i.e., the optic flow), contains spatial information about the surrounding environment. For a small, fast-moving agent, as used in search and rescue missions, it is crucial to estimate the distance to close-by objects to avoid collisions quickly. This estimation cannot be done by conventional methods, such as frame-based optic flow estimation, given the size, power, and latency constraints of the necessary hardware. A practical alternative makes use of event-based vision sensors. Contrary to the frame-based approach, they produce so-called events only when there are changes in the visual scene. We propose a novel asynchronous circuit, the spiking elementary motion detector (sEMD), composed of a single silicon neuron and synapse, to detect elementary motion from an event-based vision sensor. The sEMD encodes the time an object's image needs to travel across the retina into a burst of spikes. The number of spikes within the burst is proportional to the speed of events across the retina. A fast but imprecise estimate of the time-to-travel can already be obtained from the first two spikes of a burst and refined by subsequent interspike intervals. The latter encoding scheme is possible due to an adaptive nonlinear synaptic efficacy scaling. We show that the sEMD can be used to compute a collision avoidance direction in the context of robotic navigation in a cluttered outdoor environment and compared the collision avoidance direction to a frame-based algorithm. The proposed computational principle constitutes a generic spiking temporal correlation detector that can be applied to other sensory modalities (e.g., sound localization), and it provides a novel perspective to gating information in spiking neural networks.

Mimicking honeybee eyes with a 280 degrees field of view catadioptric imaging system.

We present a small single camera imaging system that provides a continuous 280 degrees field of view (FOV) inspired by the large FOV of insect eyes. This is achieved by combining a curved reflective surface that is machined into acrylic glass with lenses covering the frontal field that otherwise would have been obstructed by the mirror. Based on the work of Seidl (1982 PhD Thesis Technische Hochschule Darmstadt), we describe an extension of the 'bee eye optics simulation' (BEOS) model by Giger (1996 PhD Thesis Australian National University) to the full FOV which enables us to remap camera images according to the spatial resolution of honeybee eyes. This model is also useful for simulating the visual input of a bee-like agent in a virtual environment. The imaging system in combination with our bee eye model can serve as a tool for assessing the visual world from a bee's perspective which is particularly helpful for experimental setups. It is also well suited for mobile robots, in particular on flying vehicles that need light-weight sensors.

Behavioural state affects motion-sensitive neurones in the fly visual system.

The strength of stimulus-induced responses at the neuronal and the behavioural level often depends on the internal state of an animal. Within pathways processing sensory information and eventually controlling behavioural responses, such gain changes can originate at several sites. Using motion-sensitive lobula plate tangential cells (LPTCs) of blowflies, we address whether and in which way information processing changes for two different states of motor activity. We distinguish between the two states on the basis of haltere movements. Halteres are the evolutionarily transformed hindwings of flies. They oscillate when the animals walk or fly. LPTCs mediate, amongst other behaviours, head optomotor responses. These are either of large or small amplitude depending on the state of motor activity. Here we find that LPTC responses also depend on the motor activity of flies. In particular, LPTC responses are enhanced when halteres oscillate. Nevertheless, the response changes of LPTCs do not account for the corresponding large gain changes of head movements. Moreover, haltere activity itself does not change the activity of LPTCs. Instead, we propose that a central signal associated with motor activity changes the gain of head optomotor responses and the response properties of LPTCs.

Variability of blowfly head optomotor responses.

Behavioural responses of an animal are variable even when the animal experiences the same sensory input several times. This variability can arise from stochastic processes inherent to the nervous system. Also, the internal state of an animal may influence a particular behavioural response. In the present study, we analyse the variability of visually induced head pitch responses of tethered blowflies by high-speed cinematography. We found these optomotor responses to be highly variable in amplitude. Most of the variability can be attributed to two different internal states of the flies with high and low optomotor gain, respectively. Even within a given activity state, there is some variability of head optomotor responses. The amount of this variability differs for the two optomotor gain states. Moreover, these two activity states can be distinguished on a fine timescale and without visual stimulation, on the basis of the occurrence of peculiar head jitter movements. Head jitter goes along with high gain optomotor responses and haltere oscillations. Halteres are evolutionary transformed hindwings that oscillate when blowflies walk or fly. Their main function is to serve as equilibrium organs by detecting Coriolis forces and to mediate gaze stabilisation. However, their basic oscillating activity was also suggested to provide a gain-modulating signal. Our experiments demonstrate that halteres are not necessary for high gain head pitch to occur. Nevertheless, we find the halteres to be responsible for one component of head jitter movements. This component may be the inevitable consequence of their function as equilibrium and gaze-stabilising organs.

Precise timing in fly motion vision is mediated by fast components of combined graded and spike signals.

Firing of an individual neuron is determined by the activity of its presynaptic input ensemble. In this study we analyzed how presynaptic signals with different dynamics interact to control postsynaptic activity. In the blowfly's visual system we simultaneously recorded in vivo from an identified motion-sensitive neuron and from elements of the presynaptic ensemble. The presynaptic cells themselves are mutually electrically coupled and convey both graded and spike signals to their common postsynaptic target. We elicited spikes in the postsynaptic neuron by voltage-clamping one of the presynaptic neurons to various holding potentials and then analyzed the time course of the holding current. Current transients in the clamped presynaptic cell were found to coincide with postsynaptic spikes. The current transients were highly variable in amplitude and occasionally absent during postsynaptic spiking. These characteristics indicate that the current transients in the voltage-clamped neuron result from spikes in electrically coupled co-members of the presynaptic ensemble. Our results suggest that electrical coupling among presynaptic neurons mediates synchronization of spikes within the cell ensemble. Moreover, our findings demonstrate that the graded response component of the presynaptic cells effectively controls the postsynaptic firing rate on a coarse scale while the precise timing of the postsynaptic spikes is a consequence of spikes superimposed on the graded signals of the presynaptic neurons.

Encoding of naturalistic optic flow by a population of blowfly motion-sensitive neurons.

In sensory systems information is encoded by the activity of populations of neurons. To analyze the coding properties of neuronal populations sensory stimuli have usually been used that were much simpler than those encountered in real life. It has been possible only recently to stimulate visual interneurons of the blowfly with naturalistic visual stimuli reconstructed from eye movements measured during free flight. Therefore we now investigate with naturalistic optic flow the coding properties of a small neuronal population of identified visual interneurons in the blowfly, the so-called VS and HS neurons. These neurons are motion sensitive and directionally selective and are assumed to extract information about the animal's self-motion from optic flow. We could show that neuronal responses of VS and HS neurons are mainly shaped by the characteristic dynamical properties of the fly's saccadic flight and gaze strategy. Individual neurons encode information about both the rotational and the translational components of the animal's self-motion. Thus the information carried by individual neurons is ambiguous. The ambiguities can be reduced by considering neuronal population activity. The joint responses of different subpopulations of VS and HS neurons can provide unambiguous information about the three rotational and the three translational components of the animal's self-motion and also, indirectly, about the three-dimensional layout of the environment.

Representation of behaviourally relevant information by blowfly motion-sensitive visual interneurons requires precise compensatory head movements.

Flying blowflies shift their gaze by saccadic turns of body and head, keeping their gaze basically fixed between saccades. For the head, this results in almost pure translational optic flow between saccades, enabling visual interneurons in the fly motion pathway to extract information about translation of the animal and thereby about the spatial layout of the environment. There are noticeable differences between head and body movements during flight. Head saccades are faster and shorter than body saccades, and the head orientation is more stable between saccades than the body orientation. Here, we analyse the functional importance of these differences by probing visual interneurons of the blowfly motion pathway with optic flow based on either head movements or body movements, as recorded accurately with a magnetic search coil technique. We find that the precise head-body coordination is essential for the visual system to separate the translational from the rotational optic flow. If the head were tightly coupled to the body, the resulting optic flow would not contain the behaviourally important information on translation. Since it is difficult to resolve head orientation in many experimental paradigms, even when employing state-of-the-art digital video techniques, we introduce a 'headifying algorithm', which transforms the time-dependent body orientation in free flight into an estimate of head orientation. We show that application of this algorithm leads to an estimated head orientation between saccades that is sufficiently stable to enable recovering information on translation. The algorithm may therefore be of practical use when head orientation is needed but cannot be measured.

Responses of blowfly motion-sensitive neurons to reconstructed optic flow along outdoor flight paths.

The retinal image flow a blowfly experiences in its daily life on the wing is determined by both the structure of the environment and the animal's own movements. To understand the design of visual processing mechanisms, there is thus a need to analyse the performance of neurons under natural operating conditions. To this end, we recorded flight paths of flies outdoors and reconstructed what they had seen, by moving a panoramic camera along exactly the same paths. The reconstructed image sequences were later replayed on a fast, panoramic flight simulator to identified, motion sensitive neurons of the so-called horizontal system (HS) in the lobula plate of the blowfly, which are assumed to extract self-motion parameters from optic flow. We show that under real life conditions HS-cells not only encode information about self-rotation, but are also sensitive to translational optic flow and, thus, indirectly signal information about the depth structure of the environment. These properties do not require an elaboration of the known model of these neurons, because the natural optic flow sequences generate--at least qualitatively--the same depth-related response properties when used as input to a computational HS-cell model and to real neurons.

On the computations analyzing natural optic flow: quantitative model analysis of the blowfly motion vision pathway.

For many animals, including humans, the optic flow generated on the eyes during locomotion is an important source of information about self-motion and the structure of the environment. The blowfly has been used frequently as a model system for experimental analysis of optic flow processing at the microcircuit level. Here, we describe a model of the computational mechanisms implemented by these circuits in the blowfly motion vision pathway. Although this model was originally proposed based on simple experimenter-designed stimuli, we show that it is also capable to quantitatively predict the responses to the complex dynamic stimuli a blowfly encounters in free flight. In particular, the model visual system exploits the active saccadic gaze and flight strategy of blowflies in a similar way, as does its neuronal counterpart. The model circuit extracts information about translation velocity in the intersaccadic intervals and thus, indirectly, about the three-dimensional layout of the environment. By stepwise dissection of the model circuit, we determine which of its components are essential for these remarkable features. When accounting for the responses to complex natural stimuli, the model is much more robust against parameter changes than when explaining the neuronal responses to simple experimenter-defined stimuli. In contrast to conclusions drawn from experiments with simple stimuli, optimization of the parameter set for different segments of natural optic flow stimuli do not indicate pronounced adaptational changes of these parameters during long-lasting stimulation.

Motion adaptation leads to parsimonious encoding of natural optic flow by blowfly motion vision system.

Neurons sensitive to visual motion change their response properties during prolonged motion stimulation. These changes have been interpreted as adaptive and were concluded, for instance, to adjust the sensitivity of the visual motion pathway to velocity changes or to increase the reliability of encoding of motion information. These conclusions are based on experiments with experimenter-designed motion stimuli that differ substantially with respect to their dynamical properties from the optic flow an animal experiences during normal behavior. We analyze for the first time motion adaptation under natural stimulus conditions. The experiments are done on the H1-cell, an identified neuron in the blowfly visual motion pathway that has served in many previous studies as a model system for visual motion computation. We reconstructed optic flow perceived by a blowfly in free flight and used this behaviorally generated optic flow to study motion adaptation. A variety of measures (variability in spike count, response latency, jitter of spike timing) suggests that the coding quality does not improve with prolonged stimulation. However, although the number of spikes decreases considerably during stimulation with natural optic flow, the amount of information that is conveyed stays nearly constant. Thus the information per spike increases, and motion adaptation leads to parsimonious coding without sacrificing the reliability with which behaviorally relevant information is encoded.

Function and coding in the blowfly H1 neuron during naturalistic optic flow.

Naturalistic stimuli, reconstructed from measured eye movements of flying blowflies, were replayed on a panoramic stimulus device. The directional movement-sensitive H1 neuron was recorded from blowflies watching these stimuli. The response of the H1 neuron is dominated by the response to fast saccadic turns into one direction. The response between saccades is mostly inhibited by the front-to-back optic flow caused by the forward translation during flight. To unravel the functional significance of the H1 neuron, we replayed, in addition to the original behaviorally generated stimulus, two targeted stimulus modifications: (1) a stimulus in which flow resulting from translation was removed (this stimulus produced strong intersaccadic responses); and (2) a stimulus in which the saccades were removed by assuming that the head follows the smooth flight trajectory (this stimulus produced alternating zero or nearly saturating spike rates). The responses to the two modified stimuli are strongly different from the response to the original stimulus, showing the importance of translation and saccades for the H1 response to natural optic flow. The response to the original stimulus thus suggests a double function for the H1 neuron, assisting two major classes of movement-sensitive output neurons targeted by H1. First, its strong response to saccades may function as a saccadic suppressor (via one of its target neurons) for cells involved in figure-ground discrimination. Second, its intersaccadic response may increase the signal-to-noise ratio (SNR) of wide-field neurons involved in detecting translational optic flow between saccades, in particular when flying speeds are low or when object distances are large.

Synaptic transfer of dynamic motion information between identified neurons in the visual system of the blowfly.

Synaptic transmission is usually studied in vitro with electrical stimulation replacing the natural input of the system. In contrast, we analyzed in vivo transfer of visual motion information from graded-potential presynaptic to spiking postsynaptic neurons in the fly. Motion in the null direction leads to hyperpolarization of the presynaptic neuron but does not much influence the postsynaptic cell, because its firing rate is already low during rest, giving only little scope for further reductions. In contrast, preferred-direction motion leads to presynaptic depolarizations and increases the postsynaptic spike rate. Signal transfer to the postsynaptic cell is linear and reliable for presynaptic graded membrane potential fluctuations of up to approximately 10 Hz. This frequency range covers the dynamic range of velocities that is encoded with a high gain by visual motion-sensitive neurons. Hence, information about preferred-direction motion is transmitted largely undistorted ensuring a consistent dependency of neuronal signals on stimulus parameters, such as motion velocity. Postsynaptic spikes are often elicited by rapid presynaptic spike-like depolarizations which superimpose the graded membrane potential. Although the timing of most of these spike-like depolarizations is set by noise and not by the motion stimulus, it is preserved at the synapse with millisecond precision.

Visually guided orientation in flies: case studies in computational neuroethology.

To understand the functioning of nervous systems and, in particular, how they control behaviour we must bridge many levels of complexity from molecules, cells and synapses to perception behaviour. Although experimental analysis is a precondition for understanding by nervous systems, it is in no way sufficient. The understanding is aided at all levels of complexity by modelling. Modelling proved to be an inevitable tool to test the experimentally established hypotheses. In this review it will by exemplified by three case studies that the appropriate level of modelling needs to be adjusted to the particular computational problems that are to be solved. (1) Specific features of the highly virtuosic pursuit behaviour of male flies can be understood on the basis of a phenomenological model that relates the visual input to the motor output. (2) The processing of retinal image motion as is experienced by freely moving animals can be understood on the basis of a model consisting of algorithmic components and components which represent a simple equivalent circuit of nerve cells. (3) Behaviourally relevant features of the reliability of encoding of visual motion information can be understood by modelling the transformation of postsynaptic potentials into sequences of spike trains.

FliMax, a novel stimulus device for panoramic and highspeed presentation of behaviourally generated optic flow.

A high-speed panoramic visual stimulation device is introduced which is suitable to analyse visual interneurons during stimulation with rapid image displacements as experienced by fast moving animals. The responses of an identified motion sensitive neuron in the visual system of the blowfly to behaviourally generated image sequences are very complex and hard to predict from the established input circuitry of the neuron. This finding suggests that the computational significance of visual interneurons can only be assessed if they are characterised not only by conventional stimuli as are often used for systems analysis, but also by behaviourally relevant input.

Neural coding with graded membrane potential changes and spikes.

The neural encoding of sensory stimuli is usually investigated for spike responses, although many neurons are known to convey information by graded membrane potential changes. We compare by model simulations how well different dynamical stimuli can be discriminated on the basis of spiking or graded responses. Although a continuously varying membrane potential contains more information than binary spike trains, we find situations where different stimuli can be better discriminated on the basis of spike responses than on the basis of graded responses. Spikes can be superior to graded membrane potential fluctuations if spikes sharpen the temporal structure of neuronal responses by amplifying fast transients of the membrane potential. Such fast membrane potential changes can be induced deterministically by the stimulus or can be due to membrane potential noise that is influenced in its statistical properties by the stimulus. The graded response mode is superior for discrimination between stimuli on a fine time scale.

Outdoor performance of a motion-sensitive neuron in the blowfly.

We studied an identified motion-sensitive neuron of the blowfly under outdoor conditions. The neuron was stimulated by oscillating the fly in a rural environment. We analysed whether the motion-induced neuronal activity is affected by brightness changes ranging between bright sunlight and dusk. In addition, the relationship between spike rate and ambient temperature was determined. The main results are: (1) The mean spike rate elicited by visual motion is largely independent of brightness changes over several orders of magnitude as they occur as a consequence of positional changes of the sun. Even during dusk the neuron responds strongly and directionally selective to motion. (2) The neuronal spike rate is not significantly affected by short-term brightness changes caused by clouds temporarily occluding the sun. (3) In contrast, the neuronal activity is much affected by changes in ambient temperature.

Neuronal processing of behaviourally generated optic flow: experiments and model simulations.

The stimuli traditionally used for analysing visual information processing are much simpler than what an animal sees when moving in its natural environment. Therefore, we analysed in a previous study the performance of an identified neuron in the optomotor system of the fly by using as visual stimuli image sequences that were experienced by the animal while walking in a structured environment. These electrophysiological experiments revealed that the fly visual system computes from behaviourally generated optic flow a rather unambiguous representation of the animal's self-motion. In contrast to conclusions based on simple stimuli, the directions of turns are represented by an interneuron, the HSE cell, quite independent of the spatial layout of the environment and its textural properties when the cell is stimulated with behaviourally generated optic flow. This conclusion is substantiated here by further experimental evidence. Moreover, it is shown that the largely unambiguous responses of the HSE cell to behaviourally generated optic flow can be replicated to a large extent by a network model of the fly's visual motion pathway. These results stress the significance of naturalistic stimuli for analysing what is encoded by neuronal circuits under natural operating conditions.

Transfer of visual motion information via graded synapses operates linearly in the natural activity range.

Synaptic transmission between a graded potential neuron and a spiking neuron was investigated in vivo using sensory stimulation instead of artificial excitation of the presynaptic neuron. During visual motion stimulation, individual presynaptic and postsynaptic neurons in the brain of the fly were electrophysiologically recorded together with concentration changes of presynaptic calcium (Delta[Ca(2+)](pre)). Preferred-direction motion leads to depolarization of the presynaptic neuron. It also produces pronounced increases in [Ca(2+)](pre) and the postsynaptic spike rate. Motion in the opposite direction was associated with hyperpolarization of the presynaptic cell but only a weak reduction in [Ca(2+)](pre) and the postsynaptic spike rate. Apart from this rectification, the relationships between presynaptic depolarizations, Delta[Ca(2+)](pre), and postsynaptic spike rates are, on average, linear over the entire range of activity levels that can be elicited by sensory stimulation. Thus, the inevitably limited range in which the gain of overall synaptic signal transfer is constant appears to be adjusted to sensory input strengths.

Early visual experience and the receptive-field organization of optic flow processing interneurons in the fly motion pathway.

The distribution of local preferred directions and motion sensitivities within the receptive fields of so-called tangential neurons in the fly visual system was previously found to match optic flow fields as induced by certain self-motions. The complex receptive-field organization of the tangential neurons and the recent evidence showing that the orderly development of the fly's peripheral visual system depends on visual experience led us to investigate whether or not early visual input is required to establish the functional receptive-field properties of such tangential neurons. In electrophysiological investigations of two identified tangential neurons, it turned out that dark-hatched flies which were kept in complete darkness for 2 days develop basically the same receptive-field organization as flies which were raised under seasonal light/dark conditions and were free to move in their cages. We did not find any evidence that the development of the sophisticated receptive-field organization of tangential neurons depends on sensory experience. Instead, the input to the tangential neurons seems to be "hardwired" and the specificity of these cells to optic flow induced during self-motions of the animal may have evolved on a phylogenetical time scale.

Membrane potential fluctuations determine the precision of spike timing and synchronous activity: a model study.

It is much debated on what time scale information is encoded by neuronal spike activity. With a phenomenological model that transforms time-dependent membrane potential fluctuations into spike trains, we investigate constraints for the timing of spikes and for synchronous activity of neurons with common input. The model of spike generation has a variable threshold that depends on the time elapsed since the previous action potential and on the preceding membrane potential changes. To ensure that the model operates in a biologically meaningful range, the model was adjusted to fit the responses of a fly visual interneuron to motion stimuli. The dependence of spike timing on the membrane potential dynamics was analyzed. Fast membrane potential fluctuations are needed to trigger spikes with a high temporal precision. Slow fluctuations lead to spike activity with a rate about proportional to the membrane potential. Thus, for a given level of stochastic input, the frequency range of membrane potential fluctuations induced by a stimulus determines whether a neuron can use a rate code or a temporal code. The relationship between the steepness of membrane potential fluctuations and the timing of spikes has also implications for synchronous activity in neurons with common input. Fast membrane potential changes must be shared by the neurons to produce synchronous activity.