Reconstructing in-depth activity for chaotic 3D spatiotemporal excitable media models based on surface data.

Motivated by potential applications in cardiac research, we consider the task of reconstructing the dynamics within a spatiotemporal chaotic 3D excitable medium from partial observations at the surface. Three artificial neural network methods (a spatiotemporal convolutional long-short-term-memory, an autoencoder, and a diffusion model based on the U-Net architecture) are trained to predict the dynamics in deeper layers of a cube from observational data at the surface using data generated by the Barkley model on a 3D domain. The results show that despite the high-dimensional chaotic dynamics of this system, such cross-prediction is possible, but non-trivial and as expected, its quality decreases with increasing prediction depth.

Humans Predict Action using Grammar-like Structures.

Efficient action prediction is of central importance for the fluent workflow between humans and equally so for human-robot interaction. To achieve prediction, actions can be algorithmically encoded by a series of events, where every event corresponds to a change in a (static or dynamic) relation between some of the objects in the scene. These structures are similar to a context-free grammar and, importantly, within this framework the actual objects are irrelevant for prediction, only their relational changes matter. Manipulation actions and others can be uniquely encoded this way. Using a virtual reality setup and testing several different manipulation actions, here we show that humans predict actions in an event-based manner following the sequence of relational changes. Testing this with chained actions, we measure the percentage predictive temporal gain for humans and compare it to action-chains performed by robots showing that the gain is approximately equal. Event-based and, thus, object independent action recognition and prediction may be important for cognitively deducing properties of unknown objects seen in action, helping to address bootstrapping of object knowledge especially in infants.

Convolutional autoencoder and conditional random fields hybrid for predicting spatial-temporal chaos.

We present an approach for data-driven prediction of high-dimensional chaotic time series generated by spatially-extended systems. The algorithm employs a convolutional autoencoder for dimension reduction and feature extraction combined with a probabilistic prediction scheme operating in the feature space, which consists of a conditional random field. The future evolution of the spatially-extended system is predicted using a feedback loop and iterated predictions. The excellent performance of this method is illustrated and evaluated using Lorenz-96 systems and Kuramoto-Sivashinsky equations of different size generating time series of different dimensionality and complexity.

First-order and second-order statistical analysis of 3D and 2D image structure.

In the first part of this article, we analyze the relation between local image structures (i.e., homogeneous, edge-like, corner-like or texture-like structures) and the underlying local 3D structure (represented in terms of continuous surfaces and different kinds of 3D discontinuities) using range data with real-world color images. We find that homogeneous image structures correspond to continuous surfaces, and discontinuities are mainly formed by edge-like or corner-like structures, which we discuss regarding potential computer vision applications and existing assumptions about the 3D world. In the second part, we utilize the measurements developed in the first part to investigate how the depth at homogeneous image structures is related to the depth of neighbor edges. For this, we first extract the local 3D structure of regularly sampled points, and then, analyze the coplanarity relation between these local 3D structures. We show that the likelihood to find a certain depth at a homogeneous image patch depends on the distance between the image patch and a neighbor edge. We find that this dependence is higher when there is a second neighbor edge which is coplanar with the first neighbor edge. These results allow deriving statistically based prediction models for depth interpolation on homogeneous image structures.

Synaptic modifications depend on synapse location and activity: a biophysical model of STDP.

In spike-timing-dependent plasticity (STDP) the synapses are potentiated or depressed depending on the temporal order and temporal difference of the pre- and post-synaptic signals. We present a biophysical model of STDP which assumes that not only the timing, but also the shapes of these signals influence the synaptic modifications. The model is based on a Hebbian learning rule which correlates the NMDA synaptic conductance with the post-synaptic signal at synaptic location as the pre- and post-synaptic quantities. As compared to a previous paper [Saudargiene, A., Porr, B., Worgotter, F., 2004. How the shape of pre- and post-synaptic signals can influence stdp: a biophysical model. Neural Comp.], here we show that this rule reproduces the generic STDP weight change curve by using real neuronal input signals and combinations of more than two (pre- and post-synaptic) spikes. We demonstrate that the shape of the STDP curve strongly depends on the shape of the depolarising membrane potentials, which induces learning. As these potentials vary at different locations of the dendritic tree, model predicts that synaptic changes are location dependent. The model is extended to account for the patterns of more than two spikes of the pre- and post-synaptic cells. The results show that STDP weight change curve is also activity dependent.

Biologically motivated space-variant filtering for robust optic flow processing.

We describe and test a biologically motivated space-variant filtering method for decreasing the noise in optic flow fields. Our filter model adopts certain properties of a particular motion-sensitive area of the brain (area MT), which averages the incoming motion signals over receptive fields, the sizes of which increase with the distance from the center of the projection. We use heading estimation from optic flow as a criterion to evaluate the improvement of the filtered flow field. The tests are conducted on flow fields calculated with a standard flow algorithm from image sequences. We use two different sets of image sequences. The first set is recorded by a camera which is installed in a moving car. The second set is derived from a database containing three dimensional data and reflectance information from natural scenes. The latter set guarantees full control of the camera motion and ground truth about the flow field and the heading. We test the space-variant filtering method by comparing heading estimation results between space-variant filtered flow, flow filtered by averaging over domains of the visual field with constant size (constant filtering) and raw unfiltered flow. Because of noise and the aperture problem the heading estimates obtained from the raw flows are often unreliable. Estimated heading differs widely for different sub-sampled calculations. In contrast, the results obtained from the filtered flows are much less variable and therefore more consistent. Furthermore, we find a significant improvement of the results obtained from the space-variant filtered flow compared to the constant filtered flow. We suggest extensions to the space-variant filtering procedure that take other properties of motion representation in area MT into account.

Local image structures and optic flow estimation.

Different kinds of local image structures (such as homogeneous, edge-like and junction-like patches) can be distinguished by the intrinsic dimensionality of the local signals. Intrinsic dimensionality makes use of variance from a point and a line in spectral representation of the signal in order to classify it as homogeneous, edge-like or junction-like. The concept of intrinsic dimensionality has been mostly exercised using discrete formulations; however, recent work has introduced a continuous definition. The current study analyzes the distribution of local patches in natural images according to this continuous understanding of intrinsic dimensionality. This distribution reveals specific patterns than can be also associated to local image structures established in computer vision and which can be related to orientation and optic flow features. In particular, we link quantitative and qualitative properties of optic-flow error estimates to these patterns. In this way, we also introduce a new tool for better analysis of optic flow algorithms.

Neural field model of receptive field restructuring in primary visual cortex.

Receptive fields (RF) in the visual cortex can change their size depending on the state of the individual. This reflects a changing visual resolution according to different demands on information processing during drowsiness. So far, however, the possible mechanisms that underlie these size changes have not been tested rigorously. Only qualitatively has it been suggested that state-dependent lateral geniculate nucleus (LGN) firing patterns (burst versus tonic firing) are mainly responsible for the observed cortical receptive field restructuring. Here, we employ a neural field approach to describe the changes of cortical RF properties analytically. Expressions to describe the spatiotemporal receptive fields are given for pure feedforward networks. The model predicts that visual latencies increase nonlinearly with the distance of the stimulus location from the RF center. RF restructuring effects are faithfully reproduced. Despite the changing RF sizes, the model demonstrates that the width of the spatial membrane potential profile (as measured by the variance sigma of a gaussian) remains constant in cortex. In contrast, it is shown for recurrent networks that both the RF width and the width of the membrane potential profile generically depend on time and can even increase if lateral cortical excitatory connections extend further than fibers from LGN to cortex. In order to differentiate between a feedforward and a recurrent mechanism causing the experimental RF changes, we fitted the data to the analytically derived point-spread functions. Results of the fits provide estimates for model parameters consistent with the literature data and support the hypothesis that the observed RF sharpening is indeed mainly driven by input from LGN, not by recurrent intracortical connections.

Investigations on emergent spatio-temporal neural response characteristics in a small network model.

Although the functional role synchronous oscillations may play has been investigated in depth, the underlying processes and spatio-temporal aspects that establish the synchrony are still not thoroughly understood. Experimental studies suggest the existence of two types of activity: stimulus locked and stimulus induced. While stimulus-locked oscillations are systematically dependent on the stimulus (they are phase-locked to stimulus transients), stimulus-induced activity patterns (occurring in the gamma-frequency range) show no such phase-locking. We propose a unifying approach which employs very generic connection structures. The above-mentioned two types of activity patterns with different temporal properties are observed as an emergent feature of the network structure. Our model demonstrates that stimulus-locked and stimulus-induced activity patterns are two distinct states of the same system. A transition from one state to the other is observed, and the influence of structural network parameters (e.g. connections strengths) on this behavior and its dependence on stimulus situations is investigated.

Cluster update algorithm and recognition

We present a fast and robust cluster update algorithm that is especially efficient in implementing the task of image segmentation using the method of superparamagnetic clustering. We apply it to a Potts model with spin interactions that are are defined by gray-scale differences within the image. Motivated by biological systems, we introduce the concept of neural inhibition to the Potts model realization of the segmentation problem. Including the inhibition term in the Hamiltonian results in enhanced contrast and thereby significantly improves segmentation quality. As a second benefit we can, after equilibration, directly identify the image segments as the clusters formed by the clustering algorithm. To construct a subsequent spin configuration the algorithm performs the standard steps of (i) forming clusters and of (ii) updating the spins in a cluster simultaneously. As opposed to standard algorithms, however, we share the interaction energy between the two steps. Thus, the update probabilities are not independent of the interaction energies. As a consequence, we observe an acceleration of the relaxation by a factor of 10 compared to the Swendson and Wang [Phys. Rev. Lett. 58, 86 (1987)] procedure.

Context, state and the receptive fields of striatal cortex cells.

Visual cortical cells are commonly characterized by their receptive-field structure. Originally, a visual receptive field was defined in a purely spatial way as that retinal area from which a change in spiking response of the regarded cell could be elicited by visual stimulation. The first attempts to understand receptive-field structure were based entirely on the anatomical connectivity of the primary visual pathway. More recently, however, it has been discovered that the spatial and temporal context in which a stimulus is presented to a cell can strongly influence its receptive field, and this in turn is dependent on the state of arousal and attention. Accordingly, new concepts recognize that cortical receptive fields are highly dynamic entities embracing more than the sum of the full spatial and temporal response properties of a cell.

The control of low-level information flow in the visual system.

Visual information processing needs to be error free and efficient. Our visual system tries to achieve the first goal by accommodating a wide variety of visual algorithms for the extraction of the relevant features in the scene, while at the same time the second goal is addressed by controlling the amount of visual information flow in the network employing selective attention. Attentional or pre-attentional mechanisms are found throughout many visual areas and these processes may start as early as in the visual thalamus (lateral geniculate nucleus, LGN). In this review we pay particular attention to experimental and theoretical findings which indicate that even low-level structures, such as LGN and V1, can play a major role in the flow-control of visual information.

Correlated variations in EEG pattern and visual responsiveness of cat lateral geniculate relay cells.

1. Simultaneous recordings of the EEG and the visual activity of cat dorsal lateral geniculate nucleus (dLGN) relay cells were analysed for covariance. Sliding time-window analyses were performed in parallel for the EEG power spectrum and single unit visual activity. The EEG power ratio (EEG-PR) of low (1-8 Hz) to high (20-40 Hz) frequencies was chosen to achieve a quantitative measure of the EEG which could be compared with the spike rate of a dLGN unit at any time. A high EEG-PR value indicates a synchronized EEG dominated by low frequencies (delta waves and sleep spindles), a low value indicates a less synchronized EEG. 2. In the anaesthetized animal, two different underlying patterns of activity in the EEG-PR were found: slow gradual changes (slow gradations) and oscillatory changes. In many cases both were accompanied by correlated variations in dLGN spike rate, either for overall activity or for burst firing. 3. The slow gradations appear for long time periods of up to 200 s and, in most cases (76.3 %), show a negative correlation between EEG-PR and overall spike rate, but predominantly a positive correlation for burst firing (85.1 %). 4. The oscillatory changes, which have not previously been reported, appear as temporally well-coupled variations in EEG-PR and spike rate with a stable cycle length within the range 4-10 s. In about 77 % of correlated changes the temporal delay between the change in EEG-PR and that of the spike rate was less than +/- 1.0 s. 5. During simultaneous recordings from two dLGN cells the variations in spike rate tend to show the same sign of correlation with respect to the EEG pattern. This relationship is more pronounced with the slow gradations than with the oscillatory changes. 6. Slow gradations in the spectral composition of the EEG may indicate global transitions between different stages within the sleep-wake cycle, reflecting the well-known influences of the brainstem arousal system. The oscillations in the spectral composition of the EEG are accompanied by gradual variations in thalamic transmission mode and are more likely to be due to involvement of a local feedback system via the thalamo-cortico-thalamic loop. The difference between the effects on overall and burst firing activity supports the notion that phasic (burst firing) and tonic visual responses may play distinctive roles in information processing, which are functionally related to the animal's behavioural state.

The dynamic spatio-temporal behavior of visual responses in thalamus and cortex.

Due to eye and object movements the visual world changes on a rather fast time scale and the neuronal network of the primary visual pathway has to immediately react to these changes. Accordingly the neuronal activity patterns in the visual thalamus and cortex show a pronounced dynamic behavior which reenters the circuitry such that the actual cell responses are also guided by the activation history of the network. Thus, spatial and temporal aspects of visual receptive fields change not only by means of the actual visual stimulation hut also as a consequence of the state of the network. In this short review we summarize the different aspects which can influence the temporal firing patterns of cells in the visual thalamus (lateral geniculate nucleus, LGN) mainly by demonstrating how their inter-spike interval distributions will change. We then show that these firing patterns are able to change the spatial shape of receptive fields in the visual cortex (see Fig. 12 for a summary diagram). Finally, by means of a biophysical model, we will argue that the observed changes could serve to adjust the temporal and spatial resolution within the primary visual pathway to the different demands for information processing in an attentive as compared to a non-attentive state.

An ASIC-chip for stereoscopic depth analysis in video-real-time based on visual cortical cell behavior.

In a stereoscopic system both eyes or cameras have a slightly different view. As a consequence small variations between the projected images exist ("disparities") which are spatially evaluated in order to retrieve depth information. We will show that two related algorithmic versions can be designed which recover disparity. Both approaches are based on the comparison of filter outputs from filtering the left and the right image. The difference of the phase components between left and right filter responses encodes the disparity. One approach uses regular Gabor filters and computes the spatial phase differences in a conventional way as described already in 1988 by Sanger. Novel to this approach, however, is that we formulate it in a way which is fully compatible with neural operations in the visual cortex. The second approach uses the apparently paradoxical similarity between the analysis of visual disparities and the determination of the azimuth of a sound source. Animals determine the direction of the sound from the temporal delay between the left and right ear signals. Similarly, in our second approach we transpose the spatially defined problem of disparity analysis into the temporal domain and utilize two resonators implemented in the form of causal (electronic) filters to determine the disparity as local temporal phase differences between the left and right filter responses. This approach permits video real-time analysis of stereo image sequences (see movies at http://www.neurop.ruhr-uni-bochum.de/Real- Time-Stereo) and a FPGA-based PC-board has been developed which performs stereo-analysis at full PAL resolution in video real-time. An ASIC chip will be available in March 2000.

State-dependent receptive-field restructuring in the visual cortex.

To extract important information from the environment on a useful timescale, the visual system must be able to adapt rapidly to constantly changing scenes. This requires dynamic control of visual resolution, possibly at the level of the responses of single neurons. Individual cells in the visual cortex respond to light stimuli on particular locations (receptive fields) on the retina, and the structure of these receptive fields can change in different contexts. Here we show experimentally that the shape of receptive fields in the primary visual cortex of anaesthetized cats undergoes significant modifications, which are correlated with the general state of the brain as assessed by electroencephalography: receptive fields are wider during synchronized states and smaller during non-synchronized states. We also show that cortical receptive fields shrink over time when stimulated with flashing light spots. Finally, by using a network model we account for the changing size of the cortical receptive fields by dynamically rescaling the levels of excitation and inhibition in the visual thalamus and cortex. The observed dynamic changes in the sizes of the cortical receptive field could be a reflection of a process that adapts the spatial resolution within the primary visual pathway to different states of excitability.

A possible basic cortical microcircuit called "cascaded inhibition." Results from cortical network models and recording experiments from striate simple cells.

The robust behavior, the degree of response linearity, and the aspect of contrast gain control in visual cortical simple cells are (amongst others) the result of the interplay between excitatory and inhibitory afferent and intracortical connections. The goal of this study was to suggest a simple intracortical connection pattern, which could also play a role in other cortical substructures, in order to generically obtain these desired effects within large physiological parameter ranges. To this end we explored the degree of linearity of spatial summation in visual simple cells experimentally and in different models based on half-wave rectifying cells ("push-pull models"). Visual cortical push-pull connection schemes originated from antagonistic motor-control models. Thus, this model class is widely applicable but normally requires a rather specific design. On the other hand we showed that a more generic version of a push-pull model, the so-called cascaded inhibitory intracortical connection scheme, which we implemented in a biologically realistic simulation, naturally explains much of the experimental data. We investigated the influence of the afferent and intracortical connection structure on the measured linearity of spatial summation in simple cells. The analysis made use of the relative modulation measure, which is easy to apply but is limited to moving sinusoidal grating stimuli. We introduced two basic push-pull models, where the order of threshold nonlinearity and linear summation is reversed. Very little difference is observed with the relative modulation measure for these models. Alterative models, like half-wave squaring models, were also briefly discussed. Of all model parameters, the ratio of excitation to inhibition in the simple cell exerts the most crucial influence on the relative modulation. Linearity deteriorates as soon as excitatory and inhibitory inputs are imbalanced and the relative modulation drops. This prediction was tested experimentally by extracellular recordings from cat area 17 simple cells and we found that about 62% showed a significant deviation from linear behavior. The problem that individual basic push-pull models are hard to distinguish experimentally led us to suggest a different solution. In order to generically account for the observed behavior (e.g., imbalance of excitation versus inhibition), we suggested a rather generic version of a push-pull model where it no longer mattered about (the hard-to-distinguish) fine differences in connectivity. Thus, we introduced a new class of biophysically realistic models ("cascaded inhibition"). This model class requires very little connection specificity and is therefore highly robust against parameter variations. Up to 25 cells are connected to each target cell. Thereby a highly interconnected network is generated, which also leads to disinhibition at some parts of an individual receptive field. We showed that the performance of these models simulates the degree of linearity and its variability in recal simple cells with comparatively high accuracy. This behavior can be explained by the self-regulating properties of a cascaded inhibitory connection scheme by which the balance between excitation and inhibition at a given cell is improved by the joint network effects. The virtues and the generic design of this connection pattern, therefore, allow to speculate that it is used also in other parts of the cortex.

Employing the zeta-transform to optimize the calculation of the synaptic conductance of NMDA and other synaptic channels in network simulations.

Calculation of the total conductance change induced by multiple synapses at a given membrane compartment remains one of the most time-consuming processes in biophysically realistic neural network simulations. Here we show that this calculation can be achieved in a highly efficient way even for multiply converging synapses with different delays by means of the zeta-transform. Using the example of an NMDA synapse, we show that every update of the total conductance is achieved by an iterative process requiring at most three recent multiplications, which together need only the history values from the two most recent iterations. A major advantage is that this small computational load is independent of the number of synapses simulated. A benchmark comparison to other techniques demonstrates superior performance of the zeta-transform. Nonvoltage-dependent synaptic channels can be treated similarly (Olshausen, 1990; Brettle & Niebur, 1994), and the technique can also be generalized to other synaptic channels.

A Fast and Robust Cluster Update Algorithm for Image Segmentation in Spin-Lattice Models Without Annealing. Visual Latencies Revisited.

Image segmentation in spin-lattice models relies on the fast and reliable assignment of correct labels to those groups of spins that represent the same object. Commonly used local spin-update algorithms are slow because in each iteration only a single spin is flipped and a careful annealing schedule has to be designed in order to avoid local minima and correctly label larger areas. Updating of complete spin clusters is more efficient, but often clusters that should represent different objects will be conjoined. In this study, we propose a cluster update algorithm that, similar to most local update algorithms, calculates an energy function and determines the probability for flipping a whole cluster of spins by the energy gain calculated for a neighborhood of the regarded cluster. The novel algorithm, called energy-based cluster update (ECU algorithm), is compared to its predecessors. A convergence proof is derived, and it is shown that the algorithm outperforms local update algorithms by far in speed and reliability. At the same time it is more robust and noise tolerant than other versions of cluster update algorithms, making annealing completely unnecessary. The reduction in computational effort achieved this way allows us to segment real images in about 1-5 sec on a regular workstation. The ECU-algorithm can recover fine details of the images, and it is to a large degree robust with respect to luminance-gradients across objects. In a final step, we introduce luminance dependent visual latencies (Opara and Worgotter, 1996; Worgotter, Opara, Funke, and Eysel, 1996) into the spin-lattice model. This step guarantees that only spins representing pixels with similar luminance become activated at the same time. The energy function is then computed only for the interaction of the regarded cluster with the currently active spins. This latency mechanism improves the quality of the image segmentation by another 40%. The results shown are based on the evaluation of gray-level differences. It is important to realize that all algorithmic components can be transferred easily to arbitrary image features, like disparity, texture, and motion.

How to "hear" visual disparities: real-time stereoscopic spatial depth analysis using temporal resonance.

In a stereoscopic system, both eyes or cameras have a slightly different view. As a consequence, small variations between the projected images exist ('disparities') which are spatially evaluated in order to retrieve depth information (Sanger 1988; Fleet et al. 1991). A strong similarity exists between the analysis of visual disparities and the determination of the azimuth of a sound source (Wagner and Frost 1993). The direction of the sound is thereby determined from the temporal delay between the left and right ear signals (Konishi and Sullivan 1986). Similarly, here we transpose the spatially defined problem of disparity analysis into the temporal domain and utilize two resonators implemented in the form of causal (electronic) filters to determine the disparity as local temporal phase differences between the left and right filter responses. This approach permits real-time analysis and can be solved analytically for a step function contrast change, which is an important case in all real-world applications. The proposed theoretical framework for spatial depth retrieval directly utilizes a temporal algorithm borrowed from auditory signal analysis. Thus, the suggested similarity between the visual and the auditory system in the brain (Wagner and Frost 1993) finds its analogy here at the algorithmical level. We will compare the results from the temporal resonance algorithm with those obtained from several other techniques like cross-correlation or spatial phase-based disparity estimation showing that the novel algorithm achieves performances similar to the 'classical' approaches using much lower computational resources.