Natural image statistics and neural representation.

It has long been assumed that sensory neurons are adapted, through both evolutionary and developmental processes, to the statistical properties of the signals to which they are exposed. Attneave (1954)Barlow (1961) proposed that information theory could provide a link between environmental statistics and neural responses through the concept of coding efficiency. Recent developments in statistical modeling, along with powerful computational tools, have enabled researchers to study more sophisticated statistical models for visual images, to validate these models empirically against large sets of data, and to begin experimentally testing the efficient coding hypothesis for both individual neurons and populations of neurons.

Natural signal statistics and sensory gain control.

We describe a form of nonlinear decomposition that is well-suited for efficient encoding of natural signals. Signals are initially decomposed using a bank of linear filters. Each filter response is then rectified and divided by a weighted sum of rectified responses of neighboring filters. We show that this decomposition, with parameters optimized for the statistics of a generic ensemble of natural images or sounds, provides a good characterization of the nonlinear response properties of typical neurons in primary visual cortex or auditory nerve, respectively. These results suggest that nonlinear response properties of sensory neurons are not an accident of biological implementation, but have an important functional role.

Perceiving visual expansion without optic flow.

When an observer moves forward in the environment, the image on his or her retina expands. The rate of this expansion conveys information about the observer's speed and the time to collision. Psychophysical and physiological studies have provided abundant evidence that these expansionary motions are processed by specialized mechanisms in mammalian visual systems. It is commonly assumed that the rate of expansion is estimated from the divergence of the optic-flow field (the two-dimensional field of local translational velocities). But this rate might also be estimated from changes in the size (or scale) of image features. To determine whether human vision uses such scale-change information, we have synthesized stochastic texture stimuli in which the scale of image elements increases gradually over time, while the optic-flow pattern is random. Here we show, using these stimuli, that observers can estimate expansion rates from scale-change information alone, and that pure scale changes can produce motion after-effects. These two findings suggest that the visual system contains mechanisms that are explicitly sensitive to changes in scale.

Mechanisms of visual motion detection.

Visual motion is processed by neurons in primary visual cortex that are sensitive to spatial orientation and speed. Many models of local velocity computation are based on a second stage that pools the outputs of first-stage neurons selective for different orientations, but the nature of this pooling remains controversial. In a human psychophysical detection experiment, we found near-perfect summation of image energy when it was distributed uniformly across all orientations, but poor summation when it was concentrated in specific orientation bands. The data are consistent with a model that integrates uniformly over all orientations, even when this strategy is sub-optimal.

Image compression via joint statistical characterization in the wavelet domain.

We develop a probability model for natural images, based on empirical observation of their statistics in the wavelet transform domain. Pairs of wavelet coefficients, corresponding to basis functions at adjacent spatial locations, orientations, and scales, are found to be non-Gaussian in both their marginal and joint statistical properties. Specifically, their marginals are heavy-tailed, and although they are typically decorrelated, their magnitudes are highly correlated. We propose a Markov model that explains these dependencies using a linear predictor for magnitude coupled with both multiplicative and additive uncertainties, and show that it accounts for the statistics of a wide variety of images including photographic images, graphical images, and medical images. In order to directly demonstrate the power of the model, we construct an image coder called EPWIC (embedded predictive wavelet image coder), in which subband coefficients are encoded one bitplane at a time using a nonadaptive arithmetic encoder that utilizes conditional probabilities calculated from the model. Bitplanes are ordered using a greedy algorithm that considers the MSE reduction per encoded bit. The decoder uses the statistical model to predict coefficient values based on the bits it has received. Despite the simplicity of the model, the rate-distortion performance of the coder is roughly comparable to the best image coders in the literature.

Range estimation by optical differentiation.

We describe a novel formulation of the range recovery problem based on computation of the differential variation in image intensities with respect to changes in camera position. This method uses a single stationary camera and a pair of calibrated optical masks to measure this differential quantity directly. We also describe a variant based on changes in aperture size. The subsequent computation of the range image involves simple arithmetic operations and is suitable for real-time implementation. We present the theory of this technique and show results from a prototype camera that we have constructed.

A model of neuronal responses in visual area MT.

Electrophysiological studies indicate that neurons in the middle temporal (MT) area of the primate brain are selective for the velocity of visual stimuli. This paper describes a computational model of MT physiology, in which local image velocities are represented via the distribution of MT neuronal responses. The computation is performed in two stages, corresponding to neurons in cortical areas V1 and MT. Each stage computes a weighted linear sum of inputs, followed by rectification and divisive normalization. V1 receptive field weights are designed for orientation and direction selectivity. MT receptive field weights are designed for velocity (both speed and direction) selectivity. The paper includes computational simulations accounting for a wide range of physiological data, and describes experiments that could be used to further test and refine the model.

Local velocity representation: evidence from motion adaptation.

Adaptation to a moving visual pattern induces shifts in the perceived motion of subsequently viewed moving patterns. Explanations of such effects are typically based on adaptation-induced sensitivity changes in spatio-temporal frequency tuned mechanisms (STFMs). An alternative hypothesis is that adaptation occurs in mechanisms that independently encode direction and speed (DSMs). Yet a third possibility is that adaptation occurs in mechanisms that encode 2D pattern velocity (VMs). We performed a series of psychophysical experiments to examine predictions made by each of the three hypotheses. The results indicate that: (1) adaptation-induced shifts are relatively independent of spatial pattern of both adapting and test stimuli; (2) the shift in perceived direction of motion of a plaid stimulus after adaptation to a grating indicates a shift in the motion of the plaid pattern, and not a shift in the motion of the plaid components; and (3) the 2D pattern of shift in perceived velocity radiates away from the adaptation velocity, and is inseparable in speed and direction of motion. Taken together, these results are most consistent with the VM adaptation hypothesis.

Computational models of cortical visual processing.

The visual responses of neurons in the cerebral cortex were first adequately characterized in the 1960s by D. H. Hubel and T. N. Wiesel [(1962) J. Physiol. (London) 160, 106-154; (1968) J. Physiol. (London) 195, 215-243] using qualitative analyses based on simple geometric visual targets. Over the past 30 years, it has become common to consider the properties of these neurons by attempting to make formal descriptions of these transformations they execute on the visual image. Most such models have their roots in linear-systems approaches pioneered in the retina by C. Enroth-Cugell and J. R. Robson [(1966) J. Physiol. (London) 187, 517-552], but it is clear that purely linear models of cortical neurons are inadequate. We present two related models: one designed to account for the responses of simple cells in primary visual cortex (V1) and one designed to account for the responses of pattern direction selective cells in MT (or V5), an extrastriate visual area thought to be involved in the analysis of visual motion. These models share a common structure that operates in the same way on different kinds of input, and instantiate the widely held view that computational strategies are similar throughout the cerebral cortex. Implementations of these models for Macintosh microcomputers are available and can be used to explore the models' properties.

Steerable wedge filters for local orientation analysis.

Steerable filters have been used to analyze local orientation patterns in imagery. Such filters are typically based on directional derivatives, whose symmetry produces orientation responses that are periodic with period pi, independent of image structure. We present a more general set of steerable filters that alleviate this problem.