Perception of 3D surface orientation from skew symmetry.

In this paper, we investigate how symmetry can be used to perceive 3D surface orientation. When a symmetric planar object is viewed from an angle, the projected contour has skew symmetry, which provides partial information about the 3D orientation of the object. For a given skew symmetry, this information can be characterized by a constraint curve of possible slant/tilt combinations that are consistent with a mirror-symmetric interpretation. These constraint curves move around when an object is rotated within a plane, and depend on what we will term the spin of the object: the angle between its axis of symmetry and the direction of tilt. To test the influence of symmetry constraint curves, we presented subjects with stereo images of symmetric objects that varied in spin, and had them perform an orientation-matching task. We found that the judgments showed biases that depended on the spin of the objects. Since other sources of information depend only on slant and tilt, not on spin, the biases imply that skew symmetry contributed to subjects' judgments. In a second experiment, we introduced conflicts between stereo and symmetry cues, and found that the spin-dependent biases can be modulated by selectively changing stereo slant. We propose an explanation of these results involving the optimal integration of stereo and skew symmetry, and present a Bayesian model that can account for the pattern of biases.

Perceiving visual expansion without optic flow.

When an observer moves forward in the environment, the image on his or her retina expands. The rate of this expansion conveys information about the observer's speed and the time to collision. Psychophysical and physiological studies have provided abundant evidence that these expansionary motions are processed by specialized mechanisms in mammalian visual systems. It is commonly assumed that the rate of expansion is estimated from the divergence of the optic-flow field (the two-dimensional field of local translational velocities). But this rate might also be estimated from changes in the size (or scale) of image features. To determine whether human vision uses such scale-change information, we have synthesized stochastic texture stimuli in which the scale of image elements increases gradually over time, while the optic-flow pattern is random. Here we show, using these stimuli, that observers can estimate expansion rates from scale-change information alone, and that pure scale changes can produce motion after-effects. These two findings suggest that the visual system contains mechanisms that are explicitly sensitive to changes in scale.

Development of infants' sensitivity to surface contour information for spatial layout.

The development of sensitivity to a recently discovered static-monocular depth cue to surface shape, surface contours, was investigated. Twenty infants in each of three age groups (5, 5 1/2, and 7 months) viewed a display that creates an illusion, for adult viewers, that what is in fact a frontoparallel cylinder is slanted away in depth, so that one end appears closer than the other. Preferential reaching was recorded in both monocular and binocular conditions. More reaching to the apparently closer end in the monocular than in the binocular condition is evidence of sensitivity. Infants aged 7 months responded to surface contour information, but infants aged 5 and 5 1/2 months did not. In a control study, twenty 5-month-old infants reached consistently for the closer ends of cylinders that were actually rotated in depth. As findings with other static-monocular depth information suggest, infants' sensitivity to surface contour information appears to develop at approximately 6 months.

Contour into texture: information content of surface contours and texture flow.

Both surface contours and texture patterns can provide strong cues to the three-dimensional shape of a surface in space. Many of the most perceptually salient texture patterns have a strong flowlike structure, resulting from the directional nature of the surface textures from which they project. Under the minimal assumption that an oriented surface texture is homogeneous, the texture flow on a developable surface can be shown to follow parallel geodesics of the surface. The geometry of texture flow is therefore equivalent to that of an important class of surface contours: those that project from parallel geodesics of a developable surface. I derive a set of differential equations that support the estimation of surface shape from geodesic surface contours under spherical perspective, for both parallel and nonparallel contours. For perfectly oriented textures, the equations apply directly to the integrated flow lines in a texture image. For weakly oriented textures, perspective projection distorts the projected orientation of flow lines away from the idealized case of pure contours; however, simulations show that for a large class of textures, these distortions will be small and limited largely to extreme surface poses. The geometrical analysis, along with a number of phenomenal demonstrations and psychophysical results, suggests that the human visual system co-opts shape from contour mechanisms to estimate surface shape from texture flow.

Mechanisms of visual motion detection.

Visual motion is processed by neurons in primary visual cortex that are sensitive to spatial orientation and speed. Many models of local velocity computation are based on a second stage that pools the outputs of first-stage neurons selective for different orientations, but the nature of this pooling remains controversial. In a human psychophysical detection experiment, we found near-perfect summation of image energy when it was distributed uniformly across all orientations, but poor summation when it was concentrated in specific orientation bands. The data are consistent with a model that integrates uniformly over all orientations, even when this strategy is sub-optimal.

Dissociating stimulus information from internal representation--a case study in object recognition.

Human object recognition is a function of both internal memory representation(s) and stimulus input information. The role of the latter has been so far largely overlooked, and the nature of the representation is often directly equated with recognition performance. We quantify stimulus information for three classes of objects in order of decreasing object complexity: unconnected balls, balls connected with lines, and balls connected with cylinders. In an object discrimination task, subjects' performance improved with the decreasing object complexity. We show that input information also increases with decreasing object complexity. Therefore, the results could potentially be accounted for either by differences in the object representations learned for each class of objects, or by the increased information about the three-dimensional (3D) structure inherent in images of the less complex objects, or by both. We demonstrate that, when image information is taken into account, by computing efficiencies relative to a set of ideal observers, subjects were more efficient in recognizing the less complex objects. This suggests that differences in subjects' performance for different object classes is at least partly a function of the internal representations learned for the different object classes. We stress that this conclusion cannot be achieved without the quantitative analysis of stimulus input information.

Surface orientation from texture: ideal observers, generic observers and the information content of texture cues.

Perspective views of textured, planar surfaces provide a number of cues about the orientations of the surfaces. These include the information created by perspective scaling of texture elements (scaling), the information created by perspective foreshortening of texels (foreshortening) and, for textures composed of discrete elements, the information created by the effects of both scaling and foreshortening on the relative positions of texels (position). We drive a general form for ideal observers for each of these cues as they appear in images of spatially extended textures, (e.g. those composed of solid 2-D figures). As an application of the formulation, we derive a set of 'generic' observers which we show perform near optimally for images of a broad range of surface textures, without special prior knowledge about the statistics of the textures. Using simulations of ideal observers, we analyze the informational structure of texture cues, including a quantification of lower bounds on reliability for the three different cues, how cue reliability varies with slant angle and how it varies with field of view. We also quantify how strongly the reliability of the foreshortening cue depends on a prior assumption of isotropy. Finally, we extend the analysis to a naturalistic class of textures, showing that the information content of textures particularly suited to psychophysical investigation can be quantified, at least to a first-order approximation. The results provide an important computational foundation for psychophysical work on perceiving surface orientation from texture.

Discrimination of planar surface slant from texture: human and ideal observers compared.

In order to quantify the ability of the human visual system to use texture information to perceive planar surface orientation, I measured subjects' ability to discriminate planar surface slant (angle away from the fronto-parallel) for a variety of different types of textures and in a number of different viewing conditions. I measured the subjects' discrimination performance as a function of surface slant, field of view size and surface texture structure. I compared the subjects' performance with that of ideal observers derived for each of the available texture cues--texel position, scaling and foreshortening. The results can be summarized by four points: (i) subjects' discrimination performance improves dramatically with increasing surface slant, tracking the performance of the ideal observers; (ii) subjects can integrate texture information over a large range of visual angles; (iii) comparisons between human subjects and ideal observers show that the human observers rely to some degree on foreshortening information; and (iv) similar comparisons show that in using foreshortening information, subjects rely to some extent on a prior assumption of isotropy.

Ideal observer perturbation analysis reveals human strategies for inferring surface orientation from texture.

Optical texture patterns contain three quasi-independent cues to planar surface orientation: perspective scaling, projective foreshortening and density. The purpose of this work was to estimate the perceptual weights assigned to these texture cues for discriminating surface orientation and to measure the visual system's reliance on an isotropy assumption in interpreting foreshortening information. A novel analytical technique is introduced which takes advantage of the natural cue perturbations inherent in stochastic texture stimuli to estimate cue weights and measure the influence of an isotropy assumption. Ideal observers were derived which compute the exact information content of the different texture cues in the stimuli used in the experiments and which either did or did not rely on an assumption of surface texture isotropy. Simulations of the ideal observers using the same stimuli shown to subjects in a slant discrimination task provided trial-by-trial estimates of the natural cue perturbations which were inherent in the stimuli. By back-correlating subjects' judgments with the different ideal observer estimates, we were able to estimate both the weights given to each cue by subjects and the strength of subjects' prior assumptions of isotropy. In all of the conditions tested, we found that subjects relied primarily on the foreshortening cue. A small, but significant weight was given to scaling information and no significant weight was given to density information. In conditions in which the surface textures deviated from isotropy by random amounts from stimulus to stimulus, subject judgements correlated well with the estimates of an ideal observer which incorrectly assumed surface texture isotropy. This correlation was not complete, however, suggesting that a soft form of the isotropy constraint was used. Moreover, the correlation was significantly lower for textures containing higher-order information about surface orientation (skew of rectangular texture elements). The results of the analysis clearly implicate texture foreshortening as a primary cue for perceiving surface slant from texture and suggest that the visual system incorporates a strong, though not complete, bias to interpret surface textures as isotropic in its inference of surface slant from texture. They further suggest that local texture skew, when available in an image, contributes significantly to perceptual estimates of surface orientation.

The perception of cast shadows.

When an object casts its shadow on a background surface, the shadow can be informative about the shape of the object, the shape of the background surface and the spatial arrangement of the object relative to the background. Among all these roles, we found that cast shadows were perceptually most relevant for the recovery of spatial arrangement, especially when the shadow is in motion. This finding is intriguing when one considers the ambiguities in the possible ways that shadow motion can be interpreted. We reasoned that the visual system must use a priori constraints to disambiguate the cast shadow motion. One of these constraints is that the light source is stationary. Though simple, the stationary-light-source constraint supports rich, reliable inferences about the qualitative motions of objects in three dimensions.

Geometry of shadows.

Shadows provide a strong source of information about the shapes of surfaces. We analyze the local geometric structure of shadow contours on piecewise smooth surfaces. Particular attention is paid to intrinsic shadows on a surface: that is, shadows created on a surface by the surface's own shape and placement relative to a light source. Intrinsic shadow contours provide useful information about the direction of the light source and the qualitative shape of the underlying surface. We analyze the invariants relating surface shape and light-source direction to the shapes and singularities of intrinsic shadow contours. The results suggest that intrinsic shadows can be used to directly infer illuminant tilt, qualitative global surface structure, and, at intersections with surface creases, the concavity/convexity of a surface. We show that the results obtained for point sources of light generalize in a straightforward way to extended light sources, under the assumption that light sources are convex.

Moving cast shadows induce apparent motion in depth.

Phenomenally strong visual illusions are described in which the motion of an object's cast shadow determines the perceived 3-D trajectory of the object. Simply adjusting the motion of a shadow is sufficient to induce dramatically different apparent trajectories of the object casting the shadow. Psychophysical results obtained with the use of 3-D graphics are reported which show that: (i) the information provided by the motion of an object's shadow overrides other strong sources of information and perceptual biases, such as the assumption of constant object size and a general viewpoint; (ii) the natural constraint of shadow darkness plays a role in the interpretation of a moving image patch as a shadow, but under some conditions even unnatural light shadows can induce apparent motion in depth of an object; (iii) when shadow motion is caused by a moving light source, the visual system incorrectly interprets the shadow motion as consistent with a moving object, rather than a moving light source. The results support the hypothesis that the human visual system incorporates a stationary light-source constraint in the perceptual processing of spatial layout of scenes.

Categorical local-shape perception.

How well do observers perceive the local shape of an object from its shaded image? This problem was addressed by first deriving a potential representation of local solid shape. The descriptor of local shape, called shape characteristic, provides a viewpoint-independent continuum between hyperbolic (saddle-shaped) and elliptic (egg-shaped) points. The ability of human observers to make categorical judgments of local solid shape was then studied. This question was investigated by using a smooth 'croissant', a simple object made of two connected regions of elliptic and hyperbolic points. Observers decided whether the surface was locally elliptic or hyperbolic at various points on the object. The task was natural, and the observers could reliably partition the shaded image of the object into two regions, one elliptic and one hyperbolic. The ability of observers to perform this partition shows that they can, at least implicitly, localize the parabolic curves on a surface. This ability to locate the parabolic curve could in turn be exploited for other purposes, for instance to segment an object into its parts.

Object classification for human and ideal observers.

We describe a novel approach, based on ideal observer analysis, for measuring the ability of human observers to use image information for 3D object perception. We compute the statistical efficiency of subjects relative to an ideal observer for a 3D object classification task. After training to 11 different views of a randomly shaped thick wire object, subjects were asked which of a pair of noisy views of the object best matched the learned object. Efficiency relative to the actual information in the stimuli can be as high as 20%. Increases in object regularity (e.g. symmetry) lead to increases in the efficiency with which novel views of an object could be classified. Furthermore, such increases in regularity also lead to decreases in the effect of viewpoint on classification efficiency. Human statistical efficiencies relative to a 2D ideal observer exceeded 100%, thereby excluding all models which are sub-optimal relative to the 2D ideal.

Perception of surface contours and surface shape: from computation to psychophysics.

Contours projected from surface markings provide information for the perception of surface shape. The nature of this information depends on how the shapes of surface marking are constrained relative to the shapes of the surfaces upon which they lie. A natural constraint is that of figural regularity relative to the shape of an underlying surface. Such a constraint would be expressed in terms of the geodesic curvature of a marking, with markings having zero geodesic curvature (geodesics of a surface) being the prototypic regular figures. I propose a number of forms for a geodesic constraint and present psychophysical evidence from a contour-labeling experiment that the human visual system implicitly incorporates a geodesic constraint in the processing of reflectance contours.

Apparent surface curvature affects lightness perception.

The human visual system has the remarkable capacity to perceive accurately the lightness, or relative reflectance, of surfaces, even though much of the variation in image luminance may be caused by other scene attributes, such as shape and illumination. Most physiological, and computational models of lightness perception invoke early sensory mechanisms that act independently of, or before, the estimation of other scene attributes. In contrast to the modularity of lightness perception assumed in these models are experiments that show that supposedly 'higher-order' percepts of planar surface attributes, such as orientation, depth and transparency, can influence perceived lightness. Here we show that perceived surface curvature can also affect perceived lightness. The results of the earlier experiments indicate that perceiving luminance edges as changes in surface attributes other than reflectance can influence lightness. These results suggest that the interpretation of smooth variations in luminance can also affect lightness percepts.

Human discrimination of fractal images.

In order to transmit information in images efficiently, the visual system should be tuned to the statistical structure of the ensemble of images that it sees. Several authors have suggested that the ensemble of natural images exhibits fractal behavior and, therefore, has a power spectrum that drops off proportionally to 1/f beta (2 less than beta less than 4). In this paper we investigate the question of which value of the exponent beta describes the power spectrum of the ensemble of images to which the visual system is optimally tuned. An experiment in which subjects were asked to discriminate randomly generated noise textures based on their spectral drop-off was used. Whereas the discrimination-threshold function of an ideal observer was flat for different spectral drop-offs, human observers showed a broad peak in sensitivity for 2.8 less than beta less than 3.6. The results are consistent with, but do not provide direct evidence for, the theory that the visual system is tuned to an ensemble of images with Markov statistics.

Estimating illuminant direction and degree of surface relief.

Many algorithms for deriving surface shape from shading require an estimate of the direction of illumination. This paper presents a new estimator for illuminant direction, which also generates an estimate of the degree of surface relief, that is measured by the variance of surface orientation (the partial derivatives of surface depth). Surfaces are considered to be samples of a stochastic process representing depth as a function of position in the image plane. We derive an estimator for illuminant tilt that is based only on some general assumptions about the process. The assumptions are that the process is wide-sense stationary, strictly isotropic; and mean-square differentiable and that the second partial derivatives of surface depth are locally independent of the first partial derivatives. We develop an estimator of illuminant slant and degree of surface relief in two stages. In the first, we develop a general format for an estimator based on the same assumptions that are used for the tilt estimator. The second stage is the actual implementation of the estimator and requires the specification of a functional form for the local probability distribution of surface orientations. This approach contrasts with previous ones, which begin their development with an assumption of a particular distribution for surfaces. The approach has the advantage that it separates the problems of surface modeling and light-source estimation, permiting one to easily implement specific estimators for different surface models. We implement the illuminant slant estimator for surfaces that have a Gaussian distribution of surface orientations and show simulation results. Degraded performance in the presence of self-shadowing is discussed.

Associative learning of scene parameters from images.

An important problem for both biological and machine vision is the construction of scene representations from 2-D image data that are useful for recognition. One problem is that there can be more than one world out there giving rise to the image data at hand. Additional constraints regarding the nature of the environment have to be used to narrow the range of solutions. Although effort has gone into understanding these constraints, relatively little has been done to understand how neurallike learning networks may be used to solve scene-from-image problems. A paradigm is proposed in which stochastic models of scene properties are used to provide samples of image and scene representations. Distributed associative networks are taught, by example, the statistical constraints relating the image to the representation of the scene. This technique is applied to problems in optic flow, shape-from-shading, and stereo.