Visual scene memory and the guidance of saccadic eye movements.

An unresolved question is how much information can be remembered from visual scenes when they are inspected by saccadic eye movements. Subjects used saccadic eye movements to scan a computer-generated scene, and afterwards, recalled as many objects as they could. Scene memory was quite good: it improved with display duration, it persisted over time long after the display was removed, and it continued to accumulate with additional viewings of the same display (Melcher, D. (2001) The persistance of memory for scenes. Nature 412, 401). The occurrence of saccadic eye movements was important to ensure good recall performance, even though subjects often recalled non-fixated objects. Inter-saccadic intervals increased with display duration, showing an influence of duration on global scanning strategy. The choice of saccadic target was predicted by a Random Selection with Distance Weighting (RSDW) model, in which the target for each saccade is selected at random from all available objects, weighted according to distance from fixation, regardless of which objects had previously been fixated. The results show that the visual memory that was reflected in the recall reports was not utilized for the immediate decision about where to look in the scene. Visual memory can be excellent, but it is not always reflected in oculomotor measures, perhaps because the cost of rapid on-line memory retrieval is too great.

Eye movements during visual search: the costs of choosing the optimal path.

Saccadic eye movements are usually assumed to be directed to locations containing important or useful information, but such assumptions fail to take into account that planning saccades to such locations might be too costly in terms of effort or attention required. To investigate costs of saccadic planning, subjects searched for a target letter that was contained in either one of two clusters located on either side of a central fixation target. A target was present on each trial and was more likely (probability=0.8) to appear in one cluster than the other. Probabilities were disclosed by differences in cluster intensities. The distance between each cluster and central fixation varied (60'-300'). The presentation time was limited (500 ms) to ensure that a successful search would require a wisely chosen saccadic plan. The best chance of finding the target would be to direct the first saccade to the high-probability location, but only one of the six subjects tested followed this strategy consistently. The rest (to varying degrees) preferred to aim the first saccade to the closer location, often followed by an attempted search of the remaining location. Two-location searches were unsuccessful; performance at both locations was poor due to insufficient time. Preferences for such ineffective strategies were surprising. They suggest that saccadic plans were influenced by attempts to minimize the cognitive and attentional load attached to planning and to maximize the number of new foveal views that can be acquired in a limited period of time. These strategies, though disastrous in our task, may be crucial in natural scanning, when many cognitive operations are performed at once, and the risk attached to a few errant glances at unimportant places is small.

The control of saccadic adaptation: implications for the scanning of natural visual scenes.

Accurate scanning of natural scenes depends on: (1) attentional selection of the target; (2) spatial pooling over the attended target to compute the precise landing position; and (3) adaptive modification of saccades to ensure saccadic accuracy. The present experiments studied adaptation. Adaptive modifications were induced by displacing the target during saccades. Adaptation was found to be: (1) similar for a small target point and a large target circle, despite the differences in the spatial pattern of landing position errors for each; (2) unaffected by instructions to look part way to the target, even though such instructions altered landing position error relative to the target; and (3) insensitive to symbolic cues disclosing the direction of the intra-saccadic displacement. Briefly delaying the presentation of the post-saccadic target greatly reduced adaptation. Neither corrective saccades, nor the position errors that trigger corrections, were involved in adaptation because corrective saccades rarely occurred with a large target circle even though the circle produced as much adaptation as the single point. Taken together, the results do not support the traditional notion that post-saccadic retinal position error controls adaptation. We propose that adaptation relies on a comparison of the actual post-saccadic retinal image with the post-saccadic image that would be predicted based on a representation of the planned saccade. Such a comparison: (1) is consistent with our results; (2) may be more effective than retinal position error in controlling adaptation in natural visual scenes containing large targets and backgrounds; and (3) is similar to the motion-based adaptive mechanisms associated with the VOR. Similarity between the adaptive control of saccades and adaptive control of the VOR raises the possibility that the most important role of saccadic adaptation may be the coordination of eye and head movements during shifts of gaze.

Saccadic localization of occluded targets.

Saccadic eye movements are able to localize spatially-extended targets, including patterns of random dots and simple shapes, with a high degree of precision [McGowan, Kowler, Sharma & Chubb (1998). Vision Research, 38, 895-909; Melcher & Kowler (1999). Vision Research, 39, 2929-2946]. This paper investigates the representations of object shape that guide saccades. We studied saccadic localization of partially-occluded triangles (two or three vertices removed) to find out whether saccades have access to a representation of the full shape, despite the missing portions. Targets were configured so that they could be seen either as triangles, which were partially occluded by polygons, or as fragments in front of the same polygons. Subjects tried to saccade to the inferred full triangle and a discrimination paradigm was used to evaluate their success. Occlusion cues were ineffective in that saccades directed to the occluded triangles landed near the center of the visible fragment, even when it was configured as a triangle behind occluders. Removing the occluders and leaving only three segments of the triangle (vertices removed) helped somewhat, but performance never resembled that achieved with either a full triangle or a 3-dot configuration. We conclude that the saccadic system is insensitive to at least some cues that can be used to infer the shape of objects. For occluded targets, the representation used by saccades may be closer to the configuration of the retinal image.

Shapes, surfaces and saccades.

Saccadic localization of spatially extended objects requires the computation of a single saccadic landing position. What representation of the target guides saccades? Saccades were examined for various targets composed of dots to determine whether landing position corresponded to the center-of-gravity (average location) of the dots, the center-of-area of the shape, or the symmetric axis. Targets were composed of dots configured as outline drawings of circles, ellipses, cardioids, wiggly lines, or amorphous blobs. In some cases, dot spacing was varied, extraneous dot clusters were superimposed, or different distributions of dots inside the boundary were added. Quasi-random dot clusters without a well-defined contour were also studied. Instructions were to look at the target as a whole, and keep latency long enough to avoid compromising accuracy. Saccades landed with a high level of precision (S.D.s 7-10% of target eccentricity) near the center-of-area of the target shape, rather than at the center-of-gravity of the dots or on the symmetric axis. Landing position was unaffected by the spacing of dots along the boundary, the addition of dots within the boundary, or the addition of the extraneous dot clusters. When the target was a cluster of quasi-random dots, saccades landed closer to the center-of-area of the implied surface than to the average location of the dots. Overall, the positions of individual dots were important only insofar as the dots affected overall target shape. The results show that a representation of target shape guides saccades, rather than a more primitive representation of individual elements within the attended region.

Illusory shifts in visual direction accompany adaptation of saccadic eye movements.

A central problem in human vision is to explain how the visual world remains stable despite the continual displacements of the retinal image produced by rapid saccadic movements of the eyes. Perceived stability has been attributed to 'efferent-copy' signals, representing the saccadic motor commands, that cancel the effects of saccade-related retinal displacements. Here we show, by means of a perceptual illusion, that traditional cancellation theories cannot explain stability. The perceptual illusion was produced by first inducing adaptive changes in saccadic gain (ratio of saccade size to target eccentricity). Following adaptation, subjects experienced an illusory mislocalization in which widely separated targets flashed before and after saccades appeared to be in the same place. The illusion shows that the perceptual system did not take the adaptive changes into account. Perceptual localization is based on signals representing the size of the initially-intended saccade, not the size of the saccade that is ultimately executed. Signals representing intended saccades initiate a visual comparison process used to maintain perceptual stability across saccades and to generate the oculomotor error signals that ensure saccadic accuracy.

Attentional interference at small spatial separations.

The spatial characteristics of attention were studied by measuring the accuracy with which two target letters could be identified from a circular display of 24 characters. Traditional notions of spatially-limited regions of attentional enhancement predict that performance should be best when the pair of targets fall within the boundaries of a single attentional 'window'. The results were opposite to this expectation: performance was poorest when the targets were close together and improved with increasing target separation. The effects were not due to lateral sensory masking or to sensory transients and were replicated with several different types of attentional cues. Two possible models are proposed to account for the observed effects of target separation. The first model assumes that attending to one location necessarily reduces processing in the local surround. The second model proposes that the poorer performance observed at small target separations results from imprecise targeting when attention is directed to a pair of nearby locations. Both models illustrate spatially-local limits on processing capacity that attention is unable to circumvent. Enhancement at one location is achieved primarily at the expense of the immediate surround. Such spatially-local tradeoffs in processing capacity could have the useful consequence of making the attended target stand out even more against the immediate background.

Saccadic localization of random dot targets.

The targets for saccadic eye movements in natural visual scenes are spatially extended objects, yet saccades land at a single position within them. To characterize the spatial transformation that determines the saccadic goal position within attended objects, we studied saccadic localization of large patterns of random dots. Saccades landed with a high degree of precision near the center-of-gravity of the patterns (average error < 10%; SDs around the center-of-gravity = 7-11% of target eccentricity). Predictions of landing position were improved by using a weighted center-of-gravity, in which the weight assigned to each dot was reduced by the presence of neighboring dots. Weighting based either on the eccentricity of dots or their position relative to the boundary of the pattern had no effect. The results can be accounted for by a spatial transformation in which the "local signs" of an initial array of detectors, weighted by the activity of each, are averaged to yield the saccadic goal. This model can account for accurate and precise saccadic localization of large targets, while preserving sensitivity to local pattern characteristics. Unlike models of recognition, the boundary of the object has the same status as the internal details.

Gaze-shift dynamics in two kinds of sequential looking tasks.

Gaze-shift dynamics of unrestrained seated subjects were examined. The subjects participated in two tasks. In the first task, they tapped sequences of 3-D targets located on a table in front of them. In the second task, they only looked at similar sequences of targets. The purpose of the task (tapping vs only looking) affected the dynamics of gaze-shifts. Gaze and eye-in-head peak velocities were higher and gaze-shift durations were shorter during tapping than during looking-only. We conclude that task variables affect gaze-shift dynamics, altering characteristics of the so-called saccadic "main sequence".

The function of visual search and memory in sequential looking tasks.

UNLABELLED: Eye and head movements were recorded as unrestrained subjects tapped or only looked at nearby targets. Scanning patterns were the same in both tasks: subjects looked at each target before tapping it; visual search had similar speeds and gaze-shift accuracies. Looking, however, took longer and, unlike tapping, benefitted little from practice. Looking speeded up more than tapping when memory load was reduced: memory was more efficient during tapping. CONCLUSION: eye movements made when only looking are different from those made when tapping. Visual search functions as a separate process, incorporated into both tasks: it can be used to improve performance when memory load is heavy.

When push comes to shove: compensation for passive perturbation of the head during natural gaze shifts.

The effects of passive displacements to the head delivered by an abrupt push to the upper body were studied in human subjects during gaze shifts to nearby targets while the head was completely unrestrained. Accurate measurements of gaze were obtained via the Maryland Revolving Field Monitor, used to measure head and eye rotations unconfounded with translations, and by an acoustic ranging system, used to measure head translations. Compensation for head perturbations was quite good, with gaze errors much the same as gaze errors in the absence of the push. Compensation along one or both meridians was achieved by means of the vestibulo-ocular response in many of the gaze shifts. The results suggest an impressive ability to coordinate head and eye movements during natural gaze shifts, carried out by one or more different kinds of compensatory systems that the subject can access at will or according to task demands.

The role of attention in the programming of saccades.

Accurate saccadic programming in natural visual scenes requires a signal designating which of the many potential targets is to be the goal of the saccade. Is this signal controlled by the allocation of perceptual attention, or do saccades have their own independent selective filter? We found evidence for the involvement of perceptual attention, namely: (1) summoning perceptual attention to a target also facilitated saccades; (2) perceptual identification was better at the saccadic goal than elsewhere; and (3) attempts to dissociate the locus of attention from the saccadic goal were unsuccessful, i.e. it was not possible to prepare to look quickly and accurately at one target while at the same time making highly accurate perceptual judgements about targets elsewhere. We also studied the trade-off between saccadic and perceptual performance by means of a novel application of the "attentional operating characteristic" (AOC) to oculomotor performance. This analysis revealed that some attention could be diverted from the saccadic goal with virtually no cost to either saccadic latency or accuracy, showing that there is a ceiling on the attentional demands of saccades. The links we discovered between saccades and attention can be explained by a model in which perceptual attention determines the endpoint of the saccade, while a separate trigger signal initiates the saccade in response to transient changes in the attentional locus. The model will be discussed in the context of current neurophysiological work on saccadic control.

The accuracy and precision of saccades to small and large targets.

Subjects made saccades to point and spatially-extended targets located at a randomly-selected eccentricity (3.8-4.2 deg) under conditions designed to promote best possible accuracy based only on the visual information present in a single trial. Saccadic errors to point targets were small. The average difference between mean saccade size and target eccentricity was about 1% of eccentricity. Precision was excellent (SD = 5-6% of eccentricity), rivaling the precision of relative perceptual localization. This level of performance was maintained for targets up to 3 deg in diameter. Corrective saccades were infrequent and limited almost exclusively to the point targets. We conclude that the saccadic system has access to a precise representation of a central reference position within spatially-extended targets and that, when explicitly required to do so, the saccadic system is capable of demonstrating remarkably accurate and precise performance.

Slow control with eccentric targets: evidence against a position-corrective model.

Does slow control work by keeping the image at a particular retinal location (position-correction), or by keeping the image relatively stable on whatever position it occupies on the retina (velocity-correction)? The best prior evidence that slow control is not position-corrective was that a stable line of sight can be maintained anywhere on or within a small figure. This result, however, does not preclude position-correction with respect to an imagined reference position selected relative to the figure's contour. To test the importance of an imagined reference position, we compared slow control with targets for which selection of a stable reference position is easy (two points arranged symmetrically about the line of sight) and difficult (one or two points located on one side of the line of sight). We found that the stability, velocity and direction of slow control were the same with both kinds of targets. Slow drifts were in idiosyncratic directions, and not toward the eccentric target. Drift speed increased with eccentricity, but drifts did not exceed 10'/sec even at the largest eccentricity tested (4.1 degrees). The independence of slow control from the spatial configuration of the target shows that slow control does not work by bringing either a visible or an imagined reference position to the optimal fixation locus on the retina. The gradual loss of stability with increasing eccentricity is consistent with prior reports of a reduction in the number of motion detectors tuned to low velocities as eccentricity increases. We conclude that motion signals, not position signals, provide the sole sensory input to slow control.

The role of saccades in the perception of texture patterns.

We studied the perception of texture patterns when observers used saccadic eye movements to scan the display and when the line of sight was maintained in the display center without saccades. Saccades improved the discrimination of the size and the shape of a central randomly-shaped polygon for display durations > 1 sec. Saccades were more important with textures that did not readily segregate into target and background regions than with those that did. Directing saccades to the curvature extrema of the central target figure was more useful than directing them elsewhere. Saccades did not enhance texture segregation, but rather improved the discriminability of individual target and background elements by overcoming lateral interference. To the extent that strong lateral interference is inevitable with poorly-segregating textures, our results show that serial inspection is best carried out by sequences of saccades, not by sequences of attention shifts.

Saccadic localization of eccentric forms.

Saccades made to outline drawings of eccentric forms were compared with saccades made to single-point targets. Saccades could be directed to designated locations within eccentric forms nearly as accurately and precisely as they could be directed to single points. Saccades directed to the form as a whole landed at consistent locations near the center of the form. These results show that contour information is sufficient for accurate computation of a saccadic command and that this computation is constrained by the internal coding of the shape. A serial two-stage process, voluntary selection followed by a weighted-averaging process, is proposed for computation of the saccadic command based on information provided by shape.

New directions for oculomotor research.

This paper reviews major trends in the study of the oculomotor system since Westheimer published his doctoral research on this topic 35 years ago. Westheimer introduced the use of linear system analysis for the study of eye movements, an approach used a great deal by others ever since. Westheimer himself abandoned this approach within a decade, in part, because this kind of analysis becomes ambiguous when predictive properties of oculomotor system performance become prominent. We discuss the implications of ignoring the prominence of predictive eye movements and describe recent evidence for their prevelence and power. This leads us to propose that a new approach to the study of oculomotor performance is required. We also discuss the recent trend to apply the "connectionist" (or "neural network") approach in studies of the oculomotor system, and point out that the "symbolic", rather than the "adaptive", nature of predictive eye movements makes successful extension of these models to oculomotor performance unlikely. Our new approach emphasizes the use of natural stimulation in subjects free from bodily restraints. Accurate measurement of eye, head and torso movements under such conditions has become possible recently and data obtained in this manner has led to the discovery of a number of unexpected characteristics of oculomotor system performance. These developments have encouraged us to abandon the modular view of the oculomotor system, popular since Dodge launched the modern era of oculomotor research in 1903, which postulates five, or more, largely independent "subsystems". We suggest that only two subsystems (a fast saccadic and a somewhat slower smooth) are used to fixate and track a central representation of objects located in three-dimensional space. We show that this two-subsystem approach is consistent with current knowledge of oculomotor system neuroanatomy and neurophysiology.

Cognitive expectations, not habits, control anticipatory smooth oculomotor pursuit.

Human smooth pursuit eye movements anticipate the future path of moving targets. Anticipatory pursuit is sometimes attributed to cognitive expectations about future motion and other times to the habitual repetition of previous pursuit responses. Expectations and habits were separated by having subjects smoothly pursue a target moving along a randomly-selected path that was either undisclosed to the subject before each trial or disclosed by means of auditory or visual cues. When the path was undisclosed, the direction of anticipatory smooth eye movements was determined by the direction of target motion in the previous trial. In the presence of cues-the critical condition for separating habits and expectations-effects of previous trials diminished and anticipatory smooth eye movements were primarily determined by the direction of motion the subject was told to expect. These results show a strong contribution of cognitive expectations which overrides persevering smooth oculomotor habits. Smooth pursuit eye movements are driven by a signal that combines the present target motion with the target motion expected to occur several hundred milliseconds into the future. The expected motion is based on a genuine cognitive prediction, not lower-level sensory or motor memories of past events.

The role of location probability in the programming of saccades: implications for "center-of-gravity" tendencies.

Short-latency saccades to targets among nontarget backgrounds are often directed to the center of the entire (target + nontarget) stimulus configuration. This "averaging" or "center-of-gravity" tendency has been attributed to an automatic, reflexive saccadic response to a poorly-resolved visual signal. We investigated the role of high-level processes by varying the probability of the target appearing in one of two locations. Subjects were asked to make a saccade to a target "+" located above-right or above-left of a central fixation point. A nontarget ("x") was in the other location (directional separation = 30 deg). The mean latencies were short (180-230 msec) in accordance with instructions. Mean saccadic direction was shifted to the right by 24-52% of the directional separation of the stimulus pair as the probability of the target appearing on the right increased from 0.2 to 0.8. The difference in saccadic directions as a function of the actual target location was small and independent of probability, showing that probability introduced a bias without affecting the discriminability of the target from the nontarget. The effect of probability was reduced when the discrimination of the target from the nontarget was easier (square vs triangle), and abolished (saccadic accuracy near perfect with the same average latencies) when the target was presented alone. The results show that the direction of short-latency saccades, initiated before the target has been distinguished from a nearby nontarget, is based on the prior history of target locations and expectations about the future location of the target. High-level plans can account for effects of nontargets on saccades. To infer that a reflexive sensorimotor averaging mechanism exists solely on the basis of observed saccadic "centering" tendencies is unwarranted.