Multiple uses of visual motion. The case for stability in sensory cortex.

The problem of 'readout' from sensory maps has received considerable attention recently. Specifically, many experiments in different systems have suggested that the routing of sensory signals from cortical maps can be impressively flexible. In this review, we discuss many of the experiments addressing readout of motion signals from the middle temporal area (also known as V5) in the macaque monkey. We focus on two different types of output: perceptual reports (categorical decisions, usually) and motion-guided eye movements. We specifically consider situations in which multiple-motion vectors present in the stimulus are combined, as well as those in which one or more of the vectors in the stimulus is selected for output. The results of these studies suggest that in some situations multiple motions are vector averaged, while in others multiple vectors can be maintained. Interestingly, in most of the experiments producing a single (often average) vector, the output is a movement. However, many perceptual experiments involve the simultaneous processing of multiple-stimulus motions. One prosaic explanation for this pattern of apparently discrepant results is that different downstream structures impose different rules, in parallel, on the output from sensory maps such as the one in the middle temporal area. We also specifically discuss the case of motion opponency, a specific readout rule that has been posited to explain perceptual phenomena such as the waterfall illusion (motion aftereffect). We present evidence from a recent experiment showing that an opponent step must occur downstream from the middle temporal area itself. This observation is consistent with our proposal that significant processing need occur downstream from sensory structures. If a single output is to be used for multiple purposes, often at once, this necessitates a degree of task invariance on the sensory information present even at a relatively high level of cortical processing.

Extrastriate cortex: A signature of perception grows.

Neuronal activity in area MT of the extrastriate visual cortex is correlated with the choices monkeys make on perceptual tasks. New evidence suggests that this correlation is stronger on some tasks than others.

How are moving images segmented?

Visual images are segmented perceptually by a variety of cues, including color and motion. Recent experiments, using perceptual and neurophysiological approaches, have explored the complex interaction between these attributes. A full account will certainly include the effects of directed attention.

Spatial summation in the receptive fields of MT neurons.

Receptive fields (RFs) of cells in the middle temporal area (MT or V5) of monkeys will often encompass multiple objects under normal image viewing. We therefore have studied how multiple moving stimuli interact when presented within and near the RF of single MT cells. We used moving Gabor function stimuli, <1 degrees in spatial extent and approximately 100 msec in duration, presented on a grid of possible locations over the RF of the cell. Responses to these stimuli were typically robust, and their small spatial and temporal extent allowed detailed mapping of RFs and of interactions between stimuli. The responses to pairs of such stimuli were compared against the responses to the same stimuli presented singly. The responses were substantially less than the sum of the responses to the component stimuli and were well described by a power-law summation model with divisive inhibition. Such divisive inhibition is a key component of recently proposed "normalization" models of cortical physiology and is presumed to arise from lateral interconnections within a region. One open question is whether the normalization occurs only once in primary visual cortex or multiple times in different cortical areas. We addressed this question by exploring the spatial extent over which one stimulus would divide the response to another and found effective normalization from stimuli quite far removed from the RF center. This supports models under which normalization occurs both in MT and in earlier stages.

Tuning bandwidths for near-threshold stimuli in area MT.

It is not known whether psychophysical performance depends primarily on small numbers of neurons optimally tuned to specific visual stimuli, or on larger populations of neurons that vary widely in their properties. Tuning bandwidths of single cells can provide important insight into this issue, yet most bandwidth measurements have been made using suprathreshold visual stimuli, whereas psychophysical measurements are frequently obtained near threshold. We therefore examined the directional tuning of cells in the middle temporal area (MT, or V5) using perithreshold, stochastic motion stimuli that we have employed extensively in combined psychophysical and physiological studies. The strength of the motion signal (coherence) in these displays can be varied independently of its direction. For each MT neuron, we characterized the directional bandwidth by fitting Gaussian functions to directional tuning data obtained at each of several motion coherences. Directional bandwidth increased modestly as the coherence of the stimulus was reduced. We then assessed the ability of MT neurons to discriminate opposed directions of motion along six equally spaced axes of motion spanning 180 degrees. A signal detection analysis yielded neurometric functions for each axis of motion, from which neural thresholds could be extracted. Neural thresholds remained surprisingly low as the axis of motion diverged from the neuron's preferred-null axis, forming a plateau of high to medium sensitivity that extended approximately 45 degrees on either side of the preferred-null axis. We conclude that directional tuning remains broad in MT when motion signals are reduced to near-threshold values. Thus directional information is widely distributed in MT, even near the limits of psychophysical performance. These observations support models in which relatively large numbers of signals are pooled to inform psychophysical decisions.

Clustering of response selectivity in the medial superior temporal area of extrastriate cortex in the macaque monkey.

Ever since being described by Mountcastle (Mountcastle, 1957), columnar organization of sensory cortical areas has provided key leverage into understanding the functional organization of neocortex. Columnar or clustered organization of neurons sharing like properties is now known to be widespread, and probably universal in primary sensory areas. Visual cortex in primates consists of a primary area and a large number of secondary areas, which are organized in a manner both hierarchical and parallel (Felleman & Van Essen, 1991; Young, 1993; Young et al., 1995). One major component in the organization of extrastriate visual cortex appears to be the division into dorsal and ventral "streams" of processing (Ungerleider & Mishkin, 1982), each of which is organized hierarchically. Within each, columnar organization exists at early stages, but becomes less clear at higher levels. Columnar organization has been described at the highest level of the ventral stream, inferotemporal cortex (IT, Saleem et al., 1993; Fujita & Fujita, 1996; Tanaka, 1996), but has not been well characterized at the higher levels of the dorsal stream. Hints of such organization are found in the literature (Saito et al., 1986; Lagae et al., 1994), but systematic measurements are needed. In this paper, I report the existence of clustered organization in the medial superior temporal area (MST) of the dorsal stream, which is arguably the highest dominantly visual area on this pathway. I have measured the selectivity of both single- and multiple-unit activity along oblique electrode penetrations through this area to three different kinds of optic flow stimuli, and find that nearby neurons are more similar in their tuning than are more distant ones. This observation documents the existence of some form of clustered organization and supports the importance of this area in the processing of optic flow information.

Electrical microstimulation of cortical area MST biases heading perception in monkeys.

As we move through the environment, the pattern of visual motion on the retina provides rich information about our movement through the scene. Human subjects can use this information, often termed "optic flow", to accurately estimate their direction of self movement (heading) from relatively sparse displays. Physiological observations on the motion-sensitive areas of monkey visual cortex suggest that the medial superior temporal area (MST) is well suited for the analysis of optic flow information. To test whether MST is involved in extracting heading from optic flow, we perturbed its activity in monkeys trained on a heading discrimination task. Electrical microstimulation of MST frequently biased the monkeys' decisions about their heading, and these induced biases were often quite large. This result suggests that MST has a direct role in the perception of heading from optic flow.

A computational analysis of the relationship between neuronal and behavioral responses to visual motion.

We have documented previously a close relationship between neuronal activity in the middle temporal visual area (MT or V5) and behavioral judgments of motion (Newsome et al., 1989; Salzman et al., 1990; Britten et al., 1992; Britten et al., 1996). We have now used numerical simulations to try to understand how neural signals in area MT support psychophysical decisions. We developed a model that pools neuronal responses drawn from our physiological data set and compares average responses in different pools to produce psychophysical decisions. The structure of the model allows us to assess the relationship between "neuronal" input signals and simulated psychophysical performance using the same methods we have applied to real experimental data. We sought to reconcile three experimental observations: psychophysical performance (threshold sensitivity to motion stimuli embedded in noise), a trial-by-trial covariation between the neural response and the monkey's choices, and a modest correlation between pairs of MT neurons in their variable responses to identical visual stimuli. Our results can be most accurately simulated if psychophysical decisions are based on pools of at least 100 weakly correlated sensory neurons. The neurons composing the pools must include a broader range of sensitivities than we encountered in our MT recordings, presumably because of the inclusion of neurons whose optimal stimulus is different from the one being discriminated. Central sources of noise degrade the signal-to-noise ratio of the pooled signal, but this degradation is relatively small compared with the noise typically carried by single cortical neurons. This suggests that our monkeys base near-threshold psychophysical judgments on signals carried by populations of weakly interacting neurons; these populations include many neurons that are not tuned optimally for the particular stimuli being discriminated.

A relationship between behavioral choice and the visual responses of neurons in macaque MT.

We have previously documented the exquisite motion sensitivity of neurons in extrastriate area MT by studying the relationship between their responses and the direction and strength of visual motion signals delivered to their receptive fields. These results suggested that MT neurons might provide the signals supporting behavioral choice in visual discrimination tasks. To approach this question from another direction, we have now studied the relationship between the discharge of MT neurons and behavioral choice, independently of the effects of visual stimulation. We found that trial-to-trial variability in neuronal signals was correlated with the choices the monkey made. Therefore, when a directionally selective neuron in area MT fires more vigorously, the monkey is more likely to make a decision in favor of the preferred direction of the cell. The magnitude of the relationship was modest, on average, but was highly significant across a sample of 299 cells from four monkeys. The relationship was present for all stimuli (including those without a net motion signal), and for all but the weakest responses. The relationship was reduced or eliminated when the demands of the task were changed so that the directional signal carried by the cell was less informative. The relationship was evident within 50 ms of response onset, and persisted throughout the stimulus presentation. On average, neurons that were more sensitive to weak motion signals had a stronger relationship to behavior than those that were less sensitive. These observations are consistent with the idea that neuronal signals in MT are used by the monkey to determine the direction of stimulus motion. The modest relationship between behavioral choice and the discharge of any one neuron, and the prevalence of the relationship across the population, make it likely that signals from many neurons are pooled to form the data on which behavioral choices are based.

Neuronal plasticity that underlies improvement in perceptual performance.

The electrophysiological properties of sensory neurons in the adult cortex are not immutable but can change in response to alterations of sensory input caused by manipulation of afferent pathways in the nervous system or by manipulation of the sensory environment. Such plasticity creates great potential for flexible processing of sensory information, but the actual effects of neuronal plasticity on perceptual performance are poorly understood. The link between neuronal plasticity and performance was explored here by recording the responses of directionally selective neurons in the visual cortex while rhesus monkeys practiced a familiar task involving discrimination of motion direction. Each animal experienced a short-term improvement in perceptual sensitivity during daily experiments; sensitivity increased by an average of 19 percent over a few hundred trials. The increase in perceptual sensitivity was accompanied by a short-term improvement in neuronal sensitivity that mirrored the perceptual effect both in magnitude and in time course, which suggests that improved psychophysical performance can result directly from increased neuronal sensitivity within a sensory pathway.

Responses of neurons in macaque MT to stochastic motion signals.

Dynamic random-dot stimuli have been widely used to explore central mechanisms of motion processing. We have measured the responses of neurons in area MT of the alert monkey while we varied the strength and direction of the motion signal in such displays. The strength of motion is controlled by the proportion of spatiotemporally correlated dots, which we term the correlation of the stimulus. For many MT cells, responses varied approximately linearly with stimulus correlation. When they occurred, nonlinearities were equally likely to be either positively or negatively accelerated. We also explored the relationship between response magnitude and response variance for these cells and found, in general agreement with other investigators, that this relationship conforms to a power law with an exponent slightly greater than 1. The variance of the cells' discharge is little influenced by the trial-to-trial fluctuations inherent in our stochastic display, and is therefore likely to be of neural origin. Linear responses to these stochastic motion stimuli are predicted by simple, low-level motion models incorporating sensors having relatively broad spatial and temporal frequency tuning.

The analysis of visual motion: a comparison of neuronal and psychophysical performance.

We compared the ability of psychophysical observers and single cortical neurons to discriminate weak motion signals in a stochastic visual display. All data were obtained from rhesus monkeys trained to perform a direction discrimination task near psychophysical threshold. The conditions for such a comparison were ideal in that both psychophysical and physiological data were obtained in the same animals, on the same sets of trials, and using the same visual display. In addition, the psychophysical task was tailored in each experiment to the physiological properties of the neuron under study; the visual display was matched to each neuron's preference for size, speed, and direction of motion. Under these conditions, the sensitivity of most MT neurons was very similar to the psychophysical sensitivity of the animal observers. In fact, the responses of single neurons typically provided a satisfactory account of both absolute psychophysical threshold and the shape of the psychometric function relating performance to the strength of the motion signal. Thus, psychophysical decisions in our task are likely to be based upon a relatively small number of neural signals. These signals could be carried by a small number of neurons if the responses of the pooled neurons are statistically independent. Alternatively, the signals may be carried by a much larger pool of neurons if their responses are partially intercorrelated.

Microstimulation in visual area MT: effects on direction discrimination performance.

Physiological and behavioral evidence suggests that the activity of direction selective neurons in visual cortex underlies the perception of moving visual stimuli. We tested this hypothesis by measuring the effects of cortical microstimulation on perceptual judgements of motion direction. To accomplish this, rhesus monkeys were trained to discriminate the direction of motion in a near-threshold, stochastic motion display. For each experiment, we positioned a microelectrode in the middle of a cluster of neurons that shared a common preferred direction of motion. The psychophysical task was then adjusted so that the visual display was presented directly over the neurons' receptive field. The monkeys were required to discriminate between motion shown either in the direction preferred by the neurons or in the opposite direction. On half the trials of an experiment, we applied electrical microstimulation while monkeys viewed the motion display. We hypothesized that enhancing the neurons' discharge rate would introduce a directionally specific signal into the cortex and thereby influence the monkeys' choices on the discrimination task. We compared the monkeys' performance on "stimulated" and "nonstimulated" trials in 139 experiments; all trials within an experiment were presented in random order. Statistically significant effects of microstimulation were obtained in 89 experiments. In 86 of the 89 experiments with significant effects (97%), the monkeys indicated that motion was in the neurons' preferred direction more frequently on stimulated trials than on nonstimulated trials. The data demonstrate a functional link between the activity of direction selective neurons and perceptual judgements of motion direction.

Effects of inferotemporal cortex lesions on form-from-motion discrimination in monkeys.

The inferotemporal cortex of primates plays a prominent role in the learning and retention of visual form discriminations. In this experiment we investigated the role of inferotemporal (IT) cortex in the discrimination of two-dimensional forms defined by motion cues. Six monkeys were trained to a criterion level of performance on two form-from-motion problems. Three of these animals received complete bilateral lesions of IT cortex, while the other three served as unoperated controls. All animals were then retrained to criterion to evaluate the effects of IT lesions on the retention of form-from-motion learning. Compared with the control group, the lesion group was significantly impaired on both problems. Following retention testing, we trained both groups of monkeys on two new form-from-motion problems to investigate the effects of IT lesions on acquisition rates for new learning. The lesion group performed well on the new problems; the learning rates of the operated and control groups were not significantly different. When forms were defined by luminance cues, monkeys with IT lesions, like those in previous studies, were impaired both for retention and for acquisition. These findings indicate that the anterograde effects of IT lesions on learning new form discriminations are less severe for forms defined by motion cues than for forms defined by luminance cues. However, the retrograde effects of IT lesions on retention are severe for forms defined by either cue.

Cortical microstimulation influences perceptual judgements of motion direction.

Neurons in the visual cortex respond selectively to perceptually salient features of the visual scene, such as the direction and speed of moving objects, the orientation of local contours, or the colour or relative depth of a visual pattern. It is commonly assumed that the brain constructs its percept of the visual scene from information encoded in the selective responses of such neurons. We have now tested this hypothesis directly by measuring the effect on psychophysical performance of modifying the firing rates of physiologically characterized neurons. We required rhesus monkeys to report the direction of motion in a visual display while we electrically stimulated clusters of directionally selective neurons in the middle temporal visual area (MT, or V5), an extrastriate area that plays a prominent role in the analysis of visual motion information. Microstimulation biased the animals' judgements towards the direction of motion encoded by the stimulated neurons. This result indicates that physiological properties measured at the neuronal level can be causally related to a specific aspect of perceptual performance.

Neuronal correlates of a perceptual decision.

The relationship between neuronal activity and psychophysical judgement has long been of interest to students of sensory processing. Previous analyses of this problem have compared the performance of human or animal observers in detection or discrimination tasks with the signals carried by individual neurons, but have been hampered because neuronal and perceptual data were not obtained at the same time and under the same conditions. We have now measured the performance of monkeys and of visual cortical neurons while the animals performed a psychophysical task well matched to the properties of the neurons under study. Here we report that the reliability and sensitivity of most neurons on this task equalled or exceeded that of the monkeys. We therefore suggest that under our conditions, psychophysical judgements could be based on the activity of a relatively small number of neurons.