Spatially distributed computation in cortical circuits.

The traditional view of neural computation in the cerebral cortex holds that sensory neurons are specialized, i.e., selective for certain dimensions of sensory stimuli. This view was challenged by evidence of contextual interactions between stimulus dimensions in which a neuron's response to one dimension strongly depends on other dimensions. Here, we use methods of mathematical modeling, psychophysics, and electrophysiology to address shortcomings of the traditional view. Using a model of a generic cortical circuit, we begin with the simple demonstration that cortical responses are always distributed among neurons, forming characteristic waveforms, which we call neural waves. When stimulated by patterned stimuli, circuit responses arise by interference of neural waves. Results of this process depend on interaction between stimulus dimensions. Comparison of modeled responses with responses of biological vision makes it clear that the framework of neural wave interference provides a useful alternative to the standard concept of neural computation.

A perceptual scaling approach to eyewitness identification.

Eyewitness misidentification accounts for 70% of verified erroneous convictions. To address this alarming phenomenon, research has focused on factors that influence likelihood of correct identification, such as the manner in which a lineup is conducted. Traditional lineups rely on overt eyewitness responses that confound two covert factors: strength of recognition memory and the criterion for deciding what memory strength is sufficient for identification. Here we describe a lineup that permits estimation of memory strength independent of decision criterion. Our procedure employs powerful techniques developed in studies of perception and memory: perceptual scaling and signal detection analysis. Using these tools, we scale memory strengths elicited by lineup faces, and quantify performance of a binary classifier tasked with distinguishing perpetrator from innocent suspect. This approach reveals structure of memory inaccessible using traditional lineups and renders accurate identifications uninfluenced by decision bias. The approach furthermore yields a quantitative index of individual eyewitness performance.

Mechanisms of Spatiotemporal Selectivity in Cortical Area MT.

Cortical sensory neurons are characterized by selectivity to stimulation. This selectivity was originally viewed as a part of the fundamental "receptive field" characteristic of neurons. This view was later challenged by evidence that receptive fields are modulated by stimuli outside of the classical receptive field. Here, we show that even this modified view of selectivity needs revision. We measured spatial frequency selectivity of neurons in cortical area MT of alert monkeys and found that their selectivity strongly depends on luminance contrast, shifting to higher spatial frequencies as contrast increases. The changes of preferred spatial frequency are large at low temporal frequency, and they decrease monotonically as temporal frequency increases. That is, even interactions among basic stimulus dimensions of luminance contrast, spatial frequency, and temporal frequency strongly influence neuronal selectivity. This dynamic nature of neuronal selectivity is inconsistent with the notion of stimulus preference as a stable characteristic of cortical neurons.

Paradoxical perception of object identity in visual motion.

In the course of perceptual organization, incomplete optical stimulation can evoke the experience of complete objects with distinct perceptual identities. According to a well-known principle of perceptual organization, stimulus parts separated by shorter spatial distances are more likely to appear as parts of the same perceptual identity. Whereas this principle of proximity has been confirmed in many studies of perceptual grouping in static displays, we show that it does not generalize to perception of object identity in dynamic displays, where the parts are separated by spatial and temporal distances. We use ambiguous displays which contain multiple moving parts and which can be perceived two ways: as two large objects that gradually change their size or as multiple smaller objects that rotate independent of one another. Grouping over long and short distances corresponds to the perception of the respectively large and small objects. We find that grouping over long distances is often preferred to grouping over short distances, against predictions of the proximity principle. Even though these effects are observed at high luminance contrast, we show that they are consistent with results obtained at the threshold of luminance contrast, in agreement with predictions of a theory of efficient motion measurement. This is evidence that the perception of object identity can be explained by a computational principle of neural economy rather than by the empirical principle of proximity.

Adaptive Optimization of Visual Sensitivity.

Sensory systems adapt to environmental change. It has been argued that adaptation should have the effect of optimizing sensitivity to the new environment. Here we consider a framework in which this premise is made concrete using an economic normative theory of visual motion perception. In this framework, visual systems adapt to the environment by reallocating their limited neural resources. The allocation is optimal when uncertainties about different aspects of stimulation are balanced. This theory makes predictions about visual sensitivity as a function of environmental statistics. Adaptive optimization of the visual system should be manifested as a change in sensitivity for an observer and for the underlying motion-sensitive neurons. We review evidence supporting these predictions and examine effects of adaptation on the neuronal representation of visual motion.

Intermittent regime of brain activity at the early, bias-guided stage of perceptual learning.

Perceptual learning improves visual performance. Among the plausible mechanisms of learning, reduction of perceptual bias has been studied the least. Perceptual bias may compensate for lack of stimulus information, but excessive reliance on bias diminishes visual discriminability. We investigated the time course of bias in a perceptual grouping task and studied the associated cortical dynamics in spontaneous and evoked EEG. Participants reported the perceived orientation of dot groupings in ambiguous dot lattices. Performance improved over a 1-hr period as indicated by the proportion of trials in which participants preferred dot groupings favored by dot proximity. The proximity-based responses were compromised by perceptual bias: Vertical groupings were sometimes preferred to horizontal ones, independent of dot proximity. In the evoked EEG activity, greater amplitude of the N1 component for horizontal than vertical responses indicated that the bias was most prominent in conditions of reduced visual discriminability. The prominence of bias decreased in the course of the experiment. Although the bias was still prominent, prestimulus activity was characterized by an intermittent regime of alternating modes of low and high alpha power. Responses were more biased in the former mode, indicating that perceptual bias was deployed actively to compensate for stimulus uncertainty. Thus, early stages of perceptual learning were characterized by episodes of greater reliance on prior visual preferences, alternating with episodes of receptivity to stimulus information. In the course of learning, the former episodes disappeared, and biases reappeared only infrequently.

Prospective Optimization with Limited Resources.

The future is uncertain because some forthcoming events are unpredictable and also because our ability to foresee the myriad consequences of our own actions is limited. Here we studied how humans select actions under such extrinsic and intrinsic uncertainty, in view of an exponentially expanding number of prospects on a branching multivalued visual stimulus. A triangular grid of disks of different sizes scrolled down a touchscreen at a variable speed. The larger disks represented larger rewards. The task was to maximize the cumulative reward by touching one disk at a time in a rapid sequence, forming an upward path across the grid, while every step along the path constrained the part of the grid accessible in the future. This task captured some of the complexity of natural behavior in the risky and dynamic world, where ongoing decisions alter the landscape of future rewards. By comparing human behavior with behavior of ideal actors, we identified the strategies used by humans in terms of how far into the future they looked (their "depth of computation") and how often they attempted to incorporate new information about the future rewards (their "recalculation period"). We found that, for a given task difficulty, humans traded off their depth of computation for the recalculation period. The form of this tradeoff was consistent with a complete, brute-force exploration of all possible paths up to a resource-limited finite depth. A step-by-step analysis of the human behavior revealed that participants took into account very fine distinctions between the future rewards and that they abstained from some simple heuristics in assessment of the alternative paths, such as seeking only the largest disks or avoiding the smaller disks. The participants preferred to reduce their depth of computation or increase the recalculation period rather than sacrifice the precision of computation.

Prospective Optimization.

Human performance approaches that of an ideal observer and optimal actor in some perceptual and motor tasks. These optimal abilities depend on the capacity of the cerebral cortex to store an immense amount of information and to flexibly make rapid decisions. However, behavior only approaches these limits after a long period of learning while the cerebral cortex interacts with the basal ganglia, an ancient part of the vertebrate brain that is responsible for learning sequences of actions directed toward achieving goals. Progress has been made in understanding the algorithms used by the brain during reinforcement learning, which is an online approximation of dynamic programming. Humans also make plans that depend on past experience by simulating different scenarios, which is called prospective optimization. The same brain structures in the cortex and basal ganglia that are active online during optimal behavior are also active offline during prospective optimization. The emergence of general principles and algorithms for goal-directed behavior has consequences for the development of autonomous devices in engineering applications.

EEG gamma band oscillations differentiate the planning of spatially directed movements of the arm versus eye: multivariate empirical mode decomposition analysis.

The neural dynamics underlying the coordination of spatially-directed limb and eye movements in humans is not well understood. Part of the difficulty has been a lack of signal processing tools suitable for the analysis of nonstationary electroencephalographic (EEG) signals. Here, we use multivariate empirical mode decomposition (MEMD), a data-driven approach that does not employ predefined basis functions. High-density EEG, and arm and eye movements were synchronously recorded in 10 subjects performing time-constrained reaching and/or eye movements. Subjects were allowed to move both the hand and the eyes, only the hand, or only the eyes following a 500-700 ms delay interval where the hand and gaze remained on a central fixation cross. An additional condition involved a nonspatially-directed "lift" movement of the hand. The neural activity during a 500 ms delay interval was decomposed into intrinsic mode functions (IMFs) using MEMD. Classification analysis revealed that gamma band (30 Hz) IMFs produced more classifiable features differentiating the EEG according to the different upcoming movements. A benchmark test using conventional algorithms demonstrated that MEMD was the best algorithm for extracting oscillatory bands from EEG, yielding the best classification of the different movement conditions. The gamma rhythm decomposed using MEMD showed a higher correlation with the eventual movement accuracy than any other band rhythm and than any other algorithm.

Dopamine function and the efficiency of human movement.

To sustain successful behavior in dynamic environments, active organisms must be able to learn from the consequences of their actions and predict action outcomes. One of the most important discoveries in systems neuroscience over the last 15 years has been about the key role of the neurotransmitter dopamine in mediating such active behavior. Dopamine cell firing was found to encode differences between the expected and obtained outcomes of actions. Although activity of dopamine cells does not specify movements themselves, a recent study in humans has suggested that tonic levels of dopamine in the dorsal striatum may in part enable normal movement by encoding sensitivity to the energy cost of a movement, providing an implicit "motor motivational" signal for movement. We investigated the motivational hypothesis of dopamine by studying motor performance of patients with Parkinson disease who have marked dopamine depletion in the dorsal striatum and compared their performance with that of elderly healthy adults. All participants performed rapid sequential movements to visual targets associated with different risk and different energy costs, countered or assisted by gravity. In conditions of low energy cost, patients performed surprisingly well, similar to prescriptions of an ideal planner and healthy participants. As energy costs increased, however, performance of patients with Parkinson disease dropped markedly below the prescriptions for action by an ideal planner and below performance of healthy elderly participants. The results indicate that the ability for efficient planning depends on the energy cost of action and that the effect of energy cost on action is mediated by dopamine.

Sensory optimization by stochastic tuning.

Individually, visual neurons are each selective for several aspects of stimulation, such as stimulus location, frequency content, and speed. Collectively, the neurons implement the visual system's preferential sensitivity to some stimuli over others, manifested in behavioral sensitivity functions. We ask how the individual neurons are coordinated to optimize visual sensitivity. We model synaptic plasticity in a generic neural circuit and find that stochastic changes in strengths of synaptic connections entail fluctuations in parameters of neural receptive fields. The fluctuations correlate with uncertainty of sensory measurement in individual neurons: The higher the uncertainty the larger the amplitude of fluctuation. We show that this simple relationship is sufficient for the stochastic fluctuations to steer sensitivities of neurons toward a characteristic distribution, from which follows a sensitivity function observed in human psychophysics and which is predicted by a theory of optimal allocation of receptive fields. The optimal allocation arises in our simulations without supervision or feedback about system performance and independently of coupling between neurons, making the system highly adaptive and sensitive to prevailing stimulation.

Sensory adaptation as optimal resource allocation.

Visual adaptation is expected to improve visual performance in the new environment. This expectation has been contradicted by evidence that adaptation sometimes decreases sensitivity for the adapting stimuli, and sometimes it changes sensitivity for stimuli very different from the adapting ones. We hypothesize that this pattern of results can be explained by a process that optimizes sensitivity for many stimuli, rather than changing sensitivity only for those stimuli whose statistics have changed. To test this hypothesis, we measured visual sensitivity across a broad range of spatiotemporal modulations of luminance, while varying the distribution of stimulus speeds. The manipulation of stimulus statistics caused a large-scale reorganization of visual sensitivity, forming the orderly pattern of sensitivity gains and losses. This pattern is predicted by a theory of distribution of receptive field characteristics in the visual system.

Traveling waves and trial averaging: the nature of single-trial and averaged brain responses in large-scale cortical signals.

Analyzing single trial brain activity remains a challenging problem in the neurosciences. We gain purchase on this problem by focusing on globally synchronous fields in within-trial evoked brain activity, rather than on localized peaks in the trial-averaged evoked response (ER). We analyzed data from three measurement modalities, each with different spatial resolutions: magnetoencephalogram (MEG), electroencephalogram (EEG) and electrocorticogram (ECoG). We first characterized the ER in terms of summation of phase and amplitude components over trials. Both contributed to the ER, as expected, but the ER topography was dominated by the phase component. This means the observed topography of cross-trial phase will not necessarily reflect the phase topography within trials. To assess the organization of within-trial phase, traveling wave (TW) components were quantified by computing the phase gradient. TWs were intermittent but ubiquitous in the within-trial evoked brain activity. At most task-relevant times and frequencies, the within-trial phase topography was described better by a TW than by the trial-average of phase. The trial-average of the TW components also reproduced the topography of the ER; we suggest that the ER topography arises, in large part, as an average over TW behaviors. These findings were consistent across the three measurement modalities. We conclude that, while phase is critical to understanding the topography of event-related activity, the preliminary step of collating cortical signals across trials can obscure the TW components in brain activity and lead to an underestimation of the coherent motion of cortical fields.

A century of Gestalt psychology in visual perception: II. Conceptual and theoretical foundations.

Our first review article (Wagemans et al., 2012) on the occasion of the centennial anniversary of Gestalt psychology focused on perceptual grouping and figure-ground organization. It concluded that further progress requires a reconsideration of the conceptual and theoretical foundations of the Gestalt approach, which is provided here. In particular, we review contemporary formulations of holism within an information-processing framework, allowing for operational definitions (e.g., integral dimensions, emergent features, configural superiority, global precedence, primacy of holistic/configural properties) and a refined understanding of its psychological implications (e.g., at the level of attention, perception, and decision). We also review 4 lines of theoretical progress regarding the law of Prägnanz-the brain's tendency of being attracted towards states corresponding to the simplest possible organization, given the available stimulation. The first considers the brain as a complex adaptive system and explains how self-organization solves the conundrum of trading between robustness and flexibility of perceptual states. The second specifies the economy principle in terms of optimization of neural resources, showing that elementary sensors working independently to minimize uncertainty can respond optimally at the system level. The third considers how Gestalt percepts (e.g., groups, objects) are optimal given the available stimulation, with optimality specified in Bayesian terms. Fourth, structural information theory explains how a Gestaltist visual system that focuses on internal coding efficiency yields external veridicality as a side effect. To answer the fundamental question of why things look as they do, a further synthesis of these complementary perspectives is required.

Spatially invariant computations in stereoscopic vision.

PERCEPTION OF STEREOSCOPIC DEPTH REQUIRES THAT VISUAL SYSTEMS SOLVE A CORRESPONDENCE PROBLEM: find parts of the left-eye view of the visual scene that correspond to parts of the right-eye view. The standard model of binocular matching implies that similarity of left and right images is computed by inter-ocular correlation. But the left and right images of the same object are normally distorted relative to one another by the binocular projection, in particular when slanted surfaces are viewed from close distance. Correlation often fails to detect correct correspondences between such image parts. We investigate a measure of inter-ocular similarity that takes advantage of spatially invariant computations similar to the computations performed by complex cells in biological visual systems. This measure tolerates distortions of corresponding image parts and yields excellent performance over a much larger range of surface slants than the standard model. The results suggest that, rather than serving as disparity detectors, multiple binocular complex cells take part in the computation of inter-ocular similarity, and that visual systems are likely to postpone commitment to particular binocular disparities until later stages in the visual process.

Perception of time in articulated visual events.

Perceived duration of a sensory event often exceeds its actual duration. This phenomenon is called time dilation. The distortion may occur because sensory systems are optimized for perception within their respective modalities and not for perception of time. We investigated how the dilation of visual events depends on the duration and content of events. Observers compared the durations of two successive visual stimuli while the luminance of one of the stimuli was modulated at different temporal frequencies. Time dilation correlated with the frequency of modulation and the duration of the stimulus: the faster the modulation and the longer the stimulus duration, the larger the dilation. Notably, time dilation was also accompanied by a decreased sensitivity to stimulus duration. We show that these results are consistent with the notion that stimulus duration is estimated using measurement intervals of the lengths that depend on stimulus frequency content. Estimation of temporal frequency content is more precise using longer measurement intervals, whereas estimation of temporal location is more precise using shorter ones. As a result, visual perception will benefit from using longer intervals when the stimulus is modulated so that its frequency content is measured more precisely. A side effect of using longer temporal intervals is a larger uncertainty about the timing of stimulus offset (temporal location), ensuing time dilation and the reduction of sensitivity to duration. Our findings support the view that time dilation follows from basic principles of measurement and from the notion that visual systems are optimized for visual perception rather than for perception of time.

Duration of coherence intervals in electrical brain activity in perceptual organization.

We investigated the relationship between visual experience and temporal intervals of synchronized brain activity. Using high-density scalp electroencephalography, we examined how synchronized activity depends on visual stimulus information and on individual observer sensitivity. In a perceptual grouping task, we varied the ambiguity of visual stimuli and estimated observer sensitivity to this variation. We found that durations of synchronized activity in the beta frequency band were associated with both stimulus ambiguity and sensitivity: the lower the stimulus ambiguity and the higher individual observer sensitivity the longer were the episodes of synchronized activity. Durations of synchronized activity intervals followed an extreme value distribution, indicating that they were limited by the slowest mechanism among the multiple neural mechanisms engaged in the perceptual task. Because the degree of stimulus ambiguity is (inversely) related to the amount of stimulus information, the durations of synchronous episodes reflect the amount of stimulus information processed in the task. We therefore interpreted our results as evidence that the alternating episodes of desynchronized and synchronized electrical brain activity reflect, respectively, the processing of information within local regions and the transfer of information across regions.