There is a fundamental, unbridgeable gap between DNNs and the visual cortex.

Deep neural networks (DNNs) are not just inadequate models of the visual system but are so different in their structure and functionality that they are not even on the same playing field. DNN units have almost nothing in common with neurons, and, unlike visual neurons, they are often fully connected. At best, DNNs can label inputs, while our object perception is both holistic and detail preserving. A feat that no computational system can achieve.

Anatomical, physiological, and psychophysical data show that the nature of conscious perception is incompatible with the integrated information theory (IIT).

The integrated information theory (IIT) equates levels of consciousness with the amount of information integrated over the elements that constitute a system. Conscious visual perception provides two observations that contradict the IIT. First, objects are accurately perceived when presented for ≪100 ms during which time no neural integration is possible. Second, an object is seen as an integrated whole and, concurrently, all constituent elements are evident. Because integration destroys information about details, IIT cannot account for perceptual detail preservation.

Very small faces are easily discriminated under long and short exposure times.

Acuity measures related to overall face size that can be perceived have not been studied quantitatively. Consequently, experimenters use a wide range of sizes (usually large) without always providing a rationale for their choices. I studied thresholds for face discrimination by presenting both long (500 ms)- and short (17, 33, 50 ms)-duration stimuli. Face width threshold for the long presentation was ~0.2°, and thresholds for the flashed stimuli ranged from ~0.3° for the 17-ms flash to ~0.23° for the 33- and 50-ms flashes. Such thresholds indicate that face stimuli used in physiological or psychophysical experiments are often too large to tap human fine spatial capabilities, and thus interpretations of such experiments should take into account face discrimination acuity. The 0.2° threshold found in this study is incompatible with the prevalent view that faces are represented by a population of specialized "face cells" because those cells do not respond to <1° stimuli and are optimally tuned to >4° faces. Also, the ability to discriminate small, high-spatial frequency flashed face stimuli is inconsistent with models suggesting that fixational drift transforms retinal spatial patterns into a temporal code. It seems therefore that the small image motions occurring during fixation do not disrupt our perception, because all relevant processing is over with before those motions can have significant effects. NEW & NOTEWORTHY Although face perception is central to human behavior, the minimally perceived face size is not known. This study shows that humans can discriminate very small (~0.2°) faces. Furthermore, even when flashed for tens of milliseconds, ~0.25° faces can be discriminated. Such fine acuity should impact modeling of physiological mechanisms of face perception. The ability to discriminate flashed faces where there is almost no eye movement indicates that eye drift is not essential for visibility.

Consciousness weaves our internal view of the outside world.

Low-level consciousness is fundamental to our understanding of the world. Within the conscious field, the constantly changing external visual information is transformed into stable, object-based percepts. Remarkably, holistic objects are perceived while we are cognizant of all of the spatial details comprising the objects and of the relationship between individual elements. This parallel conscious association is unique to the brain. Conscious contributions to motor activity come after our understanding of the world has been established.

The anatomical and physiological properties of the visual cortex argue against cognitive penetration.

We are consciously aware of visual objects together with the minute details that characterize each object. Those details are perceived instantaneously and in parallel. V1 is the only visual area with spatial resolution and topographical exactitude matching perceptual abilities. For cognition to penetrate perception, it needs to affect V1 image representation. That is unlikely because of the detailed parallel V1 organization and the nature of top-down connections, which can influence only large parts of the visual field.

Space reconstruction by primary visual cortex activity: a parallel, non-computational mechanism of object representation.

The current view posits that objects, despite changes in appearance, are uniquely encoded by 'expert' cells. This view is untenable. First, even if cell ensemble responses are invariant and unique, we are consciously aware of all of the objects' details. Second, in addition to detail preservation, data show that the current hypothesis fails to account for uniqueness and invariance. I present an alternative view whereby objects' representation and recognition are based on parallel representation of space by primary visual cortex (V1) responses. Information necessary for invariance and other attributes is handled in series by other cortical areas through integration, interpolation, and hierarchical convergence. The parallel and serial mechanisms combine to enable our flexible space perception. Only in this alternative view is conscious perception consistent with the underlying architecture.

Saccadic amplitudes during combined saccade-vergence movements result from a weighted average of the target's locations in the two retinas.

Recent neurophysiological and behavioral studies have established that the saccadic amplitudes performed during combined saccade-vergence movements are unequal in the two eyes. These studies have not established, however, how the saccadic amplitude of each eye is determined. Our goal here is to fill this lacuna. We use three well-known metric attributes of saccadic movements as constraints and argue that the only quantitative model that obeys these constraints is one where each eye's saccadic amplitude is given by a weighted average of the target's locations in the two retinas. However, this theoretical result does not establish whether the weights in the weighted averaging operation are constant or whether they vary for different targets. To test the simpler of these two possibilities, namely the one of constant weights, we recorded combined saccade-vergence movements performed by human subjects. Our analysis of these movements shows that a constant-weights weighted averaging model provides an excellent description of their saccadic amplitudes. Overall, then, our conclusions are: (1) the two eyes' saccadic amplitudes are determined by weighted averages of the target's locations in the two retinas; (2) for targets within the oculomotor range of natural viewing, which was the range in our experiments, a weighted averaging model that uses constant weights accounts superbly for these saccadic amplitudes. We suggest that the weighted averaging operation that determines saccadic amplitudes is a by-product of a process whose purpose is to yoke the two eyes together. We provide a model explaining how this yoking may be achieved.

Evidence against the facilitation of the vergence command during saccade-vergence interactions.

Combined saccade-vergence movements result when gaze shifts are made to targets that differ both in direction and in depth from the momentary fixation point. Currently, there are two rivaling schemes to explain these eye movements. According to the first, such eye movements are due to a combination of a conjugate saccadic command and a symmetric vergence command; the two commands are not taken to be independent but instead are suggested to interact in a nonlinear manner, which leads to an intra-saccadic facilitation of the vergence command. According to the second scheme, the saccade generator is disconjugate, thus encoding vergence information in the saccadic commands themselves, and the remaining vergence requirement is provided by an asymmetric mechanism. Here, we test the scheme that suggests an intra-saccadic facilitation of the vergence command. We analyze this scheme and show that it has two fundamental properties. The first is that the vergence command is always symmetric, even during the intra-saccadic facilitation. The second is that the facilitated (and symmetric) vergence command sums linearly with the conjugate saccadic command at the final common pathway. Taking these properties together, this scheme predicts that the total magnitude of the saccadic component of combined saccade-vergence movements can be decomposed into a conjugate part and a symmetric part. When we tested this prediction in combined saccade-vergence movements of humans, we found that it was not confirmed. Thus, our results are incompatible with the facilitation of the vergence command hypothesis. Although these results do not directly verify the rivaling hypothesis, which suggests a disconjugate saccade generator, they do provide it with indirect support.

And yet it moves: perceptual illusions and neural mechanisms of pursuit compensation during smooth pursuit eye movements.

Pursuit eye movements are smooth rotations of the eye aimed at tracking moving objects. During pursuit, the visual system "compensates" for the eye movements, and transforms image movements captured by the eye from retinal to extra-retinal coordinates, for world-centered perception and action. When this function is impaired such as in schizophrenia, subjects misattribute retinal movements generated by their own eye movements to external sources. Surprisingly, even in healthy subjects pursuit compensation is incomplete, and results in illusory perception of motion. Neurophysiological, psychophysical and imaging studies elucidated many aspects of the neural substrates of visual processing during pursuit, including where and how in the cortex visual and non-visual signals interact to produce extra-retinal perception of motion. Here we review current understanding of motion processing in the visual cortex during pursuit and its relation to perception, from a broad perspective drawing from electrophysiology, fMRI, psychophysics and computational modeling. We discuss the experimental findings in the context of theories of pursuit compensation, and review some of the open questions in the field.

Physiological differences between neurons in layer 2 and layer 3 of primary visual cortex (V1) of alert macaque monkeys.

The physiological literature does not distinguish between the superficial layers 2 and 3 of the primary visual cortex even though these two layers differ in their cytoarchitecture and anatomical connections. To distinguish layer 2 from layer 3, we have analysed the response characteristics of neurons recorded during microelectrode penetrations perpendicular to the cortical surface. Extracellular responses of single neurons to sweeping bars were recorded while macaque monkeys performed a fixation task. Data were analysed from penetrations where cells could be localized to specific depths in the cortex. Although the most superficial cells (depth, 145-371 microm; presumably layer 2) responded preferentially to particular stimulus orientations, they were less selective than cells encountered immediately beneath them (depth, 386-696 microm; presumably layer 3). Layer 2 cells had smaller spikes, higher levels of ongoing activity, larger receptive field activating regions, and less finely tuned selectivity for stimulus orientation and length than layer 3 cells. Direction selectivity was found only in layer 3. These data suggest that layer 3 is involved in generating and transmitting precise, localized information about image features, while the lesser selectivity of layer 2 cells may participate in top-down influences from higher cortical areas, as well as modulatory influences from subcortical brain regions.

Saccades and drifts differentially modulate neuronal activity in V1: effects of retinal image motion, position, and extraretinal influences.

In natural vision, continuously changing input is generated by fast saccadic eye movements and slow drifts. We analyzed effects of fixational saccades, voluntary saccades, and drifts on the activity of macaque V1 neurons. Effects of fixational saccades and small voluntary saccades were equivalent. In the presence of a near-optimal stimulus, separate populations of neurons fired transient bursts after saccades, sustained discharges during drifts, or both. Strength, time course, and selectivity of activation by fast and slow eye movements were strongly correlated with responses to flashed or to externally moved stimuli. These neuronal properties support complementary functions for post-saccadic bursts and drift responses. Local post-saccadic bursts signal rapid motion or abrupt change of potentially salient stimuli within the receptive field; widespread synchronized bursts signal occurrence of a saccade. Sustained firing during drifts conveys more specific information about location and contrast of small spatial features that contribute to perception of fine detail. In addition to stimulus-driven responses, biphasic extraretinal modulation accompanying saccades was identified in one third of the cells. Brief perisaccadic suppression was followed by stronger and longer-lasting enhancement that could bias perception in favor of saccade targets. These diverse patterns of neuronal activation underlie the dynamic encoding of our visual world.

Direction selectivity in V1 of alert monkeys: evidence for parallel pathways for motion processing.

In primary visual cortex (V1) of macaque monkeys, motion selective cells form three parallel pathways. Two sets of direction selective cells, one in layer 4B, and the other in layer 6, send parallel direct outputs to area MT in the dorsal cortical stream. We show that these two outputs carry different types of spatial information. Direction selective cells in layer 4B have smaller receptive fields than those in layer 6, and layer 4B cells are more selective for orientation. We present evidence for a third direction selective pathway that flows through V1 layers 4Cm (the middle tier of layer 4C) to layer 3. Cells in layer 3 are very selective for orientation, have the smallest receptive fields in V1, and send direct outputs to area V2. Layer 3 neurons are well suited to contribute to detection and recognition of small objects by the ventral cortical stream, as well as to sense subtle motions within objects, such as changes in facial expressions.

Eye position compensation improves estimates of response magnitude and receptive field geometry in alert monkeys.

Studies of visual function in behaving subjects require that stimuli be positioned reliably on the retina in the presence of eye movements. Fixational eye movements scatter stimuli about the retina, inflating estimates of receptive field dimensions, reducing estimates of peak responses, and blurring maps of receptive field subregions. Scleral search coils are frequently used to measure eye position, but their utility for correcting the effects of fixational eye movements on receptive field maps has been questioned. Using eye coils sutured to the sclera and preamplifiers configured to minimize cable artifacts, we reexamined this issue in two rhesus monkeys. During repeated fixation trials, the eye position signal was used to adjust the stimulus position, compensating for eye movements and correcting the stimulus position to place it at the desired location on the retina. Estimates of response magnitudes and receptive field characteristics in V1 and in LGN were obtained in both compensated and uncompensated conditions. Receptive fields were narrower, with steeper borders, and response amplitudes were higher when eye movement compensation was used. In sum, compensating for eye movements facilitated more precise definition of the receptive field. We also monitored horizontal vergence over long sequences of fixation trials and found the variability to be low, as expected for this precise behavior. Our results imply that eye coil signals can be highly accurate and useful for optimizing visual physiology when rigorous precautions are observed.

High response reliability of neurons in primary visual cortex (V1) of alert, trained monkeys.

The reliability of neuronal responses determines the resources needed to represent the external world and constrains the nature of the neural code. Studies of anesthetized animals have indicated that neuronal responses become progressively more variable as information travels from the retina to the cortex. These results have been interpreted to indicate that perception must be based on pooling across relatively large numbers of cells. However, we find that in alert monkeys, responses in primary visual cortex (V1) are as reliable as the inputs from the retina and the thalamus. Moreover, when the effects of fixational eye movements were minimized, response variability (variance/mean - Fano factor, FF) in all V1 layers was low. When presenting optimal stimuli, the median FF was 0.3. High variability, FF approximately 1, was found only near threshold. Our results suggest that in natural vision, suprathreshold perception can be based on small numbers of optimally stimulated cells.

Alteration of the perceived path of a non-pursued target during smooth pursuit: analysis by a neural network model.

During pursuit of a circularly moving target, the perceived movement of a second circularly moving target is altered. The perceived movement of the non-pursued target is different from both its real movement path and its retinal path. In the present paper this phenomenon is studied using a physiologically based neural network model. Simulation results were compared to psychophysical findings in human subjects. Model simulations enabled us to suggest an explanation for this phenomenon in terms of underlying physiological mechanisms and to estimate the contribution of the efferent eye-movement signal to the perceptual process.

Orientation and direction selectivity of neurons in V1 of alert monkeys: functional relationships and laminar distributions.

We studied orientation selectivity in V1 of alert monkeys and its relationship to other physiological parameters and to anatomical organization. Single neurons were stimulated with drifting bars or with sinusoidal gratings while compensating for eye position. Orientation selectivity based on spike counts was quantified by circular variance and by the bandwidth of the orientation tuning curve. The circular variance distribution was bimodal, suggesting groups with low and with high selectivity. Orientation selectivity was clearly correlated with spontaneous activity, classical receptive field (CRF) size and the strength of surround suppression. Laminar distributions of neuronal properties were distinct. Neurons in the output layers 2/3, 4B and 5 had low spontaneous activity, small CRFs and high orientation selectivity, while the input layers had greater diversity. Direction-selective cells were among the neurons most selective for orientation and most had small CRFs. A narrow band of direction- and orientation-selective cells with small CRFs was located in the middle of layer 4C, indicating appearance of very selective cells at an early stage of cortical processing. We suggest that these results reflect interactions between excitatory and inhibitory mechanisms specific to each sublamina. Regions with less inhibition have higher spontaneous activity, larger CRFs and broader orientation tuning. Where inhibition is stronger, spontaneous activity almost disappears, CRFs shrink, and orientation selectivity is high.

Self-organizing neural network model of motion processing in the visual cortex during smooth pursuit.

A physiologically based neural network model was constructed to study cortical motion processing during pursuit eye movements. The model consists of three layers of computational units, simulating information processing by direction selective neurons in the primary visual cortex (V1), motion selective neurons in the middle-temporal area, and pursuit selective neurons in the middle-superior-temporal (MST) area. MST units integrate visual and eye-movement related information, and their connections develop during an unsupervised training process. The resulting MST units represent a transition from retinal to real-world reference frame. By analyzing the model connectivity, mechanisms underlying the functions performed by the network are studied.

Spatial organization of receptive fields of V1 neurons of alert monkeys: comparison with responses to gratings.

We studied the spatial organization of receptive fields and the responses to gratings of neurons in parafoveal V1 of alert monkeys. Activating regions (ARs) of 228 cells were mapped with increment and decrement bars while compensating for fixational eye movements. For cells with two or more ARs, the overlap between ARs responsive to increments (INC) and ARs responsive to decrements (DEC) was characterized by a quantitative overlap index (OI). The distribution of overlap indices was bimodal. The larger group (78% of cells) was composed of complex cells with strongly overlapping ARs (OI >/= 0.5). The smaller group (14%) was composed of simple cells with minimal spatial overlap of ARs (OI 1, the traditional criterion for identifying simple cells. However, unlike simple cells, even those complex cells with high RM could exhibit diverse nonlinear responses when the spatial frequency or window size was changed. Furthermore, the responses of complex cells to counterphase gratings were predominantly nonlinear even harmonics. These results show that RM is not a robust test of linearity. Our results indicate that complex cells are the most frequently encountered neurons in primate V1, and their behavior needs to receive more emphasis in models of visual function.

Efficient Biologically-based Pattern-recognizing Networks.

A biologiclly-motivated classifying neural network which is based on the feature extraction scheme found in the visual cortex is suggested. A special process is proposed for grading and automatically selecting the "best" features for specific recognition tasks. Ranking is based on a feature's calculated discriminating ability, such that a given class is separated from each and every other class by a given amount. The outcome is a net with less computational complexity than other neural nets, yet one which is more biologically plausible.The main motivation for constructing a reduced net is that the complex circuitry of the brain deals with a huge number of patterns, while a machine-based recognition system usually deals with a limited number of patterns. Results show that feature reduction is drastic and that very compact nets, of the order of tens of neurons, can be used to classify patterns, even in a noisy environment. Copyright 1996 Elsevier Science Ltd