A failure to learn object shape geometry: Implications for convolutional neural networks as plausible models of biological vision.

Here we examine the plausibility of deep convolutional neural networks (CNNs) as a theoretical framework for understanding biological vision in the context of image classification. Recent work on object recognition in human vision has shown that both global, and local, shape information is computed, and integrated, early during perceptual processing. Our goal was to compare the similarity in how object shape information is processed by CNNs and human observers. We tested the hypothesis that, unlike the human system, CNNs do not compute representations of global and local object geometry during image classification. To do so, we trained and tested six CNNs (AlexNet, VGG-11, VGG-16, ResNet-18, ResNet-50, GoogLeNet), and human observers, to discriminate geometrically possible and impossible objects. The ability to complete this task requires computation of a representational structure of shape that encodes both global and local object geometry because the detection of impossibility derives from an incongruity between well-formed local feature conjunctions and their integration into a geometrically well-formed 3D global shape. Unlike human observers, none of the tested CNNs could reliably discriminate between possible and impossible objects. Detailed analyses using gradient-weighted class activation mapping (GradCam) of CNN image feature processing showed that network classification performance was not constrained by object geometry. In contrast, if classification could be made based solely on local feature information in line drawings the CNNs were highly accurate. We argue that these findings reflect fundamental differences between CNNs and human vision in terms of underlying image processing structure. Notably, unlike human vision, CNNs do not compute representations of object geometry. The results challenge the plausibility of CNNs as a framework for understanding image classification in biological vision systems.

Early sensitivity of evoked potentials to surface and volumetric structure during the visual perception of three-dimensional object shape.

This study used event-related potentials (ERPs) to elucidate how the human visual system processes three-dimensional (3-D) object shape structure. In particular, we examined whether the perceptual mechanisms that support the analysis of 3-D shape are differentially sensitive to higher order surface and volumetric part structure. Observers performed a whole-part novel object matching task in which part stimuli comprised sub-regions of closed edge contour, surfaces or volumetric parts. Behavioural response latency data showed an advantage in matching surfaces and volumetric parts to whole objects over contours, but no difference between surfaces and volumes. ERPs were analysed using a convergence of approaches based on stimulus dependent amplitude modulations of evoked potentials, topographic segmentation, and spatial frequency oscillations. The results showed early differential perceptual processing of contours, surfaces, and volumetric part stimuli. This was first reliably observed over occipitoparietal electrodes during the N1 (140-200 ms) with a mean peak latency of 170 ms, and continued on subsequent P2 (220-260 ms) and N2 (260-320 ms) components. The differential sensitivity in perceptual processing during the N1 was accompanied by distinct microstate patterns that distinguished among contours, surfaces and volumes, and predominant theta band activity around 4-7 Hz over right occipitoparietal and orbitofrontal sites. These results provide the first evidence of early differential perceptual processing of higher order surface and volumetric shape structure within the first 200 ms of stimulus processing. The findings challenge theoretical models of object recognition that do not attribute functional significance to surface and volumetric object structure during visual perception.

A surface-based code contributes to visual shape perception.

Considerable uncertainty remains regarding the types of features human vision uses for shape representation. Visual-search experiments are reported which assessed the hypothesis of a surface-based (i.e., edge-bounded polygons) code for shape representation in human vision. The results indicate slower search rates and/or longer response times when the target shape shares its constituent surfaces with distractors (conjunction condition) than when the target surfaces are unique in the display (nonconjunction condition). This demonstration is made using test conditions that strictly control any potential artifact pertaining to target-distractor similarity. The surface-based code suggested by this surface-conjunction effect is strictly 2-D, since the effect occurs even when the surfaces are shared between the target and distractors in the 2-D image but not in their 3-D instantiation. Congruently, this latter finding is unaltered by manipulations of the richness of the depth information offered by the stimuli. It is proposed that human vision uses a 2-D surface-based code for shape representation which, considering other key findings in the field, probably coexists with an alternative representation mode based on a type of structural description that can integrate information pertaining to the 3-D aspect of shapes.

Surface diagnosticity predicts the high-level representation of regular and irregular object shape in human vision.

The human visual system has an extraordinary capacity to compute three-dimensional (3D) shape structure for both geometrically regular and irregular objects. The goal of this study was to shed new light on the underlying representational structures that support this ability. Observers (N = 85) completed two complementary perceptual tasks. Experiment 1 involved whole-part matching of image parts to whole geometrically regular and irregular novel object shapes. Image parts comprised either regions of edge contour, volumetric parts, or surfaces. Performance was better for irregular than for regular objects and interacted with part type: volumes yielded better matching performance than surfaces for regular but not for irregular objects. The basis for this effect was further explored in Experiment 2, which used implicit part-whole repetition priming. Here, we orthogonally manipulated shape regularity and a new factor of surface diagnosticity (how predictive a single surface is of object identity). The results showed that surface diagnosticity, not object shape regularity, determined the differential processing of volumes and surfaces. Regardless of shape regularity, objects with low surface diagnosticity were better primed by volumes than by surfaces. In contrast, objects with high surface diagnosticity showed the opposite pattern. These findings are the first to show that surface diagnosticity plays a fundamental role in object recognition. We propose that surface-based shape primitives-rather than volumetric parts-underlie the derivation of 3D object shape in human vision.

Human Parahippocampal Cortex Supports Spatial Binding in Visual Working Memory.

Studies investigating the functional organization of the medial temporal lobe (MTL) suggest that parahippocampal cortex (PHC) generates representations of spatial and contextual information used by the hippocampus in the formation of episodic memories. However, evidence from animal studies also implicates PHC in spatial binding of visual information held in short term, working memory. Here we examined a 46-year-old man (P.J.), after he had recovered from bilateral medial occipitotemporal cortex strokes resulting in ischemic lesions of PHC and hippocampal atrophy, and a group of age-matched healthy controls. When recalling the color of 1 of 2 objects, P.J. misidentified the target when cued by its location, but not shape. When recalling the position of 1 of 3 objects, he frequently misidentified the target, which was cued by its color. Increasing the duration of the memory delay had no impact on the proportion of binding errors, but did significantly worsen recall precision in both P.J. and controls. We conclude that PHC may play a crucial role in spatial binding during encoding of visual information in working memory.

Stereo viewing modulates three-dimensional shape processing during object recognition: A high-density ERP study.

The role of stereo disparity in the recognition of 3-dimensional (3D) object shape remains an unresolved issue for theoretical models of the human visual system. We examined this issue using high-density (128 channel) recordings of event-related potentials (ERPs). A recognition memory task was used in which observers were trained to recognize a subset of complex, multipart, 3D novel objects under conditions of either (bi-) monocular or stereo viewing. In a subsequent test phase they discriminated previously trained targets from untrained distractor objects that shared either local parts, 3D spatial configuration, or neither dimension, across both previously seen and novel viewpoints. The behavioral data showed a stereo advantage for target recognition at untrained viewpoints. ERPs showed early differential amplitude modulations to shape similarity defined by local part structure and global 3D spatial configuration. This occurred initially during an N1 component around 145-190 ms poststimulus onset, and then subsequently during an N2/P3 component around 260-385 ms poststimulus onset. For mono viewing, amplitude modulation during the N1 was greatest between targets and distracters with different local parts for trained views only. For stereo viewing, amplitude modulation during the N2/P3 was greatest between targets and distracters with different global 3D spatial configurations and generalized across trained and untrained views. The results show that image classification is modulated by stereo information about the local part, and global 3D spatial configuration of object shape. The findings challenge current theoretical models that do not attribute functional significance to stereo input during the computation of 3D object shape. (PsycINFO Database Record

Effects of stereoscopic disparity on early ERP components during classification of three-dimensional objects.

This study investigates the effects of stereo disparity on the perception of three-dimensional (3D) object shape. We tested the hypothesis that stereo input modulates the brain activity related to perceptual analyses of 3D shape configuration during image classification. High-density (256-channel) electroencephalogram (EEG) was used to record the temporal dynamics of visual shape processing under conditions of two-dimensional (2D) and 3D visual presentation. On each trial, observers made image classification judgements ('Same'/'Different') to two briefly presented, multi-part, novel objects. On different-object trials, stimuli could either share volumetric parts but not the global 3D shape configuration and have different parts but the same global 3D shape configuration or differ on both aspects. Analyses using mass univariate contrasts showed that the earliest sensitivity to 2D versus 3D viewing appeared as a negative deflection over posterior locations on the N1 component between 160 and 220 ms post-stimulus onset. Subsequently, event-related potential (ERP) modulations during the N2 time window between 240 and 370 ms were linked to image classification. N2 activity reflected two distinct components - an early N2 (240-290 ms) and a late N2 (290-370 ms) - that showed different patterns of responses to 2D and 3D input and differential sensitivity to 3D object structure. The results revealed that stereo input modulates the neural correlates of 3D object shape. We suggest that this reflects differential perceptual processing of object shape under conditions of stereo or mono input. These findings challenge current theories that attribute no functional role for stereo input during 3D shape perception.

Early differential sensitivity of evoked-potentials to local and global shape during the perception of three-dimensional objects.

Here we investigated the time course underlying differential processing of local and global shape information during the perception of complex three-dimensional (3D) objects. Observers made shape matching judgments about pairs of sequentially presented multi-part novel objects. Event-related potentials (ERPs) were used to measure perceptual sensitivity to 3D shape differences in terms of local part structure and global shape configuration - based on predictions derived from hierarchical structural description models of object recognition. There were three types of different object trials in which stimulus pairs (1) shared local parts but differed in global shape configuration; (2) contained different local parts but shared global configuration or (3) shared neither local parts nor global configuration. Analyses of the ERP data showed differential amplitude modulation as a function of shape similarity as early as the N1 component between 146-215ms post-stimulus onset. These negative amplitude deflections were more similar between objects sharing global shape configuration than local part structure. Differentiation among all stimulus types was reflected in N2 amplitude modulations between 276-330ms. sLORETA inverse solutions showed stronger involvement of left occipitotemporal areas during the N1 for object discrimination weighted towards local part structure. The results suggest that the perception of 3D object shape involves parallel processing of information at local and global scales. This processing is characterised by relatively slow derivation of 'fine-grained' local shape structure, and fast derivation of 'coarse-grained' global shape configuration. We propose that the rapid early derivation of global shape attributes underlies the observed patterns of N1 amplitude modulations.

Domain General Sequence Operations Contribute to Pre-SMA Involvement in Visuo-spatial Processing.

This study used 3T MRI to elucidate the functional role of supplementary motor area (SMA) in relation to visuo-spatial processing. A localizer task contrasting sequential number subtraction and repetitive button pressing was used to functionally delineate non-motor sequence processing in pre-SMA, and activity in SMA-proper associated with motor sequencing. Patterns of BOLD responses in these regions were then contrasted to those from two tasks of visuo-spatial processing. In one task participants performed Mental Rotation (MR) in which recognition memory judgments were made to previously memorized 2D novel patterns across image-plane rotations. The other task involved abstract grid navigation (GN) in which observers computed a series of imagined location shifts in response to directional (arrow) cues around a mental grid. The results showed overlapping activation in pre-SMA for sequential subtraction and both visuo-spatial tasks. These results suggest that visuo-spatial processing is supported by non-motor sequence operations that involve pre-SMA. More broadly, these data further highlight the functional heterogeneity of pre-SMA, and show that its role extends to processes beyond the planning and online control of movement.

The role of surface-based representations of shape in visual object recognition.

This study contrasted the role of surfaces and volumetric shape primitives in three-dimensional object recognition. Observers (N = 50) matched subsets of closed contour fragments, surfaces, or volumetric parts to whole novel objects during a whole-part matching task. Three factors were further manipulated: part viewpoint (either same or different between component parts and whole objects), surface occlusion (comparison parts contained either visible surfaces only, or a surface that was fully or partially occluded in the whole object), and target-distractor similarity. Similarity was varied in terms of systematic variation in nonaccidental (NAP) or metric (MP) properties of individual parts. Analysis of sensitivity (d') showed a whole-part matching advantage for surface-based parts and volumes over closed contour fragments--but no benefit for volumetric parts over surfaces. We also found a performance cost in matching volumetric parts to wholes when the volumes showed surfaces that were occluded in the whole object. The same pattern was found for both same and different viewpoints, and regardless of target-distractor similarity. These findings challenge models in which recognition is mediated by volumetric part-based shape representations. Instead, we argue that the results are consistent with a surface-based model of high-level shape representation for recognition.

Implicit encoding of extrinsic object properties in stored representations mediating recognition: evidence from shadow-specific repetition priming.

This study investigated whether, and under what conditions, stored shape representations mediating recognition encode extrinsic object properties that vary according to viewing conditions. This was examined in relation to cast shadow. Observers (N = 90) first memorised a subset of 3D multi-part novel objects from a limited range of viewpoints rendered with either no shadow, object internal shadow, or both object internal and external (ground) plane shadow. During a subsequent test phase previously memorised targets were discriminated from visually similar distractors across learned and novel views following brief presentation of a same-shape masked prime. The primes contained either matching or mismatching shadow rendering from the training condition. The results showed a recognition advantage for objects memorised with object internal shadow. In addition, objects encoded with internal shadow were primed more strongly by matching internal shadow primes, than by same shape primes with either no shadow or both object internal and external (ground) shadow. This pattern of priming effects generalises to previously unseen views of targets rendered with object internal shadow. The results suggest that the object recognition system contains a level of stored representation at which shape and the extrinsic object property of cast shadow are bound. We propose that this occurs when cast shadow cannot be discounted during perception on the basis of external cues to the scene lighting model.

Stereo disparity facilitates view generalization during shape recognition for solid multipart objects.

Current theories of object recognition in human vision make different predictions about whether the recognition of complex, multipart objects should be influenced by shape information about surface depth orientation and curvature derived from stereo disparity. We examined this issue in five experiments using a recognition memory paradigm in which observers (N = 134) memorized and then discriminated sets of 3D novel objects at trained and untrained viewpoints under either mono or stereo viewing conditions. In order to explore the conditions under which stereo-defined shape information contributes to object recognition we systematically varied the difficulty of view generalization by increasing the angular disparity between trained and untrained views. In one series of experiments, objects were presented from either previously trained views or untrained views rotated (15°, 30°, or 60°) along the same plane. In separate experiments we examined whether view generalization effects interacted with the vertical or horizontal plane of object rotation across 40° viewpoint changes. The results showed robust viewpoint-dependent performance costs: Observers were more efficient in recognizing learned objects from trained than from untrained views, and recognition was worse for extrapolated than for interpolated untrained views. We also found that performance was enhanced by stereo viewing but only at larger angular disparities between trained and untrained views. These findings show that object recognition is not based solely on 2D image information but that it can be facilitated by shape information derived from stereo disparity.

Impaired visuospatial transformation but intact sequence processing in Parkinson disease.

OBJECTIVE: We examined whether visuospatial deficits in Parkinson disease (PD) can be explained by a domain-general, nonspatial impairment in the sequencing or serial chaining of mental operations. BACKGROUND: PD has been shown to be associated with impaired visuospatial processing, but the mechanisms of this impairment remain unclear. METHODS: Thirteen patients with PD and 20 age-matched, neurologically normal controls performed a visuospatial grid navigation task requiring sequential spatial transformations. The participants also performed a control task of serial number subtraction designed to assess their nonvisuospatial sequencing. The tasks were matched in structure and difficulty. RESULTS: The patients were impaired on the visuospatial task but not in serial number subtraction. This finding suggests that visuospatial processing impairments in PD do not derive from a general impairment affecting sequencing or serial chaining. CONCLUSIONS: We argue that visuospatial deficits in PD result from impairments to spatial transformation routines involved in the computation of mappings between spatial locations. These routines are mediated by dopaminergic pathways linking the basal ganglia, prefrontal cortex, supplementary motor area, and parietal cortex.

Shape information mediating basic- and subordinate-level object recognition revealed by analyses of eye movements.

This study examines the kinds of shape features that mediate basic- and subordinate-level object recognition. Observers were trained to categorize sets of novel objects at either a basic (between-families) or subordinate (within-family) level of classification. We analyzed the spatial distributions of fixations and compared them to model distributions of different curvature polarity (regions of convex or concave bounding contour), as well as internal part boundaries. The results showed a robust preference for fixation at part boundaries and for concave over convex regions of bounding contour, during both basic- and subordinate-level classification. In contrast, mean saccade amplitudes were shorter during basic- than subordinate-level classification. These findings challenge models of recognition that do not posit any special functional status to part boundaries or curvature polarity. We argue that both basic- and subordinate-level classification are mediated by object representations. These representations make explicit internal part boundaries, and distinguish concave and convex regions of bounding contour. The classification task constrains how shape information in these representations is used, consistent with the hypothesis that both parts-based, and image-based, operations support object recognition in human vision.

The time course of activation of object shape and shape+colour representations during memory retrieval.

Little is known about the timing of activating memory for objects and their associated perceptual properties, such as colour, and yet this is important for theories of human cognition. We investigated the time course associated with early cognitive processes related to the activation of object shape and object shape+colour representations respectively, during memory retrieval as assessed by repetition priming in an event-related potential (ERP) study. The main findings were as follows: (1) we identified a unique early modulation of mean ERP amplitude during the N1 that was associated with the activation of object shape independently of colour; (2) we also found a subsequent early P2 modulation of mean amplitude over the same electrode clusters associated with the activation of object shape+colour representations; (3) these findings were apparent across both familiar (i.e., correctly coloured - yellow banana) and novel (i.e., incorrectly coloured - blue strawberry) objects; and (4) neither of the modulations of mean ERP amplitude were evident during the P3. Together the findings delineate the timing of object shape and colour memory systems and support the notion that perceptual representations of object shape mediate the retrieval of temporary shape+colour representations for familiar and novel objects.

Visuospatial transformation impairments in Parkinson's disease.

Some previous studies have reported impairments in Parkinson's disease (PD) that affect performance on visuospatial tasks. The aim of this study was to further elucidate the underlying cognitive impairment to visuospatial processing in PD. Nondementing PD patients and neurologically normal, age-matched controls completed two tasks of visuospatial transformation. In Task 1, observers made perceptual matching judgments about the shape equivalence of two upright or rotated 2D novel patterns. Task 2 used a recognition memory paradigm in which participants first memorized a target object at a specific orientation and then made target/nontarget judgments to targets or visually similar distractors at varying orientations. Analyses of the regression slopes between response times and angular disparity showed that PD is associated with impairments affecting spatial transformation during image normalization in both tasks. The results also showed that the patients were more impaired, relative to controls, at spatial transformation during the perceptual matching of two images, than in the recognition memory task. It is suggested that PD can result in deficits affecting spatial transformation processes that are modulated by working memory and task demands.

Eye movement patterns during the recognition of three-dimensional objects: preferential fixation of concave surface curvature minima.

This study used eye movement patterns to examine how high-level shape information is used during 3D object recognition. Eye movements were recorded while observers either actively memorized or passively viewed sets of novel objects, and then during a subsequent recognition memory task. Fixation data were contrasted against different algorithmically generated models of shape analysis based on: (1) regions of internal concave or (2) convex surface curvature discontinuity or (3) external bounding contour. The results showed a preference for fixation at regions of internal local features during both active memorization and passive viewing but also for regions of concave surface curvature during the recognition task. These findings provide new evidence supporting the special functional status of local concave discontinuities in recognition and show how studies of eye movement patterns can elucidate shape information processing in human vision.

Impaired integration of object knowledge and visual input in a case of ventral simultanagnosia with bilateral damage to area V4.

This study examines how brain damage can affect the cognitive processes that support the integration of sensory input and prior knowledge during shape perception. It is based on the first detailed study of acquired ventral simultanagnosia, which was found in a patient (M.T.) with posterior occipitotemporal lesions encompassing V4 bilaterally. Despite showing normal object recognition for single items in both accuracy and response times (RTs), and intact low-level vision assessed across an extensive battery of tests, M.T. was impaired in object identification with overlapping figures displays. Task performance was modulated by familiarity: Unlike controls, M.T. was faster with overlapping displays of abstract shapes than with overlapping displays of common objects. His performance with overlapping common object displays was also influenced by both the semantic relatedness and visual similarity of the display items. These findings challenge claims that visual perception is driven solely by feedforward mechanisms and show how brain damage can selectively impair high-level perceptual processes supporting the integration of stored knowledge and visual sensory input.