Recurrent Connections in the Primate Ventral Visual Stream Mediate a Trade-Off Between Task Performance and Network Size During Core Object Recognition.

The computational role of the abundant feedback connections in the ventral visual stream is unclear, enabling humans and nonhuman primates to effortlessly recognize objects across a multitude of viewing conditions. Prior studies have augmented feedforward convolutional neural networks (CNNs) with recurrent connections to study their role in visual processing; however, often these recurrent networks are optimized directly on neural data or the comparative metrics used are undefined for standard feedforward networks that lack these connections. In this work, we develop task-optimized convolutional recurrent (ConvRNN) network models that more correctly mimic the timing and gross neuroanatomy of the ventral pathway. Properly chosen intermediate-depth ConvRNN circuit architectures, which incorporate mechanisms of feedforward bypassing and recurrent gating, can achieve high performance on a core recognition task, comparable to that of much deeper feedforward networks. We then develop methods that allow us to compare both CNNs and ConvRNNs to finely grained measurements of primate categorization behavior and neural response trajectories across thousands of stimuli. We find that high-performing ConvRNNs provide a better match to these data than feedforward networks of any depth, predicting the precise timings at which each stimulus is behaviorally decoded from neural activation patterns. Moreover, these ConvRNN circuits consistently produce quantitatively accurate predictions of neural dynamics from V4 and IT across the entire stimulus presentation. In fact, we find that the highest-performing ConvRNNs, which best match neural and behavioral data, also achieve a strong Pareto trade-off between task performance and overall network size. Taken together, our results suggest the functional purpose of recurrence in the ventral pathway is to fit a high-performing network in cortex, attaining computational power through temporal rather than spatial complexity.

Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence.

A potentially organizing goal of the brain and cognitive sciences is to accurately explain domains of human intelligence as executable, neurally mechanistic models. Years of research have led to models that capture experimental results in individual behavioral tasks and individual brain regions. We here advocate for taking the next step: integrating experimental results from many laboratories into suites of benchmarks that, when considered together, push mechanistic models toward explaining entire domains of intelligence, such as vision, language, and motor control. Given recent successes of neurally mechanistic models and the surging availability of neural, anatomical, and behavioral data, we believe that now is the time to create integrative benchmarking platforms that incentivize ambitious, unified models. This perspective discusses the advantages and the challenges of this approach and proposes specific steps to achieve this goal in the domain of visual intelligence with the case study of an integrative benchmarking platform called Brain-Score.

Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior.

Non-recurrent deep convolutional neural networks (CNNs) are currently the best at modeling core object recognition, a behavior that is supported by the densely recurrent primate ventral stream, culminating in the inferior temporal (IT) cortex. If recurrence is critical to this behavior, then primates should outperform feedforward-only deep CNNs for images that require additional recurrent processing beyond the feedforward IT response. Here we first used behavioral methods to discover hundreds of these 'challenge' images. Second, using large-scale electrophysiology, we observed that behaviorally sufficient object identity solutions emerged ~30 ms later in the IT cortex for challenge images compared with primate performance-matched 'control' images. Third, these behaviorally critical late-phase IT response patterns were poorly predicted by feedforward deep CNN activations. Notably, very-deep CNNs and shallower recurrent CNNs better predicted these late IT responses, suggesting that there is a functional equivalence between additional nonlinear transformations and recurrence. Beyond arguing that recurrent circuits are critical for rapid object identification, our results provide strong constraints for future recurrent model development.

Predict, then simplify.

The desire to understand a given phenomenon is at the core of a scientist's mission. Yet what is meant by "understanding"? As soon as we try to operationalize this concept, I argue that understanding amounts to building models of a set of related empirical phenomena. In this commentary, I discuss three such criteria, namely, predictability, simplicity, and computability, that I find useful in guiding the quest for better models and what they imply for the use of deep nets in brain research.

Sensitivity to Nonaccidental Configurations of Two-Line Stimuli.

According to Recognition-By-Components theory, object recognition relies on a specific subset of three-dimensional shapes called geons. In particular, these configurations constitute a powerful cue to three-dimensional object reconstruction because their two-dimensional projection remains viewpoint-invariant. While a large body of literature has demonstrated sensitivity to changes in these so-called nonaccidental configurations, it remains unclear what information is used in establishing such sensitivity. In this study, we explored the possibility that nonaccidental configurations can already be inferred from the basic constituents of objects, namely, their edges. We constructed a set of stimuli composed of two lines corresponding to various nonaccidental properties and configurations underlying the distinction between geons, including collinearity, alignment, curvature of contours, curvature of configuration axis, expansion, cotermination, and junction type. Using a simple visual search paradigm, we demonstrated that participants were faster at detecting targets that differed from distractors in a nonaccidental property than in a metric property. We also found that only some but not all of the observed sensitivity could have resulted from simple low-level properties of our stimuli. Given that such sensitivity emerged from a configuration of only two lines, our results support the view that nonaccidental configurations could be encoded throughout the visual processing hierarchy even in the absence of object context.

Deep Neural Networks as a Computational Model for Human Shape Sensitivity.

Theories of object recognition agree that shape is of primordial importance, but there is no consensus about how shape might be represented, and so far attempts to implement a model of shape perception that would work with realistic stimuli have largely failed. Recent studies suggest that state-of-the-art convolutional 'deep' neural networks (DNNs) capture important aspects of human object perception. We hypothesized that these successes might be partially related to a human-like representation of object shape. Here we demonstrate that sensitivity for shape features, characteristic to human and primate vision, emerges in DNNs when trained for generic object recognition from natural photographs. We show that these models explain human shape judgments for several benchmark behavioral and neural stimulus sets on which earlier models mostly failed. In particular, although never explicitly trained for such stimuli, DNNs develop acute sensitivity to minute variations in shape and to non-accidental properties that have long been implicated to form the basis for object recognition. Even more strikingly, when tested with a challenging stimulus set in which shape and category membership are dissociated, the most complex model architectures capture human shape sensitivity as well as some aspects of the category structure that emerges from human judgments. As a whole, these results indicate that convolutional neural networks not only learn physically correct representations of object categories but also develop perceptually accurate representational spaces of shapes. An even more complete model of human object representations might be in sight by training deep architectures for multiple tasks, which is so characteristic in human development.

The occipital place area represents the local elements of scenes.

Neuroimaging studies have identified three scene-selective regions in human cortex: parahippocampal place area (PPA), retrosplenial complex (RSC), and occipital place area (OPA). However, precisely what scene information each region represents is not clear, especially for the least studied, more posterior OPA. Here we hypothesized that OPA represents local elements of scenes within two independent, yet complementary scene descriptors: spatial boundary (i.e., the layout of external surfaces) and scene content (e.g., internal objects). If OPA processes the local elements of spatial boundary information, then it should respond to these local elements (e.g., walls) themselves, regardless of their spatial arrangement. Indeed, we found that OPA, but not PPA or RSC, responded similarly to images of intact rooms and these same rooms in which the surfaces were fractured and rearranged, disrupting the spatial boundary. Next, if OPA represents the local elements of scene content information, then it should respond more when more such local elements (e.g., furniture) are present. Indeed, we found that OPA, but not PPA or RSC, responded more to multiple than single pieces of furniture. Taken together, these findings reveal that OPA analyzes local scene elements - both in spatial boundary and scene content representation - while PPA and RSC represent global scene properties.

Cue-invariant shape recognition in rats as tested with second-order contours.

Nonhuman primates are the main animal model to investigate high-level properties of human cortical vision. For one property, transformation-invariant object recognition, recent studies have revealed interesting and unknown capabilities in rats. Here we report on the ability of rats to rely upon second-order cues that are important to structure the incoming visual images into figure and background. Rats performed a visual shape discrimination task in which the shapes were not only defined by first-order luminance information but also by a variety of second-order cues such as a change in texture properties. Once the rats were acquainted with a first set of second-order stimuli, they showed a surprising degree of generalization towards new second-order stimuli. The limits of these capabilities were tested in various ways, and the ability to extract the shapes broke down only in extreme cases where no local cues were available to solve the task. These results demonstrate how rats are able to make choices based on fairly complex strategies when necessary.

Brain-decoding fMRI reveals how wholes relate to the sum of parts.

The human brain performs many nonlinear operations in order to extract relevant information from local inputs. How can we observe and quantify these effects within and across large patches of cortex? In this paper, we discuss the application of multi-voxel pattern analysis (MVPA) in functional magnetic resonance imaging (fMRI) to address this issue. Specifically, we show how MVPA (i) allows to compare various possibilities of part combinations into wholes, such as taking the mean, weighted mean, or the maximum of responses to the parts; (ii) can be used to quantify the parameters of these combinations; and (iii) can be applied in various experimental paradigms. Through these procedures, fMRI helps to obtain a computational understanding of how local information is integrated into larger wholes in various cortical regions.

Sharing code.

Sharing code is becoming increasingly important in the wake of Open Science. In this review I describe and compare two popular code-sharing utilities, GitHub and Open Science Framework (OSF). GitHub is a mature, industry-standard tool but lacks focus towards researchers. In comparison, OSF offers a one-stop solution for researchers but a lot of functionality is still under development. I conclude by listing alternative lesser-known tools for code and materials sharing.

Encoding of configural regularity in the human visual system.

The visual system is very efficient in encoding stimulus properties by utilizing available regularities in the inputs. To explore the underlying encoding strategies during visual information processing, we presented participants with two-line configurations that varied in the amount of configural regularity (or degrees of freedom in the relative positioning of the two lines) in a fMRI experiment. Configural regularity ranged from a generic configuration to stimuli resembling an "L" (i.e., a right-angle L-junction), a "T" (i.e., a right-angle midpoint T-junction), or a "+",-the latter being the most regular stimulus. We found that the response strength in the shape-selective lateral occipital area was consistently lower for a higher degree of regularity in the stimuli. In the second experiment, using multivoxel pattern analysis, we further show that regularity is encoded in terms of the fMRI signal strength but not in the distributed pattern of responses. Finally, we found that the results of these experiments could not be accounted for by low-level stimulus properties and are distinct from norm-based encoding. Our results suggest that regularity plays an important role in stimulus encoding in the ventral visual processing stream.

A conceptual framework of computations in mid-level vision.

If a picture is worth a thousand words, as an English idiom goes, what should those words-or, rather, descriptors-capture? What format of image representation would be sufficiently rich if we were to reconstruct the essence of images from their descriptors? In this paper, we set out to develop a conceptual framework that would be: (i) biologically plausible in order to provide a better mechanistic understanding of our visual system; (ii) sufficiently robust to apply in practice on realistic images; and (iii) able to tap into underlying structure of our visual world. We bring forward three key ideas. First, we argue that surface-based representations are constructed based on feature inference from the input in the intermediate processing layers of the visual system. Such representations are computed in a largely pre-semantic (prior to categorization) and pre-attentive manner using multiple cues (orientation, color, polarity, variation in orientation, and so on), and explicitly retain configural relations between features. The constructed surfaces may be partially overlapping to compensate for occlusions and are ordered in depth (figure-ground organization). Second, we propose that such intermediate representations could be formed by a hierarchical computation of similarity between features in local image patches and pooling of highly-similar units, and reestimated via recurrent loops according to the task demands. Finally, we suggest to use datasets composed of realistically rendered artificial objects and surfaces in order to better understand a model's behavior and its limitations.

A framework for streamlining research workflow in neuroscience and psychology.

Successful accumulation of knowledge is critically dependent on the ability to verify and replicate every part of scientific conduct. However, such principles are difficult to enact when researchers continue to resort on ad-hoc workflows and with poorly maintained code base. In this paper I examine the needs of neuroscience and psychology community, and introduce psychopy_ext, a unifying framework that seamlessly integrates popular experiment building, analysis and manuscript preparation tools by choosing reasonable defaults and implementing relatively rigid patterns of workflow. This structure allows for automation of multiple tasks, such as generated user interfaces, unit testing, control analyses of stimuli, single-command access to descriptive statistics, and publication quality plotting. Taken together, psychopy_ext opens an exciting possibility for a faster, more robust code development and collaboration for researchers.

Configural Gestalts remain nothing more than the sum of their parts in visual agnosia.

We report converging evidence that higher stages of the visual system are critically required for the whole to become more than the sum of its parts by studying patient DF with visual agnosia using a configural superiority paradigm. We demonstrate a clear dissociation between this patient and normal controls such that she could more easily report information about parts, demonstrating a striking reversal of the normal configural superiority effect. Furthermore, by comparing DF's performance to earlier neuroimaging and novel modeling work, we found a compelling consistency between her performance and representations in the early visual areas, which are spared in this patient. The reversed pattern of performance in this patient highlights that in some cases visual Gestalts do not emerge early on without processing in higher visual areas. More broadly, this study demonstrates how neuropsychological patients can be used to unmask representations maintained at early stages of processing.

Bistable Gestalts reduce activity in the whole of V1, not just the retinotopically predicted parts.

Activity in the primary visual cortex reduces when certain stimuli can be perceptually organized as a unified Gestalt. This reduction could offer important insights into the nature of feedback computations within the human visual system; however, the properties of this response reduction have not yet been investigated in detail. Here we replicate this reduced V1 response, but find that the modulation in V1 (and V2) to the perceived organization of the input is not specific to the retinotopic location at which the sensory input from that stimulus is represented. Instead, we find a response modulation that is equally evident across the primary visual cortex. Thus in contradiction to some models of hierarchical predictive coding, the perception of an organized Gestalt causes a broad feedback effect that does not act specifically on the part of the retinotopic map representing the sensory input.

Emergence of perceptual Gestalts in the human visual cortex: the case of the configural-superiority effect.

Many Gestalt phenomena have been described in terms of perception of a whole being not equal to the sum of its parts. It is unclear how these phenomena emerge in the brain. We used functional MRI to study the neural basis of the behavioral configural-superiority effect (i.e., visual search is more efficient when an odd element is part of a configuration than when it is presented by itself). We found that searching for the odd element in a display of four line segments (parts) was facilitated by adding two additional line segments to each of them (creating whole shapes). Functional MRI-based decoding of neural responses to the position of the odd element revealed a neural configural-superiority effect in shape-selective regions but not in low-level retinotopic areas, where decoding of parts was more pronounced. These results show how at least some Gestalt phenomena in vision emerge only at the higher stages of visual information processing and suggest that feed-forward processing might be sufficient to produce such phenomena.

Mirror-image sensitivity and invariance in object and scene processing pathways.

Electrophysiological and behavioral studies in many species have demonstrated mirror-image confusion for objects, perhaps because many objects are vertically symmetric (e.g., a cup is the same cup when seen in left or right profile). In contrast, the navigability of a scene changes when it is mirror reversed, and behavioral studies reveal high sensitivity to this change. Thus, we predicted that representations in object-selective cortex will be unaffected by mirror reversals, whereas representations in scene-selective cortex will be sensitive to such reversals. To test this hypothesis, we ran an event-related functional magnetic resonance imaging adaptation experiment in human adults. Consistent with our prediction, we found tolerance to mirror reversals in one object-selective region, the posterior fusiform sulcus, and a strong sensitivity to these reversals in two scene-selective regions, the transverse occipital sulcus and the retrosplenial complex. However, a more posterior object-selective region, the lateral occipital sulcus, showed sensitivity to mirror reversals, suggesting that the sense information that distinguishes mirror images is represented at earlier stages in the object-processing hierarchy. Moreover, one scene-selective region (the parahippocampal place area or PPA) was tolerant to mirror reversals. This last finding challenges the hypothesis that the PPA is involved in navigation and reorientation and suggests instead that scenes, like objects, are processed by distinct pathways guiding recognition and action.