Scene-selective brain regions respond to embedded objects of a scene.

Objects are fundamental to scene understanding. Scenes are defined by embedded objects and how we interact with them. Paradoxically, scene processing in the brain is typically discussed in contrast to object processing. Using the BOLD5000 dataset (Chang et al., 2019), we examined whether objects within a scene predicted the neural representation of scenes, as measured by functional magnetic resonance imaging in humans. Stimuli included 1,179 unique scenes across 18 semantic categories. Object composition of scenes were compared across scene exemplars in different semantic scene categories, and separately, in exemplars of the same scene category. Neural representations in scene- and object-preferring brain regions were significantly related to which objects were in a scene, with the effect at times stronger in the scene-preferring regions. The object model accounted for more variance when comparing scenes within the same semantic category to scenes from different categories. Here, we demonstrate the function of scene-preferring regions includes the processing of objects. This suggests visual processing regions may be better characterized by the processes, which are engaged when interacting with the stimulus kind, such as processing groups of objects in scenes, or processing a single object in our foreground, rather than the stimulus kind itself.

How associative thinking influences scene perception.

Perception of our external environment is not isolated from the influence of our internal thoughts, and past evidence points to a possible common associative mechanism underlying both the perception of scenes and our internal thought. Here, we investigated the nature of the interaction between an associative mindset and scene perception, hypothesizing a functional advantage to an associative thought pattern in the perception of scenes. Experiments 1 and 2 showed that associative thinking facilitates scene perception, which evolved over the course of the experiments. In contrast to scene perception, Experiment 3 showed that associative thinking hinders the perception of mundane objects, in which associative information is minimized. Nevertheless, object perception was facilitated when associative thinking was reduced. This double dissociation suggests that an associative mind is more receptive of externally perceived associative information, and that a match between the orientation of internal and external processing may be key for perception.

Contextual associations represented both in neural networks and human behavior.

Contextual associations facilitate object recognition in human vision. However, the role of context in artificial vision remains elusive as does the characteristics that humans use to define context. We investigated whether contextually related objects (bicycle-helmet) are represented more similarly in convolutional neural networks (CNNs) used for image understanding than unrelated objects (bicycle-fork). Stimuli were of objects against a white background and consisted of a diverse set of contexts (N = 73). CNN representations of contextually related objects were more similar to one another than to unrelated objects across all CNN layers. Critically, the similarity found in CNNs correlated with human behavior across multiple experiments assessing contextual relatedness, emerging significant only in the later layers. The results demonstrate that context is inherently represented in CNNs as a result of object recognition training, and that the representation in the later layers of the network tap into the contextual regularities that predict human behavior.

Functional Context Affects Scene Processing.

Rapid visual perception is often viewed as a bottom-up process. Category-preferred neural regions are often characterized as automatic, default processing mechanisms for visual inputs of their categorical preference. To explore the sensitivity of such regions to top-down information, we examined three scene-preferring brain regions, the occipital place area (OPA), the parahippocampal place area (PPA), and the retrosplenial complex (RSC), and tested whether the processing of outdoor scenes is influenced by the functional contexts in which they are seen. Context was manipulated by presenting real-world landscape images as if being viewed through a window or within a picture frame-manipulations that do not affect scene content but do affect one's functional knowledge regarding the scene. This manipulation influences neural scene processing (as measured by fMRI): The OPA and the PPA exhibited greater neural activity when participants viewed images as if through a window as compared with within a picture frame, whereas the RSC did not show this difference. In a separate behavioral experiment, functional context affected scene memory in predictable directions (boundary extension). Our interpretation is that the window context denotes three-dimensionality, therefore rendering the perceptual experience of viewing landscapes as more realistic. Conversely, the frame context denotes a 2-D image. As such, more spatially biased scene representations in the OPA and the PPA are influenced by differences in top-down, perceptual expectations generated from context. In contrast, more semantically biased scene representations in the RSC are likely to be less affected by top-down signals that carry information about the physical layout of a scene.

Exploring spatiotemporal neural dynamics of the human visual cortex.

The human visual cortex is organized in a hierarchical manner. Although previous evidence supporting this hypothesis has been accumulated, specific details regarding the spatiotemporal information flow remain open. Here we present detailed spatiotemporal correlation profiles of neural activity with low-level and high-level features derived from an eight-layer neural network pretrained for object recognition. These correlation profiles indicate an early-to-late shift from low-level features to high-level features and from low-level regions to higher-level regions along the visual hierarchy, consistent with feedforward information flow. Additionally, we computed three sets of features from the low- and high-level features provided by the neural network: object-category-relevant low-level features (the common components between low-level and high-level features), low-level features roughly orthogonal to high-level features (the residual Layer 1 features), and unique high-level features that were roughly orthogonal to low-level features (the residual Layer 7 features). Contrasting the correlation effects of the common components and the residual Layer 1 features, we observed that the early visual cortex (EVC) exhibited a similar amount of correlation with the two feature sets early in time, but in a later time window, the EVC exhibited a higher and longer correlation effect with the common components (i.e., the low-level object-category-relevant features) than with the low-level residual features-an effect unlikely to arise from purely feedforward information flow. Overall, our results indicate that non-feedforward processes, for example, top-down influences from mental representations of categories, may facilitate differentiation between these two types of low-level features within the EVC.

BOLD5000, a public fMRI dataset while viewing 5000 visual images.

Vision science, particularly machine vision, has been revolutionized by introducing large-scale image datasets and statistical learning approaches. Yet, human neuroimaging studies of visual perception still rely on small numbers of images (around 100) due to time-constrained experimental procedures. To apply statistical learning approaches that include neuroscience, the number of images used in neuroimaging must be significantly increased. We present BOLD5000, a human functional MRI (fMRI) study that includes almost 5,000 distinct images depicting real-world scenes. Beyond dramatically increasing image dataset size relative to prior fMRI studies, BOLD5000 also accounts for image diversity, overlapping with standard computer vision datasets by incorporating images from the Scene UNderstanding (SUN), Common Objects in Context (COCO), and ImageNet datasets. The scale and diversity of these image datasets, combined with a slow event-related fMRI design, enables fine-grained exploration into the neural representation of a wide range of visual features, categories, and semantics. Concurrently, BOLD5000 brings us closer to realizing Marr's dream of a singular vision science-the intertwined study of biological and computer vision.

Associative hallucinations result from stimulating left ventromedial temporal cortex.

Visual recognition requires connecting perceptual information with contextual information and existing knowledge. The ventromedial temporal cortex (VTC), including the medial fusiform, has been linked with object recognition, paired associate learning, contextual processing, and episodic memory, suggesting that this area may be critical in connecting visual processing, context, knowledge and experience. However, evidence for the link between associative processing, episodic memory, and visual recognition in VTC is currently lacking. Using electrocorticography (ECoG) in a single human patient, medial regions of the left VTC were found to be sensitive to the contextual associations of objects. Electrical brain stimulation (EBS) of this part of the left VTC of the patient, functionally defined as sensitive to associative processing, caused memory related, associative experiential visual phenomena. This provides evidence of a relationship between visual recognition, associative processing, and episodic memory. These results suggest a potential role for abnormalities of these processes as part of a mechanism that gives rise to some visual hallucinations.

A Neural Basis for Developmental Topographic Disorientation.

Developmental topographic disorientation (DTD) is a life-long condition in which affected individuals are severely impaired in navigating around their environment. Individuals with DTD have no apparent structural brain damage on conventional imaging and the neural mechanisms underlying DTD are currently unknown. Using functional and diffusion tensor imaging, we present a comprehensive neuroimaging study of an individual, J.N., with well defined DTD. J.N. has intact scene-selective responses in the parahippocampal place area (PPA), transverse occipital sulcus, and retrosplenial cortex (RSC), key regions associated with scene perception and navigation. However, detailed fMRI studies probing selective tuning properties of these regions, as well as functional connectivity, suggest that J.N.'s RSC has an atypical response profile and an atypical functional coupling to PPA compared with human controls. This deviant functional profile of RSC is not due to compromised structural connectivity. This comprehensive examination suggests that the RSC may play a key role in navigation-related processing and that an alteration of the RSC's functional properties may serve as the neural basis for DTD. SIGNIFICANCE STATEMENT: Individuals with developmental topographic disorientation (DTD) have a life-long impairment in spatial navigation in the absence of brain damage, neurological conditions, or basic perceptual or memory deficits. Although progress has been made in identifying brain regions that subserve normal navigation, the neural basis of DTD is unknown. Using functional and structural neuroimaging and detailed statistical analyses, we investigated the brain regions typically involved in navigation and scene processing in a representative DTD individual, J.N. Although scene-selective regions were identified, closer scrutiny indicated that these areas, specifically the retrosplenial cortex (RSC), were functionally disrupted in J.N. This comprehensive examination of a representative DTD individual provides insight into the neural basis of DTD and the role of the RSC in navigation-related processing.

Associative Processing Is Inherent in Scene Perception.

How are complex visual entities such as scenes represented in the human brain? More concretely, along what visual and semantic dimensions are scenes encoded in memory? One hypothesis is that global spatial properties provide a basis for categorizing the neural response patterns arising from scenes. In contrast, non-spatial properties, such as single objects, also account for variance in neural responses. The list of critical scene dimensions has continued to grow--sometimes in a contradictory manner--coming to encompass properties such as geometric layout, big/small, crowded/sparse, and three-dimensionality. We demonstrate that these dimensions may be better understood within the more general framework of associative properties. That is, across both the perceptual and semantic domains, features of scene representations are related to one another through learned associations. Critically, the components of such associations are consistent with the dimensions that are typically invoked to account for scene understanding and its neural bases. Using fMRI, we show that non-scene stimuli displaying novel associations across identities or locations recruit putatively scene-selective regions of the human brain (the parahippocampal/lingual region, the retrosplenial complex, and the transverse occipital sulcus/occipital place area). Moreover, we find that the voxel-wise neural patterns arising from these associations are significantly correlated with the neural patterns arising from everyday scenes providing critical evidence whether the same encoding principals underlie both types of processing. These neuroimaging results provide evidence for the hypothesis that the neural representation of scenes is better understood within the broader theoretical framework of associative processing. In addition, the results demonstrate a division of labor that arises across scene-selective regions when processing associations and scenes providing better understanding of the functional roles of each region within the cortical network that mediates scene processing.

Applying artificial vision models to human scene understanding.

How do we understand the complex patterns of neural responses that underlie scene understanding? Studies of the network of brain regions held to be scene-selective-the parahippocampal/lingual region (PPA), the retrosplenial complex (RSC), and the occipital place area (TOS)-have typically focused on single visual dimensions (e.g., size), rather than the high-dimensional feature space in which scenes are likely to be neurally represented. Here we leverage well-specified artificial vision systems to explicate a more complex understanding of how scenes are encoded in this functional network. We correlated similarity matrices within three different scene-spaces arising from: (1) BOLD activity in scene-selective brain regions; (2) behavioral measured judgments of visually-perceived scene similarity; and (3) several different computer vision models. These correlations revealed: (1) models that relied on mid- and high-level scene attributes showed the highest correlations with the patterns of neural activity within the scene-selective network; (2) NEIL and SUN-the models that best accounted for the patterns obtained from PPA and TOS-were different from the GIST model that best accounted for the pattern obtained from RSC; (3) The best performing models outperformed behaviorally-measured judgments of scene similarity in accounting for neural data. One computer vision method-NEIL ("Never-Ending-Image-Learner"), which incorporates visual features learned as statistical regularities across web-scale numbers of scenes-showed significant correlations with neural activity in all three scene-selective regions and was one of the two models best able to account for variance in the PPA and TOS. We suggest that these results are a promising first step in explicating more fine-grained models of neural scene understanding, including developing a clearer picture of the division of labor among the components of the functional scene-selective brain network.

Structurally-constrained relationships between cognitive states in the human brain.

The anatomical connectivity of the human brain supports diverse patterns of correlated neural activity that are thought to underlie cognitive function. In a manner sensitive to underlying structural brain architecture, we examine the extent to which such patterns of correlated activity systematically vary across cognitive states. Anatomical white matter connectivity is compared with functional correlations in neural activity measured via blood oxygen level dependent (BOLD) signals. Functional connectivity is separately measured at rest, during an attention task, and during a memory task. We assess these structural and functional measures within previously-identified resting-state functional networks, denoted task-positive and task-negative networks, that have been independently shown to be strongly anticorrelated at rest but also involve regions of the brain that routinely increase and decrease in activity during task-driven processes. We find that the density of anatomical connections within and between task-positive and task-negative networks is differentially related to strong, task-dependent correlations in neural activity. The space mapped out by the observed structure-function relationships is used to define a quantitative measure of separation between resting, attention, and memory states. We find that the degree of separation between states is related to both general measures of behavioral performance and relative differences in task-specific measures of attention versus memory performance. These findings suggest that the observed separation between cognitive states reflects underlying organizational principles of human brain structure and function.

The role of the parahippocampal cortex in cognition.

The parahippocampal cortex (PHC) has been associated with many cognitive processes, including visuospatial processing and episodic memory. To characterize the role of PHC in cognition, a framework is required that unifies these disparate processes. An overarching account was proposed whereby the PHC is part of a network of brain regions that processes contextual associations. Contextual associations are the principal element underlying many higher-level cognitive processes, and thus are suitable for unifying the PHC literature. Recent findings are reviewed that provide support for the contextual associations account of PHC function. In addition to reconciling a vast breadth of literature, the synthesis presented expands the implications of the proposed account and gives rise to new and general questions about context and cognition.

Structural foundations of resting-state and task-based functional connectivity in the human brain.

Magnetic resonance imaging enables the noninvasive mapping of both anatomical white matter connectivity and dynamic patterns of neural activity in the human brain. We examine the relationship between the structural properties of white matter streamlines (structural connectivity) and the functional properties of correlations in neural activity (functional connectivity) within 84 healthy human subjects both at rest and during the performance of attention- and memory-demanding tasks. We show that structural properties, including the length, number, and spatial location of white matter streamlines, are indicative of and can be inferred from the strength of resting-state and task-based functional correlations between brain regions. These results, which are both representative of the entire set of subjects and consistently observed within individual subjects, uncover robust links between structural and functional connectivity in the human brain.

Individual differences in shifting decision criterion: a recognition memory study.

An ability to flexibly shift a decision criterion can be advantageous. For example, a known change in the base rate of targets and distractors on a recognition memory test will lead optimal decision makers to shift their criterion accordingly. In the present study, 95 individuals participated in two recognition memory tests that included periodic changes in the base rate probability that the test stimulus had been presented during the study session. The results reveal a wide variability in the tendency to shift decision criterion in response to this probability information, with some appropriately shifting and others not shifting at all. However, participants were highly reliable in their tendency to shift criterion across tests. The goal of the present study was to explain what factors account for these individual differences. To accomplish this, over 50 variables were assessed for each individual (e.g., personality, cognitive style, state of mind). Using a regression model that incorporated different sets of factors, over 50% of the variance was accounted for. The results of the analysis describe the total, direct, and mediating effects on criterion shifting from factors that include memory strength, strategy, and inherent characteristics such as a fun-seeking personality, a negative affect, and military rank. The results are discussed with respect to understanding why participants rarely chose an optimal decision-making strategy and provide greater insight into the underlying mechanisms of recognition memory.

Individual differences in cognitive style and strategy predict similarities in the patterns of brain activity between individuals.

Neuroimaging is being used increasingly to make inferences about an individual. Yet, those inferences are often confounded by the fact that topographical patterns of task-related brain activity can vary greatly from person to person. This study examined two factors that may contribute to the variability across individuals in a memory retrieval task: individual differences in cognitive style and individual differences in encoding strategy. Cognitive style was probed using a battery of assessments focused on the individual's tendency to visualize or verbalize written material. Encoding strategy was probed using a series of questions designed to assess typical strategies that an individual might utilize when trying to remember a list of words. Similarity in brain activity was assessed by cross-correlating individual t-statistic maps contrasting the BOLD response during retrieval to the BOLD response during fixation. Individual differences in cognitive style and encoding strategy accounted for a significant portion of the variance in similarity. This was true above and beyond individual differences in anatomy and memory performance. These results demonstrate the need for a multidimensional approach in the use of fMRI to make inferences about an individual.

Early onset of neural synchronization in the contextual associations network.

Objects are more easily recognized in their typical context. However, is contextual information activated early enough to facilitate the perception of individual objects, or is contextual facilitation caused by postperceptual mechanisms? To elucidate this issue, we first need to study the temporal dynamics and neural interactions associated with contextual processing. Studies have shown that the contextual network consists of the parahippocampal, retrosplenial, and medial prefrontal cortices. We used functional MRI, magnetoencephalography, and phase synchrony analyses to compare the neural response to stimuli with strong or weak contextual associations. The context network was activated in functional MRI and preferentially synchronized in magnetoencephalography (MEG) for stimuli with strong contextual associations. Phase synchrony increased early (150-250 ms) only when it involved the parahippocampal cortex, whereas retrosplenial-medial prefrontal cortices synchrony was enhanced later (300-400 ms). These results describe the neural dynamics of context processing and suggest that context is activated early during object perception.

Scenes unseen: the parahippocampal cortex intrinsically subserves contextual associations, not scenes or places per se.

The parahippocampal cortex (PHC) has been implicated in both place/scene processing and episodic memory. We proposed that this region should instead be seen as intrinsically mediating contextual associations and not place/scene processing or episodic memory exclusively. Given that place/scene processing and episodic memory both rely on associations, this modified framework provides a platform for reconciling what seemed like different roles assigned to the same region. Comparing scenes with scenes, we show here that the PHC responds significantly more strongly to scenes with rich contextual associations compared with scenes of equal visual qualities but less associations. This result adds unequivocal support to the view that the PHC mediates contextual associations in general rather than places or scenes proper, and necessitates a revision of the current view that the PHC contains a dedicated place/scenes "module."

Cultural specificity in amygdala response to fear faces.

The human amygdala robustly activates to fear faces. Heightened response to fear faces is thought to reflect the amygdala's adaptive function as an early warning mechanism. Although culture shapes several facets of emotional and social experience, including how fear is perceived and expressed to others, very little is known about how culture influences neural responses to fear stimuli. Here we show that the bilateral amygdala response to fear faces is modulated by culture. We used functional magnetic resonance imaging to measure amygdala response to fear and nonfear faces in two distinct cultures. Native Japanese in Japan and Caucasians in the United States showed greater amygdala activation to fear expressed by members of their own cultural group. This finding provides novel and surprising evidence of cultural tuning in an automatic neural response.

The cortical underpinnings of context-based memory distortion.

Everyday contextual settings create associations that later afford generating predictions about what objects to expect in our environment. The cortical network that takes advantage of such contextual information is proposed to connect the representation of associated objects such that seeing one object (bed) will activate the visual representations of other objects sharing the same context (pillow). Given this proposal, we hypothesized that the cortical activity elicited by seeing a strong contextual object would predict the occurrence of false memories whereby one erroneously "remembers" having seen a new object that is related to a previously presented object. To test this hypothesis, we used functional magnetic resonance imaging during encoding of contextually related objects, and later tested recognition memory. New objects that were contextually related to previously presented objects were more often falsely judged as "old" compared with new objects that were contextually unrelated to old objects. This phenomenon was reflected by activity in the cortical network mediating contextual processing, which provides a better understanding of how the brain represents and processes context.

Famous faces activate contextual associations in the parahippocampal cortex.

The parahippocampal cortex (PHC) has been traditionally implicated both in place processing and in episodic memory. How could the same cortical region mediate these cognitive functions that seem quite different? We have recently proposed that the PHC should be seen as more generally mediating contextual associative processing, which is required for both navigation and memory. We therefore predicted that any associative objects should activate the PHC. To test this generalization, we investigated the extent to which common stimuli that are nonspatial by nature, namely faces, activate the PHC, although their perception is typically associated with other cortical structures. Specifically, we compared the activation elicited by famous faces, which are highly associated with rich pictorial and contextual information (e.g., Tom Cruise) and are not associated with a specific place, with activation elicited by unfamiliar faces. Consistent with our prediction, contrasting famous with unfamiliar faces revealed significant activation within the PHC. Taken collectively, these findings indicate that the PHC should be regarded as mediating contextual associations in general and not necessarily spatial or episodic information.