Spatial pattern of BOLD fMRI activation reveals cross-modal information in auditory cortex.

Recent findings suggest that neural representations in early auditory cortex reflect not only the physical properties of a stimulus, but also high-level, top-down, and even cross-modal information. However, the nature of cross-modal information in auditory cortex remains poorly understood. Here, we used pattern analyses of fMRI data to ask whether early auditory cortex contains information about the visual environment. Our data show that 1) early auditory cortex contained information about a visual stimulus when there was no bottom-up auditory signal, and that 2) no influence of visual stimulation was observed in auditory cortex when visual stimuli did not provide a context relevant to audition. Our findings attest to the capacity of auditory cortex to reflect high-level, top-down, and cross-modal information and indicate that the spatial patterns of activation in auditory cortex reflect contextual/implied auditory information but not visual information per se.

Pre-stimulus pattern of activity in the fusiform face area predicts face percepts during binocular rivalry.

Visual input is ambiguous, yet conscious experience is unambiguous. In binocular rivalry the two eyes receive conflicting images, but only one of them is consciously perceived at a time. Here we search for the neural sites of the competitive interactions underlying this phenomenon by testing whether neural pattern activity occurring before stimulus presentation can predict the initial dominant percept in binocular rivalry and, if so, where in the brain such predictive activity is found. Subjects were scanned while viewing an image of a face in one eye and an image of a house in the other eye with anaglyph glasses. The rivalrous stimulus was presented briefly for each trial, and the subject indicated which of the two images he or she preferentially perceived. Our results show that BOLD fMRI multivariate pattern activity in the fusiform face area (FFA) before the stimulus is presented predicts which of the two images will be dominant, suggesting that higher extrastriate areas, such as the FFA, are not only correlated with, but may also be involved in determining the initial dominant percept in binocular rivalry. Furthermore, by examining pattern activity before and after trial onset, we found that pre-trial activity in the FFA for the rivalrous face trials is no more similar to the post-trial activity for the non-rivalrous face trials than to that for the non-rivalrous house trials, indicating a dissociation between neural pattern information, which predicts a given state of awareness, and mean responses, which reflect the state of awareness ultimately achieved.

Recognition alters the spatial pattern of FMRI activation in early retinotopic cortex.

Early retinotopic cortex has traditionally been viewed as containing a veridical representation of the low-level properties of the image, not imbued by high-level interpretation and meaning. Yet several recent results indicate that neural representations in early retinotopic cortex reflect not just the sensory properties of the image, but also the perceived size and brightness of image regions. Here we used functional magnetic resonance imaging pattern analyses to ask whether the representation of an object in early retinotopic cortex changes when the object is recognized compared with when the same stimulus is presented but not recognized. Our data confirmed this hypothesis: the pattern of response in early retinotopic visual cortex to a two-tone "Mooney" image of an object was more similar to the response to the full grayscale photo version of the same image when observers knew what the two-tone image represented than when they did not. Further, in a second experiment, high-level interpretations actually overrode bottom-up stimulus information, such that the pattern of response in early retinotopic cortex to an identified two-tone image was more similar to the response to the photographic version of that stimulus than it was to the response to the identical two-tone image when it was not identified. Our findings are consistent with prior results indicating that perceived size and brightness affect representations in early retinotopic visual cortex and, further, show that even higher-level information--knowledge of object identity--also affects the representation of an object in early retinotopic cortex.

Domain specificity in visual cortex.

We investigated the prevalence and specificity of category-selective regions in human visual cortex. In the broadest survey to date of category selectivity in visual cortex, 12 participants were scanned with functional magnetic resonance imaging while viewing scenes and 19 different object categories in a blocked-design experiment. As expected, we found selectivity for faces in the fusiform face area (FFA), for scenes in the parahippocampal place area (PPA), and for bodies in the extrastriate body area (EBA). In addition, we describe 3 main new findings. First, evidence for the selectivity of the FFA, PPA, and EBA was strengthened by the finding that each area responded significantly more strongly to its preferred category than to the next most effective of the remaining 19 stimulus categories tested. Second, a region in the middle temporal gyrus that has been reported to respond significantly more strongly to tools than to animals did not respond significantly more strongly to tools than to other nontool categories (such as fruits and vegetables), casting doubt on the characterization of this region as tool selective. Finally, we did not find any new regions in the occipitotemporal pathway that were strongly selective for other categories. Taken together, these results demonstrate both the strong selectivity of a small number of regions and the scarcity of such regions in visual cortex.

A region of right posterior superior temporal sulcus responds to observed intentional actions.

Human adults and infants identify the actions of another agent based not only on its intrinsic perceptual features, but critically on the contingent relationship between its motion path and the environmental context [Trends Cogn. Sci. 7 (1995) 287; Cognition 72 (2003) 237]. Functional neuroimaging studies of the perception of agents and intentional actions, on the other hand, have mostly focussed on the perception of intrinsic cues to agency, like a face or articulated body motion (e.g. [J. Neurosci. 17 (1997) 4302; Neuroimage 8 (1998) 221; Trends Cogn. Sci. 4 (2000) 267; Nat. Neurosci. 3 (2000) 80; Neuroimage 13 (2001) 775; Proc. Natl. Acad. Sci. U.S.A. 98 (2001) 11656; Neuron 35 (2002) 1167; Neuron 34 (2002) 149, Neuroscience 15 (2003) 991; J. Neurosci. 23 (2003) 6819; Philos. Trans. R Soc. Lond. B. Biol. Sci. 358 (2003) 435]. Here we describe a region of the right posterior superior temporal sulcus that is sensitive not to articulated body motion per se, but to the relationship between the observed motion and the structure of the surrounding environment. From this and other aspects of the region's response, we hypothesize that this region is involved in the representation of observed intentional actions.

Understanding other minds: linking developmental psychology and functional neuroimaging.

Evidence from developmental psychology suggests that understanding other minds constitutes a special domain of cognition with at least two components: an early-developing system for reasoning about goals, perceptions, and emotions, and a later-developing system for representing the contents of beliefs. Neuroimaging reinforces and elaborates upon this view by providing evidence that (a) domain-specific brain regions exist for representing belief contents, (b) these regions are apparently distinct from other regions engaged in reasoning about goals and actions (suggesting that the two developmental stages reflect the emergence of two distinct systems, rather than the elaboration of a single system), and (c) these regions are distinct from brain regions engaged in inhibitory control and in syntactic processing. The clear neural distinction between these processes is evidence that belief attribution is not dependent on either inhibitory control or syntax, but is subserved by a specialized neural system for theory of mind.

People thinking about thinking people. The role of the temporo-parietal junction in "theory of mind".

Humans powerfully and flexibly interpret the behaviour of other people based on an understanding of their minds: that is, we use a "theory of mind." In this study we distinguish theory of mind, which represents another person's mental states, from a representation of the simple presence of another person per se. The studies reported here establish for the first time that a region in the human temporo-parietal junction (here called the TPJ-M) is involved specifically in reasoning about the contents of another person's mind. First, the TPJ-M was doubly dissociated from the nearby extrastriate body area (EBA; Downing et al., 2001). Second, the TPJ-M does not respond to false representations in non-social control stories. Third, the BOLD response in the TPJ-M bilaterally was higher when subjects read stories about a character's mental states, compared with stories that described people in physical detail, which did not differ from stories about nonhuman objects. Thus, the role of the TPJ-M in understanding other people appears to be specific to reasoning about the content of mental states.

Attention response functions: characterizing brain areas using fMRI activation during parametric variations of attentional load.

We derived attention response functions for different cortical areas by plotting neural activity (measured by fMRI) as a function of attentional load in a visual tracking task. In many parietal and frontal cortical areas, activation increased with load over the entire range of loads tested, suggesting that these areas are directly involved in attentional processes. However, in other areas (FEF and parietal area 7), strong activation was observed even at the lowest attentional load (compared to a passive baseline using identical stimuli), but little or no additional activation was seen with increasing load. These latter areas appear to play a different role, perhaps supporting task-relevant functions that do not vary with load, such as the suppression of eye movements.

A cortical area selective for visual processing of the human body.

Despite extensive evidence for regions of human visual cortex that respond selectively to faces, few studies have considered the cortical representation of the appearance of the rest of the human body. We present a series of functional magnetic resonance imaging (fMRI) studies revealing substantial evidence for a distinct cortical region in humans that responds selectively to images of the human body, as compared with a wide range of control stimuli. This region was found in the lateral occipitotemporal cortex in all subjects tested and apparently reflects a specialized neural system for the visual perception of the human body.

Testing cognitive models of visual attention with fMRI and MEG.

Neuroimaging techniques can be used not only to identify the neural substrates of attention, but also to test cognitive theories of attention. Here we consider four classic questions in the psychology of visual attention: (i) Are some 'special' classes of stimuli (e.g. faces) immune to attentional modulation?; (ii) What are the information units on which attention operates?; (iii) How early in stimulus processing are attentional effects observed?; and (iv) Are common mechanisms involved in different modes of attentional selection (e.g. spatial and non-spatial selection)? We describe studies from our laboratory that illustrate the ways in which fMRI and MEG can provide key evidence in answering these questions. A central methodological theme in many of our fMRI studies is the use of analyses in which the activity in certain functionally-defined regions of interest (ROIs) is used to test specific cognitive hypotheses. An analogous sensor-of-interest (SOI) approach is applied to MEG. Our results include: evidence for the modulation of face representations by attention; confirmation of the independent contributions of object-based and location-based selection; evidence for modulation of face representations by non-spatial selection within the first 170 ms of processing; and implication of the intraparietal sulcus in functions general to spatial and non-spatial visual selection.

Representation of perceived object shape by the human lateral occipital complex.

The human lateral occipital complex (LOC) has been implicated in object recognition, but it is unknown whether this region represents low-level image features or perceived object shape. We used an event-related functional magnetic resonance imaging adaptation paradigm in which the response to pairs of successively presented stimuli is lower when they are identical than when they are different. Adaptation across a change between the two stimuli in a pair provides evidence for a common neural representation invariant to that change. We found adaptation in the LOC when perceived shape was identical but contours differed, but not when contours were identical but perceived shape differed. These data indicate that the LOC represents not simple image features, but rather higher level shape information.

The midstream order deficit.

The relative order of an auditory sequence can be more difficult to apprehend when it is presented repeatedly without pause (i.e., cycling) than when it is presented only once (Warren, Obusek, Farmer, & Warren, 1969). We find that this phenomenon, referred to as the midstream order deficit (MOD), can also occur with visual stimuli. The stimuli need not form separate perceptual "streams," and the effect can occur with presentation rates as slow as five items per second, even though the identification of individual letters is very accurate at this rate. However, if the first item of the sequence is visually very distinct from the preceding items, relative order reports can be as accurate in the cycling condition as in the single-presentation condition. Our results suggest that the MOD is not due to masking, attentional blink, repetition blindness, Reeves and Sperling's (1986) order illusion, memory limitations, or decision criteria. The MOD may reflect an attentional cost to the initiation of order encoding, which is distinct from the allocation of attention is required in order to detect and identify individual items. To initiate order encoding successfully, one's attention must be set for, or captured by, an initial salient event.

The lateral occipital complex and its role in object recognition.

Here we review recent findings that reveal the functional properties of extra-striate regions in the human visual cortex that are involved in the representation and perception of objects. We characterize both the invariant and non-invariant properties of these regions and we discuss the correlation between activation of these regions and recognition. Overall, these results indicate that the lateral occipital complex plays an important role in human object recognition.

Neuroimaging of cognitive functions in human parietal cortex.

Functional neuroimaging has proven highly valuable in mapping human sensory regions, particularly visual areas in occipital cortex. Recent evidence suggests that human parietal cortex may also consist of numerous specialized subregions similar to those reported in neurophysiological studies of non-human primates. However, parietal activation generalizes across a wide variety of cognitive tasks and the extension of human brain mapping into higher-order "association cortex" may prove to be a challenge.

Neural events and perceptual awareness.

Neural correlates of perceptual awareness, until very recently an elusive quarry, are now almost commonplace findings. This article first describes a variety of neural correlates of perceptual awareness based on fMRI, ERPs, and single-unit recordings. It is then argued that our quest should ultimately focus not on mere correlates of awareness, but rather on the neural events that are both necessary and sufficient for perceptual awareness. Indeed, preliminary evidence suggests that although many of the neural correlates already reported may be necessary for the corresponding state of awareness, it is unlikely that they are sufficient for it. The final section considers three hypotheses concerning the possible sufficiency conditions for perceptual awareness.

Neuropsychological evidence for a topographical learning mechanism in parahippocampal cortex.

The Parahippocampal Place Area (PPA; Epstein & Kanwisher, 1998) is a region within posterior parahippocampal cortex that responds selectively to visual stimuli that convey information about the layout of local space. Here we describe two patients who suffered damage to the PPA after vascular incidents. Both subsequently exhibited memory problems for topographical materials and were unable to navigate unassisted in unfamiliar environments. Performance on a continuous n-back visual memory test was significantly lower for novel scene-like stimuli than for novel object-like stimuli. In contrast, performance was normal on a famous landmark recognition task and on two perceptual tasks that required on-line analysis of scene geometry. Both patients were able to produce accurate maps of premorbidly learned places but were unable to produce accurate maps of new places. These results converge with previous neuroimaging work to demonstrate that the PPA (1) is selectively involved in processing information about the geometry of surrounding space, and (2) may play a more critical role in the encoding of this information into memory than in the initial perceptual processing, recognition, or recall of this information.

Activation in human MT/MST by static images with implied motion.

A still photograph of an object in motion may convey dynamic information about the position of the object immediately before and after the photograph was taken (implied motion). Medial temporal/medial superior temporal cortex (MT/MST) is one of the main brain regions engaged in the perceptual analysis of visual motion. In two experiments we examined whether MT/MST is also involved in representing implied motion from static images. We found stronger functional magnetic resonance imaging (fMRI) activation within MT/MST during viewing of static photographs with implied motion compared to viewing of photographs without implied motion. These results suggest that brain regions involved in the visual analysis of motion are also engaged in processing implied dynamic information from static images.