Optimizing multivariate pattern classification in rapid event-related designs.

BACKGROUND: Multivariate pattern analysis (MVPA or pattern decoding) has attracted considerable attention as a sensitive analytic tool for investigations using functional magnetic resonance imaging (fMRI) data. With the introduction of MVPA, however, has come a proliferation of methodological choices confronting the researcher, with few studies to date offering guidance from the vantage point of controlled datasets detached from specific experimental hypotheses. NEW METHOD: We investigated the impact of four data processing steps on support vector machine (SVM) classification performance aimed at maximizing information capture in the presence of common noise sources. The four techniques included: trial averaging (classifying on separate trial estimates versus condition-based averages), within-run mean centering (centering the data or not), method of cost selection (using a fixed or tuned cost value), and motion-related denoising approach (comparing no denoising versus a variety of nuisance regressions capturing motion-related reference signals). The impact of these approaches was evaluated on real fMRI data from two control ROIs, as well as on simulated pattern data constructed with carefully controlled voxel- and trial-level noise components. RESULTS: We find significant improvements in classification performance across both real and simulated datasets with run-wise trial averaging and mean centering. When averaging trials within conditions of each run, we note a simultaneous increase in the between-subject variability of SVM classification accuracies which we attribute to the reduced size of the test set used to assess the classifier's prediction error. Therefore, we propose a hybrid technique whereby randomly sampled subsets of trials are averaged per run and demonstrate that it helps mitigate the tradeoff between improving signal-to-noise ratio by averaging and losing exemplars in the test set. COMPARISON WITH EXISTING METHODS: Though a handful of empirical studies have employed run-based trial averaging, mean centering, or their combination, such studies have done so without theoretical justification or rigorous testing using control ROIs. CONCLUSIONS: Therefore, we intend this study to serve as a practical guide for researchers wishing to optimize pattern decoding without risk of introducing spurious results.

Improving the accuracy of single-trial fMRI response estimates using GLMsingle.

Advances in artificial intelligence have inspired a paradigm shift in human neuroscience, yielding large-scale functional magnetic resonance imaging (fMRI) datasets that provide high-resolution brain responses to thousands of naturalistic visual stimuli. Because such experiments necessarily involve brief stimulus durations and few repetitions of each stimulus, achieving sufficient signal-to-noise ratio can be a major challenge. We address this challenge by introducing GLMsingle, a scalable, user-friendly toolbox available in MATLAB and Python that enables accurate estimation of single-trial fMRI responses (glmsingle.org). Requiring only fMRI time-series data and a design matrix as inputs, GLMsingle integrates three techniques for improving the accuracy of trial-wise general linear model (GLM) beta estimates. First, for each voxel, a custom hemodynamic response function (HRF) is identified from a library of candidate functions. Second, cross-validation is used to derive a set of noise regressors from voxels unrelated to the experiment. Third, to improve the stability of beta estimates for closely spaced trials, betas are regularized on a voxel-wise basis using ridge regression. Applying GLMsingle to the Natural Scenes Dataset and BOLD5000, we find that GLMsingle substantially improves the reliability of beta estimates across visually-responsive cortex in all subjects. Comparable improvements in reliability are also observed in a smaller-scale auditory dataset from the StudyForrest experiment. These improvements translate into tangible benefits for higher-level analyses relevant to systems and cognitive neuroscience. We demonstrate that GLMsingle: (i) helps decorrelate response estimates between trials nearby in time; (ii) enhances representational similarity between subjects within and across datasets; and (iii) boosts one-versus-many decoding of visual stimuli. GLMsingle is a publicly available tool that can significantly improve the quality of past, present, and future neuroimaging datasets sampling brain activity across many experimental conditions.

Top-Down Attention Guidance Shapes Action Encoding in the pSTS.

The posterior superior temporal sulcus (pSTS) is a brain region characterized by perceptual representations of human body actions that promote the understanding of observed behavior. Increasingly, action observation is recognized as being strongly shaped by the expectations of the observer (Kilner 2011; Koster-Hale and Saxe 2013; Patel et al. 2019). Therefore, to characterize top-down influences on action observation, we evaluated the statistical structure of multivariate activation patterns from the action observation network (AON) while observers attended to the different dimensions of action vignettes (the action kinematics, goal, or identity of avatars jumping or crouching). Decoding accuracy varied as a function of attention instruction in the right pSTS and left inferior frontal cortex (IFC), with the right pSTS classifying actions most accurately when observers attended to the action kinematics and the left IFC classifying most accurately when observed attended to the actor's goal. Functional connectivity also increased between the right pSTS and right IFC when observers attended to the actions portrayed in the vignettes. Our findings are evidence that the attentive state of the viewer modulates sensory representations in the pSTS, consistent with proposals that the pSTS occupies an interstitial zone mediating top-down context and bottom-up perceptual cues during action observation.

The relative contributions of visual and semantic information in the neural representation of object categories.

INTRODUCTION: How do multiple sources of information interact to form mental representations of object categories? It is commonly held that object categories reflect the integration of perceptual features and semantic/knowledge-based features. To explore the relative contributions of these two sources of information, we used functional magnetic resonance imaging (fMRI) to identify regions involved in the representation object categories with shared visual and/or semantic features. METHODS: Participants (N = 20) viewed a series of objects that varied in their degree of visual and semantic overlap in the MRI scanner. We used a blocked adaptation design to identify sensitivity to visual and semantic features in a priori visual processing regions and in a distributed network of object processing regions with an exploratory whole-brain analysis. RESULTS: Somewhat surprisingly, within higher-order visual processing regions-specifically lateral occipital cortex (LOC)-we did not obtain any difference in neural adaptation for shared visual versus semantic category membership. More broadly, both visual and semantic information affected a distributed network of independently identified category-selective regions. Adaptation was seen a whole-brain network of processing regions in response to visual similarity and semantic similarity; specifically, the angular gyrus (AnG) adapted to visual similarity and the dorsomedial prefrontal cortex (DMPFC) adapted to both visual and semantic similarity. CONCLUSIONS: Our findings suggest that perceptual features help organize mental categories throughout the object processing hierarchy. Most notably, visual similarity also influenced adaptation in nonvisual brain regions (i.e., AnG and DMPFC). We conclude that category-relevant visual features are maintained in higher-order conceptual representations and visual information plays an important role in both the acquisition and neural representation of conceptual object categories.

BOLD5000, a public fMRI dataset while viewing 5000 visual images.

Vision science, particularly machine vision, has been revolutionized by introducing large-scale image datasets and statistical learning approaches. Yet, human neuroimaging studies of visual perception still rely on small numbers of images (around 100) due to time-constrained experimental procedures. To apply statistical learning approaches that include neuroscience, the number of images used in neuroimaging must be significantly increased. We present BOLD5000, a human functional MRI (fMRI) study that includes almost 5,000 distinct images depicting real-world scenes. Beyond dramatically increasing image dataset size relative to prior fMRI studies, BOLD5000 also accounts for image diversity, overlapping with standard computer vision datasets by incorporating images from the Scene UNderstanding (SUN), Common Objects in Context (COCO), and ImageNet datasets. The scale and diversity of these image datasets, combined with a slow event-related fMRI design, enables fine-grained exploration into the neural representation of a wide range of visual features, categories, and semantics. Concurrently, BOLD5000 brings us closer to realizing Marr's dream of a singular vision science-the intertwined study of biological and computer vision.

Successful Reorganization of Category-Selective Visual Cortex following Occipito-temporal Lobectomy in Childhood.

Investigations of functional (re)organization in children who have undergone large cortical resections offer a unique opportunity to elucidate the nature and extent of cortical plasticity. We report findings from a 3-year investigation of a child, U.D., who underwent surgical removal of the right occipital and posterior temporal lobes at age 6 years 9 months. Relative to controls, post-surgically, U.D. showed age-appropriate intellectual performance and visuoperceptual face and object recognition skills. Using fMRI at five different time points, we observed a persistent hemianopia and no visual field remapping. In category-selective visual cortices, however, object- and scene-selective regions in the intact left hemisphere were stable early on, but regions subserving face and word recognition emerged later and evinced competition for cortical representation. These findings reveal alterations in the selectivity and topography of category-selective regions when confined to a single hemisphere and provide insights into dynamic functional changes in extrastriate cortical architecture.

Who's that Knocking at My Door? Neural Bases of Sound Source Identification.

When hearing knocking on a door, a listener typically identifies both the action (forceful and repeated impacts) and the object (a thick wooden board) causing the sound. The current work studied the neural bases of sound source identification by switching listeners' attention toward these different aspects of a set of simple sounds during functional magnetic resonance imaging scanning: participants either discriminated the action or the material that caused the sounds, or they simply discriminated meaningless scrambled versions of them. Overall, discriminating action and material elicited neural activity in a left-lateralized frontoparietal network found in other studies of sound identification, wherein the inferior frontal sulcus and the ventral premotor cortex were under the control of selective attention and sensitive to task demand. More strikingly, discriminating materials elicited increased activity in cortical regions connecting auditory inputs to semantic, motor, and even visual representations, whereas discriminating actions did not increase activity in any regions. These results indicate that discriminating and identifying material requires deeper processing of the stimuli than discriminating actions. These results are consistent with previous studies suggesting that auditory perception is better suited to comprehend the actions than the objects producing sounds in the listeners' environment.

Exploring the spatio-temporal neural basis of face learning.

Humans are experts at face individuation. Although previous work has identified a network of face-sensitive regions and some of the temporal signatures of face processing, as yet, we do not have a clear understanding of how such face-sensitive regions support learning at different time points. To study the joint spatio-temporal neural basis of face learning, we trained subjects to categorize two groups of novel faces and recorded their neural responses using magnetoencephalography (MEG) throughout learning. A regression analysis of neural responses in face-sensitive regions against behavioral learning curves revealed significant correlations with learning in the majority of the face-sensitive regions in the face network, mostly between 150-250 ms, but also after 300 ms. However, the effect was smaller in nonventral regions (within the superior temporal areas and prefrontal cortex) than that in the ventral regions (within the inferior occipital gyri (IOG), midfusiform gyri (mFUS) and anterior temporal lobes). A multivariate discriminant analysis also revealed that IOG and mFUS, which showed strong correlation effects with learning, exhibited significant discriminability between the two face categories at different time points both between 150-250 ms and after 300 ms. In contrast, the nonventral face-sensitive regions, where correlation effects with learning were smaller, did exhibit some significant discriminability, but mainly after 300 ms. In sum, our findings indicate that early and recurring temporal components arising from ventral face-sensitive regions are critically involved in learning new faces.

Associative hallucinations result from stimulating left ventromedial temporal cortex.

Visual recognition requires connecting perceptual information with contextual information and existing knowledge. The ventromedial temporal cortex (VTC), including the medial fusiform, has been linked with object recognition, paired associate learning, contextual processing, and episodic memory, suggesting that this area may be critical in connecting visual processing, context, knowledge and experience. However, evidence for the link between associative processing, episodic memory, and visual recognition in VTC is currently lacking. Using electrocorticography (ECoG) in a single human patient, medial regions of the left VTC were found to be sensitive to the contextual associations of objects. Electrical brain stimulation (EBS) of this part of the left VTC of the patient, functionally defined as sensitive to associative processing, caused memory related, associative experiential visual phenomena. This provides evidence of a relationship between visual recognition, associative processing, and episodic memory. These results suggest a potential role for abnormalities of these processes as part of a mechanism that gives rise to some visual hallucinations.

Dynamic encoding of face information in the human fusiform gyrus.

Humans' ability to rapidly and accurately detect, identify and classify faces under variable conditions derives from a network of brain regions highly tuned to face information. The fusiform face area (FFA) is thought to be a computational hub for face processing; however, temporal dynamics of face information processing in FFA remains unclear. Here we use multivariate pattern classification to decode the temporal dynamics of expression-invariant face information processing using electrodes placed directly on FFA in humans. Early FFA activity (50-75 ms) contained information regarding whether participants were viewing a face. Activity between 200 and 500 ms contained expression-invariant information about which of 70 faces participants were viewing along with the individual differences in facial features and their configurations. Long-lasting (500+ms) broadband gamma frequency activity predicted task performance. These results elucidate the dynamic computational role FFA plays in multiple face processing stages and indicate what information is used in performing these visual analyses.

Exploration of complex visual feature spaces for object perception.

The mid- and high-level visual properties supporting object perception in the ventral visual pathway are poorly understood. In the absence of well-specified theory, many groups have adopted a data-driven approach in which they progressively interrogate neural units to establish each unit's selectivity. Such methods are challenging in that they require search through a wide space of feature models and stimuli using a limited number of samples. To more rapidly identify higher-level features underlying human cortical object perception, we implemented a novel functional magnetic resonance imaging method in which visual stimuli are selected in real-time based on BOLD responses to recently shown stimuli. This work was inspired by earlier primate physiology work, in which neural selectivity for mid-level features in IT was characterized using a simple parametric approach (Hung et al., 2012). To extend such work to human neuroimaging, we used natural and synthetic object stimuli embedded in feature spaces constructed on the basis of the complex visual properties of the objects themselves. During fMRI scanning, we employed a real-time search method to control continuous stimulus selection within each image space. This search was designed to maximize neural responses across a pre-determined 1 cm(3) brain region within ventral cortex. To assess the value of this method for understanding object encoding, we examined both the behavior of the method itself and the complex visual properties the method identified as reliably activating selected brain regions. We observed: (1) Regions selective for both holistic and component object features and for a variety of surface properties; (2) Object stimulus pairs near one another in feature space that produce responses at the opposite extremes of the measured activity range. Together, these results suggest that real-time fMRI methods may yield more widely informative measures of selectivity within the broad classes of visual features associated with cortical object representation.

Comparing visual representations across human fMRI and computational vision.

Feedforward visual object perception recruits a cortical network that is assumed to be hierarchical, progressing from basic visual features to complete object representations. However, the nature of the intermediate features related to this transformation remains poorly understood. Here, we explore how well different computer vision recognition models account for neural object encoding across the human cortical visual pathway as measured using fMRI. These neural data, collected during the viewing of 60 images of real-world objects, were analyzed with a searchlight procedure as in Kriegeskorte, Goebel, and Bandettini (2006): Within each searchlight sphere, the obtained patterns of neural activity for all 60 objects were compared to model responses for each computer recognition algorithm using representational dissimilarity analysis (Kriegeskorte et al., 2008). Although each of the computer vision methods significantly accounted for some of the neural data, among the different models, the scale invariant feature transform (Lowe, 2004), encoding local visual properties gathered from "interest points," was best able to accurately and consistently account for stimulus representations within the ventral pathway. More generally, when present, significance was observed in regions of the ventral-temporal cortex associated with intermediate-level object perception. Differences in model effectiveness and the neural location of significant matches may be attributable to the fact that each model implements a different featural basis for representing objects (e.g., more holistic or more parts-based). Overall, we conclude that well-known computer vision recognition systems may serve as viable proxies for theories of intermediate visual object representation.

Fine-grained temporal coding of visually-similar categories in the ventral visual pathway and prefrontal cortex.

Humans are remarkably proficient at categorizing visually-similar objects. To better understand the cortical basis of this categorization process, we used magnetoencephalography (MEG) to record neural activity while participants learned-with feedback-to discriminate two highly-similar, novel visual categories. We hypothesized that although prefrontal regions would mediate early category learning, this role would diminish with increasing category familiarity and that regions within the ventral visual pathway would come to play a more prominent role in encoding category-relevant information as learning progressed. Early in learning we observed some degree of categorical discriminability and predictability in both prefrontal cortex and the ventral visual pathway. Predictability improved significantly above chance in the ventral visual pathway over the course of learning with the left inferior temporal and fusiform gyri showing the greatest improvement in predictability between 150 and 250 ms (M200) during category learning. In contrast, there was no comparable increase in discriminability in prefrontal cortex with the only significant post-learning effect being a decrease in predictability in the inferior frontal gyrus between 250 and 350 ms (M300). Thus, the ventral visual pathway appears to encode learned visual categories over the long term. At the same time these results add to our understanding of the cortical origins of previously reported signature temporal components associated with perceptual learning.

Explicating the face perception network with white matter connectivity.

A network of multiple brain regions is recruited in face perception. Our understanding of the functional properties of this network can be facilitated by explicating the structural white matter connections that exist between its functional nodes. We accomplished this using functional MRI (fMRI) in combination with fiber tractography on high angular resolution diffusion weighted imaging data. We identified the three nodes of the core face network: the "occipital face area" (OFA), the "fusiform face area" (mid-fusiform gyrus or mFus), and the superior temporal sulcus (STS). Additionally, a region of the anterior temporal lobe (aIT), implicated as being important for face perception was identified. Our data suggest that we can further divide the OFA into multiple anatomically distinct clusters - a partitioning consistent with several recent neuroimaging results. More generally, structural white matter connectivity within this network revealed: 1) Connectivity between aIT and mFus, and between aIT and occipital regions, consistent with studies implicating this posterior to anterior pathway as critical to normal face processing; 2) Strong connectivity between mFus and each of the occipital face-selective regions, suggesting that these three areas may subserve different functional roles; 3) Almost no connectivity between STS and mFus, or between STS and the other face-selective regions. Overall, our findings suggest a re-evaluation of the "core" face network with respect to what functional areas are or are not included in this network.

Co-analysis of brain structure and function using fMRI and diffusion-weighted imaging.

The study of complex computational systems is facilitated by network maps, such as circuit diagrams. Such mapping is particularly informative when studying the brain, as the functional role that a brain area fulfills may be largely defined by its connections to other brain areas. In this report, we describe a novel, non-invasive approach for relating brain structure and function using magnetic resonance imaging (MRI). This approach, a combination of structural imaging of long-range fiber connections and functional imaging data, is illustrated in two distinct cognitive domains, visual attention and face perception. Structural imaging is performed with diffusion-weighted imaging (DWI) and fiber tractography, which track the diffusion of water molecules along white-matter fiber tracts in the brain (Figure 1). By visualizing these fiber tracts, we are able to investigate the long-range connective architecture of the brain. The results compare favorably with one of the most widely-used techniques in DWI, diffusion tensor imaging (DTI). DTI is unable to resolve complex configurations of fiber tracts, limiting its utility for constructing detailed, anatomically-informed models of brain function. In contrast, our analyses reproduce known neuroanatomy with precision and accuracy. This advantage is partly due to data acquisition procedures: while many DTI protocols measure diffusion in a small number of directions (e.g., 6 or 12), we employ a diffusion spectrum imaging (DSI)(1, 2) protocol which assesses diffusion in 257 directions and at a range of magnetic gradient strengths. Moreover, DSI data allow us to use more sophisticated methods for reconstructing acquired data. In two experiments (visual attention and face perception), tractography reveals that co-active areas of the human brain are anatomically connected, supporting extant hypotheses that they form functional networks. DWI allows us to create a "circuit diagram" and reproduce it on an individual-subject basis, for the purpose of monitoring task-relevant brain activity in networks of interest.

Neurovirulence of H5N1 infection in ferrets is mediated by multifocal replication in distinct permissive neuronal cell regions.

Highly pathogenic avian influenza A (HPAI), subtype H5N1, remains an emergent threat to the human population. While respiratory disease is a hallmark of influenza infection, H5N1 has a high incidence of neurological sequelae in many animal species and sporadically in humans. We elucidate the temporal/spatial infection of H5N1 in the brain of ferrets following a low dose, intranasal infection of two HPAI strains of varying neurovirulence and lethality. A/Vietnam/1203/2004 (VN1203) induced mortality in 100% of infected ferrets while A/Hong Kong/483/1997 (HK483) induced lethality in only 20% of ferrets, with death occurring significantly later following infection. Neurological signs were prominent in VN1203 infection, but not HK483, with seizures observed three days post challenge and torticollis or paresis at later time points. VN1203 and HK483 replication kinetics were similar in primary differentiated ferret nasal turbinate cells, and similar viral titers were measured in the nasal turbinates of infected ferrets. Pulmonary viral titers were not different between strains and pathological findings in the lungs were similar in severity. VN1203 replicated to high titers in the olfactory bulb, cerebral cortex, and brain stem; whereas HK483 was not recovered in these tissues. VN1203 was identified adjacent to and within the olfactory nerve tract, and multifocal infection was observed throughout the frontal cortex and cerebrum. VN1203 was also detected throughout the cerebellum, specifically in Purkinje cells and regions that coordinate voluntary movements. These findings suggest the increased lethality of VN1203 in ferrets is due to increased replication in brain regions important in higher order function and explains the neurological signs observed during H5N1 neurovirulence.

Stimulus complexity modulates contrast response functions in the human middle temporal area (hMT+).

The brain systems that support motion perception are some of the most studied in the primate visual system, with apparent specialization in the middle temporal area (hMT+ in humans, MT or V5 in monkeys). Even with this specialization, it is safe to assume that the hMT+ interacts with other brain systems as visual tasks demand. Here we have measured those interactions using a specialized case of structure-from-motion, point-light biological motion. We have measured the BOLD-contrast response functions in hMT+ for translating and biological motion. Even after controlling for task and attention, we find the BOLD response for translating motion to be largely insensitive to contrast, but the BOLD response for biological motion to be strongly contrast dependent. To track the brain systems involved in these interactions, we probed for brain areas outside of the hMT+ with the same contrast dependent neural response. This analysis revealed brain systems known to support form perception (including ventral temporal cortex and the superior temporal sulcus). We conclude that the contrast dependent response in hMT+ likely reflects stimulus complexity, and may be evidence for interactions with shape-based brain systems.

fMR-Adaptation Reveals Invariant Coding of Biological Motion on the Human STS.

Neuroimaging studies of biological motion perception have found a network of coordinated brain areas, the hub of which appears to be the human posterior superior temporal sulcus (STSp). Understanding the functional role of the STSp requires characterizing the response tuning of neuronal populations underlying the BOLD response. Thus far our understanding of these response properties comes from single-unit studies of the monkey anterior STS, which has individual neurons tuned to body actions, with a small population invariant to changes in viewpoint, position and size of the action being viewed. To measure for homologous functional properties on the human STS, we used fMR-adaptation to investigate action, position and size invariance. Observers viewed pairs of point-light animations depicting human actions that were either identical, differed in the action depicted, locally scrambled, or differed in the viewing perspective, the position or the size. While extrastriate hMT+ had neural signals indicative of viewpoint specificity, the human STS adapted for all of these changes, as compared to viewing two different actions. Similar findings were observed in more posterior brain areas also implicated in action recognition. Our findings are evidence for viewpoint invariance in the human STS and related brain areas, with the implication that actions are abstracted into object-centered representations during visual analysis.

Neural adaptation for novel objects during dynamic articulation.

Human observers readily identify objects with moving parts, and recognize their underlying structure even when the component parts undergo complex movement. This suggests the existence of neural representations that are invariant to motion and state of articulation, which together allow our visual system to maintain 'object constancy'. Ventral temporal cortex has previously been implicated in object perception and in coding object identity, but it is unclear where this is achieved when objects undergo motion-driven shape changes. In the present study, we use fMRI adaptation to probe the neural response properties when subjects view dynamic novel objects. Our results reveal neural selectivity for novel objects in the LOC region of the occipito-temporal lobe, even when those objects are viewed as moving and articulating. We also identify a bilateral area of posterior fusiform outside of the LOC with neural populations invariant to changes in the articulatory state of an object, a critical feature of object constancy. These results demonstrate the functional importance of ventral temporal cortex in the perception of moving objects, and the existence of neural populations coding for object constancy across movement and articulation.

Early and sustained innate immune response defines pathology and death in nonhuman primates infected by highly pathogenic influenza virus.

The mechanisms responsible for the virulence of the highly pathogenic avian influenza (HPAI) and of the 1918 pandemic influenza virus in humans remain poorly understood. To identify crucial components of the early host response during these infections by using both conventional and functional genomics tools, we studied 34 cynomolgus macaques (Macaca fascicularis) to compare a 2004 human H5N1 Vietnam isolate with 2 reassortant viruses possessing the 1918 hemagglutinin (HA) and neuraminidase (NA) surface proteins, known conveyors of virulence. One of the reassortants also contained the 1918 nonstructural (NS1) protein, an inhibitor of the host interferon response. Among these viruses, HPAI H5N1 was the most virulent. Within 24 h, the H5N1 virus produced severe bronchiolar and alveolar lesions. Notably, the H5N1 virus targeted type II pneumocytes throughout the 7-day infection, and induced the most dramatic and sustained expression of type I interferons and inflammatory and innate immune genes, as measured by genomic and protein assays. The H5N1 infection also resulted in prolonged margination of circulating T lymphocytes and notable apoptosis of activated dendritic cells in the lungs and draining lymph nodes early during infection. While both 1918 reassortant viruses also were highly pathogenic, the H5N1 virus was exceptional for the extent of tissue damage, cytokinemia, and interference with immune regulatory mechanisms, which may help explain the extreme virulence of HPAI viruses in humans.