Decoding fMRI data with support vector machines and deep neural networks.

BACKGROUND: Multivoxel pattern analysis (MVPA) examines fMRI activation patterns associated with different cognitive conditions. Support vector machines (SVMs) are the predominant method in MVPA. While SVM is intuitive and easy to apply, it is mainly suitable for analyzing data that are linearly separable. Convolutional neural networks (CNNs) are known to have the ability to approximate nonlinear relationships. Applications of CNN to fMRI data are beginning to appear with increasing frequency, but our understanding of the similarities and differences between CNN models and SVM models is limited. NEW METHOD: We compared the two methods when they are applied to the same datasets. Two datasets were considered: (1) fMRI data collected from participants during a cued visual spatial attention task and (2) fMRI data collected from participants viewing natural images containing varying degrees of affective content. RESULTS: We found that (1) both SVM and CNN are able to achieve above-chance decoding accuracies for attention control and emotion processing in both the primary visual cortex and the whole brain, (2) the CNN decoding accuracies are consistently higher than that of the SVM, (3) the SVM and CNN decoding accuracies are generally not correlated, and (4) the heatmaps derived from SVM and CNN are not significantly overlapping. COMPARISON WITH EXISTING METHODS: By comparing SVM and CNN we pointed out the similarities and differences between the two methods. CONCLUSIONS: SVM and CNN rely on different neural features for classification. Applying both to the same data may yield a more comprehensive understanding of neuroimaging data.

Top-Down Biasing of Visual Cortical Activity Encodes Attended Information and Facilitates Behavioral Performance in Visual Spatial Attention.

Top-down attention plays a vital role in selecting relevant stimuli and suppressing distracting information. During top-down visual-spatial attention, control signals from the dorsal attention network modulate the baseline neuronal activity in the visual cortex in favor of task-relevant stimuli. While several studies have demonstrated that baseline shift during anticipatory attention occurs in multiple visual areas, such effects have not been systematically investigated across the visual hierarchy, especially when different attention conditions are matched for stimulus and task factors. In this fMRI study, we investigated anticipatory attention signals using univariate and multivariate (MVPA) analysis in multiple visual cortical areas. First, the univariate analysis yielded significant activation differences in higher-order visual areas, with the effect weaker in early visual areas. Second, however, in contrast, MVPA decoding was significant in predicting attention conditions in all visual areas and IPS, with lower-order visual areas (e.g., V1) having greater decoding accuracy than higher-order visual areas (e.g., LO1). Third, the strength of decoding accuracy predicted the behavioral performance in the discrimination task. All the results were highly replicable and consistent across two datasets with same experimental paradigms but recorded at two research sites, and two experimental conditions where the direction of spatial attention was driven either by external instructions (cue-instructed attention) or from internal decisions (free-choice attention). Our results provide clear evidence, not available in past univariate investigations, that top-down attentional control signals selectively bias neuronal processing throughout the visual hierarchy, and that this biasing is correlated with the task performance.

Decoding fMRI Data: A Comparison Between Support Vector Machines and Deep Neural Networks.

Multivoxel pattern analysis (MVPA) examines the differences in fMRI activation patterns associated with different cognitive conditions and provides information not possible with the conventional univariate analysis. Support vector machines (SVMs) are the predominant machine learning method in MVPA. SVMs are intuitive and easy to apply. The limitation is that it is a linear method and mainly suitable for analyzing data that are linearly separable. Convolutional neural networks (CNNs), a class of AI models originally developed for object recognition, are known to have the ability to approximate nonlinear relationships. CNNs are rapidly becoming an alternative to SVMs. The purpose of this study is to compare the two methods when they are applied to the same datasets. Two datasets were considered: (1) fMRI data collected from participants during a cued visual spatial attention task (the attention dataset) and (2) fMRI data collected from participants viewing natural images containing varying degrees of affective content (the emotion dataset). We found that (1) both SVM and CNN are able to achieve above chance level decoding accuracies for attention control and emotion processing in both the primary visual cortex and the whole brain with, (2) the CNN decoding accuracies are consistently higher than that of the SVM, (3) the SVM and CNN decoding accuracies are generally not correlated with each other, and (4) the heatmaps derived from SVM and CNN are not significantly overlapping. These results suggest that (1) there are both linearly separable features and nonlinearly separable features in fMRI data that distinguish cognitive conditions and (2) applying both SVM and CNN to the same data may yield a more comprehensive understanding of neuroimaging data. Key points: We compared the performance and characteristics of SVM and CNN, two major methods in MVPA analysis of neuroimaging data, by applying them to the same two fMRI datasets.Both SVM and CNN achieved decoding accuracies above chance level for both datasets in the chosen ROIs and the CNN decoding accuracies were consistently higher than those of SVM.The heatmaps derived from SVM and CNN, which assess the contribution of voxels or brain regions to MVPA decoding performance, showed no significant overlap, providing evidence that the two methods depend on distinct brain activity patterns for decoding cognitive conditions.

Imaging the neural substrate of trigeminal neuralgia pain using deep learning.

Trigeminal neuralgia (TN) is a severe and disabling facial pain condition and is characterized by intermittent, severe, electric shock-like pain in one (or more) trigeminal subdivisions. This pain can be triggered by an innocuous stimulus or can be spontaneous. Presently available therapies for TN include both surgical and pharmacological management; however, the lack of a known etiology for TN contributes to the unpredictable response to treatment and the variability in long-term clinical outcomes. Given this, a range of peripheral and central mechanisms underlying TN pain remain to be understood. We acquired functional magnetic resonance imaging (fMRI) data from TN patients who (1) rested comfortably in the scanner during a resting state session and (2) rated their pain levels in real time using a calibrated tracking ball-controlled scale in a pain tracking session. Following data acquisition, the data was analyzed using the conventional correlation analysis and two artificial intelligence (AI)-inspired deep learning methods: convolutional neural network (CNN) and graph convolutional neural network (GCNN). Each of the three methods yielded a set of brain regions related to the generation and perception of pain in TN. There were 6 regions that were identified by all three methods, including the superior temporal cortex, the insula, the fusiform, the precentral gyrus, the superior frontal gyrus, and the supramarginal gyrus. Additionally, 17 regions, including dorsal anterior cingulate cortex (dACC) and the thalamus, were identified by at least two of the three methods. Collectively, these 23 regions are taken to represent signature centers of TN pain and provide target areas for future studies seeking to understand the central mechanisms of TN.

Time Courses of Attended and Ignored Object Representations.

Selective attention prioritizes information that is relevant to behavioral goals. Previous studies have shown that attended visual information is processed and represented more efficiently, but distracting visual information is not fully suppressed, and may also continue to be represented in the brain. In natural vision, to-be-attended and to-be-ignored objects may be present simultaneously in the scene. Understanding precisely how each is represented in the visual system, and how these neural representations evolve over time, remains a key goal in cognitive neuroscience. In this study, we recorded EEG while participants performed a cued object-based attention task that involved attending to target objects and ignoring simultaneously presented and spatially overlapping distractor objects. We performed support vector machine classification on the stimulus-evoked EEG data to separately track the temporal dynamics of target and distractor representations. We found that (1) both target and distractor objects were decodable during the early phase of object processing (∼100 msec to ∼200 msec after target onset), and (2) the representations of both objects were sustained over time, remaining decodable above chance until ∼1000-msec latency. However, (3) the distractor object information faded significantly beginning after about 300-msec latency. These findings provide information about the fate of attended and ignored visual information in complex scene perception.

Top-down control of the left visual field bias in cued visual spatial attention.

A left visual field (LVF) bias in perceptual judgments, response speed, and discrimination accuracy has been reported in humans. Cognitive factors, such as visual spatial attention, are known to modulate or even eliminate this bias. We investigated this problem by recording pupillometry together with functional magnetic resonance imaging (fMRI) in a cued visual spatial attention task. We observed that (i) the pupil was significantly more dilated following attend-right than attend-left cues, (ii) the task performance (e.g. reaction time [RT]) did not differ between attend-left and attend-right trials, and (iii) the difference in cue-related pupil dilation between attend-left and attend-right trials was inversely related to the corresponding difference in RT. Neuroscientically, correlating the difference in cue-related pupil dilation with the corresponding cue-related fMRI difference yielded activations primarily in the right hemisphere, including the right intraparietal sulcus and the right ventrolateral prefrontal cortex. These results suggest that (i) there is an asymmetry in visual spatial attention control, with the rightward attention control being more effortful than the leftward attention control, (ii) this asymmetry underlies the reduction or the elimination of the LVF bias, and (iii) the components of the attentional control networks in the right hemisphere are likely part of the neural substrate of the observed asymmetry in attentional control.

Aftereffects of frontoparietal theta tACS on verbal working memory: Behavioral and neurophysiological analysis.

Verbal working memory is supported by a left-lateralized frontoparietal theta oscillatory (4-8 Hz) network. We tested whether stimulating the left frontoparietal network at theta frequency during verbal working memory can produce observable after-stimulation effects in behavior and neurophysiology. Weak theta-band alternating electric currents were delivered via two 4 × 1 HD electrode arrays centered at F3 and P3. Three stimulation configurations, including in-phase, anti-phase, or sham, were tested on three different days in a cross-over (within-subject) design. On each test day, the subject underwent three experimental sessions: pre-, during- and post-stimulation sessions. In all sessions, the subject performed a Sternberg verbal working memory task with three levels of memory load (load 2, 4 and 6), imposing three levels of cognitive demand. Analyzing behavioral and EEG data from the post-stimulation session, we report two main observations. First, in-phase stimulation improved task performance in subjects with higher working memory capacity (WMC) under higher memory load (load 6). Second, in-phase stimulation enhanced frontoparietal theta synchrony during working memory retention in subjects with higher WMC under higher memory loads (load 4 and load 6), and the enhanced frontoparietal theta synchronization is mainly driven by enhanced frontal→parietal theta Granger causality. These observations suggest that (1) in-phase theta transcranial alternating current stimulation (tACS) during verbal working memory can result in observable behavioral and neurophysiological consequences post stimulation, (2) the short-term plasticity effects are state- and individual-dependent, and (3) enhanced executive control underlies improved behavioral performance.

Anticipatory attention is a stable state induced by transient control mechanisms.

Anticipatory attention is a neurocognitive state in which attention control regions bias neural activity in sensory cortical areas to facilitate the selective processing of incoming targets. Previous electroencephalographic (EEG) studies have identified event-related potential (ERP) signatures of anticipatory attention, and implicated alpha band (8-12 Hz) EEG oscillatory activity in the selective control of neural excitability in visual cortex. However, the degree to which ERP and alpha band measures reflect related or distinct underlying neural processes remains to be further understood. To investigate this question, we analyzed EEG data from 20 human participants performing a cued object-based attention task. We used support vector machine (SVM) decoding analysis to compare the attentional time courses of ERP signals and alpha band power. We found that ERP signals encoding attentional instructions are dynamic and precede stable attention-related changes in alpha power, suggesting that ERP and alpha power reflect distinct neural processes. We proposed that the ERP patterns reflect transient attentional orienting signals originating in higher order control areas, whereas the patterns of synchronized oscillatory neural activity in the alpha band reflect a sustained attentional state. These findings support the hypothesis that anticipatory attention involves transient top-down control signals that establish more stable neural states in visual cortex, enabling selective sensory processing.

Role of Inferior Frontal Junction (IFJ) in the Control of Feature versus Spatial Attention.

Feature-based visual attention refers to preferential selection and processing of visual stimuli based on their nonspatial attributes, such as color or shape. Recent studies have highlighted the inferior frontal junction (IFJ) as a control region for feature but not spatial attention. However, the extent to which IFJ contributes to spatial versus feature attention control remains a topic of debate. We investigated in humans of both sexes the role of IFJ in the control of feature versus spatial attention in a cued visual spatial (attend-left or attend-right) and feature (attend-red or attend-green) attention task using fMRI. Analyzing cue-related fMRI using both univariate activation and multivoxel pattern analysis, we found the following results in IFJ. First, in line with some prior studies, the univariate activations were not different between feature and spatial attentional control. Second, in contrast, the multivoxel pattern analysis decoding accuracy was above chance level for feature attention (attend-red vs attend-green) but not for spatial attention (attend-left vs attend-right). Third, while the decoding accuracy for feature attention was above chance level during attentional control in the cue-to-target interval, it was not during target processing. Fourth, the right IFJ and visual cortex (V4) were observed to be functionally connected during feature but not during spatial attention control, and this functional connectivity was positively associated with subsequent attentional selection of targets in V4, as well as with behavioral performance. These results support a model in which IFJ plays a crucial role in top-down control of visual feature but not visual spatial attention.SIGNIFICANCE STATEMENT Past work has shown that the inferior frontal junction (IFJ), a prefrontal structure, is activated by both attention-to-feature (e.g., color) and attention-to-location, but the precise role of IFJ in the control of feature- versus spatial-attention is debated. We investigated this issue in a cued visual spatial (attend-left or attend-right) and feature (attend-red or attend-green) attention task using fMRI, multivoxel pattern analysis, and functional connectivity methods. The results show that (1) attend-red versus attend-green can be decoded in single-trial cue-evoked BOLD activity in IFJ but not attend-left versus attend-right and (2) only right IFJ modulates V4 to enhance task performance. This study sheds light on the function and hemispheric specialization of IFJ in the control of visual attention.

The Microstructure of Attentional Control in the Dorsal Attention Network.

The top-down control of attention involves command signals arising chiefly in the dorsal attention network (DAN) in frontal and parietal cortex and propagating to sensory cortex to enable the selective processing of incoming stimuli based on their behavioral relevance. Consistent with this view, the DAN is active during preparatory (anticipatory) attention for relevant events and objects, which, in vision, may be defined by different stimulus attributes including their spatial location, color, motion, or form. How this network is organized to support different forms of preparatory attention to different stimulus attributes remains unclear. We propose that, within the DAN, there exist functional microstructures (patterns of activity) specific for controlling attention based on the specific information to be attended. To test this, we contrasted preparatory attention to stimulus location (spatial attention) and to stimulus color (feature attention), and used multivoxel pattern analysis to characterize the corresponding patterns of activity within the DAN. We observed different multivoxel patterns of BOLD activation within the DAN for the control of spatial attention (attending left vs. right) and feature attention (attending red vs. green). These patterns of activity for spatial and feature attentional control showed limited overlap with each other within the DAN. Our findings thus support a model in which the DAN has different functional microstructures for distinctive forms of top-down control of visual attention.

Decoding Neural Representations of Affective Scenes in Retinotopic Visual Cortex.

The perception of opportunities and threats in complex visual scenes represents one of the main functions of the human visual system. The underlying neurophysiology is often studied by having observers view pictures varying in affective content. It has been shown that viewing emotionally engaging, compared with neutral, pictures (1) heightens blood flow in limbic, frontoparietal, and anterior visual structures and (2) enhances the late positive event-related potential (LPP). The role of retinotopic visual cortex in this process has, however, been contentious, with competing theories predicting the presence versus absence of emotion-specific signals in retinotopic visual areas. Recording simultaneous electroencephalography-functional magnetic resonance imaging while observers viewed pleasant, unpleasant, and neutral affective pictures, and applying multivariate pattern analysis, we found that (1) unpleasant versus neutral and pleasant versus neutral decoding accuracy were well above chance level in retinotopic visual areas, (2) decoding accuracy in ventral visual cortex (VVC), but not in early or dorsal visual cortex, was correlated with LPP, and (3) effective connectivity from amygdala to VVC predicted unpleasant versus neutral decoding accuracy, whereas effective connectivity from ventral frontal cortex to VVC predicted pleasant versus neutral decoding accuracy. These results suggest that affective scenes evoke valence-specific neural representations in retinotopic visual cortex and that these representations are influenced by reentry signals from anterior brain regions.

Decoding attention control and selection in visual spatial attention.

Event-related potentials (ERPs) are used extensively to investigate the neural mechanisms of attention control and selection. The univariate ERP approach, however, has left important questions inadequately answered. We addressed two questions by applying multivariate pattern classification to multichannel ERPs in two cued visual spatial attention experiments (N = 56): (a) impact of cueing strategies (instructional vs. probabilistic) on attention control and selection and (b) neural and behavioral effects of individual differences. Following cue onset, the decoding accuracy (cue left vs. cue right) began to rise above chance level earlier and remained higher in instructional cueing (~80 ms) than in probabilistic cueing (~160 ms), suggesting that unilateral attention focus leads to earlier and more distinct formation of the attention control set. A similar temporal sequence was also found for target-related processing (cued target vs. uncued target), suggesting earlier and stronger attention selection under instructional cueing. Across the two experiments: (a) individuals with higher cue-related decoding accuracy showed higher magnitude of attentional modulation of target-evoked N1 amplitude, suggesting that better formation of anticipatory attentional state leads to stronger modulation of target processing, and (b) individuals with higher target-related decoding accuracy showed faster reaction times (or larger cueing effects), suggesting that stronger selection of task-relevant information leads to better behavioral performance. Taken together, multichannel ERPs combined with machine learning decoding yields new insights into attention control and selection that complement the univariate ERP approach, and along with the univariate ERP approach, provides a more comprehensive methodology to the study of visual spatial attention.

Neural mechanisms of internal distraction suppression in visual attention.

When performing a demanding cognitive task, internal distraction in the form of task-irrelevant thoughts and mind wandering can shift our attention away from the task, negatively affecting task performance. Behaviorally, individuals with higher executive function indexed by higher working memory capacity (WMC) exhibit less mind wandering during cognitive tasks, but the underlying neural mechanisms are unknown. To address this problem, we recorded functional magnetic resonance imaging (fMRI) data from subjects performing a cued visual attention task, and assessed their WMC in a separate experiment. Applying machine learning and time-series analysis techniques, we showed that (1) higher WMC individuals experienced lower internal distraction through stronger suppression of posterior cingulate cortex (PCC) activity, (2) higher WMC individuals had better neural representations of attended information as evidenced by higher multivoxel decoding accuracy of cue-related activities in the dorsal attention network (DAN), (3) the positive relationship between WMC and DAN decoding accuracy was mediated by suppression of PCC activity, (4) the dorsal anterior cingulate (dACC) was a source of top-down signals that regulate PCC activity as evidenced by the negative association between Granger-causal influence dACC→PCC and PCC activity levels, and (5) higher WMC individuals exhibiting stronger dACC→PCC Granger-causal influence. These results shed light on the neural mechanisms underlying the executive suppression of internal distraction in tasks requiring externally oriented attention and provide an explanation of the individual differences in such suppression.