Human Visual Cortex and Deep Convolutional Neural Network Care Deeply about Object Background.

Deep convolutional neural networks (DCNNs) are able to partially predict brain activity during object categorization tasks, but factors contributing to this predictive power are not fully understood. Our study aimed to investigate the factors contributing to the predictive power of DCNNs in object categorization tasks. We compared the activity of four DCNN architectures with EEG recordings obtained from 62 human participants during an object categorization task. Previous physiological studies on object categorization have highlighted the importance of figure-ground segregation-the ability to distinguish objects from their backgrounds. Therefore, we investigated whether figure-ground segregation could explain the predictive power of DCNNs. Using a stimulus set consisting of identical target objects embedded in different backgrounds, we examined the influence of object background versus object category within both EEG and DCNN activity. Crucially, the recombination of naturalistic objects and experimentally controlled backgrounds creates a challenging and naturalistic task, while retaining experimental control. Our results showed that early EEG activity (< 100 msec) and early DCNN layers represent object background rather than object category. We also found that the ability of DCNNs to predict EEG activity is primarily influenced by how both systems process object backgrounds, rather than object categories. We demonstrated the role of figure-ground segregation as a potential prerequisite for recognition of object features, by contrasting the activations of trained and untrained (i.e., random weights) DCNNs. These findings suggest that both human visual cortex and DCNNs prioritize the segregation of object backgrounds and target objects to perform object categorization. Altogether, our study provides new insights into the mechanisms underlying object categorization as we demonstrated that both human visual cortex and DCNNs care deeply about object background.

Testing, explaining, and exploring models of facial expressions of emotions.

Models are the hallmark of mature scientific inquiry. In psychology, this maturity has been reached in a pervasive question-what models best represent facial expressions of emotion? Several hypotheses propose different combinations of facial movements [action units (AUs)] as best representing the six basic emotions and four conversational signals across cultures. We developed a new framework to formalize such hypotheses as predictive models, compare their ability to predict human emotion categorizations in Western and East Asian cultures, explain the causal role of individual AUs, and explore updated, culture-accented models that improve performance by reducing a prevalent Western bias. Our predictive models also provide a noise ceiling to inform the explanatory power and limitations of different factors (e.g., AUs and individual differences). Thus, our framework provides a new approach to test models of social signals, explain their predictive power, and explore their optimization, with direct implications for theory development.

Degrees of algorithmic equivalence between the brain and its DNN models.

Deep neural networks (DNNs) have become powerful and increasingly ubiquitous tools to model human cognition, and often produce similar behaviors. For example, with their hierarchical, brain-inspired organization of computations, DNNs apparently categorize real-world images in the same way as humans do. Does this imply that their categorization algorithms are also similar? We have framed the question with three embedded degrees that progressively constrain algorithmic similarity evaluations: equivalence of (i) behavioral/brain responses, which is current practice, (ii) the stimulus features that are processed to produce these outcomes, which is more constraining, and (iii) the algorithms that process these shared features, the ultimate goal. To improve DNNs as models of cognition, we develop for each degree an increasingly constrained benchmark that specifies the epistemological conditions for the considered equivalence.

A Critical Test of Deep Convolutional Neural Networks' Ability to Capture Recurrent Processing in the Brain Using Visual Masking.

Recurrent processing is a crucial feature in human visual processing supporting perceptual grouping, figure-ground segmentation, and recognition under challenging conditions. There is a clear need to incorporate recurrent processing in deep convolutional neural networks, but the computations underlying recurrent processing remain unclear. In this article, we tested a form of recurrence in deep residual networks (ResNets) to capture recurrent processing signals in the human brain. Although ResNets are feedforward networks, they approximate an excitatory additive form of recurrence. Essentially, this form of recurrence consists of repeating excitatory activations in response to a static stimulus. Here, we used ResNets of varying depths (reflecting varying levels of recurrent processing) to explain EEG activity within a visual masking paradigm. Sixty-two humans and 50 artificial agents (10 ResNet models of depths -4, 6, 10, 18, and 34) completed an object categorization task. We show that deeper networks explained more variance in brain activity compared with shallower networks. Furthermore, all ResNets captured differences in brain activity between unmasked and masked trials, with differences starting at ∼98 msec (from stimulus onset). These early differences indicated that EEG activity reflected "pure" feedforward signals only briefly (up to ∼98 msec). After ∼98 msec, deeper networks showed a significant increase in explained variance, which peaks at ∼200 msec, but only within unmasked trials, not masked trials. In summary, we provided clear evidence that excitatory additive recurrent processing in ResNets captures some of the recurrent processing in humans.

The Amsterdam Open MRI Collection, a set of multimodal MRI datasets for individual difference analyses.

We present the Amsterdam Open MRI Collection (AOMIC): three datasets with multimodal (3 T) MRI data including structural (T1-weighted), diffusion-weighted, and (resting-state and task-based) functional BOLD MRI data, as well as detailed demographics and psychometric variables from a large set of healthy participants (N = 928, N = 226, and N = 216). Notably, task-based fMRI was collected during various robust paradigms (targeting naturalistic vision, emotion perception, working memory, face perception, cognitive conflict and control, and response inhibition) for which extensively annotated event-files are available. For each dataset and data modality, we provide the data in both raw and preprocessed form (both compliant with the Brain Imaging Data Structure), which were subjected to extensive (automated and manual) quality control. All data is publicly available from the OpenNeuro data sharing platform.

Action Intentions, Predictive Processing, and Mind Reading: Turning Goalkeepers Into Penalty Killers.

The key to action control is one's ability to adequately predict the consequences of one's actions. Predictive processing theories assume that forward models enable rapid "preplay" to assess the match between predicted and intended action effects. Here we propose the novel hypothesis that "reading" another's action intentions requires a rich forward model of that agent's action. Such a forward model can be obtained and enriched through learning by either practice or simulation. Based on this notion, we ran a series of studies on soccer goalkeepers and novices, who predicted the intended direction of penalties being kicked at them in a computerized penalty-reading task. In line with hypotheses, extensive practice in penalty kicking improved performance in penalty reading among goalkeepers who had extensive prior experience in penalty blocking but not in penalty kicking. A robust benefit in penalty reading did not result from practice in kinesthetic motor imagery of penalty kicking in novice participants. To test whether goalkeepers actually use such penalty-kicking imagery in penalty reading, we trained a machine-learning classifier on multivariate fMRI activity patterns to distinguish motor-imagery-related from attention-related strategies during a penalty-imagery training task. We then applied that classifier to fMRI data related to a separate penalty-reading task and showed that 2/3 of all correctly read penalty kicks were classified as engaging the motor-imagery circuit rather than merely the attention circuit. This study provides initial evidence that, in order to read our opponent's action intention, it helps to observe their action kinematics, and use our own forward model to predict the sensory consequences of "our" penalty kick if we were to produce these action kinematics ourselves. In sum, it takes practice as a penalty kicker to become a penalty killer.

Choosing to view morbid information involves reward circuitry.

People often seek out stories, videos or images that detail death, violence or harm. Considering the ubiquity of this behavior, it is surprising that we know very little about the neural circuits involved in choosing negative information. Using fMRI, the present study shows that choosing intensely negative stimuli engages similar brain regions as those that support extrinsic incentives and "regular" curiosity. Participants made choices to view negative and positive images, based on negative (e.g., a soldier kicks a civilian against his head) and positive (e.g., children throw flower petals at a wedding) verbal cues. We hypothesized that the conflicting, but relatively informative act of choosing to view a negative image, resulted in stronger activation of reward circuitry as opposed to the relatively uncomplicated act of choosing to view a positive stimulus. Indeed, as preregistered, we found that choosing negative cues was associated with activation of the striatum, inferior frontal gyrus, anterior insula, and anterior cingulate cortex, both when contrasting against a passive viewing condition, and when contrasting against positive cues. These findings nuance models of decision-making, valuation and curiosity, and are an important starting point when considering the value of seeking out negative content.

Religious belief and cognitive conflict sensitivity: A preregistered fMRI study.

In the current preregistered fMRI study, we investigated the relationship between religiosity and behavioral and neural mechanisms of conflict processing, as a conceptual replication of the study by Inzlicht et al., (2009). Participants (N=193) performed a gender-Stroop task and afterwards completed standardized measures to assess their religiosity. As expected, the task induced cognitive conflict at the behavioral level and at a neural level this was reflected in increased activity in the anterior cingulate cortex (ACC). However, individual differences in religiosity were not related to performance on the Stroop task as measured in accuracy and interference effects, nor to neural markers of response conflict (correct responses vs. errors) or informational conflict (congruent vs. incongruent stimuli). Overall, we obtained moderate to strong evidence in favor of the null hypotheses that religiosity is unrelated to cognitive conflict sensitivity. We discuss the implications for the neuroscience of religion and emphasize the importance of designing studies that more directly implicate religious concepts and behaviors in an ecologically valid manner.

The relationship between individual differences in gray matter volume and religiosity and mystical experiences: A preregistered voxel-based morphometry study.

The neural substrates of religious belief and experience are an intriguing though contentious topic. Here, we had the unique opportunity to establish the relation between validated measures of religiosity and gray matter volume in a large sample of participants (N = 211). In this registered report, we conducted a confirmatory voxel-based morphometry analysis to test three central hypotheses regarding the relationship between religiosity and mystical experiences and gray matter volume. The preregisterered hypotheses, analysis plan, preprocessing and analysis code and statistical brain maps are all available from online repositories. By using a region-of-interest analysis, we found no evidence that religiosity is associated with a reduced volume of the orbito-frontal cortex and changes in the structure of the bilateral inferior parietal lobes. Neither did we find support for the notion that mystical experiences are associated with a reduced volume of the hippocampus, the right middle temporal gyrus or with the inferior parietal lobes. A whole-brain analysis furthermore indicated that no structural brain differences were found in association with religiosity and mystical experiences. We believe that the search for the neural correlates of religious beliefs and experiences should therefore shift focus from studying structural brain differences to a functional and multivariate approach.

How to control for confounds in decoding analyses of neuroimaging data.

Over the past decade, multivariate "decoding analyses" have become a popular alternative to traditional mass-univariate analyses in neuroimaging research. However, a fundamental limitation of using decoding analyses is that it remains ambiguous which source of information drives decoding performance, which becomes problematic when the to-be-decoded variable is confounded by variables that are not of primary interest. In this study, we use a comprehensive set of simulations as well as analyses of empirical data to evaluate two methods that were previously proposed and used to control for confounding variables in decoding analyses: post hoc counterbalancing and confound regression. In our empirical analyses, we attempt to decode gender from structural MRI data while controlling for the confound "brain size". We show that both methods introduce strong biases in decoding performance: post hoc counterbalancing leads to better performance than expected (i.e., positive bias), which we show in our simulations is due to the subsampling process that tends to remove samples that are hard to classify or would be wrongly classified; confound regression, on the other hand, leads to worse performance than expected (i.e., negative bias), even resulting in significant below chance performance in some realistic scenarios. In our simulations, we show that below chance accuracy can be predicted by the variance of the distribution of correlations between the features and the target. Importantly, we show that this negative bias disappears in both the empirical analyses and simulations when the confound regression procedure is performed in every fold of the cross-validation routine, yielding plausible (above chance) model performance. We conclude that, from the various methods tested, cross-validated confound regression is the only method that appears to appropriately control for confounds which thus can be used to gain more insight into the exact source(s) of information driving one's decoding analysis.

Porcupine: A visual pipeline tool for neuroimaging analysis.

The field of neuroimaging is rapidly adopting a more reproducible approach to data acquisition and analysis. Data structures and formats are being standardised and data analyses are getting more automated. However, as data analysis becomes more complicated, researchers often have to write longer analysis scripts, spanning different tools across multiple programming languages. This makes it more difficult to share or recreate code, reducing the reproducibility of the analysis. We present a tool, Porcupine, that constructs one's analysis visually and automatically produces analysis code. The graphical representation improves understanding of the performed analysis, while retaining the flexibility of modifying the produced code manually to custom needs. Not only does Porcupine produce the analysis code, it also creates a shareable environment for running the code in the form of a Docker image. Together, this forms a reproducible way of constructing, visualising and sharing one's analysis. Currently, Porcupine links to Nipype functionalities, which in turn accesses most standard neuroimaging analysis tools. Our goal is to release researchers from the constraints of specific implementation details, thereby freeing them to think about novel and creative ways to solve a given problem. Porcupine improves the overview researchers have of their processing pipelines, and facilitates both the development and communication of their work. This will reduce the threshold at which less expert users can generate reusable pipelines. With Porcupine, we bridge the gap between a conceptual and an implementational level of analysis and make it easier for researchers to create reproducible and shareable science. We provide a wide range of examples and documentation, as well as installer files for all platforms on our website: https://timvanmourik.github.io/Porcupine. Porcupine is free, open source, and released under the GNU General Public License v3.0.

Shared states: using MVPA to test neural overlap between self-focused emotion imagery and other-focused emotion understanding.

The present study tested whether the neural patterns that support imagining 'performing an action', 'feeling a bodily sensation' or 'being in a situation' are directly involved in understanding other people's actions, bodily sensations and situations. Subjects imagined the content of short sentences describing emotional actions, interoceptive sensations and situations (self-focused task), and processed scenes and focused on how the target person was expressing an emotion, what this person was feeling, and why this person was feeling an emotion (other-focused task). Using a linear support vector machine classifier on brain-wide multi-voxel patterns, we accurately decoded each individual class in the self-focused task. When generalizing the classifier from the self-focused task to the other-focused task, we also accurately decoded whether subjects focused on the emotional actions, interoceptive sensations and situations of others. These results show that the neural patterns that underlie self-imagined experience are involved in understanding the experience of other people. This supports the theoretical assumption that the basic components of emotion experience and understanding share resources in the brain.