THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior.

Understanding object representations requires a broad, comprehensive sampling of the objects in our visual world with dense measurements of brain activity and behavior. Here, we present THINGS-data, a multimodal collection of large-scale neuroimaging and behavioral datasets in humans, comprising densely sampled functional MRI and magnetoencephalographic recordings, as well as 4.70 million similarity judgments in response to thousands of photographic images for up to 1,854 object concepts. THINGS-data is unique in its breadth of richly annotated objects, allowing for testing countless hypotheses at scale while assessing the reproducibility of previous findings. Beyond the unique insights promised by each individual dataset, the multimodality of THINGS-data allows combining datasets for a much broader view into object processing than previously possible. Our analyses demonstrate the high quality of the datasets and provide five examples of hypothesis-driven and data-driven applications. THINGS-data constitutes the core public release of the THINGS initiative (https://things-initiative.org) for bridging the gap between disciplines and the advancement of cognitive neuroscience.

Great Expectations: A Critical Review of and Suggestions for the Study of Reward Processing as a Cause and Predictor of Depression.

Both human and animal studies support the relationship between depression and reward processing abnormalities, giving rise to the expectation that neural signals of these processes may serve as biomarkers or mechanistic treatment targets. Given the great promise of this research line, we scrutinized those findings and the theoretical claims that underlie them. To achieve this, we applied the framework provided by classical work on causality as well as contemporary approaches to prediction. We identified a number of conceptual, practical, and analytical challenges to this line of research and used a preregistered meta-analysis to quantify the longitudinal associations between reward processing abnormalities and depression. We also investigated the impact of measurement error on reported data. We found that reward processing abnormalities do not reach levels that would be useful for clinical prediction, yet the available evidence does not preclude a possible causal role in depression.

Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.

Objects can be characterized according to a vast number of possible criteria (such as animacy, shape, colour and function), but some dimensions are more useful than others for making sense of the objects around us. To identify these core dimensions of object representations, we developed a data-driven computational model of similarity judgements for real-world images of 1,854 objects. The model captured most explainable variance in similarity judgements and produced 49 highly reproducible and meaningful object dimensions that reflect various conceptual and perceptual properties of those objects. These dimensions predicted external categorization behaviour and reflected typicality judgements of those categories. Furthermore, humans can accurately rate objects along these dimensions, highlighting their interpretability and opening up a way to generate similarity estimates from object dimensions alone. Collectively, these results demonstrate that human similarity judgements can be captured by a fairly low-dimensional, interpretable embedding that generalizes to external behaviour.

Knowing What You Know in Brain Segmentation Using Bayesian Deep Neural Networks.

In this paper, we describe a Bayesian deep neural network (DNN) for predicting FreeSurfer segmentations of structural MRI volumes, in minutes rather than hours. The network was trained and evaluated on a large dataset (n = 11,480), obtained by combining data from more than a hundred different sites, and also evaluated on another completely held-out dataset (n = 418). The network was trained using a novel spike-and-slab dropout-based variational inference approach. We show that, on these datasets, the proposed Bayesian DNN outperforms previously proposed methods, in terms of the similarity between the segmentation predictions and the FreeSurfer labels, and the usefulness of the estimate uncertainty of these predictions. In particular, we demonstrated that the prediction uncertainty of this network at each voxel is a good indicator of whether the network has made an error and that the uncertainty across the whole brain can predict the manual quality control ratings of a scan. The proposed Bayesian DNN method should be applicable to any new network architecture for addressing the segmentation problem.

Subtle predictive movements reveal actions regardless of social context.

Humans have a remarkable ability to predict the actions of others. To address what information enables this prediction and how the information is modulated by social context, we used videos collected during an interactive reaching game. Two participants (an "initiator" and a "responder") sat on either side of a plexiglass screen on which two targets were affixed. The initiator was directed to tap one of the two targets, and the responder had to either beat the initiator to the target (competition) or arrive at the same time (cooperation). In a psychophysics experiment, new observers predicted the direction of the initiators' reach from brief clips, which were clipped relative to when the initiator began reaching. A machine learning classifier performed the same task. Both humans and the classifier were able to determine the direction of movement before the finger lift-off in both social conditions. Further, using an information mapping technique, the relevant information was found to be distributed throughout the body of the initiator in both social conditions. Our results indicate that we reveal our intentions during cooperation, in which communicating the future course of actions is beneficial, and also during competition despite the social motivation to reveal less information.

Efficacy of different dynamic functional connectivity methods to capture cognitively relevant information.

Given the dynamic nature of the human brain, there has been an increasing interest in investigating short-term temporal changes in functional connectivity, also known as dynamic functional connectivity (dFC), i.e., the time-varying inter-regional statistical dependence of blood oxygenation level-dependent (BOLD) signal within the constraints of a single scan. Numerous methodologies have been proposed to characterize dFC during rest and task, but few studies have compared them in terms of their efficacy to capture behavioral and clinically relevant dynamics. This is mostly due to lack of a well-defined ground truth, especially for rest scans. In this study, with a multitask dataset (rest, memory, video, and math) serving as ground truth, we investigated the efficacy of several dFC estimation techniques at capturing cognitively relevant dFC modulation induced by external tasks. We evaluated two framewise methods (dFC estimates for a single time point): dynamic conditional correlation (DCC) and jackknife correlation (JC); and five window-based methods: sliding window correlation (SWC), sliding window correlation with L1-regularization (SWC_L1), a combination of DCC and SWC called moving average DCC (DCC_MA), multiplication of temporal derivatives (MTD), and a variant of jackknife correlation called delete-d jackknife correlation (dJC). The efficacy is defined as each dFC metric's ability to successfully subdivide multitask scans into cognitively homogenous segments (even if those segments are not temporally continuous). We found that all window-based dFC methods performed well for commonly used window lengths (WL ≥ 30sec), with sliding window methods (SWC, SWC_L1) as well as the hybrid DCC_MA approach performing slightly better. For shorter window lengths (WL ≤ 15sec), DCC_MA and dJC produced the best results. Neither framewise method (i.e., DCC and JC) led to dFC estimates with high accuracy.

Distributed Weight Consolidation: A Brain Segmentation Case Study.

Collecting the large datasets needed to train deep neural networks can be very difficult, particularly for the many applications for which sharing and pooling data is complicated by practical, ethical, or legal concerns. However, it may be the case that derivative datasets or predictive models developed within individual sites can be shared and combined with fewer restrictions. Training on distributed data and combining the resulting networks is often viewed as continual learning, but these methods require networks to be trained sequentially. In this paper, we introduce distributed weight consolidation (DWC), a continual learning method to consolidate the weights of separate neural networks, each trained on an independent dataset. We evaluated DWC with a brain segmentation case study, where we consolidated dilated convolutional neural networks trained on independent structural magnetic resonance imaging (sMRI) datasets from different sites. We found that DWC led to increased performance on test sets from the different sites, while maintaining generalization performance for a very large and completely independent multi-site dataset, compared to an ensemble baseline.

Deconvolution of High Dimensional Mixtures via Boosting, with Application to Diffusion-Weighted MRI of Human Brain.

Diffusion-weighted magnetic resonance imaging (DWI) and fiber tractography are the only methods to measure the structure of the white matter in the living human brain. The diffusion signal has been modelled as the combined contribution from many individual fascicles of nerve fibers passing through each location in the white matter. Typically, this is done via basis pursuit, but estimation of the exact directions is limited due to discretization [1, 2]. The difficulties inherent in modeling DWI data are shared by many other problems involving fitting non-parametric mixture models. Ekanadaham et al. [3] proposed an approach, continuous basis pursuit, to overcome discretization error in the 1-dimensional case (e.g., spike-sorting). Here, we propose a more general algorithm that fits mixture models of any dimensionality without discretization. Our algorithm uses the principles of L2-boost [4], together with refitting of the weights and pruning of the parameters. The addition of these steps to L2-boost both accelerates the algorithm and assures its accuracy. We refer to the resulting algorithm as elastic basis pursuit, or EBP, since it expands and contracts the active set of kernels as needed. We show that in contrast to existing approaches to fitting mixtures, our boosting framework (1) enables the selection of the optimal bias-variance tradeoff along the solution path, and (2) scales with high-dimensional problems. In simulations of DWI, we find that EBP yields better parameter estimates than a non-negative least squares (NNLS) approach, or the standard model used in DWI, the tensor model, which serves as the basis for diffusion tensor imaging (DTI) [5]. We demonstrate the utility of the method in DWI data acquired in parts of the brain containing crossings of multiple fascicles of nerve fibers.