Release the Krakencoder: A unified brain connectome translation and fusion tool.

Brain connectivity can be estimated in many ways, depending on modality and processing strategy. Here we present the Krakencoder, a joint connectome mapping tool that simultaneously, bidirectionally translates between structural (SC) and functional connectivity (FC), and across different atlases and processing choices via a common latent representation. These mappings demonstrate unprecedented accuracy and individual-level identifiability; the mapping between SC and FC has identifiability 42-54% higher than existing models. The Krakencoder combines all connectome flavors via a shared low-dimensional latent space. This "fusion" representation i) better reflects familial relatedness, ii) preserves age- and sex-relevant information and iii) enhances cognition-relevant information. The Krakencoder can be applied without retraining to new, out-of-age-distribution data while still preserving inter-individual differences in the connectome predictions and familial relationships in the latent representations. The Krakencoder is a significant leap forward in capturing the relationship between multi-modal brain connectomes in an individualized, behaviorally- and demographically-relevant way.

Human brain responses are modulated when exposed to optimized natural images or synthetically generated images.

Understanding how human brains interpret and process information is important. Here, we investigated the selectivity and inter-individual differences in human brain responses to images via functional MRI. In our first experiment, we found that images predicted to achieve maximal activations using a group level encoding model evoke higher responses than images predicted to achieve average activations, and the activation gain is positively associated with the encoding model accuracy. Furthermore, anterior temporal lobe face area (aTLfaces) and fusiform body area 1 had higher activation in response to maximal synthetic images compared to maximal natural images. In our second experiment, we found that synthetic images derived using a personalized encoding model elicited higher responses compared to synthetic images from group-level or other subjects' encoding models. The finding of aTLfaces favoring synthetic images than natural images was also replicated. Our results indicate the possibility of using data-driven and generative approaches to modulate macro-scale brain region responses and probe inter-individual differences in and functional specialization of the human visual system.

Modulating human brain responses via optimal natural image selection and synthetic image generation.

One of the main goals of neuroscience is to understand how biological brains interpret and process incoming environmental information. Building computational encoding models that map images to neural responses is one way to pursue this goal. Moreover, generating or selecting visual stimuli designed to achieve specific patterns of responses allows exploration and control of neuronal firing rates or regional brain activity responses. Here, we investigated the brain's regional activation selectivity and inter-individual differences in human brain responses to various sets of natural and synthetic (generated) images via two functional MRI (fMRI) studies. For our first fMRI study, we used a pre-trained group-level neural model for selecting or synthesizing images that are predicted to maximally activate targeted brain regions. We then presented these images to subjects while collecting their fMRI data. Our results show that optimized images indeed evoke larger magnitude responses than other images predicted to achieve average levels of activation.Furthermore, the activation gain is positively associated with the encoding model accuracy. While most regions' activations in response to maximal natural images and maximal synthetic images were not different, two regions, namely anterior temporal lobe faces (aTLfaces) and fusiform body area 1 (FBA1), had significantly higher activation in response to maximal synthetic images compared to maximal natural images. On the other hand, three regions; medial temporal lobe face area (mTLfaces), ventral word form area 1 (VWFA1) and ventral word form area 2 (VWFA2), had higher activation in response to maximal natural images compared to maximal synthetic images. In our second fMRI experiment, we focused on probing inter-individual differences in face regions' responses and found that individual-specific synthetic (and not natural) images derived using a personalized encoding model elicited significantly higher responses compared to synthetic images derived from the group-level or other subjects' encoding models. Finally, we replicated the finding showing synthetic images elicited larger activation responses in the aTLfaces region compared to natural image responses in that region. Here, for the first time, we leverage our data-driven and generative modeling framework NeuroGen to probe inter-individual differences in and functional specialization of the human visual system. Our results indicate that NeuroGen can be used to modulate macro-scale brain regions in specific individuals using synthetically generated visual stimuli.

Graph-matching distance between individuals' functional connectomes varies with relatedness, age, and cognitive score.

Functional connectomes (FCs), represented by networks or graphs that summarize coactivation patterns between pairs of brain regions, have been related at a population level to age, sex, cognitive/behavioral scores, life experience, genetics, and disease/disorders. However, quantifying FC differences between individuals also provides a rich source of information with which to map to differences in those individuals' biology, experience, genetics or behavior. In this study, graph matching is used to create a novel inter-individual FC metric, called swap distance, that quantifies the distance between pairs of individuals' partial FCs, with a smaller swap distance indicating the individuals have more similar FC. We apply graph matching to align FCs between individuals from the the Human Connectome Project N = 997 and find that swap distance (i) increases with increasing familial distance, (ii) increases with subjects' ages, (iii) is smaller for pairs of females compared to pairs of males, and (iv) is larger for females with lower cognitive scores compared to females with larger cognitive scores. Regions that contributed most to individuals' swap distances were in higher-order networks, that is, default-mode and fronto-parietal, that underlie executive function and memory. These higher-order networks' regions also had swap frequencies that varied monotonically with familial relatedness of the individuals in question. We posit that the proposed graph matching technique provides a novel way to study inter-subject differences in FC and enables quantification of how FC may vary with age, relatedness, sex, and behavior.

Personalized visual encoding model construction with small data.

Quantifying population heterogeneity in brain stimuli-response mapping may allow insight into variability in bottom-up neural systems that can in turn be related to individual's behavior or pathological state. Encoding models that predict brain responses to stimuli are one way to capture this relationship. However, they generally need a large amount of fMRI data to achieve optimal accuracy. Here, we propose an ensemble approach to create encoding models for novel individuals with relatively little data by modeling each subject's predicted response vector as a linear combination of the other subjects' predicted response vectors. We show that these ensemble encoding models trained with hundreds of image-response pairs, achieve accuracy not different from models trained on 20,000 image-response pairs. Importantly, the ensemble encoding models preserve patterns of inter-individual differences in the image-response relationship. We also show the proposed approach is robust against domain shift by validating on data with a different scanner and experimental setup. Additionally, we show that the ensemble encoding models are able to discover the inter-individual differences in various face areas' responses to images of animal vs human faces using a recently developed NeuroGen framework. Our approach shows the potential to use existing densely-sampled data, i.e. large amounts of data collected from a single individual, to efficiently create accurate, personalized encoding models and, subsequently, personalized optimal synthetic images for new individuals scanned under different experimental conditions.

NeuroGen: Activation optimized image synthesis for discovery neuroscience.

Functional MRI (fMRI) is a powerful technique that has allowed us to characterize visual cortex responses to stimuli, yet such experiments are by nature constructed based on a priori hypotheses, limited to the set of images presented to the individual while they are in the scanner, are subject to noise in the observed brain responses, and may vary widely across individuals. In this work, we propose a novel computational strategy, which we call NeuroGen, to overcome these limitations and develop a powerful tool for human vision neuroscience discovery. NeuroGen combines an fMRI-trained neural encoding model of human vision with a deep generative network to synthesize images predicted to achieve a target pattern of macro-scale brain activation. We demonstrate that the reduction of noise that the encoding model provides, coupled with the generative network's ability to produce images of high fidelity, results in a robust discovery architecture for visual neuroscience. By using only a small number of synthetic images created by NeuroGen, we demonstrate that we can detect and amplify differences in regional and individual human brain response patterns to visual stimuli. We then verify that these discoveries are reflected in the several thousand observed image responses measured with fMRI. We further demonstrate that NeuroGen can create synthetic images predicted to achieve regional response patterns not achievable by the best-matching natural images. The NeuroGen framework extends the utility of brain encoding models and opens up a new avenue for exploring, and possibly precisely controlling, the human visual system.

Estimated connectivity networks outperform observed connectivity networks when classifying people with multiple sclerosis into disability groups.

BACKGROUND: Multiple Sclerosis (MS), a neurodegenerative and neuroinflammatory disease, causing lesions that disrupt the brain's anatomical and physiological connectivity networks, resulting in cognitive, visual and/or motor disabilities. Advanced imaging techniques like diffusion and functional MRI allow measurement of the brain's structural connectivity (SC) and functional connectivity (FC) networks, and can enable a better understanding of how their disruptions cause disability in people with MS (pwMS). However, advanced MRI techniques are used mainly for research purposes as they are expensive, time-consuming and require high-level expertise to acquire and process. As an alternative, the Network Modification (NeMo) Tool can be used to estimate SC and FC using lesion masks derived from pwMS and a reference set of controls' connectivity networks. OBJECTIVE: Here, we test the hypothesis that estimated SC and FC (eSC and eFC) from the NeMo Tool, based only on an individual's lesion masks, can be used to classify pwMS into disability categories just as well as SC and FC extracted from advanced MRI directly in pwMS. We also aim to find the connections most important for differentiating between no disability vs evidence of disability groups. MATERIALS AND METHODS: One hundred pwMS (age:45.5 ± 11.4 years, 66% female, disease duration: 12.97 ± 8.07 years) were included in this study. Expanded Disability Status Scale (EDSS) was used to assess disability, 67 pwMS had no disability (EDSS < 2). Observed SC and FC were extracted from diffusion and functional MRI directly in pwMS, respectively. The NeMo Tool was used to estimate the remaining structural connectome (eSC), by removing streamlines in a reference set of tractograms that intersected the lesion mask. The NeMo Tool's eSC was used then as input to a deep neural network to estimate the corresponding FC (eFC). Logistic regression with ridge regularization was used to classify pwMS into disability categories (no disability vs evidence of disability), based on demographics/clinical information (sex, age, race, disease duration, clinical phenotype, and spinal lesion burden) and either pairwise entries or regional summaries from one of the following matrices: SC, FC, eSC, and eFC. The area under the ROC curve (AUC) was used to assess the classification performance. Both univariate statistics and parameter coefficients from the classification models were used to identify features important to differentiating between the groups. RESULTS: The regional eSC and eFC models outperformed their observed FC and SC counterparts (p-value < 0.05), while the pairwise eSC and SC performed similarly (p = 0.10). Regional eSC and eFC models had higher AUC (0.66-0.68) than the pairwise models (0.60-0.65), with regional eFC having highest classification accuracy across all models. Ridge regression coefficients for the regional eFC and regional observed FC models were significantly correlated (Pearson's r = 0.52, p-value < 10e-7). Decreased estimated SC node strength in default mode and ventral attention networks and increased eFC node strength in visual networks was associated with evidence of disability. DISCUSSION: Here, for the first time, we use clinically acquired lesion masks to estimate both structural and functional connectomes in patient populations to better understand brain lesion-dysfunction mapping in pwMS. Models based on the NeMo Tool's estimates of SC and FC better classified pwMS by disability level than SC and FC observed directly in the individual using advanced MRI. This work provides a viable alternative to performing high-cost, advanced MRI in patient populations, bringing the connectome one step closer to the clinic.

Heritability and interindividual variability of regional structure-function coupling.

White matter structural connections are likely to support flow of functional activation or functional connectivity. While the relationship between structural and functional connectivity profiles, here called SC-FC coupling, has been studied on a whole-brain, global level, few studies have investigated this relationship at a regional scale. Here we quantify regional SC-FC coupling in healthy young adults using diffusion-weighted MRI and resting-state functional MRI data from the Human Connectome Project and study how SC-FC coupling may be heritable and varies between individuals. We show that regional SC-FC coupling strength varies widely across brain regions, but was strongest in highly structurally connected visual and subcortical areas. We also show interindividual regional differences based on age, sex and composite cognitive scores, and that SC-FC coupling was highly heritable within certain networks. These results suggest regional structure-function coupling is an idiosyncratic feature of brain organisation that may be influenced by genetic factors.

Multiple just-noticeable-difference-based no-reference stereoscopic image quality assessment.

Just-noticeable difference (JND) is an important characteristic of the human visual system (HVS), and some established JND models imitating the perception of human eyes already exist. However, their utilization in stereoscopic image quality assessment (SIQA) remains limited. To better simulate how HVS senses 3D images under a no-reference situation, a novel SIQA method based on multiple JND models is proposed in this paper. In our metric, the stereoscopic image pairs are decomposed into multi-scale monocular views and binocular views. Then, texture and edge information of these multi-scale images is extracted. Next, a monocular JND model, a binocular JND model, and a depth JND model are separately applied to the extracted features and the depth map. Finally, these features are synthesized and mapped to objective scores. Through experiment and comparison on public 3D image databases, the proposed method shows a competitive advantage over most state-of-the-art SIQA methods, which indicates that it has a promising prospect in practical applications.