MCAN: Multimodal Causal Adversarial Networks for Dynamic Effective Connectivity Learning from fMRI and EEG Data.

Dynamic effective connectivity (DEC) is the accumulation of effective connectivity in the time dimension, which can describe the continuous neural activities in the brain. Recently, learning DEC from functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) data has attracted the attention of neuroinformatics researchers. However, the current methods fail to consider the gap between the fMRI and EEG modality, which can not precisely learn the DEC network from multimodal data. In this paper, we propose a multimodal causal adversarial network for DEC learning, named MCAN. The MCAN contains two modules: multimodal causal generator and multimodal causal discriminator. First, MCAN employs a multimodal causal generator with an attention-guided layer to produce a posterior signal and output a set of DEC networks. Then, the proposed method uses a multimodal causal discriminator to unsupervised calculate the joint gradient, which directs the update of the whole network. The experimental results on simulated data sets show that MCAN is superior to other state-of-the-art methods in learning the network structure of DEC and can effectively estimate the brain states. The experimental results on real data sets show that MCAN can better reveal abnormal patterns of brain activity and has good application potential in brain network analysis.

Brain Network Evaluation by Functional-Guided Effective Connectivity Reinforcement Learning Method Indicates Therapeutic Effect for Tinnitus.

Using functional connectivity (FC) or effective connectivity (EC) alone cannot effectively delineate brain networks based on functional magnetic resonance imaging (fMRI) data, limiting the understanding of the mechanism of tinnitus and its treatment. Investigating brain FC is a foundational step in exploring EC. This study proposed a functionally guided EC (FGEC) method based on reinforcement learning (FGECRL) to enhance the precision of identifying EC between distinct brain regions. An actor-critic framework with an encoder-decoder model was adopted as the actor network. The encoder utilizes a transformer model; the decoder employs a bidirectional long short-term memory network with attention. An FGEC network was constructed for the enrolled participants per fMRI scan, including 65 patients with tinnitus and 28 control participants healthy at the enrollment time. After 6 months of sound therapy for tinnitus and prospective follow-up, fMRI data were acquired again and retrospectively categorized into an effective group (EG) and an ineffective group (IG) according to the treatment effect. Compared with FC and EC, the FGECRL method demonstrated better accuracy in discriminating between different groups, highlighting the advantage of FGECRL in identifying brain network features. For the FGEC network of the EG and IG per state (before and after treatment) and healthy controls, effective therapy is characterized by a similar pattern of FGEC network between patients with tinnitus after treatment and healthy controls. Deactivated information output in the motor network, somatosensory network, and medioventral occipital cortex may biologically indicate effective treatment. The maintenance of decreased EC in the primary auditory cortex may represent a failure of sound therapy, further supporting the Bayesian inference theory for tinnitus perception. The FGEC network can provide direct evidence for the mechanism of sound therapy in patients with tinnitus with distinct outcomes.

MetaCAE: Causal autoencoder with meta-knowledge transfer for brain effective connectivity estimation.

Using machine learning methods to estimate brain effective connectivity networks from functional magnetic resonance imaging (fMRI) data has gradually become one of the hot subjects in the fields of neuroscience. In particular, the encoder-decoder based methods can effectively extract the connections in fMRI time series, which have achieved promising performance. However, these methods generally use Granger causality model, which may identify false directions due to the non-stationary characteristic of fMRI data. Additionally, fMRI datasets have limited sample sizes, which significantly constrains the development of these methods. In this paper, we propose a novel brain effective connectivity estimation method based on causal autoencoder with meta-knowledge transfer, called MetaCAE. The proposed approach employs a causal autoencoder (CAE) to extract causal dependencies from non-stationary fMRI time series, and leverages meta-knowledge transfer to improve the estimation accuracy on small-sample data. More specifically, MetaCAE first employs a temporal convolutional encoder to extract non-stationary temporal information from fMRI time series. Then it uses a structural equation model-based decoder to decode causal relationships between brain regions. Finally, it utilizes a model-agnostic meta-learning method to learn the meta-knowledge of the shared brain effective connectivity among different subjects, and transfers the meta-knowledge to the CAE to enhance its estimation ability on small-sample fMRI data. Comprehensive experiments on both simulated and real-world data demonstrate the efficacy of MetaCAE in estimating brain effective connectivity.

Weakly guided attention model with hierarchical interaction for brain CT report generation.

Brain Computed Tomography (CT) report generation, which aims to assist radiologists in diagnosing cerebrovascular diseases efficiently, is challenging in feature representation for dozens of images and language descriptions with several sentences. Existing report generation methods have achieved significant achievement based on the encoder-decoder framework and attention mechanism. However, current research has limitations in solving the many-to-many alignment between the multi-images of Brain CT imaging and the multi-sentences of Brain CT report, and fails to attend to critical images and lesion areas, resulting in inaccurate descriptions. In this paper, we propose a novel Weakly Guided Attention Model with Hierarchical Interaction, named WGAM-HI, to improve Brain CT report generation. Specifically, WGAM-HI conducts many-to-many matching for multiple visual images and semantic sentences via a hierarchical interaction framework with a two-layer attention model and a two-layer report generator. In addition, two weakly guided mechanisms are proposed to facilitate the attention model to focus more on important images and lesion areas under the guidance of pathological events and Gradient-weighted Class Activation Mapping (Grad-CAM) respectively. The pathological event acts as a bridge between the essential serial images and the corresponding sentence, and the Grad-CAM bridges the lesion areas and pathology words. Therefore, under the hierarchical interaction with the weakly guided attention model, the report generator generates more accurate words and sentences. Experiments on the Brain CT dataset demonstrate the effectiveness of WGAM-HI in attending to important images and lesion areas gradually, and generating more accurate reports.

Learning Causal Biological Networks with Parallel Ant Colony Optimization Algorithm.

A wealth of causal relationships exists in biological systems, both causal brain networks and causal protein signaling networks are very classical causal biological networks (CBNs). Learning CBNs from biological signal data reliably is a critical problem today. However, most of the existing methods are not excellent enough in terms of accuracy and time performance, and tend to fall into local optima because they do not take full advantage of global information. In this paper, we propose a parallel ant colony optimization algorithm to learn causal biological networks from biological signal data, called PACO. Specifically, PACO first maps the construction of CBNs to ants, then searches for CBNs in parallel by simulating multiple groups of ants foraging, and finally obtains the optimal CBN through pheromone fusion and CBNs fusion between different ant colonies. Extensive experimental results on simulation data sets as well as two real-world data sets, the fMRI signal data set and the Single-cell data set, show that PACO can accurately and efficiently learn CBNs from biological signal data.

Amortization Transformer for Brain Effective Connectivity Estimation from fMRI Data.

Using machine learning methods to estimate brain effective connectivity networks from functional magnetic resonance imaging (fMRI) data has garnered significant attention in the fields of neuroinformatics and bioinformatics. However, existing methods usually require retraining the model for each subject, which ignores the knowledge shared across subjects. In this paper, we propose a novel framework for estimating effective connectivity based on an amortization transformer, named AT-EC. In detail, AT-EC first employs an amortization transformer to model the dynamics of fMRI time series and infer brain effective connectivity across different subjects, which can train an amortized model that leverages the shared knowledge from different subjects. Then, an assisted learning mechanism based on functional connectivity is designed to assist the estimation of the brain effective connectivity network. Experimental results on both simulated and real-world data demonstrate the efficacy of our method.

Deep Hashing Mutual Learning for Brain Network Classification.

Recently, clinical phenotypic semantic information has begun to play an important role in some brain network classification methods based on deep learning. However, most current methods only consider the phenotypic semantic information of individual brain networks but ignore the potential phenotypic characteristics among group brain networks. To address this problem, we present a deep hashing mutual learning (DHML)-based brain network classification method. Specifically, we first design a separable CNN-based deep hashing learning to extract individual topological features of brain networks and map them into hash codes. Secondly, we construct a group brain network relationship graph based on the similarity of phenotypic semantic information, in which each node is a brain network, and the properties of the nodes are the individual features extracted in the previous step. Then, we adopt a GCN-based deep hashing learning to extract the group topological features of the brain network and map them to hash codes. Finally, the two deep hashing learning models perform mutual learning by measuring the distribution differences between the hash codes to achieve the interaction of individual and group features. The experimental results on the three commonly used brain atlases (AAL Atlas, Dosenbach160 Atlas, and CC200 Atlas) of the ABIDE I dataset show that our proposed DHML method achieves optimal classification performance compared with some state-of-the-art methods.

Brain effective connectivity analysis facilitates the treatment outcome expectation of sound therapy in patients with tinnitus.

Tinnitus is associated with abnormal functional connectivity of multiple regions of the brain. However, previous analytic methods have disregarded information on the direction of functional connectivity, leading to only a moderate efficacy of pretreatment planning. We hypothesized that the pattern of directional functional connectivity can provide key information on treatment outcomes. Sixty-four participants were enrolled in this study: eighteen patients with tinnitus were categorized into the effective group, twenty-two patients into the ineffective group, and twenty-four healthy participants into the healthy control group. We acquired resting-state functional magnetic resonance images prior to sound therapy and constructed an effective connectivity network of the three groups using an artificial bee colony algorithm and transfer entropy. The key feature of patients with tinnitus was the significantly increased signal output of the sensory network, including the auditory, visual, and somatosensory networks, and parts of the motor network. This provided critical insights into the gain theory of tinnitus development. The altered pattern of functional information orchestration, represented by a higher degree of hypervigilance-driven attention and enhanced multisensory integration, may explain poor clinical outcomes. The activated gating function of the thalamus is one of the key factors for a good prognosis in tinnitus treatment. We developed a novel method for analyzing effective connectivity, facilitating an understanding of the tinnitus mechanism and treatment outcome expectation based on the direction of information flow.

A Survey on Brain Effective Connectivity Network Learning.

Human brain effective connectivity characterizes the causal effects of neural activities among different brain regions. Studies of brain effective connectivity networks (ECNs) for different populations contribute significantly to the understanding of the pathological mechanism associated with neuropsychiatric diseases and facilitate finding new brain network imaging markers for the early diagnosis and evaluation for the treatment of cerebral diseases. A deeper understanding of brain ECNs also greatly promotes brain-inspired artificial intelligence (AI) research in the context of brain-like neural networks and machine learning. Thus, how to picture and grasp deeper features of brain ECNs from functional magnetic resonance imaging (fMRI) data is currently an important and active research area of the human brain connectome. In this survey, we first show some typical applications and analyze existing challenging problems in learning brain ECNs from fMRI data. Second, we give a taxonomy of ECN learning methods from the perspective of computational science and describe some representative methods in each category. Third, we summarize commonly used evaluation metrics and conduct a performance comparison of several typical algorithms both on simulated and real datasets. Finally, we present the prospects and references for researchers engaged in learning ECNs.

Exploring Brain Effective Connectivity Networks Through Spatiotemporal Graph Convolutional Models.

Learning brain effective connectivity networks (ECN) from functional magnetic resonance imaging (fMRI) data has gained much attention in recent years. With the successful applications of deep learning in numerous fields, several brain ECN learning methods based on deep learning have been reported in the literature. However, current methods ignore the deep temporal features of fMRI data and fail to fully employ the spatial topological relationship between brain regions. In this article, we propose a novel method for learning brain ECN based on spatiotemporal graph convolutional models (STGCM), named STGCMEC, in which we first adopt the temporal convolutional network to extract the deep temporal features of fMRI data and utilize the graph convolutional network to update the spatial features of each brain region by aggregating information from neighborhoods, which makes the features of brain regions more discriminative. Then, based on such features of brain regions, we design a joint loss function to guide STGCMEC to learn the brain ECN, which includes a task prediction loss and a graph regularization loss. The experimental results on a simulated dataset and a real Alzheimer's disease neuroimaging initiative (ADNI) dataset show that the proposed STGCMEC is able to better learn brain ECN compared with some state-of-the-art methods.

Deep Forest With Multi-Channel Message Passing and Neighborhood Aggregation Mechanisms for Brain Network Classification.

As a novel deep learning method, deep forest has achieved excellent classification performance on many small-scale datasets, thus providing a new opportunity to accurately classify brain networks (BNs) on limited fMRI data. Though there are a few explorations about classifying BNs using deep forest, they only adopt sliding windows to extract adjacent features of BNs and fail to use prior knowledge to strengthen the features more relevant to brain diseases. In this paper, we propose a deep forest framework with multi-channel message passing and neighborhood aggregation mechanisms (DF-MCMPNA) to extract and aggregate long-range multi-channel topological features. Firstly, we use the three intrinsic connectivity networks (ICNs) and the whole-brain to form four feature extraction channels. Secondly, we present a multi-channel message passing mechanism and a channel-shared neighborhood aggregation mechanism to recursively extract long-range multi-channel topological features, where the first mechanism can learn local topological features in each channel and the second mechanism can fuse multi-channel topological features. Finally, the extracted features are fed into the casForst to perform further feature learning and classification. Experimental results on ABIDE I, ABIDE II, and ADHD-200 datasets show that the DF-MCMPNA outperforms several state-of-the-art methods on classification performance and accurately identifies abnormal brain regions.

Functional Brain Network Classification Based on Deep Graph Hashing Learning.

Brain network classification using resting-state functional magnetic resonance imaging (rs-fMRI) is an effective analytical method for diagnosing brain diseases. In recent years, brain network classification methods based on deep learning have attracted increasing attention. However, these methods only consider the spatial topological characteristics of the brain network but ignore its proximity relationships in semantic space. To overcome this problem, we propose a novel brain network classification method based on deep graph hashing learning named BNC-DGHL. Specifically, we first extract the deep features of the brain network and then learn a graph hash function based on clinical phenotype labels and the similarity of diagnostic labels. Secondly, we use the learned graph hash function to convert deep features into hash codes, which can maintain the original semantic spatial relationships. Finally, we calculate the distance between hash codes to obtain the predicted category of the brain network. Experimental results on ABIDE I, ABIDE II, and ADHD-200 datasets demonstrate that our method achieves better classification performance of brain diseases compared with some state-of-the-art methods, and the abnormal functional connectivities between brain regions identified may serve as biomarkers associated with related brain diseases.

Inferring Effective Connectivity Networks From fMRI Time Series With a Temporal Entropy-Score.

Inferring brain-effective connectivity networks from neuroimaging data has become a very hot topic in neuroinformatics and bioinformatics. In recent years, the search methods based on Bayesian network score have been greatly developed and become an emerging method for inferring effective connectivity. However, the previous score functions ignore the temporal information from functional magnetic resonance imaging (fMRI) series data and may not be able to determine all orientations in some cases. In this article, we propose a novel score function for inferring effective connectivity from fMRI data based on the conditional entropy and transfer entropy (TE) between brain regions. The new score employs the TE to capture the temporal information and can effectively infer connection directions between brain regions. Experimental results on both simulated and real-world data demonstrate the efficacy of our proposed score function.

Convolutional Neural Network With Sparse Strategies to Classify Dynamic Functional Connectivity.

Classification of dynamic functional connectivity (DFC) is becoming a promising approach for diagnosing various neurodegenerative diseases. However, the existing methods generally face the problem of overfitting. To solve it, this paper proposes a convolutional neural network with three sparse strategies named SCNN to classify DFC. Firstly, an element-wise filter is designed to impose sparse constraints on the DFC matrix by replacing the redundant elements with zeroes, where the DFC matrix is specially constructed to quantify the spatial and temporal variation of DFC. Secondly, a 1×1 convolutional filter is adopted to reduce the dimensionality of the sparse DFC matrix, and remove meaningless features resulted from zero elements in the subsequent convolution process. Finally, an extra sparse optimization classifier is employed to optimize the parameters of the above two filters, which can effectively improve the ability of SCNN to extract discriminative features. Experimental results on multiple resting-state fMRI datasets demonstrate that the proposed model provides a better classification performance of DFC compared with several state-of-the-art methods, and can identify the abnormal brain functional connectivity.

Deep attributed graph clustering with self-separation regularization and parameter-free cluster estimation.

Detecting clusters over attributed graphs is a fundamental task in the graph analysis field. The goal is to partition nodes into dense clusters based on both their attributes and structures. Modern graph neural networks provide facilitation to jointly capture the above information in attributed graphs with a feature aggregation manner, and have achieved great success in attributed graph clustering. However, existing methods mainly focus on capturing the proximity information in graphs and often fail to learn cluster-friendly features during the training of models. Besides, similar to many deep clustering frameworks, current methods based on graph neural networks require a preassigned cluster number before estimating the clusters. To address these limitations, we propose in this paper a deep attributed clustering method based on self-separated graph neural networks and parameter-free cluster estimation. First, to learn cluster-friendly features, we jointly optimize a jumping graph convolutional auto-encoder with a self-separation regularizer, which learns clusters with changing sizes while keeping dense intra-cluster structures and sparse inter structures. Second, an additional softmax auto-encoder is trained to determine the natural cluster number from the data. The hidden units capture cluster structures and can be used to estimate the number of clusters. Extensive experiments show the effectiveness of the proposed model.

Estimating Effective Connectivity by Recurrent Generative Adversarial Networks.

Estimating effective connectivity from functional magnetic resonance imaging (fMRI) time series data has become a very hot topic in neuroinformatics and brain informatics. However, it is hard for the current methods to accurately estimate the effective connectivity due to the high noise and small sample size of fMRI data. In this paper, we propose a novel framework for estimating effective connectivity based on recurrent generative adversarial networks, called EC-RGAN. The proposed framework employs the generator that consists of a set of effective connectivity generators based on recurrent neural networks to generate the fMRI time series of each brain region, and uses the discriminator to distinguish between the joint distributions of the real and generated fMRI time series. When the model is well-trained and generated fMRI data is similar to real fMRI data, EC-RGAN outputs the effective connectivity by means of the causal parameters of the effective connectivity generators. Experimental results on both simulated and real-world fMRI time series data demonstrate the efficacy of our proposed framework.

Convolutional Neural Network With Graphical Lasso to Extract Sparse Topological Features for Brain Disease Classification.

The functional connectivity provides new insights into the mechanisms of the human brain at network-level, which has been proved to be an effective biomarker for brain disease classification. Recently, machine learning methods have played an important role in functional connectivity classification, among which convolutional neural network (CNN) based methods become a new hot topic since they can extract topological features in the brain network. However, the conventional CNN-based methods haven't taken sparse connectivity patterns (SCPs) of the human brain into consideration, which may lead to redundancy of the topological features, and limit their performance and generalization. To solve it, we propose a novel CNN-based model with graphical Lasso (CNNGLasso) to extract sparse topological features for brain disease classification. First, we develop a novel graphical Lasso model for revealing the SCPs at group-level. Then, the SCPs are used to guide the topological feature extraction. Finally, the obtained sparse topological features are used to classify the patients from normal controls. The experiment results on the ABIDE dataset demonstrate that the CNNGLasso outperforms the others on various performances. Besides, the abnormal brain regions derived from the trained model are consistent with the previous investigations, which further proves the application prospect of the CNNGLasso.

Deep scaled dot-product attention based domain adaptation model for biomedical question answering.

Biomedical text mining is becoming increasingly important as the number of biomedical documents grow rapidly. Deep learning has boosted the development of biomedical text mining models. However, as deep learning models require a large amount of training data, a hierarchical attention based transfer learning model is proposed in this paper for the question answering task in biomedical field which lacks of sufficient training data. We adopt BERT (Bidirectional Encoder Representation Transformers), which has the ability to learn from large-scale unsupervised data, to enrich the semantic representation in our model. Especially, the scaled dot-product attention mechanism captures the question interaction clues for passage encoding. The domain adaptation technique of fine-tuning is used to reinforce the performance, which penalizes the deviations from the source model's parameters and remembers the knowledge of source domain. We evaluate the system performance on the open data set of BioASQ-Task B. The results show that our system achieves the state-of-the-art performance without any handcrafted features and outperforms the best solution for factoid questions in 2016 and 2017 BioASQ-Task B.

Learning Brain Effective Connectivity Network Structure Using Ant Colony Optimization Combining With Voxel Activation Information.

Learning brain effective connectivity (EC) networks from functional magnetic resonance imaging (fMRI) data has become a new hot topic in the neuroinformatics field. However, how to accurately and efficiently learn brain EC networks is still a challenging problem. In this paper, we propose a new algorithm to learn the brain EC network structure using ant colony optimization (ACO) algorithm combining with voxel activation information, named as VACOEC. First, VACOEC uses the voxel activation information to measure the independence between each pair of brain regions and effectively restricts the space of candidate solutions, which makes many unnecessary searches of ants be avoided. Then, by combining the global score increase of a solution with the voxel activation information, a new heuristic function is designed to guide the process of ACO to search for the optimal solution. The experimental results on simulated datasets show that the proposed method can accurately and efficiently identify the directions of the brain EC networks. Moreover, the experimental results on real-world data show that patients with Alzheimers disease (AD) exhibit decreased effective connectivity not only in the intra-network within the default mode network (DMN) and salience network (SN), but also in the inter-network between DMN and SN, compared with normal control (NC) subjects. The experimental results demonstrate that VACOEC is promising for practical applications in the neuroimaging studies of geriatric subjects and neurological patients.

Network Representations of Facial and Bodily Expressions: Evidence From Multivariate Connectivity Pattern Classification.

Emotions can be perceived from both facial and bodily expressions. Our previous study has found the successful decoding of facial expressions based on the functional connectivity (FC) patterns. However, the role of the FC patterns in the recognition of bodily expressions remained unclear, and no neuroimaging studies have adequately addressed the question of whether emotions perceiving from facial and bodily expressions are processed rely upon common or different neural networks. To address this, the present study collected functional magnetic resonance imaging (fMRI) data from a block design experiment with facial and bodily expression videos as stimuli (three emotions: anger, fear, and joy), and conducted multivariate pattern classification analysis based on the estimated FC patterns. We found that in addition to the facial expressions, bodily expressions could also be successfully decoded based on the large-scale FC patterns. The emotion classification accuracies for the facial expressions were higher than that for the bodily expressions. Further contributive FC analysis showed that emotion-discriminative networks were widely distributed in both hemispheres, containing regions that ranged from primary visual areas to higher-level cognitive areas. Moreover, for a particular emotion, discriminative FCs for facial and bodily expressions were distinct. Together, our findings highlight the key role of the FC patterns in the emotion processing, indicating how large-scale FC patterns reconfigure in processing of facial and bodily expressions, and suggest the distributed neural representation for the emotion recognition. Furthermore, our results also suggest that the human brain employs separate network representations for facial and bodily expressions of the same emotions. This study provides new evidence for the network representations for emotion perception and may further our understanding of the potential mechanisms underlying body language emotion recognition.