Brain-Inspired Spatio-Temporal Associative Memories for Neuroimaging Data Classification: EEG and fMRI.

Humans learn from a lot of information sources to make decisions. Once this information is learned in the brain, spatio-temporal associations are made, connecting all these sources (variables) in space and time represented as brain connectivity. In reality, to make a decision, we usually have only part of the information, either as a limited number of variables, limited time to make the decision, or both. The brain functions as a spatio-temporal associative memory. Inspired by the ability of the human brain, a brain-inspired spatio-temporal associative memory was proposed earlier that utilized the NeuCube brain-inspired spiking neural network framework. Here we applied the STAM framework to develop STAM for neuroimaging data, on the cases of EEG and fMRI, resulting in STAM-EEG and STAM-fMRI. This paper showed that once a NeuCube STAM classification model was trained on a complete spatio-temporal EEG or fMRI data, it could be recalled using only part of the time series, or/and only part of the used variables. We evaluated both temporal and spatial association and generalization accuracy accordingly. This was a pilot study that opens the field for the development of classification systems on other neuroimaging data, such as longitudinal MRI data, trained on complete data but recalled on partial data. Future research includes STAM that will work on data, collected across different settings, in different labs and clinics, that may vary in terms of the variables and time of data collection, along with other parameters. The proposed STAM will be further investigated for early diagnosis and prognosis of brain conditions and for diagnostic/prognostic marker discovery.

Spiking neural networks for predictive and explainable modelling of multimodal streaming data with a case study on financial time series and online news.

In a first study, this paper argues and demonstrates that spiking neural networks (SNN) can be successfully used for predictive and explainable modelling of multimodal streaming data. The paper proposes a new method, where both time series and on-line news are integrated as numerical streaming data in the same time domain and then used to train incrementally a SNN model. The connectivity and the spiking activity of the SNN are then analyzed through clustering and dynamic graph extraction to reveal on-line interaction between all input variables in regard to the predicted one. The paper answers the main research question of how to understand the dynamic interaction of time series and on-line news through their integrative modelling. It offers a new method to evaluate the efficiency of using on-line news on the predictive modelling of time series. Results on financial stock time series and online news are presented. In contrast to traditional machine learning techniques, the method reveals the dynamic interaction between stock variables and news and their dynamic impact on model accuracy when compared to models that do not use news information. Along with the used financial data, the method is applicable to a wide range of other multimodal time series and news data, such as economic, medical, environmental and social. The proposed method, being based on SNN, promotes the use of massively parallel and low energy neuromorphic hardware for multivariate on-line data modelling.

GeSeNet: A General Semantic-Guided Network With Couple Mask Ensemble for Medical Image Fusion.

At present, multimodal medical image fusion technology has become an essential means for researchers and doctors to predict diseases and study pathology. Nevertheless, how to reserve more unique features from different modal source images on the premise of ensuring time efficiency is a tricky problem. To handle this issue, we propose a flexible semantic-guided architecture with a mask-optimized framework in an end-to-end manner, termed as GeSeNet. Specifically, a region mask module is devised to deepen the learning of important information while pruning redundant computation for reducing the runtime. An edge enhancement module and a global refinement module are presented to modify the extracted features for boosting the edge textures and adjusting overall visual performance. In addition, we introduce a semantic module that is cascaded with the proposed fusion network to deliver semantic information into our generated results. Sufficient qualitative and quantitative comparative experiments (i.e., MRI-CT, MRI-PET, and MRI-SPECT) are deployed between our proposed method and ten state-of-the-art methods, which shows our generated images lead the way. Moreover, we also conduct operational efficiency comparisons and ablation experiments to prove that our proposed method can perform excellently in the field of multimodal medical image fusion. The code is available at https://github.com/lok-18/GeSeNet.

Integrating Spatial and Temporal Information for Violent Activity Detection from Video Using Deep Spiking Neural Networks.

Increasing violence in workplaces such as hospitals seriously challenges public safety. However, it is time- and labor-consuming to visually monitor masses of video data in real time. Therefore, automatic and timely violent activity detection from videos is vital, especially for small monitoring systems. This paper proposes a two-stream deep learning architecture for video violent activity detection named SpikeConvFlowNet. First, RGB frames and their optical flow data are used as inputs for each stream to extract the spatiotemporal features of videos. After that, the spatiotemporal features from the two streams are concatenated and fed to the classifier for the final decision. Each stream utilizes a supervised neural network consisting of multiple convolutional spiking and pooling layers. Convolutional layers are used to extract high-quality spatial features within frames, and spiking neurons can efficiently extract temporal features across frames by remembering historical information. The spiking neuron-based optical flow can strengthen the capability of extracting critical motion information. This method combines their advantages to enhance the performance and efficiency for recognizing violent actions. The experimental results on public datasets demonstrate that, compared with the latest methods, this approach greatly reduces parameters and achieves higher inference efficiency with limited accuracy loss. It is a potential solution for applications in embedded devices that provide low computing power but require fast processing speeds.

Denoising of Fluorescence Image on the Surface of Quantum Dot/Nanoporous Silicon Biosensors.

In the process of biological detection of porous silicon photonic crystals based on quantum dots, the concentration of target organisms can be indirectly measured via the change in the gray value of the fluorescence emitted from the quantum dots in the porous silicon pores before and after the biological reaction on the surface of the device. However, due to the disordered nanostructures in porous silicon and the roughness of the surface, the fluorescence images on the surface contain noise. This paper analyzes the type of noise and its influence on the gray value of fluorescent images. The change in the gray value caused by noise greatly reduces the detection sensitivity. To reduce the influence of noise on the gray value of quantum dot fluorescence images, this paper proposes a denoising method based on gray compression and nonlocal anisotropic diffusion filtering. We used the proposed method to denoise the quantum dot fluorescence image after DNA hybridization in a Bragg structure porous silicon device. The experimental results show that the sensitivity of digital image detection improved significantly after denoising.

DNA Matrix Operation Based on the Mechanism of the DNAzyme Binding to Auxiliary Strands to Cleave the Substrate.

Numerical computation is a focus of DNA computing, and matrix operations are among the most basic and frequently used operations in numerical computation. As an important computing tool, matrix operations are often used to deal with intensive computing tasks. During calculation, the speed and accuracy of matrix operations directly affect the performance of the entire computing system. Therefore, it is important to find a way to perform matrix calculations that can ensure the speed of calculations and improve the accuracy. This paper proposes a DNA matrix operation method based on the mechanism of the DNAzyme binding to auxiliary strands to cleave the substrate. In this mechanism, the DNAzyme binding substrate requires the connection of two auxiliary strands. Without any of the two auxiliary strands, the DNAzyme does not cleave the substrate. Based on this mechanism, the multiplication operation of two matrices is realized; the two types of auxiliary strands are used as elements of the two matrices, to participate in the operation, and then are combined with the DNAzyme to cut the substrate and output the result of the matrix operation. This research provides a new method of matrix operations and provides ideas for more complex computing systems.

Prediction of Acoustic Residual Inhibition of Tinnitus Using a Brain-Inspired Spiking Neural Network Model.

Auditory Residual Inhibition (ARI) is a temporary suppression of tinnitus that occurs in some people following the presentation of masking sounds. Differences in neural response to ARI stimuli may enable classification of tinnitus and a tailored approach to intervention in the future. In an exploratory study, we investigated the use of a brain-inspired artificial neural network to examine the effects of ARI on electroencephalographic function, as well as the predictive ability of the model. Ten tinnitus patients underwent two auditory stimulation conditions (constant and amplitude modulated broadband noise) at two time points and were then characterised as responders or non-responders, based on whether they experienced ARI or not. Using a spiking neural network model, we evaluated concurrent neural patterns generated across space and time from features of electroencephalographic data, capturing the neural dynamic changes before and after stimulation. Results indicated that the model may be used to predict the effect of auditory stimulation on tinnitus on an individual basis. This approach may aid in the development of predictive models for treatment selection.

Detail Preserving Low Illumination Image and Video Enhancement Algorithm Based on Dark Channel Prior.

In low illumination situations, insufficient light in the monitoring device results in poor visibility of effective information, which cannot meet practical applications. To overcome the above problems, a detail preserving low illumination video image enhancement algorithm based on dark channel prior is proposed in this paper. First, a dark channel refinement method is proposed, which is defined by imposing a structure prior to the initial dark channel to improve the image brightness. Second, an anisotropic guided filter (AnisGF) is used to refine the transmission, which preserves the edges of the image. Finally, a detail enhancement algorithm is proposed to avoid the problem of insufficient detail in the initial enhancement image. To avoid video flicker, the next video frames are enhanced based on the brightness of the first enhanced frame. Qualitative and quantitative analysis shows that the proposed algorithm is superior to the contrast algorithm, in which the proposed algorithm ranks first in average gradient, edge intensity, contrast, and patch-based contrast quality index. It can be effectively applied to the enhancement of surveillance video images and for wider computer vision applications.

Pulsewidth Modulation-Based Algorithm for Spike Phase Encoding and Decoding of Time-Dependent Analog Data.

This article proposes a new spike encoding and decoding algorithm for analog data. The algorithm uses the pulsewidth modulation principles to achieve a high reconstruction accuracy of the signal, along with a high level of data compression. Two benchmark data sets are used to illustrate the method: stock index time series and human voice data. Applications of the method for spiking neural network (SNN) modeling and neuromorphic implementations are discussed. The proposed method would allow the development of new applications of SNNs as regression techniques for predictive time-series modeling.

Speckle Noise Removal in Image-based Detection of Refractive Index Changes in Porous Silicon Microarrays.

Based on porous silicon (PSi) microarray images, we propose a new method called the phagocytosis algorithm (PGY) for removing the influence of speckle noise on image gray values. In a theoretical analysis, speckle noise of different intensities is added to images, and a suitable denoising method is developed to restore the image gray level. This method can be used to reduce the influence of speckle noise on the gray values of PSi microarray images to improve the accuracy of detection and increase detection sensitivity. In experiments, the method is applied to detect refractive index changes in PSi microcavity images, and a good linear relationship between the gray level change and the refractive index change is obtained. In addition, the algorithm is applied to a PSi microarray image, and good results are obtained.

Anytime multipurpose emotion recognition from EEG data using a Liquid State Machine based framework.

Recent technological advances in machine learning offer the possibility of decoding complex datasets and discern latent patterns. In this study, we adopt Liquid State Machines (LSM) to recognize the emotional state of an individual based on EEG data. LSM were applied to a previously validated EEG dataset where subjects view a battery of emotional film clips and then rate their degree of emotion during each film based on valence, arousal, and liking levels. We introduce LSM as a model for an automatic feature extraction and prediction from raw EEG with potential extension to a wider range of applications. We also elaborate on how to exploit the separation property in LSM to build a multipurpose and anytime recognition framework, where we used one trained model to predict valence, arousal and liking levels at different durations of the input. Our simulations showed that the LSM-based framework achieve outstanding results in comparison with other works using different emotion prediction scenarios with cross validation.

Mapping, Learning, Visualization, Classification, and Understanding of fMRI Data in the NeuCube Evolving Spatiotemporal Data Machine of Spiking Neural Networks.

This paper introduces a new methodology for dynamic learning, visualization, and classification of functional magnetic resonance imaging (fMRI) as spatiotemporal brain data. The method is based on an evolving spatiotemporal data machine of evolving spiking neural networks (SNNs) exemplified by the NeuCube architecture [1]. The method consists of several steps: mapping spatial coordinates of fMRI data into a 3-D SNN cube (SNNc) that represents a brain template; input data transformation into trains of spikes; deep, unsupervised learning in the 3-D SNNc of spatiotemporal patterns from data; supervised learning in an evolving SNN classifier; parameter optimization; and 3-D visualization and model interpretation. Two benchmark case study problems and data are used to illustrate the proposed methodology-fMRI data collected from subjects when reading affirmative or negative sentences and another one-on reading a sentence or seeing a picture. The learned connections in the SNNc represent dynamic spatiotemporal relationships derived from the fMRI data. They can reveal new information about the brain functions under different conditions. The proposed methodology allows for the first time to analyze dynamic functional and structural connectivity of a learned SNN model from fMRI data. This can be used for a better understanding of brain activities and also for online generation of appropriate neurofeedback to subjects for improved brain functions. For example, in this paper, tracing the 3-D SNN model connectivity enabled us for the first time to capture prominent brain functional pathways evoked in language comprehension. We found stronger spatiotemporal interaction between left dorsolateral prefrontal cortex and left temporal while reading a negated sentence. This observation is obviously distinguishable from the patterns generated by either reading affirmative sentences or seeing pictures. The proposed NeuCube-based methodology offers also a superior classification accuracy when compared with traditional AI and statistical methods. The created NeuCube-based models of fMRI data are directly and efficiently implementable on high performance and low energy consumption neuromorphic platforms for real-time applications.

Network-Based Method for Inferring Cancer Progression at the Pathway Level from Cross-Sectional Mutation Data.

Large-scale cancer genomics projects are providing a wealth of somatic mutation data from a large number of cancer patients. However, it is difficult to obtain several samples with a temporal order from one patient in evaluating the cancer progression. Therefore, one of the most challenging problems arising from the data is to infer the temporal order of mutations across many patients. To solve the problem efficiently, we present a Network-based method (NetInf) to Infer cancer progression at the pathway level from cross-sectional data across many patients, leveraging on the exclusive property of driver mutations within a pathway and the property of linear progression between pathways. To assess the robustness of NetInf, we apply it on simulated data with the addition of different levels of noise. To verify the performance of NetInf, we apply it to analyze somatic mutation data from three real cancer studies with large number of samples. Experimental results reveal that the pathways detected by NetInf show significant enrichment. Our method reduces computational complexity by constructing gene networks without assigning the number of pathways, which also provides new insights on the temporal order of somatic mutations at the pathway level rather than at the gene level.

A Spiking Neural Network Methodology and System for Learning and Comparative Analysis of EEG Data From Healthy Versus Addiction Treated Versus Addiction Not Treated Subjects.

This paper introduces a method utilizing spiking neural networks (SNN) for learning, classification, and comparative analysis of brain data. As a case study, the method was applied to electroencephalography (EEG) data collected during a GO/NOGO cognitive task performed by untreated opiate addicts, those undergoing methadone maintenance treatment (MMT) for opiate dependence and a healthy control group. METHODS: the method is based on an SNN architecture called NeuCube, trained on spatiotemporal EEG data. OBJECTIVE: NeuCube was used to classify EEG data across subject groups and across GO versus NOGO trials, but also facilitated a deeper comparative analysis of the dynamic brain processes. RESULTS: This analysis results in a better understanding of human brain functioning across subject groups when performing a cognitive task. In terms of the EEG data classification, a NeuCube model obtained better results (the maximum obtained accuracy: 90.91%) when compared with traditional statistical and artificial intelligence methods (the maximum obtained accuracy: 50.55%). SIGNIFICANCE: more importantly, new information about the effects of MMT on cognitive brain functions is revealed through the analysis of the SNN model connectivity and its dynamics. CONCLUSION: this paper presented a new method for EEG data modeling and revealed new knowledge on brain functions associated with mental activity which is different from the brain activity observed in a resting state of the same subjects.

NeuCube: a spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data.

The brain functions as a spatio-temporal information processing machine. Spatio- and spectro-temporal brain data (STBD) are the most commonly collected data for measuring brain response to external stimuli. An enormous amount of such data has been already collected, including brain structural and functional data under different conditions, molecular and genetic data, in an attempt to make a progress in medicine, health, cognitive science, engineering, education, neuro-economics, Brain-Computer Interfaces (BCI), and games. Yet, there is no unifying computational framework to deal with all these types of data in order to better understand this data and the processes that generated it. Standard machine learning techniques only partially succeeded and they were not designed in the first instance to deal with such complex data. Therefore, there is a need for a new paradigm to deal with STBD. This paper reviews some methods of spiking neural networks (SNN) and argues that SNN are suitable for the creation of a unifying computational framework for learning and understanding of various STBD, such as EEG, fMRI, genetic, DTI, MEG, and NIRS, in their integration and interaction. One of the reasons is that SNN use the same computational principle that generates STBD, namely spiking information processing. This paper introduces a new SNN architecture, called NeuCube, for the creation of concrete models to map, learn and understand STBD. A NeuCube model is based on a 3D evolving SNN that is an approximate map of structural and functional areas of interest of the brain related to the modeling STBD. Gene information is included optionally in the form of gene regulatory networks (GRN) if this is relevant to the problem and the data. A NeuCube model learns from STBD and creates connections between clusters of neurons that manifest chains (trajectories) of neuronal activity. Once learning is applied, a NeuCube model can reproduce these trajectories, even if only part of the input STBD or the stimuli data is presented, thus acting as an associative memory. The NeuCube framework can be used not only to discover functional pathways from data, but also as a predictive system of brain activities, to predict and possibly, prevent certain events. Analysis of the internal structure of a model after training can reveal important spatio-temporal relationships 'hidden' in the data. NeuCube will allow the integration in one model of various brain data, information and knowledge, related to a single subject (personalized modeling) or to a population of subjects. The use of NeuCube for classification of STBD is illustrated in a case study problem of EEG data. NeuCube models result in a better accuracy of STBD classification than standard machine learning techniques. They are robust to noise (so typical in brain data) and facilitate a better interpretation of the results and understanding of the STBD and the brain conditions under which data was collected. Future directions for the use of SNN for STBD are discussed.

LDA merging and splitting with applications to multiagent cooperative learning and system alteration.

To adapt linear discriminant analysis (LDA) to real-world applications, there is a pressing need to equip it with an incremental learning ability to integrate knowledge presented by one-pass data streams, a functionality to join multiple LDA models to make the knowledge sharing between independent learning agents more efficient, and a forgetting functionality to avoid reconstruction of the overall discriminant eigenspace caused by some irregular changes. To this end, we introduce two adaptive LDA learning methods: LDA merging and LDA splitting. These provide the benefits of ability of online learning with one-pass data streams, retained class separability identical to the batch learning method, high efficiency for knowledge sharing due to condensed knowledge representation by the eigenspace model, and more preferable time and storage costs than traditional approaches under common application conditions. These properties are validated by experiments on a benchmark face image data set. By a case study on the application of the proposed method to multiagent cooperative learning and system alternation of a face recognition system, we further clarified the adaptability of the proposed methods to complex dynamic learning tasks.

Computational neurogenetic modelling: a pathway to new discoveries in genetic neuroscience.

The paper presents a methodology for using computational neurogenetic modelling (CNGM) to bring new original insights into how genes influence the dynamics of brain neural networks. CNGM is a novel computational approach to brain neural network modelling that integrates dynamic gene networks with artificial neural network model (ANN). Interaction of genes in neurons affects the dynamics of the whole ANN model through neuronal parameters, which are no longer constant but change as a function of gene expression. Through optimization of interactions within the internal gene regulatory network (GRN), initial gene/protein expression values and ANN parameters, particular target states of the neural network behaviour can be achieved, and statistics about gene interactions can be extracted. In such a way, we have obtained an abstract GRN that contains predictions about particular gene interactions in neurons for subunit genes of AMPA, GABAA and NMDA neuro-receptors. The extent of sequence conservation for 20 subunit proteins of all these receptors was analysed using standard bioinformatics multiple alignment procedures. We have observed abundance of conserved residues but the most interesting observation has been the consistent conservation of phenylalanine (F at position 269) and leucine (L at position 353) in all 20 proteins with no mutations. We hypothesise that these regions can be the basis for mutual interactions. Existing knowledge on evolutionary linkage of their protein families and analysis at molecular level indicate that the expression of these individual subunits should be coordinated, which provides the biological justification for our optimized GRN.