Snn and sound: a comprehensive review of spiking neural networks in sound.

The rapid advancement of AI and machine learning has significantly enhanced sound and acoustic recognition technologies, moving beyond traditional models to more sophisticated neural network-based methods. Among these, Spiking Neural Networks (SNNs) are particularly noteworthy. SNNs mimic biological neurons and operate on principles similar to the human brain, using analog computing mechanisms. This capability allows for efficient sound processing with low power consumption and minimal latency, ideal for real-time applications in embedded systems. This paper reviews recent developments in SNNs for sound recognition, underscoring their potential to overcome the limitations of digital computing and suggesting directions for future research. The unique attributes of SNNs could lead to breakthroughs in mimicking human auditory processing more closely.

Review on spiking neural network-based ECG classification methods for low-power environments.

This paper reviews arrhythmia classification studies using electrocardiogram (ECG) signals. Research on automatically diagnosing arrhythmia in daily life has been actively underway for early detection and treatment of heart disease. Development of automatic arrhythmia classification using ECG signal began based on handcrafted morphological feature extraction and machine learning-based classification methods. As deep neural networks (DNN) show excellent performance in the signal processing field, studies using various types of DNN are also being conducted in ECG classification. However, these DNN-based studies have extremely high computational complexity, making it challenging to perform real-time classification, and are unsuitable for low-power environments such as wearable devices due to high power consumption. Currently, research based on spiking neural network (SNN), which mimics the low-power operation of the human nervous system, is attracting attention as a method that can dramatically reduce complexity and power consumption. The classification accuracy of the SNN-based ECG classification studies is close to that of the DNN-based studies. When combined with neuromorphic hardware, it shows ultra-low-power performance, suggesting the possibility of use in lightweight devices. In this paper, the SNN-based ECG classification studies for low-power environments are mainly reviewed, and prior to this, conventional and DNN-based ECG classification studies are also reviewed. We hope that this review will be helpful to researchers and engineers interested in the field of ECG classification.

Spiking Neural Network Integrated with Impact Ionization Field-Effect Transistor Neuron and a Ferroelectric Field-Effect Transistor Synapse.

The integration of artificial spiking neurons based on steep-switching logic devices and artificial synapses with neuromorphic functions enables an energy-efficient computer architecture that mimics the human brain well, known as a spiking neural network (SNN). 2D materials with impact ionization or ferroelectric characteristics have the potential for use in such devices. However, research on 2D spiking neurons remains limited and investigations of 2D artificial synapses far more common. An innovative 2D spiking neuron is implemented using a WSe2 impact ionization transistor (I2FET), while a spiking neural network is formed by combining it with a 2D ferroelectric synaptic device (FeFET). The suggested 2D spiking neuron demonstrates precise spiking behavior that closely resembles that of actual neurons. In addition, it achieves a low energy consumption of 2 pJ/spike. The better impact ionization properties of WSe2 are responsible for this efficiency. Furthermore, an all-2D SNN consisting of 2D I2FET neurons and 2D FeFET synapses is constructed, which achieves high accuracy of 87.5% in a face classification task by unsupervised learning. The integration of a 2D SNN with 2D steep-switching spiking neuronal devices and 2D synaptic devices shows great potential for the development of neuromorphic systems with improved energy efficiency and computational capabilities.

Predicting the temporal-dynamic trajectories of cortical neuronal responses in non-human primates based on deep spiking neural network.

Deep convolutional neural networks (CNNs) are commonly used as computational models for the primate ventral stream, while deep spiking neural networks (SNNs) incorporated with both the temporal and spatial spiking information still lack investigation. We compared performances of SNN and CNN in prediction of visual responses to the naturalistic stimuli in area V4, inferior temporal (IT), and orbitofrontal cortex (OFC). The accuracies based on SNN were significantly higher than that of CNN in prediction of temporal-dynamic trajectory and averaged firing rate of visual response in V4 and IT. The temporal dynamics were captured by SNN for neurons with diverse temporal profiles and category selectivities, and most sensitively captured around the time of peak responses for each brain region. Consistently, SNN activities showed significantly stronger correlations with IT, V4 and OFC responses. In SNN, correlations with neural activities were stronger for later time-step features than early time-step features. The temporal-dynamic prediction was also significantly improved by considering preceding neural activities during the prediction. Thus, our study demonstrated SNN as a powerful temporal-dynamic model for cortical responses to complex naturalistic stimuli.

Auto-Spikformer: Spikformer architecture search.

Introduction: The integration of self-attention mechanisms into Spiking Neural Networks (SNNs) has garnered considerable interest in the realm of advanced deep learning, primarily due to their biological properties. Recent advancements in SNN architecture, such as Spikformer, have demonstrated promising outcomes. However, we observe that Spikformer may exhibit excessive energy consumption, potentially attributable to redundant channels and blocks. Methods: To mitigate this issue, we propose a one-shot Spiking Transformer Architecture Search method, namely Auto-Spikformer. Auto-Spikformer extends the search space to include both transformer architecture and SNN inner parameters. We train and search the supernet based on weight entanglement, evolutionary search, and the proposed Discrete Spiking Parameters Search (DSPS) methods. Benefiting from these methods, the performance of subnets with weights inherited from the supernet without even retraining is comparable to the original Spikformer. Moreover, we propose a new fitness function aiming to find a Pareto optimal combination balancing energy consumption and accuracy. Results and discussion: Our experimental results demonstrate the effectiveness of Auto-Spikformer, which outperforms the original Spikformer and most CNN or ViT models with even fewer parameters and lower energy consumption.

BayesianSpikeFusion: accelerating spiking neural network inference via Bayesian fusion of early prediction.

Spiking neural networks (SNNs) have garnered significant attention due to their notable energy efficiency. However, conventional SNNs rely on spike firing frequency to encode information, necessitating a fixed sampling time and leaving room for further optimization. This study presents a novel approach to reduce sampling time and conserve energy by extracting early prediction results from the intermediate layer of the network and integrating them with the final layer's predictions in a Bayesian fashion. Experimental evaluations conducted on image classification tasks using MNIST, CIFAR-10, and CIFAR-100 datasets demonstrate the efficacy of our proposed method when applied to VGGNets and ResNets models. Results indicate a substantial energy reduction of 38.8% in VGGNets and 48.0% in ResNets, illustrating the potential for achieving significant efficiency gains in spiking neural networks. These findings contribute to the ongoing research in enhancing the performance of SNNs, facilitating their deployment in resource-constrained environments. Our code is available on GitHub: https://github.com/hanebarla/BayesianSpikeFusion.

Research on low-power driving fatigue monitoring method based on spiking neural network.

Fatigue driving is one of the leading causes of traffic accidents, and the rapid and accurate detection of driver fatigue is of paramount importance for enhancing road safety. However, the application of deep learning models in fatigue driving detection has long been constrained by high computational costs and power consumption. To address this issue, this study proposes an approach that combines Self-Organizing Map (SOM) and Spiking Neural Networks (SNN) to develop a low-power model capable of accurately recognizing the driver's mental state. Initially, spatial features are extracted from electroencephalogram (EEG) signals using the SOM network. Subsequently, the extracted weight vectors are encoded and fed into the SNN for fatigue driving classification. The research results demonstrate that the proposed method effectively considers the spatiotemporal characteristics of EEG signals, achieving efficient fatigue detection. Simultaneously, this approach successfully reduces the model's power consumption. When compared to traditional artificial neural networks, our method reduces energy consumption by approximately 12.21-42.59%.

Direct training high-performance deep spiking neural networks: a review of theories and methods.

Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks (ANNs), in virtue of their high biological plausibility, rich spatial-temporal dynamics, and event-driven computation. The direct training algorithms based on the surrogate gradient method provide sufficient flexibility to design novel SNN architectures and explore the spatial-temporal dynamics of SNNs. According to previous studies, the performance of models is highly dependent on their sizes. Recently, direct training deep SNNs have achieved great progress on both neuromorphic datasets and large-scale static datasets. Notably, transformer-based SNNs show comparable performance with their ANN counterparts. In this paper, we provide a new perspective to summarize the theories and methods for training deep SNNs with high performance in a systematic and comprehensive way, including theory fundamentals, spiking neuron models, advanced SNN models and residual architectures, software frameworks and neuromorphic hardware, applications, and future trends.

Real-time execution of SNN models with synaptic plasticity for handwritten digit recognition on SIMD hardware.

Recent advancements in neuromorphic computing have led to the development of hardware architectures inspired by Spiking Neural Networks (SNNs) to emulate the efficiency and parallel processing capabilities of the human brain. This work focuses on testing the HEENS architecture, specifically designed for high parallel processing and biological realism in SNN emulation, implemented on a ZYNQ family FPGA. The study applies this architecture to the classification of digits using the well-known MNIST database. The image resolutions were adjusted to match HEENS' processing capacity. Results were compared with existing work, demonstrating HEENS' performance comparable to other solutions. This study highlights the importance of balancing accuracy and efficiency in the execution of applications. HEENS offers a flexible solution for SNN emulation, allowing for the implementation of programmable neural and synaptic models. It encourages the exploration of novel algorithms and network architectures, providing an alternative for real-time processing with efficient energy consumption.

Multiscale modeling of neuronal dynamics in hippocampus CA1.

The development of biologically realistic models of brain microcircuits and regions constitutes currently a very relevant topic in computational neuroscience. One of the main challenges of such models is the passage between different scales, going from the microscale (cellular) to the meso (microcircuit) and macroscale (region or whole-brain level), while keeping at the same time a constraint on the demand of computational resources. In this paper we introduce a multiscale modeling framework for the hippocampal CA1, a region of the brain that plays a key role in functions such as learning, memory consolidation and navigation. Our modeling framework goes from the single cell level to the macroscale and makes use of a novel mean-field model of CA1, introduced in this paper, to bridge the gap between the micro and macro scales. We test and validate the model by analyzing the response of the system to the main brain rhythms observed in the hippocampus and comparing our results with the ones of the corresponding spiking network model of CA1. Then, we analyze the implementation of synaptic plasticity within our framework, a key aspect to study the role of hippocampus in learning and memory consolidation, and we demonstrate the capability of our framework to incorporate the variations at synaptic level. Finally, we present an example of the implementation of our model to study a stimulus propagation at the macro-scale level, and we show that the results of our framework can capture the dynamics obtained in the corresponding spiking network model of the whole CA1 area.

SNN-BERT: Training-efficient Spiking Neural Networks for energy-efficient BERT.

Spiking Neural Networks (SNNs) are naturally suited to process sequence tasks such as NLP with low power, due to its brain-inspired spatio-temporal dynamics and spike-driven nature. Current SNNs employ "repeat coding" that re-enter all input tokens at each timestep, which fails to fully exploit temporal relationships between the tokens and introduces memory overhead. In this work, we align the number of input tokens with the timestep and refer to this input coding as "individual coding". To cope with the increase in training time for individual encoded SNNs due to the dramatic increase in timesteps, we design a Bidirectional Parallel Spiking Neuron (BPSN) with following features: First, BPSN supports spike parallel computing and effectively avoids the issue of uninterrupted firing; Second, BPSN excels in handling adaptive sequence length tasks, which is a capability that existing work does not have; Third, the fusion of bidirectional information enhances the temporal information modeling capabilities of SNNs; To validate the effectiveness of our BPSN, we present the SNN-BERT, a deep direct training SNN architecture based on the BERT model in NLP. Compared to prior repeat 4-timestep coding baseline, our method achieves a 6.46× reduction in energy consumption and a significant 16.1% improvement, raising the performance upper bound of the SNN domain on the GLUE dataset to 74.4%. Additionally, our method achieves 3.5× training acceleration and 3.8× training memory optimization. Compared with artificial neural networks of similar architecture, we obtain comparable performance but up to 22.5× energy efficiency. We would provide the codes.

Spike-Based Neuromorphic Hardware for Dynamic Tactile Perception with a Self-Powered Mechanoreceptor Array.

A self-powered mechanoreceptor array is demonstrated using four mechanoreceptor cells for recognition of dynamic touch gestures. Each cell consists of a triboelectric nanogenerator (TENG) for touch sensing and a bi-stable resistor (biristor) for spike encoding. It produces informative spike signals by sensing a force of an external touch and encoding the force into the number of spikes. An array of the mechanoreceptor cells is utilized to monitor various touch gestures and it successfully generated spike signals corresponding to all the gestures. To validate the practicality of the mechanoreceptor array, a spiking neural network (SNN), highly attractive for power consumption compared to the conventional von Neumann architecture, is used for the identification of touch gestures. The measured spiking signals are reflected as inputs for the SNN simulations. Consequently, touch gestures are classified with a high accuracy rate of 92.5%. The proposed mechanoreceptor array emerges as a promising candidate for a building block of tactile in-sensor computing in the era of the Internet of Things (IoT), due to the low cost and high manufacturability of the TENG. This eliminates the need for a power supply, coupled with the intrinsic high throughput of the Si-based biristor employing complementary metal-oxide-semiconductor (CMOS) technology.

Directly training temporal Spiking Neural Network with sparse surrogate gradient.

Brain-inspired Spiking Neural Networks (SNNs) have attracted much attention due to their event-based computing and energy-efficient features. However, the spiking all-or-none nature has prevented direct training of SNNs for various applications. The surrogate gradient (SG) algorithm has recently enabled spiking neural networks to shine in neuromorphic hardware. However, introducing surrogate gradients has caused SNNs to lose their original sparsity, thus leading to the potential performance loss. In this paper, we first analyze the current problem of direct training using SGs and then propose Masked Surrogate Gradients (MSGs) to balance the effectiveness of training and the sparseness of the gradient, thereby improving the generalization ability of SNNs. Moreover, we introduce a temporally weighted output (TWO) method to decode the network output, reinforcing the importance of correct timesteps. Extensive experiments on diverse network structures and datasets show that training with MSG and TWO surpasses the SOTA technique.

Automatic detection of sleep apnea from a single-lead ECG signal based on spiking neural network model.

BACKGROUND: Sleep apnea (SLA) is a commonly encountered sleep disorder characterized by repetitive cessation of respiration while sleeping. In the past few years, researchers have focused on developing less complex and more cost-effective diagnostic approaches for identifying SLA recipients, in contrast to the cumbersome, complicated, and expensive conventional methods. METHOD: This study presents a biologically plausible learning approach of spiking neural networks (SNN) with temporal coding and a tempotron learning model for diagnosing SLA disorder using single-lead electrocardiogram (ECG) data information. The proposed framework utilizes temporal encoding and the leaky integrate and fire model to transform the ECG signal into spikes for capturing the signal's dynamic pattern nature and to simulate input response behaviors. The tempoton learning technique, a spike-based algorithm, trains the SNN model to identify SLA event patterns from encoded output spike trains. This study utilized ECG data to extract heart rate variability (HRV) and ECG-derived respiration (EDR) signals from 1-min segment data of ECG records for input to SNN model. Thirty-five recordings of both released and withheld data from the Apnea-ECG databases from Physionet have been applied to train the SNN model and validate the model's efficacy in identifying SLA occurrences. RESULTS: The proposed method demonstrated substantial improvements compared to other SLA detection techniques, achieving a significant accuracy of 94.63 % for per-segment detection, along with specificity, sensitivity, F1-score and AUC values of 96.21 %, 92.04 %, 0.9285, and 0.9851 respectively. The accuracy for per-recording detection achieved 100 %, with a correlation coefficient value of 0.986. Additionally, the experiment used UCD data for validation methods, achieving an accuracy of 84.573 %. CONCLUSIONS: These results suggest the effectiveness and accessibility of the presented approach for accurately identifying SLA cases. The suggested model enhances the performance of SLA detection when contrasted with various techniques based on feature engineering and feature learning.

Leveraging spiking neural networks for topic modeling.

This article investigates the application of spiking neural networks (SNNs) to the problem of topic modeling (TM): the identification of significant groups of words that represent human-understandable topics in large sets of documents. Our research is based on the hypothesis that an SNN that implements the Hebbian learning paradigm is capable of becoming specialized in the detection of statistically significant word patterns in the presence of adequately tailored sequential input. To support this hypothesis, we propose a novel spiking topic model (STM) that transforms text into a sequence of spikes and uses that sequence to train single-layer SNNs. In STM, each SNN neuron represents one topic, and each of the neuron's weights corresponds to one word. STM synaptic connections are modified according to spike-timing-dependent plasticity; after training, the neurons' strongest weights are interpreted as the words that represent topics. We compare the performance of STM with four other TM methods Latent Dirichlet Allocation (LDA), Biterm Topic Model (BTM), Embedding Topic Model (ETM) and BERTopic on three datasets: 20Newsgroups, BBC news, and AG news. The results demonstrate that STM can discover high-quality topics and successfully compete with comparative classical methods. This sheds new light on the possibility of the adaptation of SNN models in unsupervised natural language processing.

Energy-Efficient PPG-Based Respiratory Rate Estimation Using Spiking Neural Networks.

Respiratory rate (RR) is a vital indicator for assessing the bodily functions and health status of patients. RR is a prominent parameter in the field of biomedical signal processing and is strongly associated with other vital signs such as blood pressure, heart rate, and heart rate variability. Various physiological signals, such as photoplethysmogram (PPG) signals, are used to extract respiratory information. RR is also estimated by detecting peak patterns and cycles in the signals through signal processing and deep-learning approaches. In this study, we propose an end-to-end RR estimation approach based on a third-generation artificial neural network model-spiking neural network. The proposed model employs PPG segments as inputs, and directly converts them into sequential spike events. This design aims to reduce information loss during the conversion of the input data into spike events. In addition, we use feedback-based integrate-and-fire neurons as the activation functions, which effectively transmit temporal information. The network is evaluated using the BIDMC respiratory dataset with three different window sizes (16, 32, and 64 s). The proposed model achieves mean absolute errors of 1.37 ± 0.04, 1.23 ± 0.03, and 1.15 ± 0.07 for the 16, 32, and 64 s window sizes, respectively. Furthermore, it demonstrates superior energy efficiency compared with other deep learning models. This study demonstrates the potential of the spiking neural networks for RR monitoring, offering a novel approach for RR estimation from the PPG signal.

Spiking generative adversarial network with attention scoring decoding.

Generative models based on neural networks present a substantial challenge within deep learning. As it stands, such models are primarily limited to the domain of artificial neural networks. Spiking neural networks, as the third generation of neural networks, offer a closer approximation to brain-like processing due to their rich spatiotemporal dynamics. However, generative models based on spiking neural networks are not well studied. Particularly, previous works on generative adversarial networks based on spiking neural networks are conducted on simple datasets and do not perform well. In this work, we pioneer constructing a spiking generative adversarial network capable of handling complex images and having higher performance. Our first task is to identify the problems of out-of-domain inconsistency and temporal inconsistency inherent in spiking generative adversarial networks. We address these issues by incorporating the Earth-Mover distance and an attention-based weighted decoding method, significantly enhancing the performance of our algorithm across several datasets. Experimental results reveal that our approach outperforms existing methods on the MNIST, FashionMNIST, CIFAR10, and CelebA. In addition to our examination of static datasets, this study marks our inaugural investigation into event-based data, through which we achieved noteworthy results. Moreover, compared with hybrid spiking generative adversarial networks, where the discriminator is an artificial analog neural network, our methodology demonstrates closer alignment with the information processing patterns observed in the mouse. Our code can be found at https://github.com/Brain-Cog-Lab/sgad.

Electrocardiography Classification with Leaky Integrate-and-Fire Neurons in an Artificial Neural Network-Inspired Spiking Neural Network Framework.

Monitoring heart conditions through electrocardiography (ECG) has been the cornerstone of identifying cardiac irregularities. Cardiologists often rely on a detailed analysis of ECG recordings to pinpoint deviations that are indicative of heart anomalies. This traditional method, while effective, demands significant expertise and is susceptible to inaccuracies due to its manual nature. In the realm of computational analysis, Artificial Neural Networks (ANNs) have gained prominence across various domains, which can be attributed to their superior analytical capabilities. Conversely, Spiking Neural Networks (SNNs), which mimic the neural activity of the brain more closely through impulse-based processing, have not seen widespread adoption. The challenge lies primarily in the complexity of their training methodologies. Despite this, SNNs offer a promising avenue for energy-efficient computational models capable of displaying a high-level performance. This paper introduces an innovative approach employing SNNs augmented with an attention mechanism to enhance feature recognition in ECG signals. By leveraging the inherent efficiency of SNNs, coupled with the precision of attention modules, this model aims to refine the analysis of cardiac signals. The novel aspect of our methodology involves adapting the learned parameters from ANNs to SNNs using leaky integrate-and-fire (LIF) neurons. This transfer learning strategy not only capitalizes on the strengths of both neural network models but also addresses the training challenges associated with SNNs. The proposed method is evaluated through extensive experiments on two publicly available benchmark ECG datasets. The results show that our model achieves an overall accuracy of 93.8% on the MIT-BIH Arrhythmia dataset and 85.8% on the 2017 PhysioNet Challenge dataset. This advancement underscores the potential of SNNs in the field of medical diagnostics, offering a path towards more accurate, efficient, and less resource-intensive analyses of heart diseases.

DT-SCNN: dual-threshold spiking convolutional neural network with fewer operations and memory access for edge applications.

The spiking convolutional neural network (SCNN) is a kind of spiking neural network (SNN) with high accuracy for visual tasks and power efficiency on neuromorphic hardware, which is attractive for edge applications. However, it is challenging to implement SCNNs on resource-constrained edge devices because of the large number of convolutional operations and membrane potential (Vm) storage needed. Previous works have focused on timestep reduction, network pruning, and network quantization to realize SCNN implementation on edge devices. However, they overlooked similarities between spiking feature maps (SFmaps), which contain significant redundancy and cause unnecessary computation and storage. This work proposes a dual-threshold spiking convolutional neural network (DT-SCNN) to decrease the number of operations and memory access by utilizing similarities between SFmaps. The DT-SCNN employs dual firing thresholds to derive two similar SFmaps from one Vm map, reducing the number of convolutional operations and decreasing the volume of Vms and convolutional weights by half. We propose a variant spatio-temporal back propagation (STBP) training method with a two-stage strategy to train DT-SCNNs to decrease the inference timestep to 1. The experimental results show that the dual-thresholds mechanism achieves a 50% reduction in operations and data storage for the convolutional layers compared to conventional SCNNs while achieving not more than a 0.4% accuracy loss on the CIFAR10, MNIST, and Fashion MNIST datasets. Due to the lightweight network and single timestep inference, the DT-SCNN has the least number of operations compared to previous works, paving the way for low-latency and power-efficient edge applications.

Photoprogrammed Multifunctional Optoelectronic Synaptic Transistor Arrays Based on Photosensitive Polymer-Sorted Semiconducting Single-Walled Carbon Nanotubes for Image Recognition.

The development of neuromorphic optoelectronic systems opens up the possibility of the next generation of artificial vision. In this work, the novel broadband (from 365 to 940 nm) and multilevel storage optoelectronic synaptic thin-film transistor (TFT) arrays are reported using the photosensitive conjugated polymer (poly[(9,9-dioctylfluorenyl-2,7-diyl)-co-(bithiophene)], F8T2) sorted semiconducting single-walled carbon nanotubes (sc-SWCNTs) as channel materials. The broadband synaptic responses are inherited to absorption from both photosensitive F8T2 and sorted sc-SWCNTs, and the excellent optoelectronic synaptic behaviors with 200 linearly increasing conductance states and long retention time > 103 s are attributed to the superior charge trapping at the AlOx dielectric layer grown by atomic layer deposition. Furthermore, the synaptic TFTs can achieve IOn/IOff ratios up to 106 and optoelectronic synaptic plasticity with the low power consumption (59 aJ per single pulse), which can simulate not only basic biological synaptic functions but also optical write and electrical erase, multilevel storage, and image recognition. Further, a novel Spiking Neural Network algorithm based on hardware characteristics is designed for the recognition task of Caltech 101 dataset and multiple features of the images are successfully extracted with higher accuracy (97.92%) of the recognition task from the multi-frequency curves of the optoelectronic synaptic devices.