Learning a Deep Demosaicing Network for Spike Camera With Color Filter Array.

For capturing dynamic scenes with ultra-fast motion, neuromorphic cameras with extremely high temporal resolution have demonstrated their great capability and potential. Different from the event cameras that only record relative changes in light intensity, spike camera fires a stream of spikes according to a full-time accumulation of photons so that it can recover the texture details for both static areas and dynamic areas. Recently, color spike camera has been invented to record color information of dynamic scenes using a color filter array (CFA). However, demosaicing for color spike cameras is an open and challenging problem. In this paper, we develop a demosaicing network, called CSpkNet, to reconstruct dynamic color visual signals from the spike stream captured by the color spike camera. Firstly, we develop a light inference module to convert binary spike streams to intensity estimates. In particular, a feature-based channel attention module is proposed to reduce the noises caused by quantization errors. Secondly, considering both the Bayer configuration and object motion, we propose a motion-guided filtering module to estimate the missing pixels of each color channel, without undesired motion blur. Finally, we design a refinement module to improve the intensity and details, utilizing the color correlation. Experimental results demonstrate that CSpkNet can reconstruct color images from the Bayer-pattern spike stream with promising visual quality.

Fully Spiking Actor Network With Intralayer Connections for Reinforcement Learning.

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (DRL). In this article, we focus on the task where the agent needs to learn multidimensional deterministic policies to control, which is very common in real scenarios. Recently, the surrogate gradient method has been utilized for training multilayer SNNs, which allows SNNs to achieve comparable performance with the corresponding deep networks in this task. Most existing spike-based reinforcement learning (RL) methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully connected (FC) layer. However, the decimal characteristic of the firing rate brings the floating-point matrix operations to the FC layer, making the whole SNN unable to deploy on the neuromorphic hardware directly. To develop a fully spiking actor network (SAN) without any floating-point matrix operations, we draw inspiration from the nonspiking interneurons found in insects and employ the membrane voltage of the nonspiking neurons to represent the action. Before the nonspiking neurons, multiple population neurons are introduced to decode different dimensions of actions. Since each population is used to decode a dimension of action, we argue that the neurons in each population should be connected in time domain and space domain. Hence, the intralayer connections are used in output populations to enhance the representation capacity. This mechanism exists extensively in animals and has been demonstrated effectively. Finally, we propose a fully SAN with intralayer connections (ILC-SAN). Extensive experimental results demonstrate that the proposed method outperforms the state-of-the-art performance on continuous control tasks from OpenAI gym. Moreover, we estimate the theoretical energy consumption when deploying ILC-SAN on neuromorphic chips to illustrate its high energy efficiency.

Robust Decoding of Rich Dynamical Visual Scenes With Retinal Spikes.

Sensory information transmitted to the brain activates neurons to create a series of coping behaviors. Understanding the mechanisms of neural computation and reverse engineering the brain to build intelligent machines requires establishing a robust relationship between stimuli and neural responses. Neural decoding aims to reconstruct the original stimuli that trigger neural responses. With the recent upsurge of artificial intelligence, neural decoding provides an insightful perspective for designing novel algorithms of brain-machine interface. For humans, vision is the dominant contributor to the interaction between the external environment and the brain. In this study, utilizing the retinal neural spike data collected over multi trials with visual stimuli of two movies with different levels of scene complexity, we used a neural network decoder to quantify the decoded visual stimuli with six different metrics for image quality assessment establishing comprehensive inspection of decoding. With the detailed and systematical study of the effect and single and multiple trials of data, different noise in spikes, and blurred images, our results provide an in-depth investigation of decoding dynamical visual scenes using retinal spikes. These results provide insights into the neural coding of visual scenes and services as a guideline for designing next-generation decoding algorithms of neuroprosthesis and other devices of brain-machine interface.

Bioinformatic Identification of the Pyroptosis-Related Transcription Factor-MicroRNA-Target Gene Regulatory Network in Angiotensin II-Induced Cardiac Remodeling and Validation of Key Components.

BACKGROUND: Accumulative evidence suggests that pyroptosis plays a key role in mediating angiotensin II (Ang II)-induced cardiac remodeling However, the potential role of pyroptosis-related transcription factor (TF)-microRNA (miRNA)-gene regulatory networks in mediating Ang II-associated cardiac remodeling remains largely unknown. Therefore, we identified the pyroptosis-related hub genes and constructed a transcription factor (TF)-miRNA-target gene regulatory network using bioinformatic tools to elucidate the pathogenesis of Ang II-induced cardiac remodeling. METHODS: The pyroptosis-related differentially expressed genes (DEGs) were identified from the cardiac remodeling-related dataset GSE47420. Gene ontology (GO), Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway enrichment analysis, and protein-protein interaction (PPI) analysis were performed to identify the pyroptosis-related hub DEGs. A TF-miRNA-target gene network was constructed and further validated by quantitative real-time polymerase chain reaction (qRT-PCR) in animal experiments. The correlation between the pyroptosis-related hub DEGs and cardiac remodeling was evaluated using comparative toxicogenomics database. The drug-gene interaction analysis was performed to identify potential drugs that target the pyroptosis-related hub DEGs. RESULTS: A total of 32 pyroptosis-related DEGs were identified and enriched in the inflammation-related pathways by KEGG analysis. 13 of the 32 pyroptosis-related DEGs were identified as hub DEGs. Furthermore, a TF-miRNA-target gene regulatory network containing 16 TFs, 6 miRNAs, and 5 hub target genes was constructed. The five pyroptosis-related hub target genes (DDX3X, ELAVL1, YWHAZ, STAT3, and EED) were identified as crucial cardiac remodeling-related genes using the comparative toxicogenomics database (CTD) database. Five drugs including celecoxib were identified as potential drugs for the treatment of cardiac remodeling. Finally, the expression levels of two top-ranked TF-miRNA-target genes axis were verified by qRT-PCR in mice with Ang II-induced cardiac remodeling and found to be generally consistent with the microarray results. CONCLUSIONS: This study constructed a pyroptosis-related TF-miRNA-target gene regulatory network for Ang II-induced cardiac remodeling. Five pyroptosis-related genes (DDX3X, ELAVL1, YWHAZ, STAT3, and EED) can be considered the core genes associated with pyrotposis-related cardiac remodeling. The findings of this study provide new insights into the molecular mechanisms of Ang II-induced cardiac remodeling and may serve as potential biomarkers or therapeutic targets for Ang II-induced cardiac remodeling.

SPI1-Mediated Upregulation of the CST1 Gene as an Independent Poor Prognostic Factor Accelerates Metastasis in Esophageal Squamous Cell Carcinoma (ESCC) by Interacting with MMP2.

BACKGROUND: Esophageal squamous cell carcinoma (ESCC) is a highly lethal tumor type, but studies on the ESCC tumor microenvironment are limited. We found that cystatin SN (CST1) plays an important role in the ESCC tumor microenvironment. CST1 has been reported to act as an oncogene in multiple human cancers, but its clinical significance and underlying mechanism in ESCC remain elusive. METHODS: We performed ESCC gene expression profiling with data from RNA-sequencing and public databases and found CST1 upregulation in ESCC. Then, we assessed CST1 expression in ESCC by RT‒qPCR and Western blot analysis. In addition, immunohistochemistry (IHC) and enzyme-linked immunosorbent assay (ELISA) were used to estimate the expression of CST1 in ESCC tissue and serum. Moreover, further functional experiments were conducted to verify that the gain and loss of CST1 in ESCC cell lines significantly influenced the proliferation and metastasis of ESCC. Mass spectrometry, coimmunoprecipitation, and gelatin zymography experiments were used to validate the interaction between CST1 and matrix metalloproteinase 2 (MMP2) and the mechanism of CST1 influence on metastasis in ESCC. RESULTS: Here, we found that CST1 expression was significantly elevated in ESCC tissues and serum. Moreover, compared with patients with low CST1 expression, patients with high CST1 expression had a worse prognosis. Overall survival (OS) and disease-free survival (DFS) were significantly unfavorable in the high CST1 expression subgroup. Likewise, the CST1 level was significantly increased in ESCC serum compared with healthy control serum, indicating that CST1 may be a potential serum biomarker for diagnosis, with an area under the curve (AUC) = 0.9702 and p < 0.0001 by receiver operating curve (ROC) analysis. Furthermore, upregulated CST1 can promote the motility and metastatic capacity of ESCC in vitro and in vivo by influencing epithelial mesenchymal transition (EMT) and interacting with MMP2 in the tumor microenvironment (TME). CONCLUSIONS: Collectively, the results of this study indicated that high CST1 expression mediated by SPI1 in ESCC may serve as a potentially prognostic and diagnostic predictor and as an oncogene to promote motility and metastatic capacity of ESCC by influencing EMT and interacting with MMP2 in the TME.

A GPU-based computational framework that bridges neuron simulation and artificial intelligence.

Biophysically detailed multi-compartment models are powerful tools to explore computational principles of the brain and also serve as a theoretical framework to generate algorithms for artificial intelligence (AI) systems. However, the expensive computational cost severely limits the applications in both the neuroscience and AI fields. The major bottleneck during simulating detailed compartment models is the ability of a simulator to solve large systems of linear equations. Here, we present a novel Dendritic Hierarchical Scheduling (DHS) method to markedly accelerate such a process. We theoretically prove that the DHS implementation is computationally optimal and accurate. This GPU-based method performs with 2-3 orders of magnitude higher speed than that of the classic serial Hines method in the conventional CPU platform. We build a DeepDendrite framework, which integrates the DHS method and the GPU computing engine of the NEURON simulator and demonstrate applications of DeepDendrite in neuroscience tasks. We investigate how spatial patterns of spine inputs affect neuronal excitability in a detailed human pyramidal neuron model with 25,000 spines. Furthermore, we provide a brief discussion on the potential of DeepDendrite for AI, specifically highlighting its ability to enable the efficient training of biophysically detailed models in typical image classification tasks.

NeuroZoom: Denoising and Super Resolving Neuromorphic Events and Spikes.

Neuromorphic cameras are emerging imaging technology that has advantages over conventional imaging sensors in several aspects including dynamic range, sensing latency, and power consumption. However, the signal-to-noise level and the spatial resolution still fall behind the state of conventional imaging sensors. In this article, we address the denoising and super-resolution problem for modern neuromorphic cameras. We employ 3D U-Net as the backbone neural architecture for such a task. The networks are trained and tested on two types of neuromorphic cameras: a dynamic vision sensor and a spike camera. Their pixels generate signals asynchronously, the former is based on perceived light changes and the latter is based on accumulated light intensity. To collect the datasets for training such networks, we design a display-camera system to record high frame-rate videos at multiple resolutions, providing supervision for denoising and super-resolution. The networks are trained in a noise-to-noise fashion, where the two ends of the network are unfiltered noisy data. The output of the networks has been tested for downstream applications including event-based visual object tracking and image reconstruction. Experimental results demonstrate the effectiveness of improving the quality of neuromorphic events and spikes, and the corresponding improvement to downstream applications with state-of-the-art performance.

Visual information processing through the interplay between fine and coarse signal pathways.

Object recognition is often viewed as a feedforward, bottom-up process in machine learning, but in real neural systems, object recognition is a complicated process which involves the interplay between two signal pathways. One is the parvocellular pathway (P-pathway), which is slow and extracts fine features of objects; the other is the magnocellular pathway (M-pathway), which is fast and extracts coarse features of objects. It has been suggested that the interplay between the two pathways endows the neural system with the capacity of processing visual information rapidly, adaptively, and robustly. However, the underlying computational mechanism remains largely unknown. In this study, we build a two-pathway model to elucidate the computational properties associated with the interactions between two visual pathways. Specifically, we model two visual pathways using two convolution neural networks: one mimics the P-pathway, referred to as FineNet, which is deep, has small-size kernels, and receives detailed visual inputs; the other mimics the M-pathway, referred to as CoarseNet, which is shallow, has large-size kernels, and receives blurred visual inputs. We show that CoarseNet can learn from FineNet through imitation to improve its performance, FineNet can benefit from the feedback of CoarseNet to improve its robustness to noise; and the two pathways interact with each other to achieve rough-to-fine information processing. Using visual backward masking as an example, we further demonstrate that our model can explain visual cognitive behaviors that involve the interplay between two pathways. We hope that this study gives us insight into understanding the interaction principles between two visual pathways.

PS-Net: human perception-guided segmentation network for EM cell membrane.

MOTIVATION: Cell membrane segmentation in electron microscopy (EM) images is a crucial step in EM image processing. However, while popular approaches have achieved performance comparable to that of humans on low-resolution EM datasets, they have shown limited success when applied to high-resolution EM datasets. The human visual system, on the other hand, displays consistently excellent performance on both low and high resolutions. To better understand this limitation, we conducted eye movement and perceptual consistency experiments. Our data showed that human observers are more sensitive to the structure of the membrane while tolerating misalignment, contrary to commonly used evaluation criteria. Additionally, our results indicated that the human visual system processes images in both global-local and coarse-to-fine manners. RESULTS: Based on these observations, we propose a computational framework for membrane segmentation that incorporates these characteristics of human perception. This framework includes a novel evaluation metric, the perceptual Hausdorff distance (PHD), and an end-to-end network called the PHD-guided segmentation network (PS-Net) that is trained using adaptively tuned PHD loss functions and a multiscale architecture. Our subjective experiments showed that the PHD metric is more consistent with human perception than other criteria, and our proposed PS-Net outperformed state-of-the-art methods on both low- and high-resolution EM image datasets as well as other natural image datasets. AVAILABILITY AND IMPLEMENTATION: The code and dataset can be found at https://github.com/EmmaSRH/PS-Net.

Capture the Moment: High-Speed Imaging With Spiking Cameras Through Short-Term Plasticity.

High-speed imaging can help us understand some phenomena that are too fast to be captured by our eyes. Although ultra-fast frame-based cameras (e.g., Phantom) can record millions of fps at reduced resolution, they are too expensive to be widely used. Recently, a retina-inspired vision sensor, spiking camera, has been developed to record external information at 40, 000 Hz. The spiking camera uses the asynchronous binary spike streams to represent visual information. Despite this, how to reconstruct dynamic scenes from asynchronous spikes remains challenging. In this paper, we introduce novel high-speed image reconstruction models based on the short-term plasticity (STP) mechanism of the brain, termed TFSTP and TFMDSTP. We first derive the relationship between states of STP and spike patterns. Then, in TFSTP, by setting up the STP model at each pixel, the scene radiance can be inferred by the states of the models. In TFMDSTP, we use the STP to distinguish the moving and stationary regions, and then use two sets of STP models to reconstruct them respectively. In addition, we present a strategy for correcting error spikes. Experimental results show that the STP-based reconstruction methods can effectively reduce noise with less computing time, and achieve the best performances on both real-world and simulated datasets.

Hybrid High Dynamic Range Imaging fusing Neuromorphic and Conventional Images.

Reconstruction of high dynamic range image from a single low dynamic range image captured by a conventional RGB camera, which suffers from over- or under-exposure, is an ill-posed problem. In contrast, recent neuromorphic cameras like event camera and spike camera can record high dynamic range scenes in the form of intensity maps, but with much lower spatial resolution and no color information. In this article, we propose a hybrid imaging system (denoted as NeurImg) that captures and fuses the visual information from a neuromorphic camera and ordinary images from an RGB camera to reconstruct high-quality high dynamic range images and videos. The proposed NeurImg-HDR+ network consists of specially designed modules, which bridges the domain gaps on resolution, dynamic range, and color representation between two types of sensors and images to reconstruct high-resolution, high dynamic range images and videos. We capture a test dataset of hybrid signals on various HDR scenes using the hybrid camera, and analyze the advantages of the proposed fusing strategy by comparing it to state-of-the-art inverse tone mapping methods and merging two low dynamic range images approaches. Quantitative and qualitative experiments on both synthetic data and real-world scenarios demonstrate the effectiveness of the proposed hybrid high dynamic range imaging system. Code and dataset can be found at: https://github.com/hjynwa/NeurImg-HDR.

Heuristic Tree-Partition-Based Parallel Method for Biophysically Detailed Neuron Simulation.

Biophysically detailed neuron simulation is a powerful tool to explore the mechanisms behind biological experiments and bridge the gap between various scales in neuroscience research. However, the extremely high computational complexity of detailed neuron simulation restricts the modeling and exploration of detailed network models. The bottleneck is solving the system of linear equations. To accelerate detailed simulation, we propose a heuristic tree-partition-based parallel method (HTP) to parallelize the computation of the Hines algorithm, the kernel for solving linear equations, and leverage the strong parallel capability of the graphic processing unit (GPU) to achieve further speedup. We formulate the problem of how to get a fine parallel process as a tree-partition problem. Next, we present a heuristic partition algorithm to obtain an effective partition to efficiently parallelize the equation-solving process in detailed simulation. With further optimization on GPU, our HTP method achieves 2.2 to 8.5 folds speedup compared to the state-of-the-art GPU method and 36 to 660 folds speedup compared to the typical Hines algorithm.

Asynchronous Spatiotemporal Spike Metric for Event Cameras.

Event cameras as bioinspired vision sensors have shown great advantages in high dynamic range and high temporal resolution in vision tasks. Asynchronous spikes from event cameras can be depicted using the marked spatiotemporal point processes (MSTPPs). However, how to measure the distance between asynchronous spikes in the MSTPPs still remains an open issue. To address this problem, we propose a general asynchronous spatiotemporal spike metric considering both spatiotemporal structural properties and polarity attributes for event cameras. Technically, the conditional probability density function is first introduced to describe the spatiotemporal distribution and polarity prior in the MSTPPs. Besides, a spatiotemporal Gaussian kernel is defined to capture the spatiotemporal structure, which transforms discrete spikes into the continuous function in a reproducing kernel Hilbert space (RKHS). Finally, the distance between asynchronous spikes can be quantified by the inner product in the RKHS. The experimental results demonstrate that the proposed approach outperforms the state-of-the-art methods and achieves significant improvement in computational efficiency. Especially, it is able to better depict the changes involving spatiotemporal structural properties and polarity attributes.

Ultra-High Temporal Resolution Visual Reconstruction From a Fovea-Like Spike Camera via Spiking Neuron Model.

Neuromorphic vision sensor is a new bio-inspired imaging paradigm emerged in recent years. It uses the asynchronous spike signals instead of the traditional frame-based manner to achieve ultra-high speed sampling. Unlike the dynamic vision sensor (DVS) that perceives movement by imitating the retinal periphery, the spike camera was developed recently to perceive fine textures by simulating a small retinal region called the fovea. For this new type of neuromorphic camera, how to reconstruct ultra-high speed visual images from spike data becomes an important yet challenging issue in visual scene perception, analysis, and recognition applications. In this paper, a bio-inspired visual reconstruction framework for the spike camera is proposed for the first time. Its core idea is to use the biologically inspired adaptive adjustment mechanisms, combined with the spatiotemporal spike information extracted by the proposed model, to reconstruct the full texture of natural scenes in an ultra-high temporal resolution. Specifically, the proposed model consists of a motion local excitation layer, a spike refining layer and a visual reconstruction layer motivated by the bio-realistic leaky integrate-and-fire (LIF) neurons and synapse connection with spike-timing dependent plasticity (STDP) rule. To evaluate the performance, a spike dataset was constructed for normal and high-speed scenes in real-world recorded by the spike camera. The experimental results show that the proposed approach can reconstruct the visual images with 40,000 frames per second in both normal and high-speed scenes, while achieving high dynamic range and high image quality.

Improving Multispike Learning With Plastic Synaptic Delays.

Emulating the spike-based processing in the brain, spiking neural networks (SNNs) are developed and act as a promising candidate for the new generation of artificial neural networks that aim to produce efficient cognitions as the brain. Due to the complex dynamics and nonlinearity of SNNs, designing efficient learning algorithms has remained a major difficulty, which attracts great research attention. Most existing ones focus on the adjustment of synaptic weights. However, other components, such as synaptic delays, are found to be adaptive and important in modulating neural behavior. How could plasticity on different components cooperate to improve the learning of SNNs remains as an interesting question. Advancing our previous multispike learning, we propose a new joint weight-delay plasticity rule, named TDP-DL, in this article. Plastic delays are integrated into the learning framework, and as a result, the performance of multispike learning is significantly improved. Simulation results highlight the effectiveness and efficiency of our TDP-DL rule compared to baseline ones. Moreover, we reveal the underlying principle of how synaptic weights and delays cooperate with each other through a synthetic task of interval selectivity and show that plastic delays can enhance the selectivity and flexibility of neurons by shifting information across time. Due to this capability, useful information distributed away in the time domain can be effectively integrated for a better accuracy performance, as highlighted in our generalization tasks of the image, speech, and event-based object recognitions. Our work is thus valuable and significant to improve the performance of spike-based neuromorphic computing.

Novel long noncoding RNA LINC02820 augments TNF signaling pathway to remodel cytoskeleton and potentiate metastasis in esophageal squamous cell carcinoma.

Esophageal squamous cell carcinoma (ESCC) is one of the most common malignant tumors in China. However, there are no targets to treat ESCC because the molecular mechanism behind the cancer is still unclear. Here, we found a novel long noncoding RNA LINC02820 was upregulated in ESCC and associated with the ESCC clinicopathological stage. Through a series of functional experiments, we observed that LINC02820 only promoted the migration and invasion capabilities of ESCC cell lines. Mechanically, we found that LINC02820 may affect the cytoskeletal remodeling, interact with splice factor 3B subunit 3 (SF3B3), and cooperate with TNFα to amplify the NF-κB signaling pathway, which can lead to ESCC metastasis. Overall, our findings revealed that LINC02820 is a potential biomarker and therapeutic target for the diagnosis and treatment of ESCC.

A theoretical model reveals specialized synaptic depressions and temporal frequency tuning in retinal parallel channels.

In the Outer Plexiform Layer of a retina, a cone pedicle provides synaptic inputs for multiple cone bipolar cell (CBC) subtypes so that each subtype formats a parallelized processing channel to filter visual features from the environment. Due to the diversity of short-term depressions among cone-CBC contacts, these channels have different temporal frequency tunings. Here, we propose a theoretical model based on the hierarchy Linear-Nonlinear-Synapse framework to link the synaptic depression and the neural activities of the cone-CBC circuit. The model successfully captures various frequency tunings of subtype-specialized channels and infers synaptic depression recovery time constants inside circuits. Furthermore, the model can predict frequency-tuning behaviors based on synaptic activities. With the prediction of region-specialized UV cone parallel channels, we suggest the acute zone in the zebrafish retina supports detecting light-off events at high temporal frequencies.

GraphReg: Dynamical Point Cloud Registration With Geometry-Aware Graph Signal Processing.

This study presents a high-accuracy, efficient, and physically induced method for 3D point cloud registration, which is the core of many important 3D vision problems. In contrast to existing physics-based methods that merely consider spatial point information and ignore surface geometry, we explore geometry aware rigid-body dynamics to regulate the particle (point) motion, which results in more precise and robust registration. Our proposed method consists of four major modules. First, we leverage the graph signal processing (GSP) framework to define a new signature, i.e., point response intensity for each point, by which we succeed in describing the local surface variation, resampling keypoints, and distinguishing different particles. Then, to address the shortcomings of current physics-based approaches that are sensitive to outliers, we accommodate the defined point response intensity to median absolute deviation (MAD) in robust statistics and adopt the X84 principle for adaptive outlier depression, ensuring a robust and stable registration. Subsequently, we propose a novel geometric invariant under rigid transformations to incorporate higher-order features of point clouds, which is further embedded for force modeling to guide the correspondence between pairwise scans credibly. Finally, we introduce an adaptive simulated annealing (ASA) method to search for the global optimum and substantially accelerate the registration process. We perform comprehensive experiments to evaluate the proposed method on various datasets captured from range scanners to LiDAR. Results demonstrate that our proposed method outperforms representative state-of-the-art approaches in terms of accuracy and is more suitable for registering large-scale point clouds. Furthermore, it is considerably faster and more robust than most competitors. Our implementation is publicly available at https://github.com/zikai1/GraphReg.

An FPGA Accelerator for High-Speed Moving Objects Detection and Tracking With a Spike Camera.

Ultra-high-speed object detection and tracking are crucial in fields such as fault detection and scientific observation. Existing solutions to this task have deficiencies in processing speeds. To deal with this difficulty, we propose a neural-inspired ultra-high-speed moving object filtering, detection, and tracking scheme, as well as a corresponding accelerator based on a high-speed spike camera. We parallelize the filtering module and divide the detection module to accelerate the algorithm and balance latency among modules for the benefit of the task-level pipeline. To be specific, a block-based parallel computation model is proposed to accelerate the filtering module, and the detection module is accelerated by a parallel connected component labeling algorithm modeling spike sparsity and spatial connectivity of moving objects with a searching tree. The hardware optimizations include processing the LIF layer with a group of multiplexers to reduce ADD operations and replacing expensive exponential operations with multiplications of preprocessed fixed-point values to increase processing speed and minimize resource consumption. We design an accelerator with the above techniques, achieving 19 times acceleration over the serial version after 25-way parallelization. A processing system for the accelerator is also implemented on the Xilinx ZCU-102 board to validate its functionality and performance. Our accelerator can process more than 20,000 spike images with 250 × 400 resolution per second with 1.618 W dynamic power consumption.

Decoding Pixel-Level Image Features From Two-Photon Calcium Signals of Macaque Visual Cortex.

Images of visual scenes comprise essential features important for visual cognition of the brain. The complexity of visual features lies at different levels, from simple artificial patterns to natural images with different scenes. It has been a focus of using stimulus images to predict neural responses. However, it remains unclear how to extract features from neuronal responses. Here we address this question by leveraging two-photon calcium neural data recorded from the visual cortex of awake macaque monkeys. With stimuli including various categories of artificial patterns and diverse scenes of natural images, we employed a deep neural network decoder inspired by image segmentation technique. Consistent with the notation of sparse coding for natural images, a few neurons with stronger responses dominated the decoding performance, whereas decoding of ar tificial patterns needs a large number of neurons. When natural images using the model pretrained on artificial patterns are decoded, salient features of natural scenes can be extracted, as well as the conventional category information. Altogether, our results give a new perspective on studying neural encoding principles using reverse-engineering decoding strategies.