Molecular Mechanisms for Changing Brain Connectivity in Mice and Humans.

The goal of this study was to examine commonalities in the molecular basis of learning in mice and humans. In previous work we have demonstrated that the anterior cingulate cortex (ACC) and hippocampus (HC) are involved in learning a two-choice visuospatial discrimination task. Here, we began by looking for candidate genes upregulated in mouse ACC and HC with learning. We then determined which of these were also upregulated in mouse blood. Finally, we used RT-PCR to compare candidate gene expression in mouse blood with that from humans following one of two forms of learning: a working memory task (network training) or meditation (a generalized training shown to change many networks). Two genes were upregulated in mice following learning: caspase recruitment domain-containing protein 6 (Card6) and inosine monophosphate dehydrogenase 2 (Impdh2). The Impdh2 gene product catalyzes the first committed step of guanine nucleotide synthesis and is tightly linked to cell proliferation. The Card6 gene product positively modulates signal transduction. In humans, Card6 was significantly upregulated, and Impdh2 trended toward upregulation with training. These genes have been shown to regulate pathways that influence nuclear factor kappa B (NF-κB), a factor previously found to be related to enhanced synaptic function and learning.

Bulk-Switching Memristor-Based Compute-In-Memory Module for Deep Neural Network Training.

The constant drive to achieve higher performance in deep neural networks (DNNs) has led to the proliferation of very large models. Model training, however, requires intensive computation time and energy. Memristor-based compute-in-memory (CIM) modules can perform vector-matrix multiplication (VMM) in place and in parallel, and have shown great promises in DNN inference applications. However, CIM-based model training faces challenges due to non-linear weight updates, device variations, and low-precision. In this work, a mixed-precision training scheme is experimentally implemented to mitigate these effects using a bulk-switching memristor-based CIM module. Low-precision CIM modules are used to accelerate the expensive VMM operations, with high-precision weight updates accumulated in digital units. Memristor devices are only changed when the accumulated weight update value exceeds a pre-defined threshold. The proposed scheme is implemented with a system-onchip of fully integrated analog CIM modules and digital sub-systems, showing fast convergence of LeNet training to 97.73%. The efficacy of training larger models is evaluated using realistic hardware parameters and verifies that CIM modules can enable efficient mix-precision DNN training with accuracy comparable to full-precision software-trained models. Additionally, models trained on chip are inherently robust to hardware variations, allowing direct mapping to CIM inference chips without additional re-training.

A mathematical framework for improved weight initialization of neural networks using Lagrange multipliers.

A good weight initialization is crucial to accelerate the convergence of the weights in a neural network. However, training a neural network is still time-consuming, despite recent advances in weight initialization approaches. In this paper, we propose a mathematical framework for the weight initialization in the last layer of a neural network. We first derive analytically a tight constraint on the weights that accelerates the convergence of the weights during the back-propagation algorithm. We then use linear regression and Lagrange multipliers to analytically derive the optimal initial weights and initial bias of the last layer, that minimize the initial training loss given the derived tight constraint. We also show that the restrictive assumption of traditional weight initialization algorithms that the expected value of the weights is zero is redundant for our approach. We first apply our proposed weight initialization approach to a Convolutional Neural Network that predicts the Remaining Useful Life of aircraft engines. The initial training and validation loss are relatively small, the weights do not get stuck in a local optimum, and the convergence of the weights is accelerated. We compare our approach with several benchmark strategies. Compared to the best performing state-of-the-art initialization strategy (Kaiming initialization), our approach needs 34% less epochs to reach the same validation loss. We also apply our approach to ResNets for the CIFAR-100 dataset, combined with transfer learning. Here, the initial accuracy is already at least 53%. This gives a faster weight convergence and a higher test accuracy than the benchmark strategies.

Attribute-driven streaming edge partitioning with reconciliations for distributed graph neural network training.

Current distributed graph training frameworks evenly partition a large graph into small chunks to suit distributed storage, leverage a uniform interface to access neighbors, and train graph neural networks in a cluster of machines to update weights. Nevertheless, they consider a separate design of storage and training, resulting in huge communication costs for retrieving neighborhoods. During the storage phase, traditional heuristic graph partitioning not only suffers from memory overhead because of loading the full graph into the memory but also damages semantically related structures because of its neglecting meaningful node attributes. What is more, in the weight-update phase, directly averaging synchronization is difficult to tackle with heterogeneous local models where each machine's data are loaded from different subgraphs, resulting in slow convergence. To solve these problems, we propose a novel distributed graph training approach, attribute-driven streaming edge partitioning with reconciliations (ASEPR), where the local model loads only the subgraph stored on its own machine to make fewer communications. ASEPR firstly clusters nodes with similar attributes in the same partition to maintain semantic structure and keep multihop neighbor locality. Then streaming partitioning combined with attribute clustering is applied to subgraph assignment to alleviate memory overhead. After local graph neural network training on distributed machines, we deploy cross-layer reconciliation strategies for heterogeneous local models to improve the averaged global model by knowledge distillation and contrastive learning. Extensive experiments conducted on four large graph datasets on node classification and link prediction tasks show that our model outperforms DistDGL, with fewer resource requirements and up to quadruple the convergence speed.

The centrality of population-level factors to network computation is demonstrated by a versatile approach for training spiking networks.

Neural activity is often described in terms of population-level factors extracted from the responses of many neurons. Factors provide a lower-dimensional description with the aim of shedding light on network computations. Yet, mechanistically, computations are performed not by continuously valued factors but by interactions among neurons that spike discretely and variably. Models provide a means of bridging these levels of description. We developed a general method for training model networks of spiking neurons by leveraging factors extracted from either data or firing-rate-based networks. In addition to providing a useful model-building framework, this formalism illustrates how reliable and continuously valued factors can arise from seemingly stochastic spiking. Our framework establishes procedures for embedding this property in network models with different levels of realism. The relationship between spikes and factors in such networks provides a foundation for interpreting (and subtly redefining) commonly used quantities such as firing rates.

Model-based Deep Learning Reconstruction Using a Folded Image Training Strategy for Abdominal 3D T1-weighted Imaging.

PURPOSE: To evaluate the feasibility of folded image training strategy (FITS) and the quality of images reconstructed using the improved model-based deep learning (iMoDL) network trained with FITS (FITS-iMoDL) for abdominal MR imaging. METHODS: This retrospective study included abdominal 3D T1-weighted images of 122 patients. In the experimental analyses, peak SNR (PSNR) and structure similarity index (SSIM) of images reconstructed with FITS-iMoDL were compared with those with the following reconstruction methods: conventional model-based deep learning (conv-MoDL), MoDL trained with FITS (FITS-MoDL), total variation regularized compressed sensing (CS), and parallel imaging (CG-SENSE). In the clinical analysis, SNR and image contrast were measured on the reference, FITS-iMoDL, and CS images. Three radiologists evaluated the image quality using a 5-point scale to determine the mean opinion score (MOS). RESULTS: The PSNR of FITS-iMoDL was significantly higher than that of FITS-MoDL, conv-MoDL, CS, and CG-SENSE (P < 0.001). The SSIM of FITS-iMoDL was significantly higher than those of the others (P < 0.001), except for FITS-MoDL (P = 0.056). In the clinical analysis, the SNR of FITS-iMoDL was significantly higher than that of the reference and CS (P < 0.0001). Image contrast was equivalent within an equivalence margin of 10% among these three image sets (P < 0.0001). MOS was significantly improved in FITS-iMoDL (P < 0.001) compared with CS images in terms of liver edge and vessels conspicuity, lesion depiction, artifacts, blurring, and overall image quality. CONCLUSION: The proposed method, FITS-iMoDL, allowed a deeper MoDL reconstruction network without increasing memory consumption and improved image quality on abdominal 3D T1-weighted imaging compared with CS images.

Improving Network Training on Resource-Constrained Devices via Habituation Normalization.

As a technique for accelerating and stabilizing training, the batch normalization (BN) is widely used in deep learning. However, BN cannot effectively estimate the mean and the variance of samples when training/fine-tuning with small batches of data on resource-constrained devices. It will lead to a decrease in the accuracy of the deep learning model. In the fruit fly olfactory system, the algorithm based on the "negative image" habituation model can filter redundant information and improve numerical stability. Inspired by the circuit mechanism, we propose a novel normalization method, the habituation normalization (HN). HN first eliminates the "negative image" obtained by habituation and then calculates the statistics for normalizing. It solves the problem of accuracy degradation of BN when the batch size is small. The experiment results show that HN can speed up neural network training and improve the model accuracy on vanilla LeNet-5, VGG16, and ResNet-50 in the Fashion MNIST and CIFAR10 datasets. Compared with four standard normalization methods, HN keeps stable and high accuracy in different batch sizes, which shows that HN has strong robustness. Finally, the applying HN to the deep learning-based EEG signal application system indicates that HN is suitable for the network fine-tuning and neural network applications under limited computing power and memory.

Analysis of the Application Efficiency of TensorFlow and PyTorch in Convolutional Neural Network.

In this paper, we present an analysis of important aspects that arise during the development of neural network applications. Our aim is to determine if the choice of library can impact the system's overall performance, either during training or design, and to extract a set of criteria that could be used to highlight the advantages and disadvantages of each library under consideration. To do so, we first extracted the previously mentioned aspects by comparing two of the most popular neural network libraries-PyTorch and TensorFlow-and then we performed an analysis on the obtained results, with the intent of determining if our initial hypothesis was correct. In the end, the results of the analysis are gathered, and an overall picture of what tasks are better suited for what library is presented.

A Novel Deep Supervised Learning-Based Approach for Intrusion Detection in IoT Systems.

The Internet of Things (IoT) has become one of the most important concepts in various aspects of our modern life in recent years. However, the most critical challenge for the world-wide use of the IoT is to address its security issues. One of the most important tasks to address the security challenges in the IoT is to detect intrusion in the network. Although the machine/deep learning-based solutions have been repeatedly used to detect network intrusion through recent years, there is still considerable potential to improve the accuracy and performance of the classifier (intrusion detector). In this paper, we develop a novel training algorithm to better tune the parameters of the used deep architecture. To specifically do so, we first introduce a novel neighborhood search-based particle swarm optimization (NSBPSO) algorithm to improve the exploitation/exploration of the PSO algorithm. Next, we use the advantage of NSBPSO to optimally train the deep architecture as our network intrusion detector in order to obtain better accuracy and performance. For evaluating the performance of the proposed classifier, we use two network intrusion detection datasets named UNSW-NB15 and Bot-IoT to rate the accuracy and performance of the proposed classifier.

Dataset of manually classified images obtained from a construction site.

A manually classified dataset of images obtained by four static cameras located around a construction site is presented. Eight object classes, typically found in a construction environment, were considered. The dataset consists of 1046 images selected from video footage by a frame extraction algorithm and txt files containing the objects' class and coordinates information. These data can be used to develop computer vision techniques in the engineering and construction fields.

Adversarially Robust Learning via Entropic Regularization.

In this paper we propose a new family of algorithms, ATENT, for training adversarially robust deep neural networks. We formulate a new loss function that is equipped with an additional entropic regularization. Our loss function considers the contribution of adversarial samples that are drawn from a specially designed distribution in the data space that assigns high probability to points with high loss and in the immediate neighborhood of training samples. Our proposed algorithms optimize this loss to seek adversarially robust valleys of the loss landscape. Our approach achieves competitive (or better) performance in terms of robust classification accuracy as compared to several state-of-the-art robust learning approaches on benchmark datasets such as MNIST and CIFAR-10.

Streaming Batch Eigenupdates for Hardware Neural Networks.

Neural networks based on nanodevices, such as metal oxide memristors, phase change memories, and flash memory cells, have generated considerable interest for their increased energy efficiency and density in comparison to graphics processing units (GPUs) and central processing units (CPUs). Though immense acceleration of the training process can be achieved by leveraging the fact that the time complexity of training does not scale with the network size, it is limited by the space complexity of stochastic gradient descent, which grows quadratically. The main objective of this work is to reduce this space complexity by using low-rank approximations of stochastic gradient descent. This low spatial complexity combined with streaming methods allows for significant reductions in memory and compute overhead, opening the door for improvements in area, time and energy efficiency of training. We refer to this algorithm and architecture to implement it as the streaming batch eigenupdate (SBE) approach.

Effect of fresnel press-on prisms combined with mental image network training on binocular visual function of the postoperative concomitant strabismus / 国际眼科杂志(Guoji Yanke Zazhi)

@#AIM: To discuss the effect of fresnel press-on prisms combined with mental image network training on binocular visual function recovery of the postoperative concomitant strabismus(CS). <p>METHODS: Totally 120 postoperative children with CS were selected from January 2015 to January 2018 in our hospital. According to the random digital table method, they were divided into the combination group and the prism group, 60 cases in each group, the prism group was given fresnel press-on prisms treatment, the combination group was given mental image network training on the basis, the visual function recovery of the two groups was compared. <p>RESULTS: After 6mo treatment, binocular visual machine visual function and Titmus near stereopsis visual function in both groups were significantly higher than before, and the binocular visual machine visual function and Titmus near stereopsis visual function in the combination group were significantly higher than those in the single press group, the difference was statistically significant(all <i>P</i><0.05). The long distance fusion function normal rates in the combination group and the prism group were significantly higher than those before treatment, the long distance fusion function normal rate in the combination group was significantly higher than that in the prism press group, the difference was statistically significant(93.3% <i>vs</i> 70.0%, <i>P</i><0.05). <p>CONCLUSION: Fresnel press-on prisms combined with mental image network training can effectively promote the binocular visual function recovery of the postoperative children with CS, which is worth for further clinical promotion.

Effective neural network training with adaptive learning rate based on training loss.

A method that uses an adaptive learning rate is presented for training neural networks. Unlike most conventional updating methods in which the learning rate gradually decreases during training, the proposed method increases or decreases the learning rate adaptively so that the training loss (the sum of cross-entropy losses for all training samples) decreases as much as possible. It thus provides a wider search range for solutions and thus a lower test error rate. The experiments with some well-known datasets to train a multilayer perceptron show that the proposed method is effective for obtaining a better test accuracy under certain conditions.

Establishment and application of the training system for laboratory animal employees / 中国比较医学杂志

Laboratory animals are an important part of life sciences and medical researches, as well an important support for the science and technology innovation in our country. Laboratory animal science is of great significance to the protection of human health,food safety and biological safety. Laboratory animals are indispensable in the development of food safety,drugs,vaccines and biological products and the studies of human disease pathogenesis. In order to adapt to the requirements for overall development of the laboratory animal industry in China, our institute has independently developed the Network Training System for Laboratory Animal Managers. This system is an online education and training platform which integrates the practical operation and theoretical knowledge of laboratory animals,including seven knowledge modules such as animal welfare,animal breeding,animal surgery and so on. The training subjects of the system include managers, experiment operators, laboratory animal doctors and breeders, aimed at accelerating the personnel training and team building of laboratory animal sciences,and promoting the transformation and development of personnel training in laboratory animal industry.

Effect of mental image network training on binocular visual function recovery in children with concomitant strabismus / 国际眼科杂志(Guoji Yanke Zazhi)

· AIM:To discuss the effect of mental image network training on binocular visual function recovery in children with concomitant strabismus.· METHODS:Totally 100 children with concomitant strabismus were selected from March 2013 to March 2017 in our Hospital.According to the random distribution,they were divided into mental group and control group,50 cases in each group.Mental group was given the mental image network training,control group was given no training.The visual function of the two groups of binocular vision,the near stereoscopic visual acuity and the eye position of the two groups were compared.· RESULTS:The proportion of patients with vision function by synoptophore at Ⅰ,Ⅱ,Ⅲ after treatment of mental group and control group were significantly higher than those before treatment,the after treatment of mental group was significantly higher than that of control group,the difference was statistically significant (P＜0.05).In the aspect of rate of ceses without stereovision by Titmus near stereoacuity,that after treatment of mental group and control group were significantly lower than those of the before treatment,that after treatment of mental group was significantly lower than control group;in the aspect of central fovea,macular hole,peripheral stereoscopic vision,those after treatment of mental group and control group were significantly higher than those of the before treatment,those after treatment of mental group was significantly higher than that of control group,the difference was statistically significant (P＜ 0.05).During the follow-up for 6mo,the ocular position maintenance rate of mental group was significantly higher than that of control group,the difference was statistically significant (P＜0.05).· CONCLUSION:Mental image network training can effectively promote the recovery of visual function in children after concomitant strabismus surgery.It is beneficial to maintain the position of the eye of children.

Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations.

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

Exploration on Domestic Cerebral Palsy Rehabilitation from Experiences of American Association of China Partner Network Training Project / 中国康复理论与实践

@#China Partner Network (CPN) training project would train professionals for better management of children rehabilitation, including advanced theory and manipulation in treatment and functional assessment of children with cerebral palsy, psychological education,and orthotics as well as operation and non-operation protocols. Domestic status quo in this area and some other relevant experiences were also compared with CPN.