Computational analysis of electrode structure and configuration for efficient and localized neural stimulation.

Neuromodulation technique using electric stimulation is widely applied in neural prosthesis, therapy, and neuroscience research. Various stimulation techniques have been developed to enhance stimulation efficiency and to precisely target the specific area of the brain which involves optimizing the geometry and the configuration of the electrode, stimulation pulse type and shapes, and electrode materials. Although the effects of electrode shape, size, and configuration on the performance of neural stimulation have individually been characterized, to date, there is no integrative investigation of how this factor affects neural stimulation. In this study, we computationally modeled the various types of electrodes with varying shapes, sizes, and configurations and simulated the electric field to calculate the activation function. The electrode geometry is then integratively assessed in terms of stimulation efficiency and stimulation focality. We found that stimulation efficiency is enhanced by making the electrode sharper and smaller. A center-to-vertex distance exceeding 100 µm shows enhanced stimulation efficiency in the bipolar configuration. Additionally, the separation distance of less than 1 mm between the reference and stimulation electrodes exhibits higher stimulation efficiency compared to the monopolar configuration. The region of neurons to be stimulated can also be modified. We found that sharper electrodes can locally activate the neuron. In most cases, except for the rectangular electrode shape with a center-to-vertex distance smaller than 100 µm, the bipolar electrode configuration can locally stimulate neurons as opposed to the monopolar configuration. These findings shed light on the optimal selection of neural electrodes depending on the target applications.

A lightweight network architecture for traffic sign recognition based on enhanced LeNet-5 network.

As an important part of the unmanned driving system, the detection and recognition of traffic sign need to have the characteristics of excellent recognition accuracy, fast execution speed and easy deployment. Researchers have applied the techniques of machine learning, deep learning and image processing to traffic sign recognition successfully. Considering the hardware conditions of the terminal equipment in the unmanned driving system, in this research work, the goal was to achieve a convolutional neural network (CNN) architecture that is lightweight and easily implemented for an embedded application and with excellent recognition accuracy and execution speed. As a classical CNN architecture, LeNet-5 network model was chosen to be improved, including image preprocessing, improving spatial pool convolutional neural network, optimizing neurons, optimizing activation function, etc. The test experiment of the improved network architecture was carried out on German Traffic Sign Recognition Benchmark (GTSRB) database. The experimental results show that the improved network architecture can obtain higher recognition accuracy in a short interference time, and the algorithm loss is significantly reduced with the progress of training. At the same time, compared with other lightweight network models, this network architecture gives a good recognition result, with a recognition accuracy of 97.53%. The network structure is simple, the algorithm complexity is low, and it is suitable for all kinds of terminal equipment, which can have a wider application in unmanned driving system.

A Neural-Network-Based Watermarking Method Approximating JPEG Quantization.

We propose a neural-network-based watermarking method that introduces the quantized activation function that approximates the quantization of JPEG compression. Many neural-network-based watermarking methods have been proposed. Conventional methods have acquired robustness against various attacks by introducing an attack simulation layer between the embedding network and the extraction network. The quantization process of JPEG compression is replaced by the noise addition process in the attack layer of conventional methods. In this paper, we propose a quantized activation function that can simulate the JPEG quantization standard as it is in order to improve the robustness against the JPEG compression. Our quantized activation function consists of several hyperbolic tangent functions and is applied as an activation function for neural networks. Our network was introduced in the attack layer of ReDMark proposed by Ahmadi et al. to compare it with their method. That is, the embedding and extraction networks had the same structure. We compared the usual JPEG compressed images and the images applying the quantized activation function. The results showed that a network with quantized activation functions can approximate JPEG compression with high accuracy. We also compared the bit error rate (BER) of estimated watermarks generated by our network with those generated by ReDMark. We found that our network was able to produce estimated watermarks with lower BERs than those of ReDMark. Therefore, our network outperformed the conventional method with respect to image quality and BER.

Improving Surgical Scene Semantic Segmentation through a Deep Learning Architecture with Attention to Class Imbalance.

This article addresses the semantic segmentation of laparoscopic surgery images, placing special emphasis on the segmentation of structures with a smaller number of observations. As a result of this study, adjustment parameters are proposed for deep neural network architectures, enabling a robust segmentation of all structures in the surgical scene. The U-Net architecture with five encoder-decoders (U-Net5ed), SegNet-VGG19, and DeepLabv3+ employing different backbones are implemented. Three main experiments are conducted, working with Rectified Linear Unit (ReLU), Gaussian Error Linear Unit (GELU), and Swish activation functions. The applied loss functions include Cross Entropy (CE), Focal Loss (FL), Tversky Loss (TL), Dice Loss (DiL), Cross Entropy Dice Loss (CEDL), and Cross Entropy Tversky Loss (CETL). The performance of Stochastic Gradient Descent with momentum (SGDM) and Adaptive Moment Estimation (Adam) optimizers is compared. It is qualitatively and quantitatively confirmed that DeepLabv3+ and U-Net5ed architectures yield the best results. The DeepLabv3+ architecture with the ResNet-50 backbone, Swish activation function, and CETL loss function reports a Mean Accuracy (MAcc) of 0.976 and Mean Intersection over Union (MIoU) of 0.977. The semantic segmentation of structures with a smaller number of observations, such as the hepatic vein, cystic duct, Liver Ligament, and blood, verifies that the obtained results are very competitive and promising compared to the consulted literature. The proposed selected parameters were validated in the YOLOv9 architecture, which showed an improvement in semantic segmentation compared to the results obtained with the original architecture.

Dynamics of heterogeneous Hopfield neural network with adaptive activation function based on memristor.

Memristor and activation function are two important nonlinear factors of the memristive Hopfield neural network. The effects of different memristors on the dynamics of Hopfield neural networks have been studied by many researchers. However, less attention has been paid to the activation function. In this paper, we present a heterogeneous memristive Hopfield neural network with neurons using different activation functions. The activation functions include fixed activation functions and an adaptive activation function, where the adaptive activation function is based on a memristor. The theoretical and experimental study of the neural network's dynamics has been conducted using phase portraits, bifurcation diagrams, and Lyapunov exponents spectras. Numerical results show that complex dynamical behaviors such as multi-scroll chaos, transient chaos, state jumps and multi-type coexisting attractors can be observed in the heterogeneous memristive Hopfield neural network. In addition, the hardware implementation of memristive Hopfield neural network with adaptive activation function is designed and verified. The experimental results are in good agreement with those obtained using numerical simulations.

Improved weight initialization for deep and narrow feedforward neural network.

Appropriate weight initialization settings, along with the ReLU activation function, have become cornerstones of modern deep learning, enabling the training and deployment of highly effective and efficient neural network models across diverse areas of artificial intelligence. The problem of "dying ReLU," where ReLU neurons become inactive and yield zero output, presents a significant challenge in the training of deep neural networks with ReLU activation function. Theoretical research and various methods have been introduced to address the problem. However, even with these methods and research, training remains challenging for extremely deep and narrow feedforward networks with ReLU activation function. In this paper, we propose a novel weight initialization method to address this issue. We establish several properties of our initial weight matrix and demonstrate how these properties enable the effective propagation of signal vectors. Through a series of experiments and comparisons with existing methods, we demonstrate the effectiveness of the novel initialization method.

OOA-modified Bi-LSTM network: An effective intrusion detection framework for IoT systems.

Currently, the Internet of Things (IoT) generates a huge amount of traffic data in communication and information technology. The diversification and integration of IoT applications and terminals make IoT vulnerable to intrusion attacks. Therefore, it is necessary to develop an efficient Intrusion Detection System (IDS) that guarantees the reliability, integrity, and security of IoT systems. The detection of intrusion is considered a challenging task because of inappropriate features existing in the input data and the slow training process. In order to address these issues, an effective meta heuristic based feature selection and deep learning techniques are developed for enhancing the IDS. The Osprey Optimization Algorithm (OOA) based feature selection is proposed for selecting the highly informative features from the input which leads to an effective differentiation among the normal and attack traffic of network. Moreover, the traditional sigmoid and tangent activation functions are replaced with the Exponential Linear Unit (ELU) activation function to propose the modified Bi-directional Long Short Term Memory (Bi-LSTM). The modified Bi-LSTM is used for classifying the types of intrusion attacks. The ELU activation function makes gradients extremely large during back-propagation and leads to faster learning. This research is analysed in three different datasets such as N-BaIoT, Canadian Institute for Cybersecurity Intrusion Detection Dataset 2017 (CICIDS-2017), and ToN-IoT datasets. The empirical investigation states that the proposed framework obtains impressive detection accuracy of 99.98 %, 99.97 % and 99.88 % on the N-BaIoT, CICIDS-2017, and ToN-IoT datasets, respectively. Compared to peer frameworks, this framework obtains high detection accuracy with better interpretability and reduced processing time.

An analytical approach for unsupervised learning rate estimation using rectified linear units.

Unsupervised learning based on restricted Boltzmann machine or autoencoders has become an important research domain in the area of neural networks. In this paper mathematical expressions to adaptive learning step calculation for RBM with ReLU transfer function are proposed. As a result, we can automatically estimate the step size that minimizes the loss function of the neural network and correspondingly update the learning step in every iteration. We give a theoretical justification for the proposed adaptive learning rate approach, which is based on the steepest descent method. The proposed technique for adaptive learning rate estimation is compared with the existing constant step and Adam methods in terms of generalization ability and loss function. We demonstrate that the proposed approach provides better performance.

Efficient Neural Networks on the Edge with FPGAs by Optimizing an Adaptive Activation Function.

The implementation of neural networks (NNs) on edge devices enables local processing of wireless data, but faces challenges such as high computational complexity and memory requirements when deep neural networks (DNNs) are used. Shallow neural networks customized for specific problems are more efficient, requiring fewer resources and resulting in a lower latency solution. An additional benefit of the smaller network size is that it is suitable for real-time processing on edge devices. The main concern with shallow neural networks is their accuracy performance compared to DNNs. In this paper, we demonstrate that a customized adaptive activation function (AAF) can meet the accuracy of a DNN. We designed an efficient FPGA implementation for a customized segmented spline curve neural network (SSCNN) structure to replace the traditional fixed activation function with an AAF. We compared our SSCNN with different neural network structures such as a real-valued time-delay neural network (RVTDNN), an augmented real-valued time-delay neural network (ARVTDNN), and deep neural networks with different parameters. Our proposed SSCNN implementation uses 40% fewer hardware resources and no block RAMs compared to the DNN with similar accuracy. We experimentally validated this computationally efficient and memory-saving FPGA implementation of the SSCNN for digital predistortion of radio-frequency (RF) power amplifiers using the AMD/Xilinx RFSoC ZCU111. The implemented solution uses less than 3% of the available resources. The solution also enables an increase of the clock frequency to 221.12 MHz, allowing the transmission of wide bandwidth signals.

Physics-informed kernel function neural networks for solving partial differential equations.

This paper proposes an improved version of physics-informed neural networks (PINNs), the physics-informed kernel function neural networks (PIKFNNs), to solve various linear and some specific nonlinear partial differential equations (PDEs). It can also be considered as a novel radial basis function neural network (RBFNN). In the proposed PIKFNNs, it employs one-hidden-layer shallow neural network with the physics-informed kernel functions (PIKFs) as the customized activation functions. The PIKFs fully or partially contain PDE information, which can be chosen as fundamental solutions, green's functions, T-complete functions, harmonic functions, radial Trefftz functions, probability density functions and even the solutions of some linear simplified PDEs and so on. The main difference between the PINNs and the proposed PIKFNNs is that the PINNs add PDE constraints to the loss function, and the proposed PIKFNNs embed PDE information into the activation functions of the neural network. The feasibility and accuracy of the proposed PIKFNNs are validated by some benchmark examples.

IremulbNet: Rethinking the inverted residual architecture for image recognition.

An increasing need of running Convolutional Neural Network (CNN) models on mobile devices encourages the studies on efficient and lightweight neural network model. In this paper, an Inverse Residual Multi-Branch Network named IremulbNet is proposed to solve the problem of insufficient classification accuracy in existing lightweight network models. The core module of this model is to reconstruct an inverse residual structure, in which a special feature fusion method, multi-branch feature extraction, and depthwise separable convolution techniques are used to improve the classification accuracy. The performance of model is tested using image databases. Experimental results show that for the fine-grained image dataset Imagenet-woof, IremulbNet achieved 10.9%, 12.2%, and 15.3% higher accuracy than that of MobileNet V3, ShuffleNet V2, and PeleeNet, respectively. Moreover, it can reduce inference time (GPU) about 42.09% and 75.56% compared to classic ResNet50 and DenseNet121.

Implementation of Convolutional Neural Networks in Memristor Crossbar Arrays with Binary Activation and Weight Quantization.

We propose a hardware-friendly architecture of a convolutional neural network using a 32 × 32 memristor crossbar array having an overshoot suppression layer. The gradual switching characteristics in both set and reset operations enable the implementation of a 3-bit multilevel operation in a whole array that can be utilized as 16 kernels. Moreover, a binary activation function mapped to the read voltage and ground is introduced to evaluate the result of training with a boundary of 0.5 and its estimated gradient. Additionally, we adopt a fixed kernel method, where inputs are sequentially applied to a crossbar array with a differential memristor pair scheme, reducing unused cell waste. The binary activation has robust characteristics against device state variations, and a neuron circuit is experimentally demonstrated on a customized breadboard. Thanks to the analogue switching characteristics of the memristor device, the accurate vector-matrix multiplication (VMM) operations can be experimentally demonstrated by combining sequential inputs and the weights obtained through tuning operations in the crossbar array. In addition, the feature images extracted by VMM during the hardware inference operations on 100 test samples are classified, and the classification performance by off-chip training is compared with the software results. Finally, inference results depending on the tolerance are statistically verified through several tuning cycles.

Cross-Parallel Transformer: Parallel ViT for Medical Image Segmentation.

Medical image segmentation primarily utilizes a hybrid model consisting of a Convolutional Neural Network and sequential Transformers. The latter leverage multi-head self-attention mechanisms to achieve comprehensive global context modelling. However, despite their success in semantic segmentation, the feature extraction process is inefficient and demands more computational resources, which hinders the network's robustness. To address this issue, this study presents two innovative methods: PTransUNet (PT model) and C-PTransUNet (C-PT model). The C-PT module refines the Vision Transformer by substituting a sequential design with a parallel one. This boosts the feature extraction capabilities of Multi-Head Self-Attention via self-correlated feature attention and channel feature interaction, while also streamlining the Feed-Forward Network to lower computational demands. On the Synapse public dataset, the PT and C-PT models demonstrate improvements in DSC accuracy by 0.87% and 3.25%, respectively, in comparison with the baseline model. As for the parameter count and FLOPs, the PT model aligns with the baseline model. In contrast, the C-PT model shows a decrease in parameter count by 29% and FLOPs by 21.4% relative to the baseline model. The proposed segmentation models in this study exhibit benefits in both accuracy and efficiency.

Structure and functions of the N-terminal domain of steroid hormone receptors.

The steroid hormone receptors (SHRs) belong to the large superfamily of nuclear receptors that selectively modulate gene expression in response to specific hormone ligands. The SHRs are required in a broad range of normal physiological processes as well as associated with numerous pathological conditions. Over years, the understanding of the SHR biology and mechanisms of their actions on target cells have found many clinical applications and management of various endocrine-related disorders. However, the effectiveness of SHR-based therapies in endocrine-related cancers remain a clinical challenge. This, in part, is due to the lack of in-depth understanding of structural dynamics and functions of SHRs' intrinsically disordered N-terminal domain (NTD). Recent progress in delineating SHR structural information and their correlations with receptor action in a highly dynamic environment is ultimately helping to explain how diverse SHR signaling mechanisms can elicit selective biological effects. Recent developments are providing new insights of how NTD's structural flexibility plays an important role in SHRs' allosteric regulation leading to the fine tuning of target gene expression to more precisely control SHRs' cell/tissue-specific functions. In this review article, we are discussing the up-to-date knowledge about the SHR actions with a particular emphasis on the structure and functions of the NTD.

A varying-parameter fixed-time gradient-based dynamic network for convex optimization.

We focus on the fixed-time convergence and robustness of gradient-based dynamic networks for solving convex optimization. Most of the existing gradient-based dynamic networks with fixed-time convergence have limited ability to resist interferences of noises. To improve the convergence of the gradient-based dynamic networks, we design a new activation function and propose a gradient-based dynamic network with fixed-time convergence. The proposed dynamic network has a smaller upper bound of the convergence time than the existing dynamic networks with fixed-time convergence. A time-varying scaling parameter is employed to speed up the convergence. Our gradient-based dynamic network is proved to be robust against bounded noises and is able to resist the interference of unbounded noises. The numerical tests illustrate the effectiveness and superiority of the proposed network.

Deep Learning Techniques for Vehicle Detection and Classification from Images/Videos: A Survey.

Detecting and classifying vehicles as objects from images and videos is challenging in appearance-based representation, yet plays a significant role in the substantial real-time applications of Intelligent Transportation Systems (ITSs). The rapid development of Deep Learning (DL) has resulted in the computer-vision community demanding efficient, robust, and outstanding services to be built in various fields. This paper covers a wide range of vehicle detection and classification approaches and the application of these in estimating traffic density, real-time targets, toll management and other areas using DL architectures. Moreover, the paper also presents a detailed analysis of DL techniques, benchmark datasets, and preliminaries. A survey of some vital detection and classification applications, namely, vehicle detection and classification and performance, is conducted, with a detailed investigation of the challenges faced. The paper also addresses the promising technological advancements of the last few years.

Design of continuous-time recurrent neural networks with piecewise-linear activation function for generation of prescribed sequences of bipolar vectors.

A recurrent neural network (RNN) can generate a sequence of patterns as the temporal evolution of the output vector. This paper focuses on a continuous-time RNN model with a piecewise-linear activation function that has neither external inputs nor hidden neurons, and studies the problem of finding the parameters of the model so that it generates a given sequence of bipolar vectors. First, a sufficient condition for the model to generate the desired sequence is derived, which is expressed as a system of linear inequalities in the parameters. Next, three approaches to finding solutions of the system of linear inequalities are proposed: One is formulated as a convex quadratic programming problem and others are linear programming problems. Then, two types of sequences of bipolar vectors that can be generated by the model are presented. Finally, the case where the model generates a periodic sequence of bipolar vectors is considered, and a sufficient condition for the trajectory of the state vector to converge to a limit cycle is provided.

Local Linear Regression and the problem of dimensionality: a remedial strategy via a new locally adaptive bandwidths selector.

Local Linear Regression (LLR) is a nonparametric regression model applied in the modeling phase of Response Surface Methodology (RSM). LLR does not make reference to any fixed parametric model. Hence, LLR is flexible and can capture local trends in the data that might be too complicated for the OLS. However, besides the small sample size and sparse data which characterizes RSM, the performance of the LLR model nosedives as the number of explanatory variables considered in the study increases. This phenomenon, popularly referred to as curse of dimensionality, results in the scanty application of LLR in RSM. In this paper, we propose a novel locally adaptive bandwidths selector, unlike the fixed bandwidths and existing locally adaptive bandwidths selectors, takes into account both the number of the explanatory variables in the study and their individual values at each data point. Single and multiple response problems from the literature and simulated data were used to compare the performance of the L L R P A B with those of the OLS, L L R F B and L L R A B . Neural network activation functions such ReLU, Leaky-ReLU, SELU and SPOCU was considered and give a remarkable improvement on the loss function (Mean Squared Error) over the regression models utilized in the three data.

A Novel Image Classification Method Based on Residual Network, Inception, and Proposed Activation Function.

In deeper layers, ResNet heavily depends on skip connections and Relu. Although skip connections have demonstrated their usefulness in networks, a major issue arises when the dimensions between layers are not consistent. In such cases, it is necessary to use techniques such as zero-padding or projection to match the dimensions between layers. These adjustments increase the complexity of the network architecture, resulting in an increase in parameter number and a rise in computational costs. Another problem is the vanishing gradient caused by utilizing Relu. In our model, after making appropriate adjustments to the inception blocks, we replace the deeper layers of ResNet with modified inception blocks and Relu with our non-monotonic activation function (NMAF). To reduce parameter number, we use symmetric factorization and 1×1 convolutions. Utilizing these two techniques contributed to reducing the parameter number by around 6 M parameters, which has helped reduce the run time by 30 s/epoch. Unlike Relu, NMAF addresses the deactivation problem of the non-positive number by activating the negative values and outputting small negative numbers instead of zero in Relu, which helped in enhancing the convergence speed and increasing the accuracy by 5%, 15%, and 5% for the non-noisy datasets, and 5%, 6%, 21% for non-noisy datasets.

An Optimized Convolutional Neural Network for the 3D Point-Cloud Compression.

Due to the tremendous volume taken by the 3D point-cloud models, knowing how to achieve the balance between a high compression ratio, a low distortion rate, and computing cost in point-cloud compression is a significant issue in the field of virtual reality (VR). Convolutional neural networks have been used in numerous point-cloud compression research approaches during the past few years in an effort to progress the research state. In this work, we have evaluated the effects of different network parameters, including neural network depth, stride, and activation function on point-cloud compression, resulting in an optimized convolutional neural network for compression. We first have analyzed earlier research on point-cloud compression based on convolutional neural networks before designing our own convolutional neural network. Then, we have modified our model parameters using the experimental data to further enhance the effect of point-cloud compression. Based on the experimental results, we have found that the neural network with the 4 layers and 2 strides parameter configuration using the Sigmoid activation function outperforms the default configuration by 208% in terms of the compression-distortion rate. The experimental results show that our findings are effective and universal and make a great contribution to the research of point-cloud compression using convolutional neural networks.