A bio-inspired positional embedding network for transformer-based models.

Owing to the progress of transformer-based networks, there have been significant improvements in the performance of vision models in recent years. However, there is further potential for improvement in positional embeddings that play a crucial role in distinguishing information across different positions. Based on the biological mechanisms of human visual pathways, we propose a positional embedding network that adaptively captures position information by modeling the dorsal pathway, which is responsible for spatial perception in human vision. Our proposed double-stream architecture leverages large zero-padding convolutions to learn local positional features and utilizes transformers to learn global features, effectively capturing the interaction between dorsal and ventral pathways. To evaluate the effectiveness of our method, we implemented experiments on various datasets, employing differentiated designs. Our statistical analysis demonstrates that the simple implementation significantly enhances image classification performance, and the observed trends demonstrate its biological plausibility.

Enhanced Gradient for Differentiable Architecture Search.

In recent years, neural architecture search (NAS) methods have been proposed for the automatic generation of task-oriented network architecture in image classification. However, the architectures obtained by existing NAS approaches are optimized only for classification performance and do not adapt to devices with limited computational resources. To address this challenge, we propose a neural network architecture search algorithm aiming to simultaneously improve the network performance and reduce the network complexity. The proposed framework automatically builds the network architecture at two stages: block-level search and network-level search. At the stage of block-level search, a gradient-based relaxation method is proposed, using an enhanced gradient to design high-performance and low-complexity blocks. At the stage of network-level search, an evolutionary multiobjective algorithm is utilized to complete the automatic design from blocks to the target network. The experimental results demonstrate that our method outperforms all evaluated hand-crafted networks in image classification, with an error rate of 3.18% on Canadian Institute for Advanced Research (CIFAR10) and an error rate of 19.16% on CIFAR100, both at network parameter size less than 1 M. Obviously, compared with other NAS methods, our method offers a tremendous reduction in designed network architecture parameters.

Remix: Towards the transferability of adversarial examples.

Deep neural networks (DNNs) are susceptible to adversarial examples, which are crafted by deliberately adding some human-imperceptible perturbations on original images. To explore the vulnerability of models of DNNs, transfer-based black-box attacks are attracting increasing attention of researchers credited to their high practicality. The transfer-based approaches can launch attacks against models easily in the black-box setting by resultant adversarial examples, whereas the success rates are not satisfactory. To boost the adversarial transferability, we propose a Remix method with multiple input transformations, which could achieve multiple data augmentation by utilizing gradients from previous iterations and images from other categories in the same iteration. Extensive experiments on the NeurIPS 2017 adversarial dataset and the ILSVRC 2012 validation dataset demonstrate that the proposed approach could drastically enhance the adversarial transferability and maintain similar success rates of white-box attacks on both undefended models and defended models. Furthermore, extended experiments based on LPIPS show that our method could maintain a similar perceived distance compared to other baselines.

Variational Bayesian Inference for Robust Identification of PWARX Systems With Time-Varying Time-Delays.

This article presents a robust variational Bayesian (VB) algorithm for identifying piecewise autoregressive exogenous (PWARX) systems with time-varying time-delays. To alleviate the adverse effects caused by outliers, the probability distribution of noise is taken to follow a t -distribution. Meanwhile, a solution strategy for more accurately classifying undecidable data points is proposed, and the hyperplanes used to split data are determined by a support vector machine (SVM). In addition, maximum-likelihood estimation (MLE) is adopted to re-estimate the unknown parameters through the classification results. The time-delay is regarded as a hidden variable and identified through the VB algorithm. The effectiveness of the proposed algorithm is illustrated by two simulation examples.

Adaptive Prototypical Networks With Label Words and Joint Representation Learning for Few-Shot Relation Classification.

Relation classification (RC) task is one of fundamental tasks of information extraction, aiming to detect the relation information between entity pairs in unstructured natural language text and generate structured data in the form of entity-relation triple. Although distant supervision methods can effectively alleviate the problem of lack of training data in supervised learning, they also introduce noise into the data and still cannot fundamentally solve the long-tail distribution problem of the training instances. In order to enable the neural network to learn new knowledge through few instances such as humans, this work focuses on few-shot relation classification (FSRC), where a classifier should generalize to new classes that have not been seen in the training set, given only a number of samples for each class. To make full use of the existing information and get a better feature representation for each instance, we propose to encode each class prototype in an adaptive way from two aspects. First, based on the prototypical networks, we propose an adaptive mixture mechanism to add label words to the representation of the class prototype, which, to the best of our knowledge, is the first attempt to integrate the label information into features of the support samples of each class so as to get more interactive class prototypes. Second, to more reasonably measure the distances between samples of each category, we introduce a loss function for joint representation learning (JRL) to encode each support instance in an adaptive manner. Extensive experiments have been conducted on FewRel under different few-shot (FS) settings, and the results show that the proposed adaptive prototypical networks with label words and JRL has not only achieved significant improvements in accuracy but also increased the generalization ability of FSRC.

Boosting the transferability of adversarial examples via stochastic serial attack.

Deep neural networks (DNNs) are vulnerable to adversarial examples, which are crafted by imposing mild perturbation on clean ones. An intriguing property of adversarial examples is that they are efficient among different DNNs. Thus transfer-based attacks against DNNs become an increasing concern. In this scenario, attackers devise adversarial instances based on the local model without feedback information from the target one. Unfortunately, most existing transfer-based attack methods only employ a single local model to generate adversarial examples. It results in poor transferability because of overfitting to the local model. Although several ensemble attacks have been proposed, the transferability of adversarial examples merely have a slight increase. Meanwhile, these methods need high memory cost during the training process. To this end, we propose a novel attack strategy called stochastic serial attack (SSA). It adopts a serial strategy to attack local models, which reduces memory consumption compared to parallel attacks. Moreover, since local models are stochastically selected from a large model set, SSA can ensure that the adversarial examples do not overfit specific weaknesses of local source models. Extensive experiments on the ImageNet dataset and NeurIPS 2017 adversarial competition dataset show the effectiveness of SSA in improving the transferability of adversarial examples and reducing the memory consumption of the training process.

Computational Modeling of Structural Synaptic Plasticity in Echo State Networks.

Most existing studies on computational modeling of neural plasticity have focused on synaptic plasticity. However, regulation of the internal weights in the reservoir based on synaptic plasticity often results in unstable learning dynamics. In this article, a structural synaptic plasticity learning rule is proposed to train the weights and add or remove neurons within the reservoir, which is shown to be able to alleviate the instability of the synaptic plasticity, and to contribute to increase the memory capacity of the network as well. Our experimental results also reveal that a few stronger connections may last for a longer period of time in a constantly changing network structure, and are relatively resistant to decay or disruptions in the learning process. These results are consistent with the evidence observed in biological systems. Finally, we show that an echo state network (ESN) using the proposed structural plasticity rule outperforms an ESN using synaptic plasticity and three state-of-the-art ESNs on four benchmark tasks.

Visual interaction networks: A novel bio-inspired computational model for image classification.

Inspired by biological mechanisms and structures in neuroscience, many biologically inspired visual computational models have been presented to provide new solutions for visual recognition task. For example, convolutional neural network (CNN) was proposed according to the hierarchical structure of biological vision, which could achieve superior performance in large-scale image classification. In this paper, we propose a new framework called visual interaction networks (VIN-Net), which is inspired by visual interaction mechanisms. More specifically, self-interaction, mutual-interaction, multi-interaction, and adaptive interaction are proposed in VIN-Net, forming the first interactive completeness of the visual interaction model. To further enhance the representation ability of visual features, the adaptive adjustment mechanism is integrated into the VIN-Net model. Finally, our model is evaluated on three benchmark datasets and two self-built textile defect datasets. The experimental results demonstrate that the proposed model exhibits its efficiency on visual classification tasks. Furthermore, a textile industrial application shows that the proposed architecture outperforms the state-of-the-art approaches in classification performance.

Evolving Local Plasticity Rules for Synergistic Learning in Echo State Networks.

Existing synaptic plasticity rules for optimizing the connections between neurons within the reservoir of echo state networks (ESNs) remain to be global in that the same type of plasticity rule with the same parameters is applied to all neurons. However, this is biologically implausible and practically inflexible for learning the structures in the input signals, thereby limiting the learning performance of ESNs. In this paper, we propose to use local plasticity rules that allow different neurons to use different types of plasticity rules and different parameters, which are achieved by optimizing the parameters of the local plasticity rules using the evolution strategy (ES) with covariance matrix adaptation (CMA-ES). We show that evolving neural plasticity will result in a synergistic learning of different plasticity rules, which plays an important role in improving the learning performance. Meanwhile, we show that the local plasticity rules can effectively alleviate synaptic interferences in learning the structure in sensory inputs. The proposed local plasticity rules are compared with a number of the state-of-the-art ESN models and the canonical ESN using a global plasticity rule on a set of widely used prediction and classification benchmark problems to demonstrate its competitive learning performance.

Immune-Endocrine System Inspired Hierarchical Coevolutionary Multiobjective Optimization Algorithm for IoT Service.

The intelligent devices in Internet of Things (IoT) not only provide services but also consider how to allocate heterogeneous resources and reduce resource consumption and service time as far as possible. This issue becomes crucial in the case of large-scale IoT environments. In order for the IoT service system to respond to multiple requests simultaneously and provide Pareto optimal decisions, we propose an immune-endocrine system inspired hierarchical coevolutionary multiobjective optimization algorithm (IE-HCMOA) in this paper. In IE-HCMOA, a multiobjective immune algorithm based on global ranking with vaccine is designed to choose superior antibodies. Meanwhile, we adopt clustering in top population to make the operations more directional and purposeful and realize self-adaptive searching. And we use the human forgetting memory mechanism to design two-level memory storage for the choice problem of solutions to achieve promising performance. In order to validate the practicability and effectiveness of IE-HCMOA, we apply it to the field of agricultural IoT service. The simulation results demonstrate that the proposed algorithm can obtain the best Pareto, the strongest exploration ability, and excellent performance than nondominated neighbor immune algorithms and NSGA-II.

A Self-Learning Immune Co-Evolutionary Network for Multiple Escaping Targets Search With Random Observable Conditions.

The search for multiple escaping targets is a significant issue of cooperative control in multi-agent systems since targets consciously seek to avoid being captured. Moreover, the assumption of continuous observations in existing works is not always suitable due to the limit of measuring equipment and uncertain movement of targets. Therefore, the problem with searching for escaping targets, which can be more aptly labeled "multiple escaping-targets search with random observation conditions" (MESROC), is difficult to address by conventional methods. Inspired by machine learning and the immune response mechanism of human bodies, a self-learning immune co-evolutionary network (SLICEN) is proposed. The SLICEN consists mainly of an immune cellular network (ICN) and an immune learning algorithm (ILA). The ICN provides feasible solutions to MESROC. Different kinds of network models are introduced to work as an ICN, such as convolutional neural networks, extreme learning machines, and support vector machines. The ILA evaluates the performance of feasible solutions and selects the optimal ones to further strengthen ICN reversely. Solutions are repeatedly improved through the co-evolution of ICN and ILA. An essential distinction to conventional machine learning approaches is that SLICEN works well without training samples. Simulations and comparisons demonstrate that patterns of advanced cooperative behavior among searchers function properly. SLICEN is an efficient method for solving MESROC.

A Clustering-Based Adaptive Evolutionary Algorithm for Multiobjective Optimization With Irregular Pareto Fronts.

Existing multiobjective evolutionary algorithms (MOEAs) perform well on multiobjective optimization problems (MOPs) with regular Pareto fronts in which the Pareto optimal solutions distribute continuously over the objective space. When the Pareto front is discontinuous or degenerated, most existing algorithms cannot achieve good results. To remedy this issue, a clustering-based adaptive MOEA (CA-MOEA) is proposed in this paper for solving MOPs with irregular Pareto fronts. The main idea is to adaptively generate a set of cluster centers for guiding selection at each generation to maintain diversity and accelerate convergence. We investigate the performance of CA-MOEA on 18 widely used benchmark problems. Our results demonstrate the competitiveness of CA-MOEA for multiobjective optimization, especially for problems with irregular Pareto fronts. In addition, CA-MOEA is shown to perform well on the optimization of the stretching parameters in the carbon fiber formation process.

Degeneration Recognizing Clonal Selection Algorithm for Multimodal Optimization.

In this paper, a computing speed improvement for the clonal selection algorithm (CSA) is proposed based on a degeneration recognizing (DR) method. The degeneration recognizing clonal selection algorithm (DR-CSA) is designed for solving complex engineering multimodal optimization problems. On each iteration of CSA, there is a large amount of eliminated solutions which are usually neglected. But these solutions do contain the knowledge of the nonoptimal area. By storing and utilizing these data, the DR-CSA is aimed to identify part of the new population as degenerated and eliminate them before the evaluation operation, so that a number of evaluation times can be avoided. This pre-elimination operation is able to save computing time because the evaluation is the main reason for the time cost in the complex engineering optimization problem. Experiments on both test function and a real-world engineering optimization problem (wet spinning coagulating process) are conducted. The results show that the proposed DR-CSA is as accurate as regular CSA and is effective in reducing a considerable amount of computing time.

Global nonlinear kernel prediction for large data set with a particle swarm-optimized interval support vector regression.

A new global nonlinear predictor with a particle swarm-optimized interval support vector regression (PSO-ISVR) is proposed to address three issues (viz., kernel selection, model optimization, kernel method speed) encountered when applying SVR in the presence of large data sets. The novel prediction model can reduce the SVR computing overhead by dividing input space and adaptively selecting the optimized kernel functions to obtain optimal SVR parameter by PSO. To quantify the quality of the predictor, its generalization performance and execution speed are investigated based on statistical learning theory. In addition, experiments using synthetic data as well as the stock volume weighted average price are reported to demonstrate the effectiveness of the developed models. The experimental results show that the proposed PSO-ISVR predictor can improve the computational efficiency and the overall prediction accuracy compared with the results produced by the SVR and other regression methods. The proposed PSO-ISVR provides an important tool for nonlinear regression analysis of big data.

Intelligent Optimization of the Film-to-Fiber Ratio of a Degradable Braided Bicomponent Ureteral Stent.

A hierarchical support vector regression (SVR) model (HSVRM) was employed to correlate the compositions and mechanical properties of bicomponent stents composed of poly(lactic-co-glycolic acid) (PGLA) film and poly(glycolic acid) (PGA) fibers for urethral repair for the first time. PGLA film and PGA fibers could provide ureteral stents with good compressive and tensile properties, respectively. In bicomponent stents, high film content led to high stiffness, while high fiber content resulted in poor compressional properties. To simplify the procedures to optimize the ratio of PGLA film and PGA fiber in the stents, a hierarchical support vector regression model (HSVRM) and particle swarm optimization (PSO) algorithm were used to construct relationships between the film-to-fiber weight ratio and the measured compressional/tensile properties of the stents. The experimental data and simulated data fit well, proving that the HSVRM could closely reflect the relationship between the component ratio and performance properties of the ureteral stents.

Bidirectional optimization of the melting spinning process.

A bidirectional optimizing approach for the melting spinning process based on an immune-enhanced neural network is proposed. The proposed bidirectional model can not only reveal the internal nonlinear relationship between the process configuration and the quality indices of the fibers as final product, but also provide a tool for engineers to develop new fiber products with expected quality specifications. A neural network is taken as the basis for the bidirectional model, and an immune component is introduced to enlarge the searching scope of the solution field so that the neural network has a larger possibility to find the appropriate and reasonable solution, and the error of prediction can therefore be eliminated. The proposed intelligent model can also help to determine what kind of process configuration should be made in order to produce satisfactory fiber products. To make the proposed model practical to the manufacturing, a software platform is developed. Simulation results show that the proposed model can eliminate the approximation error raised by the neural network-based optimizing model, which is due to the extension of focusing scope by the artificial immune mechanism. Meanwhile, the proposed model with the corresponding software can conduct optimization in two directions, namely, the process optimization and category development, and the corresponding results outperform those with an ordinary neural network-based intelligent model. It is also proved that the proposed model has the potential to act as a valuable tool from which the engineers and decision makers of the spinning process could benefit.

The Bi-Directional Prediction of Carbon Fiber Production Using a Combination of Improved Particle Swarm Optimization and Support Vector Machine.

This paper creates a bi-directional prediction model to predict the performance of carbon fiber and the productive parameters based on a support vector machine (SVM) and improved particle swarm optimization (IPSO) algorithm (SVM-IPSO). In the SVM, it is crucial to select the parameters that have an important impact on the performance of prediction. The IPSO is proposed to optimize them, and then the SVM-IPSO model is applied to the bi-directional prediction of carbon fiber production. The predictive accuracy of SVM is mainly dependent on its parameters, and IPSO is thus exploited to seek the optimal parameters for SVM in order to improve its prediction capability. Inspired by a cell communication mechanism, we propose IPSO by incorporating information of the global best solution into the search strategy to improve exploitation, and we employ IPSO to establish the bi-directional prediction model: in the direction of the forward prediction, we consider productive parameters as input and property indexes as output; in the direction of the backward prediction, we consider property indexes as input and productive parameters as output, and in this case, the model becomes a scheme design for novel style carbon fibers. The results from a set of the experimental data show that the proposed model can outperform the radial basis function neural network (RNN), the basic particle swarm optimization (PSO) method and the hybrid approach of genetic algorithm and improved particle swarm optimization (GA-IPSO) method in most of the experiments. In other words, simulation results demonstrate the effectiveness and advantages of the SVM-IPSO model in dealing with the problem of forecasting.

A novel automatic detection system for ECG arrhythmias using maximum margin clustering with immune evolutionary algorithm.

This paper presents a novel maximum margin clustering method with immune evolution (IEMMC) for automatic diagnosis of electrocardiogram (ECG) arrhythmias. This diagnostic system consists of signal processing, feature extraction, and the IEMMC algorithm for clustering of ECG arrhythmias. First, raw ECG signal is processed by an adaptive ECG filter based on wavelet transforms, and waveform of the ECG signal is detected; then, features are extracted from ECG signal to cluster different types of arrhythmias by the IEMMC algorithm. Three types of performance evaluation indicators are used to assess the effect of the IEMMC method for ECG arrhythmias, such as sensitivity, specificity, and accuracy. Compared with K-means and iterSVR algorithms, the IEMMC algorithm reflects better performance not only in clustering result but also in terms of global search ability and convergence ability, which proves its effectiveness for the detection of ECG arrhythmias.

Performance Prediction of Differential Fibers with a Bi-Directional Optimization Approach.

This paper develops a bi-directional prediction approach to predict the production parameters and performance of differential fibers based on neural networks and a multi-objective evolutionary algorithm. The proposed method does not require accurate description and calculation for the multiple processes, different modes and complex conditions of fiber production. The bi-directional prediction approach includes the forward prediction and backward reasoning. Particle swam optimization algorithms with K-means algorithm are used to minimize the prediction error of the forward prediction results. Based on the forward prediction, backward reasoning uses the multi-objective evolutionary algorithm to find the reasoning results. Experiments with polyester filament parameters of differential production conditions indicate that the proposed approach obtains good prediction results. The results can be used to optimize fiber production and to design differential fibers. This study also has important value and widespread application prospects regarding the spinning of differential fiber optimization.

[The study on the characters of membrane protein interaction and its network based on integrated intelligence method].

Membrane protein and its interaction network have become a novel research direction in bioinformatics. In this paper, a novel membrane protein interaction network simulator is proposed for system biology studies by integrated intelligence method including spectrum analysis, fuzzy K-Nearest Neighbor(KNN) algorithm and so on. We consider biological system as a set of active computational components interacting with each other and with the external environment. Then we can use the network simulator to construct membrane protein interaction networks. Based on the proposed approach, we found that the membrane protein interaction network almost has some dynamic and collective characteristics, such as small-world network, scale free distributing, and hierarchical module structure. These properties are similar to those of other extensively studied protein interaction networks. The present studies on the characteristics of the membrane protein interaction network will be valuable for its relatively biological and medical studies.