Online sequential fuzzy extreme learning machine for function approximation and classification problems.

In this correspondence, an online sequential fuzzy extreme learning machine (OS-Fuzzy-ELM) has been developed for function approximation and classification problems. The equivalence of a Takagi-Sugeno-Kang (TSK) fuzzy inference system (FIS) to a generalized single hidden-layer feedforward network is shown first, which is then used to develop the OS-Fuzzy-ELM algorithm. This results in a FIS that can handle any bounded nonconstant piecewise continuous membership function. Furthermore, the learning in OS-Fuzzy-ELM can be done with the input data coming in a one-by-one mode or a chunk-by-chunk (a block of data) mode with fixed or varying chunk size. In OS-Fuzzy-ELM, all the antecedent parameters of membership functions are randomly assigned first, and then, the corresponding consequent parameters are determined analytically. Performance comparisons of OS-Fuzzy-ELM with other existing algorithms are presented using real-world benchmark problems in the areas of nonlinear system identification, regression, and classification. The results show that the proposed OS-Fuzzy-ELM produces similar or better accuracies with at least an order-of-magnitude reduction in the training time.

Neural adaptive control for vibration suppression in composite fin-tip of aircraft.

In this paper, we present a neural adaptive control scheme for active vibration suppression of a composite aircraft fin tip. The mathematical model of a composite aircraft fin tip is derived using the finite element approach. The finite element model is updated experimentally to reflect the natural frequencies and mode shapes very accurately. Piezo-electric actuators and sensors are placed at optimal locations such that the vibration suppression is a maximum. Model-reference direct adaptive neural network control scheme is proposed to force the vibration level within the minimum acceptable limit. In this scheme, Gaussian neural network with linear filters is used to approximate the inverse dynamics of the system and the parameters of the neural controller are estimated using Lyapunov based update law. In order to reduce the computational burden, which is critical for real-time applications, the number of hidden neurons is also estimated in the proposed scheme. The global asymptotic stability of the overall system is ensured using the principles of Lyapunov approach. Simulation studies are carried-out using sinusoidal force functions of varying frequency. Experimental results show that the proposed neural adaptive control scheme is capable of providing significant vibration suppression in the multiple bending modes of interest. The performance of the proposed scheme is better than the H(infinity) control scheme.

Multi-category classification using an Extreme Learning Machine for microarray gene expression cancer diagnosis.

In this paper, the recently developed Extreme Learning Machine (ELM) is used for direct multicategory classification problems in the cancer diagnosis area. ELM avoids problems like local minima, improper learning rate and overfitting commonly faced by iterative learning methods and completes the training very fast. We have evaluated the multi-category classification performance of ELM on three benchmark microarray datasets for cancer diagnosis, namely, the GCM dataset, the Lung dataset and the Lymphoma dataset. The results indicate that ELM produces comparable or better classification accuracies with reduced training time and implementation complexity compared to artificial neural networks methods like conventional back-propagation ANN, Linder's SANN, and Support Vector Machine methods like SVM-OVO and Ramaswamy's SVM-OVA. ELM also achieves better accuracies for classification of individual categories.

A fast and accurate online sequential learning algorithm for feedforward networks.

In this paper, we develop an online sequential learning algorithm for single hidden layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes in a unified framework. The algorithm is referred to as online sequential extreme learning machine (OS-ELM) and can learn data one-by-one or chunk-by-chunk (a block of data) with fixed or varying chunk size. The activation functions for additive nodes in OS-ELM can be any bounded nonconstant piecewise continuous functions and the activation functions for RBF nodes can be any integrable piecewise continuous functions. In OS-ELM, the parameters of hidden nodes (the input weights and biases of additive nodes or the centers and impact factors of RBF nodes) are randomly selected and the output weights are analytically determined based on the sequentially arriving data. The algorithm uses the ideas of ELM of Huang et al. developed for batch learning which has been shown to be extremely fast with generalization performance better than other batch training methods. Apart from selecting the number of hidden nodes, no other control parameters have to be manually chosen. Detailed performance comparison of OS-ELM is done with other popular sequential learning algorithms on benchmark problems drawn from the regression, classification and time series prediction areas. The results show that the OS-ELM is faster than the other sequential algorithms and produces better generalization performance.

A generalized growing and pruning RBF (GGAP-RBF) neural network for function approximation.

This paper presents a new sequential learning algorithm for radial basis function (RBF) networks referred to as generalized growing and pruning algorithm for RBF (GGAP-RBF). The paper first introduces the concept of significance for the hidden neurons and then uses it in the learning algorithm to realize parsimonious networks. The growing and pruning strategy of GGAP-RBF is based on linking the required learning accuracy with the significance of the nearest or intentionally added new neuron. Significance of a neuron is a measure of the average information content of that neuron. The GGAP-RBF algorithm can be used for any arbitrary sampling density for training samples and is derived from a rigorous statistical point of view. Simulation results for bench mark problems in the function approximation area show that the GGAP-RBF outperforms several other sequential learning algorithms in terms of learning speed, network size and generalization performance regardless of the sampling density function of the training data.

An efficient sequential learning algorithm for growing and pruning RBF (GAP-RBF) networks.

This paper presents a simple sequential growing and pruning algorithm for radial basis function (RBF) networks. The algorithm referred to as growing and pruning (GAP)-RBF uses the concept of "Significance" of a neuron and links it to the learning accuracy. "Significance" of a neuron is defined as its contribution to the network output averaged over all the input data received so far. Using a piecewise-linear approximation for the Gaussian function, a simple and efficient way of computing this significance has been derived for uniformly distributed input data. In the GAP-RBF algorithm, the growing and pruning are based on the significance of the "nearest" neuron. In this paper, the performance of the GAP-RBF learning algorithm is compared with other well-known sequential learning algorithms like RAN, RANEKF, and MRAN on an artificial problem with uniform input distribution and three real-world nonuniform, higher dimensional benchmark problems. The results indicate that the GAP-RBF algorithm can provide comparable generalization performance with a considerably reduced network size and training time.

Text-independent speaker verification using Minimal Resource Allocation Networks.

This paper presents a text-independent speaker verification system based on an online Radial Basis Function (RBF) network referred to as Minimal Resource Allocation Network (MRAN). MRAN is a sequential learning RBF, in which hidden neurons are added or removed as training progresses. LP-derived cepstral coefficients are used as feature vectors during training and verification phases. The performance of MRAN is compared with other well-known RBF and Elliptical Basis Function (EBF) based speaker verification methods in terms of error rates and computational complexity on a series of speaker verification experiments. The experiments use data from 258 speakers from the phonetically balancedcontinuous speech corpus TIMIT. The results show that MRAN produces comparable error rates to other methods with much less computational complexity.

Novel neutral network approach to call admission control in high-speed networks.

This paper presents a novel Call Admission Control (CAC) scheme which adopts the neural network approach, namely Minimal Resource Allocation Network (MRAN) and its extended version EMRAN. Though the current focus is on the Call Admission Control (CAC) for Asynchronous Transfer Mode (ATM) networks, the scheme is applicable to most high-speed networks. As there is a need for accurate estimation of the required bandwidth for different services, the proposed scheme can offer a simple design procedure and provide a better control in fulfilling the Quality of Service (QoS) requirements. MRAN and EMRAN are on-line learning algorithms to facilitate efficient admission control in different traffic environments. Simulation results show that the proposed CAC schemes are more efficient than the two conventional CAC approaches, the Peak Bandwidth Allocation scheme and the Cell Loss Ratio (CLR) upperbound formula scheme. The prediction precision and computational time of MRAN and EMRAN algorithms are also investigated. Both MRAN and EMRAN algorithms yield similar performance results, but the EMRAN algorithm has less computational load.

Communication channel equalization using complex-valued minimal radial basis function neural networks.

A complex radial basis function neural network is proposed for equalization of quadrature amplitude modulation (QAM) signals in communication channels. The network utilizes a sequential learning algorithm referred to as complex minimal resource allocation network (CMRAN) and is an extension of the MRAN algorithm originally developed for online learning in real-valued radial basis function (RBF) networks. CMRAN has the ability to grow and prune the (complex) RBF network's hidden neurons to ensure a parsimonious network structure. The performance of the CMRAN equalizer for nonlinear channel equalization problems has been evaluated by comparing it with the functional link artificial neural network (FLANN) equalizer of J.C. Patra et al. (1999) and the Gaussian stochastic gradient (SG) RBF equalizer of I. Cha and S. Kassam (1995). The results clearly show that CMRANs performance is superior in terms of symbol error rates and network complexity.

EEG source localization: comparative study of classical and neural network methods.

We address the problem of estimating biopotential sources within the brain, based on EEG signals observed on the scalp. This problem, known as the inverse problem of electrophysiology, has no closed-form solution, and requires iterative techniques such as the Levenberg-Marquardt (LM) algorithm. Considering the nonlinear nature of the inverse problem, and the low signal to noise ratio inherent in EEG signals, a backpropagation neural network (BPN) has been recently proposed as a solution. The technique has not been properly compared with classical techniques such as the LM method, or with more recent neural network techniques such as the Radial Basis Function (RBF) network. In this paper, we provide improved strategies based on BPN and consider RBF networks in solving the inverse problem. We compare the performances of BPN, RBF and a hybrid technique with that of the classical LM method.

Nonlinear magnetic storage channel equalization using minimal resource allocation network (MRAN).

This letter presents the application of the recently developed minimal radial basis function neural network called minimal resource allocation network (MRAN) for equalization in highly nonlinear magnetic data storage channels. Using a realistic magnetic channel model, MRAN equalizer's performance is compared with the nonlinear neural equalizer of Nair and Moon (1997), referred to as maximum signal-to-distortion ratio (MSDR) equalizer. MSDR equalizer uses a specially designed neural architecture where all the parameters are determined theoretically. Simulation results indicate that MRAN equalizer has better performance than that of MSDR equalizer in terms of higher signal-to-distortion ratios.

RBF networks for source localization in quantitative electrophysiology.

The backpropagation neural network methods have been proposed recently to solve the inverse problem in quantitative electrophysiology. A major advantage of the technique is that once a neural network is trained, it no longer requires iterations or access to sophisticated computations. We propose to use RBF networks for source localization in the brain, and systematically compare their performance to those of Levenberg-Marquardt (LM) algorithms. We show the use of two types of Radial Basis Function Networks (RBF) network: a classic network with fixed number of hidden layer neurons and an improved network, Minimal Resource Allocation Network (MRAN), recently proposed by one of the authors, capable for dynamically configuring its structure so as to obtain a compact topology to match the data presented to it.

Complex-valued minimal resource allocation network for nonlinear signal processing.

This paper presents a sequential learning algorithm and evaluates its performance on complex valued signal processing problems. The algorithm is referred to as Complex Minimal Resource Allocation Network (CMRAN) algorithm and it is an extension of the MRAN algorithm originally developed for online learning in real valued RBF networks. CMRAN has the ability to grow and prune the (complex) RBF network's hidden neurons to ensure a parsimonious network structure. The performance of the learning algorithm is illustrated using two applications from signal processing of communication systems. The first application considers identification of a nonlinear complex channel. The second application considers the use of CMRAN to QAM digital channel equalization problems. Simulation results presented clearly show that CMRAN is very effective in modeling and equalization with performance achieved often being superior to that of some of the well known methods.

A complex valued radial basis function network for equalization of fast time varying channels.

This paper presents a complex valued radial basis function (RBF) network for equalization of fast time varying channels. A new method for calculating the centers of the RBF network is given. The method allows fixing the number of RBF centers even as the equalizer order is increased so that a good performance is obtained by a high-order RBF equalizer with small number of centers. Simulations are performed on time varying channels using a Rayleigh fading channel model to compare the performance of our RBF with an adaptive maximum-likelihood sequence estimator (MLSE) consisting of a channel estimator and a MLSE implemented by the Viterbi algorithm. The results show that the RBF equalizer produces superior performance with less computational complexity.

Performance evaluation of a sequential minimal radial basis function (RBF) neural network learning algorithm.

This paper presents a detailed performance analysis of the minimal resource allocation network (M-RAN) learning algorithm, M-RAN is a sequential learning radial basis function neural network which combines the growth criterion of the resource allocating network (RAN) of Platt (1991) with a pruning strategy based on the relative contribution of each hidden unit to the overall network output. The resulting network leads toward a minimal topology for the RAN. The performance of this algorithm is compared with the multilayer feedforward networks (MFNs) trained with 1) a variant of the standard backpropagation algorithm, known as RPROP and 2) the dependence identification (DI) algorithm of Moody and Antsaklis on several benchmark problems in the function approximation and pattern classification areas. For all these problems, the M-RAN algorithm is shown to realize networks with far fewer hidden neurons with better or same approximation/classification accuracy. Further, the time taken for learning (training) is also considerably shorter as M-RAN does not require repeated presentation of the training data.

A sequential learning scheme for function approximation using minimal radial basis function neural networks.

This article presents a sequential learning algorithm for function approximation and time-series prediction using a minimal radial basis function neural network (RBFNN). The algorithm combines the growth criterion of the resource-allocating network (RAN) of Platt (1991) with a pruning strategy based on the relative contribution of each hidden unit to the overall network output. The resulting network leads toward a minimal topology for the RBFNN. The performance of the algorithm is compared with RAN and the enhanced RAN algorithm of Kadirkamanathan and Niranjan (1993) for the following benchmark problems: (1) hearta from the benchmark problems database PROBEN1, (2) Hermite polynomial, and (3) Mackey-Glass chaotic time series. For these problems, the proposed algorithm is shown to realize RBFNNs with far fewer hidden neurons with better or same accuracy.

Parallel implementation of backpropagation neural networks on a heterogeneous array of transputers.

This paper analyzes parallel implementation of the backpropagation training algorithm on a heterogeneous transputer network (i.e., transputers of different speed and memory) connected in a pipelined ring topology. Training-set parallelism is employed as the parallelizing paradigm for the backpropagation algorithm. It is shown through analysis that finding the optimal allocation of the training patterns amongst the processors to minimize the time for a training epoch is a mixed integer programming problem. Using mixed integer programming optimal pattern allocations for heterogeneous processor networks having a mixture of T805-20 (20 MHz) and T805-25 (25 MHz) transputers are theoretically found for two benchmark problems. The time for an epoch corresponding to the optimal pattern allocations is then obtained experimentally for the benchmark problems from the T805-20, TS805-25 heterogeneous networks. A Monte Carlo simulation study is carried out to statistically verify the optimality of the epoch time obtained from the mixed integer programming based allocations. In this study pattern allocations are randomly generated and the corresponding time for an epoch is experimentally obtained from the heterogeneous network. The mean and standard deviation for the epoch times from the random allocations are then compared with the optimal epoch time. The results show the optimal epoch time to be always lower than the mean epoch times by more than three standard deviations (3sigma) for all the sample sizes used in the study thus giving validity to the theoretical analysis.

Analysis of training set parallelism for backpropagation neural networks.

Training set parallelism and network based parallelism are two popular paradigms for parallelizing a feedforward (artificial) neural network. Training set parallelism is particularly suited to feedforward neural networks with backpropagation learning where the size of the training set is large in relation to the size of the network. This paper analyzes training set parallelism for feedforward neural networks when implemented on a transputer array configured in a pipelined ring topology. Theoretical expressions for the time per epoch (iteration) and optimal size of a processor network are derived when the training set is equally distributed among the processing nodes. These show that the speed up is a function of the number of patterns per processor, communication overhead per epoch and the total number of processors in the topology. Further analysis of how to optimally distribute the training set on a given processor network when the number of patterns in the training set is not an integer multiple of the number of processors, is also carried out. It is shown that optimal allocation of patterns in such cases is a mixed integer programming problem. Using this analysis it is found that equal distribution of training patterns among the processors is not the optimal way to allocate the patterns even when the training set is an integer multiple of the number of processors. Extension of the analysis to processor networks comprising processors of different speeds is also carried out. Experimental results from a T805 transputer array are presented to verify all the theoretical results.

Dynamic programming approach to optimal weight selection in multilayer neural networks.

A novel algorithm for weight adjustments in a multilayer neural network is derived using the principles of dynamic programming. The algorithm computes the optimal values for weights on a layer-by-layer basis starting from the output layer of the network. The advantage of this algorithm is that it provides an error function for every hidden layer expressed entirely in terms of the weights and outputs of the hidden layer, and minimization of this error function yields the optimum weights for the hidden layer.

An improved mathematical model of human thyroid hormone regulation.

1. A mathematical model has been constructed of human thyroid hormone regulation by the anterior pituitary gland, which takes account of most of the currently available experimental data. 2. Successful simulation of data on the stimulation of thyrotrophin (TSH) secretion by thyrotrophin releasing hormone (TRH) was achieved assuming that the TSH secretion rate is proportional to the logarithm of the concurrent blood TRH level. 3. Data on the regulation of triiodothyronine (T3) secretion by TSH and the inhibition of TSH secretion by thyroid hormones in contrast could not be simulated on the assumption of instantaneous proportional responses. A mixture of proportional and integral control--the latter taking account of the past history of plasma levels of the regulatory hormone--appeared to be operating at both levels. 4. The pituitary gland appears to be more sensitive to a given fractional change in TRH secretion rate than to the same fractional change in T3 plasma concentration.