Clustering-based algorithms for single-hidden-layer sigmoid perceptron.

Gradient-descent type supervised learning is the most commonly used algorithm for design of the standard sigmoid perceptron (SP). However, it is computationally expensive (slow) and has the local-minima problem. Moody and Darken (1989) proposed an input-clustering based hierarchical algorithm for fast learning in networks of locally tuned neurons in the context of radial basis function networks. We propose and analyze input clustering (IC) and input-output clustering (IOC)-based algorithms for fast learning in networks of globally tuned neurons in the context of the SP. It is shown that "localizing'' the input layer weights of the SP by the IC and the IOC minimizes an upper bound to the SP output error. The proposed algorithms could possibly be used also to initialize the SP weights for the conventional gradient-descent learning. Simulation results offer that the SPs designed by the IC and the IOC yield comparable performance in comparison with its radial basis function network counterparts.

Analysis of input-output clustering for determining centers of RBFN.

The key point in design of radial basis function networks is to specify the number and the locations of the centers. Several heuristic hybrid learning methods, which apply a clustering algorithm for locating the centers and subsequently a linear leastsquares method for the linear weights, have been previously suggested. These hybrid methods can be put into two groups, which will be called as input clustering (IC) and input-output clustering (IOC), depending on whether the output vector is also involved in the clustering process. The idea of concatenating the output vector to the input vector in the clustering process has independently been proposed by several papers in the literature although none of them presented a theoretical analysis on such procedures, but rather demonstrated their effectiveness in several applications. The main contribution of this paper is to present an approach for investigating the relationship between clustering process on input-output training samples and the mean squared output error in the context of a radial basis function netowork (RBFN). We may summarize our investigations in that matter as follows: 1) A weighted mean squared input-output quantization error, which is to be minimized by IOC, yields an upper bound to the mean squared output error. 2) This upper bound and consequently the output error can be made arbitrarily small (zero in the limit case) by decreasing the quantization error which can be accomplished through increasing the number of hidden units.