Fast and Robust Attribute Reduction Based on the Separability in Fuzzy Decision Systems.

Attribute reduction is one of the most important preprocessing steps in machine learning and data mining. As a key step of attribute reduction, attribute evaluation directly affects classification performance, search time, and stopping criterion. The existing evaluation functions are greatly dependent on the relationship between objects, which makes its computational time and space more costly. To solve this problem, we propose a novel separability-based evaluation function and reduction method by using the relationship between objects and decision categories directly. The degree of aggregation (DA) of intraclass objects and the degree of dispersion (DD) of between-class objects are first defined to measure the significance of an attribute subset. Then, the separability of attribute subsets is defined by DA and DD in fuzzy decision systems, and we design a sequentially forward selection based on the separability (SFSS) algorithm to select attributes. Furthermore, a postpruning strategy is introduced to prevent overfitting and determine a termination parameter. Finally, the SFSS algorithm is compared with some typical reduction algorithms using some public datasets from UCI and ELVIRA Biomedical repositories. The interpretability of SFSS is directly presented by the performance on MNIST handwritten digits. The experimental comparisons show that SFSS is fast and robust, which has higher classification accuracy and compression ratio, with extremely low computational time.

Data-Distribution-Aware Fuzzy Rough Set Model and its Application to Robust Classification.

Fuzzy rough sets (FRSs) are considered to be a powerful model for analyzing uncertainty in data. This model encapsulates two types of uncertainty: 1) fuzziness coming from the vagueness in human concept formation and 2) roughness rooted in the granulation coming with human cognition. The rough set theory has been widely applied to feature selection, attribute reduction, and classification. However, it is reported that the classical FRS model is sensitive to noisy information. To address this problem, several robust models have been developed in recent years. Nevertheless, these models do not consider a statistical distribution of data, which is an important type of uncertainty. Data distribution serves as crucial information for designing an optimal classification or regression model. Thus, we propose a data-distribution-aware FRS model that considers distribution information and incorporates it in computing lower and upper fuzzy approximations. The proposed model considers not only the similarity between samples, but also the probability density of classes. In order to demonstrate the effectiveness of the proposed model, we design a new sample evaluation index for prototype-based classification based on the model, and a prototype selection algorithm is developed using this index. Furthermore, a robust classification algorithm is constructed with prototype covering and nearest neighbor classification. Experimental results confirm the robustness and effectiveness of the proposed model.

Nesting one-against-one algorithm based on SVMs for pattern classification.

Support vector machines (SVMs), which were originally designed for binary classifications, are an excellent tool for machine learning. For the multiclass classifications, they are usually converted into binary ones before they can be used to classify the examples. In the one-against-one algorithm with SVMs, there exists an unclassifiable region where the data samples cannot be classified by its decision function. This paper extends the one-against-one algorithm to handle this problem. We also give the convergence and computational complexity analysis of the proposed method. Finally, one-against-one, fuzzy decision function (FDF), and decision-directed acyclic graph (DDAG) algorithms and our proposed method are compared using five University of California at Irvine (UCI) data sets. The results report that the proposed method can handle the unclassifiable region better than others.

Localized generalization error model and its application to architecture selection for radial basis function neural network.

The generalization error bounds found by current error models using the number of effective parameters of a classifier and the number of training samples are usually very loose. These bounds are intended for the entire input space. However, support vector machine (SVM), radial basis function neural network (RBFNN), and multilayer perceptron neural network (MLPNN) are local learning machines for solving problems and treat unseen samples near the training samples to be more important. In this paper, we propose a localized generalization error model which bounds from above the generalization error within a neighborhood of the training samples using stochastic sensitivity measure. It is then used to develop an architecture selection technique for a classifier with maximal coverage of unseen samples by specifying a generalization error threshold. Experiments using 17 University of California at Irvine (UCI) data sets show that, in comparison with cross validation (CV), sequential learning, and two other ad hoc methods, our technique consistently yields the best testing classification accuracy with fewer hidden neurons and less training time.

Weighted mahalanobis distance kernels for support vector machines.

The support vector machine (SVM) has been demonstrated to be a very effective classifier in many applications, but its performance is still limited as the data distribution information is underutilized in determining the decision hyperplane. Most of the existing kernels employed in nonlinear SVMs measure the similarity between a pair of pattern images based on the Euclidean inner product or the Euclidean distance of corresponding input patterns, which ignores data distribution tendency and makes the SVM essentially a "local" classifier. In this paper, we provide a step toward a paradigm of kernels by incorporating data specific knowledge into existing kernels. We first find the data structure for each class adaptively in the input space via agglomerative hierarchical clustering (AHC), and then construct the weighted Mahalanobis distance (WMD) kernels using the detected data distribution information. In WMD kernels, the similarity between two pattern images is determined not only by the Mahalanobis distance (MD) between their corresponding input patterns but also by the sizes of the clusters they reside in. Although WMD kernels are not guaranteed to be positive definite (pd) or conditionally positive definite (cpd), satisfactory classification results can still be achieved because regularizers in SVMs with WMD kernels are empirically positive in pseudo-Euclidean (pE) spaces. Experimental results on both synthetic and real-world data sets show the effectiveness of "plugging" data structure into existing kernels.

Structured one-class classification.

The one-class classification problem aims to distinguish a target class from outliers. The spherical one-class classifier (SOCC) solves this problem by finding a hypersphere with minimum volume that contains the target data while keeping outlier samples outside. SOCC achieves satisfactory performance only when the target samples have the same distribution tendency in all orientations. Therefore, the performance of the SOCC is limited in the way that many superfluous outliers might be mistakenly enclosed. The authors propose to exploit target data structures obtained via unsupervised methods such as agglomerative hierarchical clustering and use them in calculating a set of hyperellipsoidal separating boundaries. This method is named the structured one-class classifier (TOCC). The optimization problem in TOCC can be formulated as a series of second-order cone programming problems that can be solved with acceptable efficiency by primal-dual interior-point methods. The experimental results on artificially generated data sets and benchmark data sets demonstrate the advantages of TOCC.

Nonlinear Canonical Correlation Analysis of fMRI Signals Using HDR Models.

A nonlinear canonical correlation analysis (CCA) for detecting neural activation in fMRI data is proposed in this paper. We use the BOLD response based on the HDR models with various parameters as reference signals. Instead of characterizing the relationship between the paradigm and time series using the oversimplified linear model, we employ the kernel trick that maps the intensities of the voxels within a small cubic at each time point into a high-dimensional kernel space, where the linear combinations correspond to nonlinear ones in the original space. The experimental results show that the proposed nonlinear CCA can improve the detection performance of traditional linear CCA.

Support vector clustering for brain activation detection.

In this paper, we propose a new approach to detect activated time series in functional MRI using support vector clustering (SVC). We extract Fourier coefficients as the features of fMRI time series and cluster these features by SVC. In SVC, these features are mapped from their original feature space to a very high dimensional kernel space. By finding a compact sphere that encloses the mapped features in the kernel space, one achieves a set of cluster boundaries in the feature space. The SVC is an effective and robust fMRI activation detection method because of its advantages in (1) better discovery of real data structure since there is no cluster shape restriction, (2) high quality detection results without explicitly specifying the number of clusters, (3) the stronger robustness due to the mechanism in outlier elimination. Experimental results on simulated and real fMRI data demonstrate the effectiveness of SVC.

Handling interaction in fuzzy production rule reasoning.

When fuzzy production rules are used to approximate reasoning, interaction exists among rules that have the same consequent. Due to this interaction, the weighted average model frequently used in approximate reasoning does not work well in many real-world problems. In order to model and handle this interaction, this paper proposes to use a nonadditive nonnegative set function to replace the weights assigned to rules having the same consequent, and to draw the reasoning conclusion based on an integral with respect to the nonadditive nonnegative set function, rather than on the weighted average model. Handling interaction in fuzzy production rule reasoning in this way can lead to a good understanding of the rules base and an improvement of reasoning accuracy. This paper also investigates how to determine from data the nonadditive set function that cannot be specified by a domain expert.

Refinement of generated fuzzy production rules by using a fuzzy neural network.

Fuzzy production rules (FPRs) have been used for years to capture and represent fuzzy, vague, imprecise and uncertain domain knowledge in many fuzzy systems. There have been a lot of researches on how to generate or obtain FPRs. There exist two methods to obtain FPRs. One is by painstakingly, repeatedly and time-consuming interviewing domain experts to extract the domain knowledge. The other is by using some machine learning techniques to generate and extract FPRs from some training samples. These extracted rules, however, are found to be nonoptimal and sometimes redundant. Furthermore, these generated rules suffer from the problem of low accuracy of classifying or recognizing unseen examples. The reasons for having these problems are 1) the FPRs generated are not powerful enough to represent the domain knowledge, 2) the techniques used to generate FPRs are pre-matured, ad-hoc or may not be suitable for the problem, and 3) further refinement of the extracted rules has not been done. In this paper we look into the solutions of the above problems by 1) enhancing the representation power of FPRs by including local and global weights, 2) developing a fuzzy neural network (FNN) with enhanced learning algorithm, and 3) using this FNN to refine the local and global weights of FPRs. By experimenting our method with some existing benchmark examples, the proposed method is found to have high accuracy in classifying unseen samples without increasing the number of the FPRs extracted and the time required to consult with domain experts is greatly reduced.