Hybrid Gromov-Wasserstein Embedding for Capsule Learning.

Capsule networks (CapsNets) aim to parse images into a hierarchy of objects, parts, and their relationships using a two-step process involving part-whole transformation and hierarchical component routing. However, this hierarchical relationship modeling is computationally expensive, which has limited the wider use of CapsNet despite its potential advantages. The current state of CapsNet models primarily focuses on comparing their performance with capsule baselines, falling short of achieving the same level of proficiency as deep convolutional neural network (CNN) variants in intricate tasks. To address this limitation, we present an efficient approach for learning capsules that surpasses canonical baseline models and even demonstrates superior performance compared with high-performing convolution models. Our contribution can be outlined in two aspects: first, we introduce a group of subcapsules onto which an input vector is projected. Subsequently, we present the hybrid Gromov-Wasserstein (HGW) framework, which initially quantifies the dissimilarity between the input and the components modeled by the subcapsules, followed by determining their alignment degree through optimal transport (OT). This innovative mechanism capitalizes on new insights into defining alignment between the input and subcapsules, based on the similarity of their respective component distributions. This approach enhances CapsNets' capacity to learn from intricate, high-dimensional data while retaining their interpretability and hierarchical structure. Our proposed model offers two distinct advantages: 1) its lightweight nature facilitates the application of capsules to more intricate vision tasks, including object detection; and 2) it outperforms baseline approaches in these demanding tasks. Our empirical findings illustrate that HGW capsules (HGWCapsules) exhibit enhanced robustness against affine transformations, scale effectively to larger datasets, and surpass CNN and CapsNet models across various vision tasks.

GLGAN-VC: A Guided Loss-Based Generative Adversarial Network for Many-to-Many Voice Conversion.

Many-to-many voice conversion (VC) is a technique aimed at mapping speech features between multiple speakers during training and transferring the vocal characteristics of one source speaker to another target speaker, all while maintaining the content of the source speech unchanged. Existing research highlights a notable gap between the original and generated speech samples in terms of naturalness within many-to-many VC. Therefore, there is substantial room for improvement in achieving more natural-sounding speech samples for both parallel and nonparallel VC scenarios. In this study, we introduce a generative adversarial network (GAN) system with a guided loss (GLGAN-VC) designed to enhance many-to-many VC by focusing on architectural improvements and the integration of alternative loss functions. Our approach includes a pair-wise downsampling and upsampling (PDU) generator network for effective speech feature mapping (FM) in multidomain VC. In addition, we incorporate an FM loss to preserve content information and a residual connection (RC)-based discriminator network to improve learning. A guided loss (GL) function is introduced to efficiently capture differences in latent feature representations between source and target speakers, and an enhanced reconstruction loss is proposed for better contextual information preservation. We evaluate our model on various datasets, including VCC 2016, VCC 2018, VCC 2020, and an emotional speech dataset (ESD). Our results, based on both subjective and objective evaluation metrics, demonstrate that our model outperforms state-of-the-art (SOTA) many-to-many GAN-based VC models in terms of speech quality and speaker similarity in the generated speech samples.

Robust Principal Component Analysis: A Median of Means Approach.

Principal component analysis (PCA) is a fundamental tool for data visualization, denoising, and dimensionality reduction. It is widely popular in statistics, machine learning, computer vision, and related fields. However, PCA is well-known to fall prey to outliers and often fails to detect the true underlying low-dimensional structure within the dataset. Following the Median of Means (MoM) philosophy, recent supervised learning methods have shown great success in dealing with outlying observations without much compromise to their large sample theoretical properties. This article proposes a PCA procedure based on the MoM principle. Called the MoMPCA, the proposed method is not only computationally appealing but also achieves optimal convergence rates under minimal assumptions. In particular, we explore the nonasymptotic error bounds of the obtained solution via the aid of the Rademacher complexities while granting absolutely no assumption on the outlying observations. The derived concentration results are not dependent on the dimension because the analysis is conducted in a separable Hilbert space, and the results only depend on the fourth moment of the underlying distribution in the corresponding norm. The proposal's efficacy is also thoroughly showcased through simulations and real data applications.

An exhaustive review of the metaheuristic algorithms for search and optimization: taxonomy, applications, and open challenges.

As the world moves towards industrialization, optimization problems become more challenging to solve in a reasonable time. More than 500 new metaheuristic algorithms (MAs) have been developed to date, with over 350 of them appearing in the last decade. The literature has grown significantly in recent years and should be thoroughly reviewed. In this study, approximately 540 MAs are tracked, and statistical information is also provided. Due to the proliferation of MAs in recent years, the issue of substantial similarities between algorithms with different names has become widespread. This raises an essential question: can an optimization technique be called 'novel' if its search properties are modified or almost equal to existing methods? Many recent MAs are said to be based on 'novel ideas', so they are discussed. Furthermore, this study categorizes MAs based on the number of control parameters, which is a new taxonomy in the field. MAs have been extensively employed in various fields as powerful optimization tools, and some of their real-world applications are demonstrated. A few limitations and open challenges have been identified, which may lead to a new direction for MAs in the future. Although researchers have reported many excellent results in several research papers, review articles, and monographs during the last decade, many unexplored places are still waiting to be discovered. This study will assist newcomers in understanding some of the major domains of metaheuristics and their real-world applications. We anticipate this resource will also be useful to our research community.

Self-Adaptive Spherical Search With a Low-Precision Projection Matrix for Real-World Optimization.

Since the last three decades, numerous search strategies have been introduced within the framework of different evolutionary algorithms (EAs). Most of the popular search strategies operate on the hypercube (HC) search model, and search models based on other hypershapes, such as hyper-spherical (HS), are not investigated well yet. The recently developed spherical search (SS) algorithm utilizing the HS search model has been shown to perform very well for the bound-constrained and constrained optimization problems compared to several state-of-the-art algorithms. Nevertheless, the computational burdens for generating an HS locus are higher than that for an HC locus. We propose an efficient technique to construct an HS locus by approximating the orthogonal projection matrix to resolve this issue. As per our empirical experiments, this technique significantly improves the performance of the original SS with less computational effort. Moreover, to enhance SS's search capability, we put forth a self-adaptation technique for choosing the effective values of the control parameters dynamically during the optimization process. We validate the proposed algorithm's performance on a plethora of real-world and benchmark optimization problems with and without constraints. Experimental results suggest that the proposed algorithm remains better than or at least comparable to the best-known state-of-the-art algorithms on a wide spectrum of problems.

GEN: Generative Equivariant Networks for Diverse Image-to-Image Translation.

Image-to-image (I2I) translation has become a key asset for generative adversarial networks. Convolutional neural networks (CNNs), despite having a significant performance, are not able to capture the spatial relationships among different parts of an object and, thus, do not qualify as the ideal representative model for image translation tasks. As a remedy to this problem, capsule networks have been proposed to represent patterns for a visual object in such a way that preserves hierarchical spatial relationships. The training of capsules is constrained by learning all pairwise relationships between capsules of consecutive layers. This design would be prohibitively expensive both in time and memory. In this article, we present a new framework for capsule networks to provide a full description of the input components at various levels of semantics, which can successfully be applied to the generator-discriminator architectures without incurring computational overhead compared to the CNNs. To successfully apply the proposed capsules in the generative adversarial network, we put forth a novel Gromov-Wasserstein (GW) distance as a differentiable loss function that compares the dissimilarity between two distributions and then guides the learned distribution toward target properties, using optimal transport (OT) discrepancy. The proposed method-which is called generative equivariant network (GEN)-is an alternative architecture for GANs with equivariance capsule layers. The proposed model is evaluated through a comprehensive set of experiments on I2I translation and image generation tasks and compared with several state-of-the-art models. Results indicate that there is a principled connection between generative and capsule models that allows extracting discriminant and invariant information from image data.

Implicit Annealing in Kernel Spaces: A Strongly Consistent Clustering Approach.

Kernel k-means clustering is a powerful tool for unsupervised learning of non-linearly separable data. Its merits are thoroughly validated on a suite of simulated datasets and real data benchmarks that feature nonlinear and multi-view separation. Since the earliest attempts, researchers have noted that such algorithms often become trapped by local minima arising from the non-convexity of the underlying objective function. In this paper, we generalize recent results leveraging a general family of means to combat sub-optimal local solutions to the kernel and multi-kernel settings. Called Kernel Power k-Means, our algorithm uses majorization-minimization (MM) to better solve this non-convex problem. We show that the method implicitly performs annealing in kernel feature space while retaining efficient, closed-form updates. We rigorously characterize its convergence properties both from computational and statistical points of view. In particular, we characterize the large sample behavior of the proposed method by establishing strong consistency guarantees as well as finite-sample bounds on the excess risk of the estimates through modern tools in learning theory. The proposal's efficacy is demonstrated through an array of simulated and real data experiments.

On Consistent Entropy-Regularized k-Means Clustering With Feature Weight Learning: Algorithm and Statistical Analyses.

Clusters in real data are often restricted to low-dimensional subspaces rather than the entire feature space. Recent approaches to circumvent this difficulty are often computationally inefficient and lack theoretical justification in terms of their large-sample behavior. This article deals with the problem by introducing an entropy incentive term to efficiently learn the feature importance within the framework of center-based clustering. A scalable block-coordinate descent algorithm, with closed-form updates, is incorporated to minimize the proposed objective function. We establish theoretical guarantees on our method by Vapnik-Chervonenkis (VC) theory to establish strong consistency along with uniform concentration bounds. The merits of our method are showcased through detailed experimental analysis on toy examples as well as real data clustering benchmarks.

A υ-Constrained Matrix Adaptation Evolution Strategy With Broyden-Based Mutation for Constrained Optimization.

To solve the nonconvex constrained optimization problems (COPs) over continuous search spaces by using a population-based optimization algorithm, balancing between the feasible and infeasible solutions in the population plays an important role over different stages of the optimization process. To keep this balance, we propose a constraint handling technique, called the υ -level penalty function, which works by transforming a COP into an unconstrained one. Also, to improve the ability of the algorithm in handling several complex constraints, especially nonlinear inequality and equality constraints, we suggest a Broyden-based mutation that finds a feasible solution to replace an infeasible solution. By incorporating these techniques with the matrix adaptation evolution strategy (MA-ES), we develop a new constrained optimization algorithm. An extensive comparative analysis undertaken using a broad range of benchmark problems indicates that the proposed algorithm can outperform several state-of-the-art constrained evolutionary optimizers.

A Reference Vector-Based Simplified Covariance Matrix Adaptation Evolution Strategy for Constrained Global Optimization.

During the last two decades, the notion of multiobjective optimization (MOO) has been successfully adopted to solve the nonconvex constrained optimization problems (COPs) in their most general forms. However, such works mainly utilized the Pareto dominance-based MOO framework while the other successful MOO frameworks, such as the reference vector (RV) and the decomposition-based ones, have not drawn sufficient attention from the COP researchers. In this article, we utilize the concepts of the RV-based MOO to design a ranking strategy for the solutions of a COP. We first transform the COP into a biobjective optimization problem (BOP) and then solve it by using the covariance matrix adaptation evolution strategy (CMA-ES), which is arguably one of the most competitive evolutionary algorithms of current interest. We propose an RV-based ranking strategy to calculate the mean and update the covariance matrix in CMA-ES. Besides, the RV is explicitly tuned during the optimization process based on the characteristics of COPs in a RV-based MOO framework. We also propose a repair mechanism for the infeasible solutions and a restart strategy to facilitate the population to escape from the infeasible region. We test the proposal extensively on two well-known benchmark suites comprised of 36 and 112 test problems (at different scales) from the IEEE CEC (Congress on Evolutionary Computation) 2010 and 2017 competitions along with a real-world problem related to power flow. Our experimental results suggest that the proposed algorithm can meet or beat several other state-of-the-art constrained optimizers in terms of the performance on a wide variety of problems.

Detecting Meaningful Clusters From High-Dimensional Data: A Strongly Consistent Sparse Center-Based Clustering Approach.

In context to high-dimensional clustering, the concept of feature weighting has gained considerable importance over the years to capture the relative degrees of importance of different features in revealing the cluster structure of the dataset. However, the popular techniques in this area either fail to perform feature selection or do not preserve the simplicity of Lloyd's heuristic to solve the k-means problem and the like. In this paper, we propose a Lasso Weighted k-means ( LW- k-means) algorithm, as a simple yet efficient sparse clustering procedure for high-dimensional data where the number of features ( p) can be much higher than the number of observations ( n). The LW- k-means method imposes an l1 regularization term involving the feature weights directly to induce feature selection in a sparse clustering framework. We develop a simple block-coordinate descent type algorithm with time-complexity resembling that of Lloyd's method, to optimize the proposed objective. In addition, we establish the strong consistency of the LW- k-means procedure. Such an analysis of the large sample properties is not available for the conventional sparse k-means algorithms, in general. LW- k-means is tested on a number of synthetic and real-life datasets and through a detailed experimental analysis, we find that the performance of the method is highly competitive against the baselines as well as the state-of-the-art procedures for center-based high-dimensional clustering, not only in terms of clustering accuracy but also with respect to computational time.

A Deep Learning Framework Integrating the Spectral and Spatial Features for Image-Assisted Medical Diagnostics.

The development of a computer-aided disease detection system to ease the long and arduous manual diagnostic process is an emerging research interest. Living through the recent outbreak of the COVID-19 virus, we propose a machine learning and computer vision algorithms-based automatic diagnostic solution for detecting the COVID-19 infection. Our proposed method applies to chest radiograph that uses readily available infrastructure. No studies in this direction have considered the spatial aspect of the medical images. This motivates us to investigate the role of spectral-domain information of medical images along with the spatial content towards improved disease detection ability. Successful integration of spatial and spectral features is demonstrated on the COVID-19 infection detection task. Our proposed method comprises three stages - Feature extraction, Dimensionality reduction via projection, and prediction. At first, images are transformed into spectral and spatio-spectral domains by using Discrete cosine transform (DCT) and Discrete Wavelet transform (DWT), two powerful image processing algorithms. Next, features from spatial, spectral, and spatio-spectral domains are projected into a lower dimension through the Convolutional Neural Network (CNN), and those three types of projected features are then fed to Multilayer Perceptron (MLP) for final prediction. The combination of the three types of features yielded superior performance than any of the features when used individually. This indicates the presence of complementary information in the spectral domain of the chest radiograph to characterize the considered medical condition. Moreover, saliency maps corresponding to classes representing different medical conditions demonstrate the reliability of the proposed method. The study is further extended to identify different medical conditions using diverse medical image datasets and shows the efficiency of leveraging the combined features. Altogether, the proposed method exhibits potential as a generalized and robust medical image-assisted diagnostic solution.

Fuzzy Clustering to Identify Clusters at Different Levels of Fuzziness: An Evolutionary Multiobjective Optimization Approach.

Fuzzy clustering methods identify naturally occurring clusters in a dataset, where the extent to which different clusters are overlapped can differ. Most methods have a parameter to fix the level of fuzziness. However, the appropriate level of fuzziness depends on the application at hand. This paper presents an entropy c -means (ECM), a method of fuzzy clustering that simultaneously optimizes two contradictory objective functions, resulting in the creation of fuzzy clusters with different levels of fuzziness. This allows ECM to identify clusters with different degrees of overlap. ECM optimizes the two objective functions using two multiobjective optimization methods, nondominated sorting genetic algorithm II (NSGA-II) and multiobjective evolutionary algorithm based on decomposition (MOEA/D). We also propose a method to select a suitable tradeoff clustering from the Pareto front. Experiments on challenging synthetic datasets as well as real-world datasets show that ECM leads to better cluster detection compared to the conventional fuzzy clustering methods as well as previously used multiobjective methods for fuzzy clustering.

Reusing the Past Difference Vectors in Differential Evolution-A Simple But Significant Improvement.

Differential evolution (DE) has established itself as a simple but efficient population-based, nonconvex optimization algorithm for continuous search spaces. Unlike the conventional real-coded genetic algorithms (GAs) and evolution strategies (ESs), DE uses a mandatory self-referential mutation for its population members, each of which are perturbed with the scaled difference(s) of the individuals from the current generation (iteration). These difference vectors determine the direction of the search moves for the individuals. However, unlike the better individuals, they are not retained in the elitist evolution cycle of DE. In this paper, we show that by archiving the most promising difference vectors from past generations and then by reusing them for generating offspring in the subsequent generations, we can strikingly improve the performance of DE. This strategy can be integrated with any classical or advanced DE variant with no serious overhead in time or space complexity. We demonstrate that when combined with the DE-based winners of the IEEE Congress on Evolutionary Computation (CEC) 2013, 2014, and 2017 competitions on real parameter optimization, the simple reuse strategy leads to a statistically significant performance improvement in the majority of test cases. We further showcase the efficacy of our proposal on a practical optimization problem concerning the design of circular antenna arrays with a prespecified radiation pattern.

Multiobjective Support Vector Machines: Handling Class Imbalance With Pareto Optimality.

Support vector machines (SVMs) seek to optimize three distinct objectives: maximization of margin, minimization of regularization from the positive class, and minimization of regularization from the negative class. The right choice of weightage for each of these objectives is critical to the quality of the classifier learned, especially in case of the class imbalanced data sets. Therefore, costly parameter tuning has to be undertaken to find a set of suitable relative weights. In this brief, we propose to train SVMs, on two-class as well as multiclass data sets, in a multiobjective optimization framework called radial boundary intersection to overcome this shortcoming. The experimental results suggest that the radial boundary intersection-based scheme is indeed effective in finding the best tradeoff among the objectives compared with parameter-tuning schemes.

Stronger Convergence Results for the Center-Based Fuzzy Clustering With Convex Divergence Measure.

We present a novel alternative convergence theory of the fuzzy C -means (FCM) clustering algorithm with a super-class of the so-called "distance like functions" which emerged from the earlier attempts of unifying the theories of center-based clustering methods. This super-class does not assume the existence of double derivative of the distance measure with respect to the coordinate of the cluster representative (first coordinate in this formulation). The convergence result does not require the separability of the distance measures. Moreover, it provides us with a stronger convergence property comparable (same to be precise, but in terms of the generalized distance measure) to that of the classical FCM with squared Euclidean distance. The crux of the convergence analysis lies in the development of a fundamentally novel mathematical proof of the continuity of the clustering operator even in absence of the closed form upgrading rule, without necessitating the separability and double differentiability of the distance function and still providing us with a convergence result comparable to that of the classical FCM. The implication of our novel proof technique goes way beyond the realm of FCM and provides a general setup for convergence analysis of the similar iterative algorithms.

Adaptive Learning-Based -Nearest Neighbor Classifiers With Resilience to Class Imbalance.

The classification accuracy of a -nearest neighbor ( NN) classifier is largely dependent on the choice of the number of nearest neighbors denoted by . However, given a data set, it is a tedious task to optimize the performance of NN by tuning . Moreover, the performance of NN degrades in the presence of class imbalance, a situation characterized by disparate representation from different classes. We aim to address both the issues in this paper and propose a variant of NN called the Adaptive NN (Ada- NN). The Ada- NN classifier uses the density and distribution of the neighborhood of a test point and learns a suitable point-specific for it with the help of artificial neural networks. We further improve our proposal by replacing the neural network with a heuristic learning method guided by an indicator of the local density of a test point and using information about its neighboring training points. The proposed heuristic learning algorithm preserves the simplicity of NN without incurring serious computational burden. We call this method Ada- NN2. Ada- NN and Ada- NN2 perform very competitive when compared with NN, five of NN's state-of-the-art variants, and other popular classifiers. Furthermore, we propose a class-based global weighting scheme (Global Imbalance Handling Scheme or GIHS) to compensate for the effect of class imbalance. We perform extensive experiments on a wide variety of data sets to establish the improvement shown by Ada- NN and Ada- NN2 using the proposed GIHS, when compared with NN, and its 12 variants specifically tailored for imbalanced classification.

A Fuzzy Rule-Based Penalty Function Approach for Constrained Evolutionary Optimization.

This paper proposes a novel fuzzy rule-based penalty function approach for solving single-objective nonlinearly constrained optimization problems. Of all the existing state-of-the-art constraint handling techniques, the conventional method of penalty can be easily implemented because of its simplicity but suffers from the lack of robustness. To mitigate the problem of parameter dependency, several forms of adaptive penalties have been suggested in literature. Instead of identifying a complex mathematical function to compute the penalty for constraint violation, we propose a Mamdani type IF-THEN rule-based fuzzy inference system that incorporates all the required criteria of self-adaptive penalty without formulating an explicit mapping. Effectiveness of the proposed constrained optimization algorithm has been empirically validated on the basis of the standard optimality theorems from the literature on mathematical programming. Simulation results show that fuzzy penalty not only surpasses its existing counterpart i.e., self adaptive penalty, but also remain competitive against several other standard as well as currently developed complex constraint handling strategies.

Near-Bayesian Support Vector Machines for imbalanced data classification with equal or unequal misclassification costs.

Support Vector Machines (SVMs) form a family of popular classifier algorithms originally developed to solve two-class classification problems. However, SVMs are likely to perform poorly in situations with data imbalance between the classes, particularly when the target class is under-represented. This paper proposes a Near-Bayesian Support Vector Machine (NBSVM) for such imbalanced classification problems, by combining the philosophies of decision boundary shift and unequal regularization costs. Based on certain assumptions which hold true for most real-world datasets, we use the fractions of representation from each of the classes, to achieve the boundary shift as well as the asymmetric regularization costs. The proposed approach is extended to the multi-class scenario and also adapted for cases with unequal misclassification costs for the different classes. Extensive comparison with standard SVM and some state-of-the-art methods is furnished as a proof of the ability of the proposed approach to perform competitively on imbalanced datasets. A modified Sequential Minimal Optimization (SMO) algorithm is also presented to solve the NBSVM optimization problem in a computationally efficient manner.

An improved parent-centric mutation with normalized neighborhoods for inducing niching behavior in differential evolution.

In real life, we often need to find multiple optimally sustainable solutions of an optimization problem. Evolutionary multimodal optimization algorithms can be very helpful in such cases. They detect and maintain multiple optimal solutions during the run by incorporating specialized niching operations in their actual framework. Differential evolution (DE) is a powerful evolutionary algorithm (EA) well-known for its ability and efficiency as a single peak global optimizer for continuous spaces. This article suggests a niching scheme integrated with DE for achieving a stable and efficient niching behavior by combining the newly proposed parent-centric mutation operator with synchronous crowding replacement rule. The proposed approach is designed by considering the difficulties associated with the problem dependent niching parameters (like niche radius) and does not make use of such control parameter. The mutation operator helps to maintain the population diversity at an optimum level by using well-defined local neighborhoods. Based on a comparative study involving 13 well-known state-of-the-art niching EAs tested on an extensive collection of benchmarks, we observe a consistent statistical superiority enjoyed by our proposed niching algorithm.