A manifold learning approach for gesture recognition from micro-Doppler radar measurements.

A recent paper (Mhaskar (2020)) introduces a straightforward and simple kernel based approximation for manifold learning that does not require the knowledge of anything about the manifold, except for its dimension. In this paper, we examine how the pointwise error in approximation using least squares optimization based on similarly localized kernels depends upon the data characteristics and deteriorates as one goes away from the training data. The theory is presented with an abstract localized kernel, which can utilize any prior knowledge about the data being located on an unknown sub-manifold of a known manifold. We demonstrate the performance of our approach using a publicly available micro-Doppler data set, and investigate the use of different preprocessing measures, kernels, and manifold dimensions. Specifically, it is shown that the localized kernel introduced in the above mentioned paper when used with PCA components leads to a near-competitive performance to deep neural networks, and offers significant improvements in training speed and memory requirements. To demonstrate the fact that our methods are agnostic to the domain knowledge, we examine the classification problem in a simple video data set.

Theory-Inspired Deep Network for Instantaneous-Frequency Extraction and Subsignals Recovery From Discrete Blind-Source Data.

In the mathematical and engineering literature on signal processing and time-series analysis, there are two opposite points of view concerning the extraction of time-varying frequencies (commonly called instantaneous frequencies, IFs). One is to consider the given signal as a composite signal consisting of a finite number of subsignals that are oscillating, and the goal is to decompose the signal into the sum of the (unknown) subsignals, followed by extracting the IF from each subsignal; the other is first to extract from the given signal, the IFs of the (unknown) subsignals, from which the subsignals that constitute the given signal are recovered. Let us call the first the ``signal decomposition approach'' and the second the ``signal resolution approach.'' For the ``signal decomposition approach,'' rigorous mathematical theories on function decomposition have been well developed in the mathematical literature, with the most relevant one, called ``atomic decomposition'' initiated by R. Coifman, with various extensions by others, notably by D. Donoho, with the goal of extracting the signal building blocks, but without concern of which building blocks constitute any of the subsignals, and consequently, the subsignals along with their IFs cannot be recovered. On the other hand, the most popular of the decomposition approach is the ``empirical mode decomposition (EMD),'' proposed by N. Huang et al., with many variations by others. In contrast to atomic decomposition, all variations of EMD are ad hoc algorithms, without any rigorous mathematical theory. Unfortunately, all existing versions of EMD fail to resolve the inverse problem on the recovery of the subsignals that constitute the given composite signal, and consequently, extracting the IFs is not satisfactory. For example, EMD fails to extract even two IFs that are not far apart from each other. In contrast to the signal decomposition approach, the ``signal resolution approach'' has a very long history dated back to the Prony method, introduced by G. de Prony in 1795, for solving the inverse problem of time-invariant linear systems. On the other hand, for nonstationary signals, the synchrosqueezed wavelet transform (SST), proposed by I. Daubechies over a decade ago, with various extensions and variations by others, was introduced to resolving the inverse problem, by first extracting the IFs from some reference frequency, followed by recovering the subsignals. Unfortunately, the SST approximate IFs could not be separated when the target IFs are close to one another at certain time instants, and even if they could be separated, the approximation is usually not sufficiently accurate. For these reasons, some signal components could not be recovered, and those that could be recovered are usually inexact. More recently, we introduced and developed a more direct method, called signal separation operation (SSO), published in 2016, to accurately compute the IFs and to accurately recover all signal components even if some of the target IFs are close to each other. The main contributions of this article are twofold. First, the SSO method is extended from uniformly sampled data to arbitrarily sampled data. This method is localized as illustrated by a number of numerical examples, including components with different subsignal arrival and departure times. It also yields a short-term prediction of the digital components along with their IFs. Second, we present a novel theory-inspired implementation of our method as a deep neural network (DNN). We have proved that a major advantage of DNN over shallow networks is that DNN can take advantage of any inherent compositional structure in the target function, while shallow networks are necessarily blind to such structure. Therefore, DNN can avoid the so-called curse of dimensionality using what we have called the blessing of compositionality. However, the compositional structure of the target function is not uniquely defined, and the constituent functions are typically not known so that the networks still need to be trained end-to-end. In contrast, the DNN introduced in this article implements a mathematical procedure so that no training is required at all, and the compositional structure is evident from the procedure. We will disclose the extension of the SSO method in Sections II and III and explain the construction of the deep network in Section IV.

A direct approach for function approximation on data defined manifolds.

In much of the literature on function approximation by deep networks, the function is assumed to be defined on some known domain, such as a cube or a sphere. In practice, the data might not be dense on these domains, and therefore, the approximation theory results are observed to be too conservative. In manifold learning, one assumes instead that the data is sampled from an unknown manifold; i.e., the manifold is defined by the data itself. Function approximation on this unknown manifold is then a two stage procedure: first, one approximates the Laplace-Beltrami operator (and its eigen-decomposition) on this manifold using a graph Laplacian, and next, approximates the target function using the eigen-functions. Alternatively, one estimates first some atlas on the manifold and then uses local approximation techniques based on the local coordinate charts. In this paper, we propose a more direct approach to function approximation on unknown, data defined manifolds without computing the eigen-decomposition of some operator or an atlas for the manifold, and without any kind of training in the classical sense. Our constructions are universal; i.e., do not require the knowledge of any prior on the target function other than continuity on the manifold. We estimate the degree of approximation. For smooth functions, the estimates do not suffer from the so-called saturation phenomenon. We demonstrate via a property called good propagation of errors how the results can be lifted for function approximation using deep networks where each channel evaluates a Gaussian network on a possibly unknown manifold.

Dimension independent bounds for general shallow networks.

This paper proves an abstract theorem addressing in a unified manner two important problems in function approximation: avoiding curse of dimensionality and estimating the degree of approximation for out-of-sample extension in manifold learning. We consider an abstract (shallow) network that includes, for example, neural networks, radial basis function networks, and kernels on data defined manifolds used for function approximation in various settings. A deep network is obtained by a composition of the shallow networks according to a directed acyclic graph, representing the architecture of the deep network. In this paper, we prove dimension independent bounds for approximation by shallow networks in the very general setting of what we have called G-networks on a compact metric measure space, where the notion of dimension is defined in terms of the cardinality of maximal distinguishable sets, generalizing the notion of dimension of a cube or a manifold. Our techniques give bounds that improve without saturation with the smoothness of the kernel involved in an integral representation of the target function. In the context of manifold learning, our bounds provide estimates on the degree of approximation for an out-of-sample extension of the target function to the ambient space. One consequence of our theorem is that without the requirement of robust parameter selection, deep networks using a non-smooth activation function such as the ReLU, do not provide any significant advantage over shallow networks in terms of the degree of approximation alone.

An analysis of training and generalization errors in shallow and deep networks.

This paper is motivated by an open problem around deep networks, namely, the apparent absence of over-fitting despite large over-parametrization which allows perfect fitting of the training data. In this paper, we analyze this phenomenon in the case of regression problems when each unit evaluates a periodic activation function. We argue that the minimal expected value of the square loss is inappropriate to measure the generalization error in approximation of compositional functions in order to take full advantage of the compositional structure. Instead, we measure the generalization error in the sense of maximum loss, and sometimes, as a pointwise error. We give estimates on exactly how many parameters ensure both zero training error as well as a good generalization error. We prove that a solution of a regularization problem is guaranteed to yield a good training error as well as a good generalization error and estimate how much error to expect at which test data.

Locally learning biomedical data using diffusion frames.

Diffusion geometry techniques are useful to classify patterns and visualize high-dimensional datasets. Building upon ideas from diffusion geometry, we outline our mathematical foundations for learning a function on high-dimension biomedical data in a local fashion from training data. Our approach is based on a localized summation kernel, and we verify the computational performance by means of exact approximation rates. After these theoretical results, we apply our scheme to learn early disease stages in standard and new biomedical datasets.

A generalized diffusion frame for parsimonious representation of functions on data defined manifolds.

One of the now standard techniques in semi-supervised learning is to think of a high dimensional data as a subset of a low dimensional manifold embedded in a high dimensional ambient space, and to use projections of the data on eigenspaces of a diffusion map. This paper is motivated by a recent work of Coifman and Maggioni on diffusion wavelets to accomplish such projections approximately using iterates of the heat kernel. In greater generality, we consider a quasi-metric measure space X (in place of the manifold), and a very general operator T defined on the class of integrable functions on X (in place of the diffusion map). We develop a representation of functions on X in terms of linear combinations of iterates of T. Our construction obviates the need to compute the eigenvalues and eigenfunctions of the operator. In addition, the local smoothness of a function f is characterized by the local norm behavior of the terms in our representation of f. This property is similar to that of the classical wavelet representations. Although the operator T utilizes the values of the target function on the entire space, this ability results in automatic "feature detection", leading to a parsimonious representation of the target function. In the case when X is a smooth compact manifold (without boundary), our theory allows T to be any operator that commutes with the heat operator, subject to certain conditions on its eigenvalues. In particular, T can be chosen to be the heat operator itself, or a Green's operator corresponding to a suitable pseudo-differential operator.

When is approximation by Gaussian networks necessarily a linear process?

Let s > or = 1 be an integer. A Gaussian network is a function on Rs of the form [Formula: see text]. The minimal separation among the centers, defined by (1/2) min(1 < or = j not = k < or = N) [Formula: see text], is an important characteristic of the network that determines the stability of interpolation by Gaussian networks, the degree of approximation by such networks, etc. Let (within this abstract only) the set of all Gaussian networks with minimal separation exceeding 1/m be denoted by Gm. We prove that for functions [Formula: see text] such that [Formula: see text], if the degree of L2(nonlinear) approximation of [Formula: see text] from Gm is [Formula: see text] then necessarily the degree of approximation of [Formula: see text] by (rectangular) partial sums of degree m2 of the Hermite expansion of [Formula: see text] is also [Formula: see text]. Moreover, Gaussian networks in Gm having fixed centers in a ball of radius [Formula: see text] and coefficients being linear functionals of [Formula: see text] can be constructed to yield the same degree of approximation. Similar results are proved for the Lp norms, 1 < or = p < or =[Formula: see text] but with the condition that the number of neurons N, should satisfy logN = [Formula: see text](m2).

Zonal function network frames on the sphere.

We introduce a class of zonal function network frames suitable for analyzing data collected at scattered sites on the surface of the unit sphere of a Euclidean space. Our frames consist of zonal function networks and are well localized. The frames belonging to higher and higher scale wavelet spaces have more and more vanishing polynomial moments. The main technique is applicable in the general setting of separable Hilbert spaces, in which context, we study the construction of new frames by perturbing an orthonormal basis.

Neural networks for functional approximation and system identification.

We construct generalized translation networks to approximate uniformly a class of nonlinear, continuous functionals defined on Lp ([-1, 1]s) for integer s > or = 1, 1 < or = p < infinity, or C ([-1, 1]s). We obtain lower bounds on the possible order of approximation for such functionals in terms of any approximation process depending continuously on a given number of parameters. Our networks almost achieve this order of approximation in terms of the number of parameters (neurons) involved in the network. The training is simple and noniterative; in particular, we avoid any optimization such as that involved in the usual backpropagation.