Learning transfer operators by kernel density estimation.

Inference of transfer operators from data is often formulated as a classical problem that hinges on the Ulam method. The conventional description, known as the Ulam-Galerkin method, involves projecting onto basis functions represented as characteristic functions supported over a fine grid of rectangles. From this perspective, the Ulam-Galerkin approach can be interpreted as density estimation using the histogram method. In this study, we recast the problem within the framework of statistical density estimation. This alternative perspective allows for an explicit and rigorous analysis of bias and variance, thereby facilitating a discussion on the mean square error. Through comprehensive examples utilizing the logistic map and a Markov map, we demonstrate the validity and effectiveness of this approach in estimating the eigenvectors of the Frobenius-Perron operator. We compare the performance of histogram density estimation (HDE) and kernel density estimation (KDE) methods and find that KDE generally outperforms HDE in terms of accuracy. However, it is important to note that KDE exhibits limitations around boundary points and jumps. Based on our research findings, we suggest the possibility of incorporating other density estimation methods into this field and propose future investigations into the application of KDE-based estimation for high-dimensional maps. These findings provide valuable insights for researchers and practitioners working on estimating the Frobenius-Perron operator and highlight the potential of density estimation techniques in this area of study.

Machine learning enhanced Hankel dynamic-mode decomposition.

While the acquisition of time series has become more straightforward, developing dynamical models from time series is still a challenging and evolving problem domain. Within the last several years, to address this problem, there has been a merging of machine learning tools with what is called the dynamic-mode decomposition (DMD). This general approach has been shown to be an especially promising avenue for accurate model development. Building on this prior body of work, we develop a deep learning DMD based method, which makes use of the fundamental insight of Takens' embedding theorem to build an adaptive learning scheme that better approximates higher dimensional and chaotic dynamics. We call this method the Deep Learning Hankel DMD. We likewise explore how our method learns mappings, which tend, after successful training, to significantly change the mutual information between dimensions in the dynamics. This appears to be a key feature in enhancing DMD overall, and it should help provide further insight into developing other deep learning methods for time series analysis and model generation.

Fractal basins as a mechanism for the nimble brain.

An interesting feature of the brain is its ability to respond to disparate sensory signals from the environment in unique ways depending on the environmental context or current brain state. In dynamical systems, this is an example of multi-stability, the ability to switch between multiple stable states corresponding to specific patterns of brain activity/connectivity. In this article, we describe chimera states, which are patterns consisting of mixed synchrony and incoherence, in a brain-inspired dynamical systems model composed of a network with weak individual interactions and chaotic/periodic local dynamics. We illustrate the mechanism using synthetic time series interacting on a realistic anatomical brain network derived from human diffusion tensor imaging. We introduce the so-called vector pattern state (VPS) as an efficient way of identifying chimera states and mapping basin structures. Clustering similar VPSs for different initial conditions, we show that coexisting attractors of such states reveal intricately "mingled" fractal basin boundaries that are immediately reachable. This could explain the nimble brain's ability to rapidly switch patterns between coexisting attractors.

Data-driven learning of Boolean networks and functions by optimal causation entropy principle.

Boolean functions, and networks thereof, are useful for analysis of complex data systems, including from biological systems, bioinformatics, decision making, medical fields, and finance. However, automated learning of a Boolean networked function, from data, is a challenging task due in part to the large number of unknown structures of the network and the underlying functions. In this paper, we develop a new information theoretic methodology, called Boolean optimal causation entropy, that we show is significantly more efficient than previous approaches. Our method is computationally efficient and also resilient to noise. Furthermore, it allows for selection of features that best explains the process, described as a networked Boolean function reduced-order model. We highlight our method to the feature selection in several real-world examples: (1) diagnosis of urinary diseases, (2) cardiac single proton emission computed tomography diagnosis, (3) informative positions in the game Tic-Tac-Toe, and (4) risk causality analysis of loans in default status.

Stability analysis of intralayer synchronization in time-varying multilayer networks with generic coupling functions.

The stability analysis of synchronization patterns on generalized network structures is of immense importance nowadays. In this article, we scrutinize the stability of intralayer synchronous state in temporal multilayer hypernetworks, where each dynamic units in a layer communicate with others through various independent time-varying connection mechanisms. Here, dynamical units within and between layers may be interconnected through arbitrary generic coupling functions. We show that intralayer synchronous state exists as an invariant solution. Using fast-switching stability criteria, we derive the condition for stable coherent state in terms of associated time-averaged network structure, and in some instances we are able to separate the transverse subspace optimally. Using simultaneous block diagonalization of coupling matrices, we derive the synchronization stability condition without considering time-averaged network structure. Finally, we verify our analytically derived results through a series of numerical simulations on synthetic and real-world neuronal networked systems.

Spanning trees of recursive scale-free graphs.

We present a link-by-link rule-based method for constructing all members of the ensemble of spanning trees for any recursively generated, finitely articulated graph, such as the Dorogovtsev-Goltsev-Mendes (DGM) net. The recursions allow for many large-scale properties of the ensemble of spanning trees to be analytically solved exactly. We show how a judicious application of the prescribed growth rules selects for certain subsets of the spanning trees with particular desired properties (small world, extended diameter, degree distribution, etc.), and thus approximates and/or provides solutions to several optimization problems on undirected and unweighted networks. The analysis of spanning trees enhances the usefulness of recursive graphs as sophisticated models for everyday life complex networks.

Beach-level 24-hour forecasts of Florida red tide-induced respiratory irritation.

An accurate forecast of the red tide respiratory irritation level would improve the lives of many people living in areas affected by algal blooms. Using a decades-long database of daily beach conditions, two conceptually different models to forecast the respiratory irritation risk level one day ahead of time are trained. One model is wind-based, using the current days' respiratory level and the predicted wind direction of the following day. The other model is a probabilistic self-exciting Hawkes process model. Both models are trained on beaches in Florida during 2011--2017 and applied to the red tide bloom during 2018-2019. For beaches where there is enough historical data to develop a model, the model which performs best depends on the beach. The wind-based model is the most accurate at half the beaches, correctly predicting the respiratory risk level on average about 84% of the time. The Hawkes model is the most accurate (81% accuracy) at nearly all of the remaining beaches.

Entropic regression with neurologically motivated applications.

The ultimate goal of cognitive neuroscience is to understand the mechanistic neural processes underlying the functional organization of the brain. The key to this study is understanding the structure of both the structural and functional connectivity between anatomical regions. In this paper, we use an information theoretic approach, which defines direct information flow in terms of causation entropy, to improve upon the accuracy of the recovery of the true network structure over popularly used methods for this task such as correlation and least absolute shrinkage and selection operator regression. The method outlined above is tested on synthetic data, which is produced by following previous work in which a simple dynamical model of the brain is used, simulated on top of a real network of anatomical brain regions reconstructed from diffusion tensor imaging. We demonstrate the effectiveness of the method of AlMomani et al. [Chaos 30, 013107 (2020)] when applied to data simulated on the realistic diffusion tensor imaging network, as well as on randomly generated small-world and Erdös-Rényi networks.

The essential synchronization backbone problem.

Network optimization strategies for the process of synchronization have generally focused on the re-wiring or re-weighting of links in order to (1) expand the range of coupling strengths that achieve synchronization, (2) expand the basin of attraction for the synchronization manifold, or (3) lower the average time to synchronization. A new optimization goal is proposed in seeking the minimum subset of the edge set of the original network that enables the same essential ability to synchronize in that the synchronization manifolds have conjugate stability. We call this type of minimal spanning subgraph an essential synchronization backbone of the original system, and we present two algorithms: one is a strategy for an exhaustive search for a true solution, while the other is a method of approximation for this combinatorial problem. The solution spaces that result from different choices of dynamical systems and coupling schemes vary with the level of a hierarchical structure present and also the number of interwoven central cycles. Applications can include the important problem in civil engineering of power grid hardening, where new link creation may be costly, and the defense of certain key links to the functional process may be prioritized.

Next generation reservoir computing.

Reservoir computing is a best-in-class machine learning algorithm for processing information generated by dynamical systems using observed time-series data. Importantly, it requires very small training data sets, uses linear optimization, and thus requires minimal computing resources. However, the algorithm uses randomly sampled matrices to define the underlying recurrent neural network and has a multitude of metaparameters that must be optimized. Recent results demonstrate the equivalence of reservoir computing to nonlinear vector autoregression, which requires no random matrices, fewer metaparameters, and provides interpretable results. Here, we demonstrate that nonlinear vector autoregression excels at reservoir computing benchmark tasks and requires even shorter training data sets and training time, heralding the next generation of reservoir computing.

Spontaneous emergence of leadership patterns drives synchronization in complex human networks.

Synchronization of human networks is fundamental in many aspects of human endeavour. Recently, much research effort has been spent on analyzing how motor coordination emerges in human groups (from rocking chairs to violin players) and how it is affected by coupling structure and strength. Here we uncover the spontaneous emergence of leadership (based on physical signaling during group interaction) as a crucial factor steering the occurrence of synchronization in complex human networks where individuals perform a joint motor task. In two experiments engaging participants in an arm movement synchronization task, in the physical world as well as in the digital world, we found that specific patterns of leadership emerged and increased synchronization performance. Precisely, three patterns were found, involving a subtle interaction between phase of the motion and amount of influence. Such patterns were independent of the presence or absence of physical interaction, and persisted across manipulated spatial configurations. Our results shed light on the mechanisms that drive coordination and leadership in human groups, and are consequential for the design of interactions with artificial agents, avatars or robots, where social roles can be determinant for a successful interaction.

On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD.

Machine learning has become a widely popular and successful paradigm, especially in data-driven science and engineering. A major application problem is data-driven forecasting of future states from a complex dynamical system. Artificial neural networks have evolved as a clear leader among many machine learning approaches, and recurrent neural networks are considered to be particularly well suited for forecasting dynamical systems. In this setting, the echo-state networks or reservoir computers (RCs) have emerged for their simplicity and computational complexity advantages. Instead of a fully trained network, an RC trains only readout weights by a simple, efficient least squares method. What is perhaps quite surprising is that nonetheless, an RC succeeds in making high quality forecasts, competitively with more intensively trained methods, even if not the leader. There remains an unanswered question as to why and how an RC works at all despite randomly selected weights. To this end, this work analyzes a further simplified RC, where the internal activation function is an identity function. Our simplification is not presented for the sake of tuning or improving an RC, but rather for the sake of analysis of what we take to be the surprise being not that it does not work better, but that such random methods work at all. We explicitly connect the RC with linear activation and linear readout to well developed time-series literature on vector autoregressive (VAR) averages that includes theorems on representability through the Wold theorem, which already performs reasonably for short-term forecasts. In the case of a linear activation and now popular quadratic readout RC, we explicitly connect to a nonlinear VAR, which performs quite well. Furthermore, we associate this paradigm to the now widely popular dynamic mode decomposition; thus, these three are in a sense different faces of the same thing. We illustrate our observations in terms of popular benchmark examples including Mackey-Glass differential delay equations and the Lorenz63 system.

On Geometry of Information Flow for Causal Inference.

Causal inference is perhaps one of the most fundamental concepts in science, beginning originally from the works of some of the ancient philosophers, through today, but also weaved strongly in current work from statisticians, machine learning experts, and scientists from many other fields. This paper takes the perspective of information flow, which includes the Nobel prize winning work on Granger-causality, and the recently highly popular transfer entropy, these being probabilistic in nature. Our main contribution will be to develop analysis tools that will allow a geometric interpretation of information flow as a causal inference indicated by positive transfer entropy. We will describe the effective dimensionality of an underlying manifold as projected into the outcome space that summarizes information flow. Therefore, contrasting the probabilistic and geometric perspectives, we will introduce a new measure of causal inference based on the fractal correlation dimension conditionally applied to competing explanations of future forecasts, which we will write G e o C y → x . This avoids some of the boundedness issues that we show exist for the transfer entropy, T y → x . We will highlight our discussions with data developed from synthetic models of successively more complex nature: these include the Hénon map example, and finally a real physiological example relating breathing and heart rate function.

Stochastic and mixed flower graphs.

Stochasticity is introduced to a well studied class of recursively grown graphs: (u,v)-flower nets, which have power-law degree distributions as well as small-world properties (when u=1). The stochastic variant interpolates between different (deterministic) flower graphs thus adding flexibility to the model. The random multiplicative growth process involved, however, leads to a spread ensemble of networks with finite variance for the number of links, nodes, and loops. Nevertheless, the degree exponent and loopiness exponent attain unique values in the thermodynamic limit of infinitely large graphs. We also study a class of mixed flower networks, closely related to the stochastic flowers, but which are grown recursively in a deterministic way. The deterministic growth of mixed flower-nets eliminates ensemble spreads, and their recursive growth allows for exact analysis of their (uniquely defined) mixed properties.

Manifold learning for organizing unstructured sets of process observations.

Data mining is routinely used to organize ensembles of short temporal observations so as to reconstruct useful, low-dimensional realizations of an underlying dynamical system. In this paper, we use manifold learning to organize unstructured ensembles of observations ("trials") of a system's response surface. We have no control over where every trial starts, and during each trial, operating conditions are varied by turning "agnostic" knobs, which change system parameters in a systematic, but unknown way. As one (or more) knobs "turn," we record (possibly partial) observations of the system response. We demonstrate how such partial and disorganized observation ensembles can be integrated into coherent response surfaces whose dimension and parametrization can be systematically recovered in a data-driven fashion. The approach can be justified through the Whitney and Takens embedding theorems, allowing reconstruction of manifolds/attractors through different types of observations. We demonstrate our approach by organizing unstructured observations of response surfaces, including the reconstruction of a cusp bifurcation surface for hydrogen combustion in a continuous stirred tank reactor. Finally, we demonstrate how this observation-based reconstruction naturally leads to informative transport maps between the input parameter space and output/state variable spaces.

How entropic regression beats the outliers problem in nonlinear system identification.

In this work, we developed a nonlinear System Identification (SID) method that we called Entropic Regression. Our method adopts an information-theoretic measure for the data-driven discovery of the underlying dynamics. Our method shows robustness toward noise and outliers, and it outperforms many of the current state-of-the-art methods. Moreover, the method of Entropic Regression overcomes many of the major limitations of the current methods such as sloppy parameters, diverse scale, and SID in high-dimensional systems such as complex networks. The use of information-theoretic measures in entropic regression has unique advantages, due to the Asymptotic Equipartition Property of probability distributions, that outliers and other low-occurrence events are conveniently and intrinsically de-emphasized as not-typical, by definition. We provide a numerical comparison with the current state-of-the-art methods in sparse regression, and we apply the methods to different chaotic systems such as the Lorenz System, the Kuramoto-Sivashinsky equations, and the Double-Well Potential.

Inferring causation from time series in Earth system sciences.

The heart of the scientific enterprise is a rational effort to understand the causes behind the phenomena we observe. In large-scale complex dynamical systems such as the Earth system, real experiments are rarely feasible. However, a rapidly increasing amount of observational and simulated data opens up the use of novel data-driven causal methods beyond the commonly adopted correlation techniques. Here, we give an overview of causal inference frameworks and identify promising generic application cases common in Earth system sciences and beyond. We discuss challenges and initiate the benchmark platform causeme.net to close the gap between method users and developers.

Open or closed? Information flow decided by transfer operators and forecastability quality metric.

A basic systems question concerns the concept of closure, meaning autonomy (closed) in the sense of describing the (sub)system as fully consistent within itself. Alternatively, the system may be nonautonomous (open), meaning it receives influence from an outside subsystem. We assert here that the concept of information flow and the related concept of causation inference are summarized by this simple question of closure as we define herein. We take the forecasting perspective of Weiner-Granger causality that describes a causal relationship exists if a subsystem's forecast quality depends on considering states of another subsystem. Here, we develop a new direct analytic discussion, rather than a data oriented approach. That is, we refer to the underlying Frobenius-Perron (FP) transfer operator that moderates evolution of densities of ensembles of orbits, and two alternative forms of the restricted Frobenius-Perron operator, interpreted as if either closed (deterministic FP) or not closed (the unaccounted outside influence seems stochastic and we show correspondingly requires the stochastic FP operator). Thus follows contrasting the kernels of the variants of the operators, as if densities in their own rights. However, the corresponding differential entropy comparison by Kullback-Leibler divergence, as one would typically use when developing transfer entropy, becomes ill-defined. Instead, we build our Forecastability Quality Metric (FQM) upon the "symmetrized" variant known as Jensen-Shannon divergence, and we are also able to point out several useful resulting properties. We illustrate the FQM by a simple coupled chaotic system. Our analysis represents a new theoretical direction, but we do describe data oriented directions for the future.

Anatomy of leadership in collective behaviour.

Understanding the mechanics behind the coordinated movement of mobile animal groups (collective motion) provides key insights into their biology and ecology, while also yielding algorithms for bio-inspired technologies and autonomous systems. It is becoming increasingly clear that many mobile animal groups are composed of heterogeneous individuals with differential levels and types of influence over group behaviors. The ability to infer this differential influence, or leadership, is critical to understanding group functioning in these collective animal systems. Due to the broad interpretation of leadership, many different measures and mathematical tools are used to describe and infer "leadership," e.g., position, causality, influence, and information flow. But a key question remains: which, if any, of these concepts actually describes leadership? We argue that instead of asserting a single definition or notion of leadership, the complex interaction rules and dynamics typical of a group imply that leadership itself is not merely a binary classification (leader or follower), but rather, a complex combination of many different components. In this paper, we develop an anatomy of leadership, identify several principal components, and provide a general mathematical framework for discussing leadership. With the intricacies of this taxonomy in mind, we present a set of leadership-oriented toy models that should be used as a proving ground for leadership inference methods going forward. We believe this multifaceted approach to leadership will enable a broader understanding of leadership and its inference from data in mobile animal groups and beyond.