Optimal criteria and their asymptotic form for data selection in data-driven reduced-order modelling with Gaussian process regression.

We derive criteria for the selection of datapoints used for data-driven reduced-order modelling and other areas of supervised learning based on Gaussian process regression (GPR). While this is a well-studied area in the fields of active learning and optimal experimental design, most criteria in the literature are empirical. Here we introduce an optimality condition for the selection of a new input defined as the minimizer of the distance between the approximated output probability density function (pdf) of the reduced-order model and the exact one. Given that the exact pdf is unknown, we define the selection criterion as the supremum over the unit sphere of the native Hilbert space for the GPR. The resulting selection criterion, however, has a form that is difficult to compute. We combine results from GPR theory and asymptotic analysis to derive a computable form of the defined optimality criterion that is valid in the limit of small predictive variance. The derived asymptotic form of the selection criterion leads to convergence of the GPR model that guarantees a balanced distribution of data resources between probable and large-deviation outputs, resulting in an effective way of sampling towards data-driven reduced-order modelling. This article is part of the theme issue 'Data-driven prediction in dynamical systems'.

Discovering and forecasting extreme events via active learning in neural operators.

Extreme events in society and nature, such as pandemic spikes, rogue waves or structural failures, can have catastrophic consequences. Characterizing extremes is difficult, as they occur rarely, arise from seemingly benign conditions, and belong to complex and often unknown infinite-dimensional systems. Such challenges render attempts at characterizing them moot. We address each of these difficulties by combining output-weighted training schemes in Bayesian experimental design (BED) with an ensemble of deep neural operators. This model-agnostic framework pairs a BED scheme that actively selects data for quantifying extreme events with an ensemble of deep neural operators that approximate infinite-dimensional nonlinear operators. We show that not only does this framework outperform Gaussian processes, but that (1) shallow ensembles of just two members perform best; (2) extremes are uncovered regardless of the state of the initial data (that is, with or without extremes); (3) our method eliminates 'double-descent' phenomena; (4) the use of batches of suboptimal acquisition samples compared to step-by-step global optima does not hinder BED performance; and (5) Monte Carlo acquisition outperforms standard optimizers in high dimensions. Together, these conclusions form a scalable artificial intelligence (AI)-assisted experimental infrastructure that can efficiently infer and pinpoint critical situations across many domains, from physical to societal systems.

Output-weighted optimal sampling for Bayesian regression and rare event statistics using few samples.

For many important problems the quantity of interest is an unknown function of the parameters, which is a random vector with known statistics. Since the dependence of the output on this random vector is unknown, the challenge is to identify its statistics, using the minimum number of function evaluations. This problem can be seen in the context of active learning or optimal experimental design. We employ Bayesian regression to represent the derived model uncertainty due to finite and small number of input-output pairs. In this context we evaluate existing methods for optimal sample selection, such as model error minimization and mutual information maximization. We show that for the case of known output variance, the commonly employed criteria in the literature do not take into account the output values of the existing input-output pairs, while for the case of unknown output variance this dependence can be very weak. We introduce a criterion that takes into account the values of the output for the existing samples and adaptively selects inputs from regions of the parameter space which have an important contribution to the output. The new method allows for application to high-dimensional inputs, paving the way for optimal experimental design in high dimensions.

Learning the tangent space of dynamical instabilities from data.

For a large class of dynamical systems, the optimally time-dependent (OTD) modes, a set of deformable orthonormal tangent vectors that track directions of instabilities along any trajectory, are known to depend "pointwise" on the state of the system on the attractor but not on the history of the trajectory. We leverage the power of neural networks to learn this "pointwise" mapping from the phase space to OTD space directly from data. The result of the learning process is a cartography of directions associated with strongest instabilities in the phase space. Implications for data-driven prediction and control of dynamical instabilities are discussed.

Closed-loop adaptive control of extreme events in a turbulent flow.

Extreme events that arise spontaneously in chaotic dynamical systems often have an adverse impact on the system or the surrounding environment. As such, their mitigation is highly desirable. Here, we introduce a control strategy for mitigating extreme events in a turbulent shear flow. The controller combines a probabilistic prediction of the extreme events with a deterministic actuator. The predictions are used to actuate the controller only when an extreme event is imminent. When actuated, the controller only acts on the degrees of freedom that are involved in the formation of the extreme events, exerting minimal interference with the flow dynamics. As a result, the attractors of the controlled and uncontrolled systems share the same chaotic core (containing the nonextreme events) and only differ in the tail of their distributions. We propose that such adaptive low-dimensional controllers should be used to mitigate extreme events in general chaotic dynamical systems, beyond the shear flow considered here.

Predicting ocean rogue waves from point measurements: An experimental study for unidirectional waves.

Rogue waves are strong localizations of the wave field that can develop in different branches of physics and engineering, such as water or electromagnetic waves. Here, we experimentally quantify the prediction potentials of a comprehensive rogue-wave reduced-order precursor tool that has been recently developed to predict extreme events due to spatially localized modulation instability. The laboratory tests have been conducted in two different water wave facilities and they involve unidirectional water waves; in both cases we show that the deterministic and spontaneous emergence of extreme events is well predicted through the reported scheme. Due to the interdisciplinary character of the approach, similar studies may be motivated in other nonlinear dispersive media, such as nonlinear optics, plasma, and solids, governed by similar equations, allowing the early stage of extreme wave detection.

Sequential sampling strategy for extreme event statistics in nonlinear dynamical systems.

We develop a method for the evaluation of extreme event statistics associated with nonlinear dynamical systems from a small number of samples. From an initial dataset of design points, we formulate a sequential strategy that provides the "next-best" data point (set of parameters) that when evaluated results in improved estimates of the probability density function (pdf) for a scalar quantity of interest. The approach uses Gaussian process regression to perform Bayesian inference on the parameter-to-observation map describing the quantity of interest. We then approximate the desired pdf along with uncertainty bounds using the posterior distribution of the inferred map. The next-best design point is sequentially determined through an optimization procedure that selects the point in parameter space that maximally reduces uncertainty between the estimated bounds of the pdf prediction. Since the optimization process uses only information from the inferred map, it has minimal computational cost. Moreover, the special form of the metric emphasizes the tails of the pdf. The method is practical for systems where the dimensionality of the parameter space is of moderate size and for problems where each sample is very expensive to obtain. We apply the method to estimate the extreme event statistics for a very high-dimensional system with millions of degrees of freedom: an offshore platform subjected to 3D irregular waves. It is demonstrated that the developed approach can accurately determine the extreme event statistics using a limited number of samples.

New perspectives for the prediction and statistical quantification of extreme events in high-dimensional dynamical systems.

We discuss extreme events as random occurrences of strongly transient dynamics that lead to nonlinear energy transfers within a chaotic attractor. These transient events are the result of finite-time instabilities and therefore are inherently connected with both statistical and dynamical properties of the system. We consider two classes of problems related to extreme events and nonlinear energy transfers, namely (i) the derivation of precursors for the short-term prediction of extreme events, and (ii) the efficient sampling of random realizations for the fastest convergence of the probability density function in the tail region. We summarize recent methods on these problems that rely on the simultaneous consideration of the statistical and dynamical characteristics of the system. This is achieved by combining available data, in the form of second-order statistics, with dynamical equations that provide information for the transient events that lead to extreme responses. We present these methods through two high-dimensional, prototype systems that exhibit strongly chaotic dynamics and extreme responses due to transient instabilities, the Kolmogorov flow and unidirectional nonlinear water waves.This article is part of the theme issue 'Nonlinear energy transfer in dynamical and acoustical systems'.

Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks.

We introduce a data-driven forecasting method for high-dimensional chaotic systems using long short-term memory (LSTM) recurrent neural networks. The proposed LSTM neural networks perform inference of high-dimensional dynamical systems in their reduced order space and are shown to be an effective set of nonlinear approximators of their attractor. We demonstrate the forecasting performance of the LSTM and compare it with Gaussian processes (GPs) in time series obtained from the Lorenz 96 system, the Kuramoto-Sivashinsky equation and a prototype climate model. The LSTM networks outperform the GPs in short-term forecasting accuracy in all applications considered. A hybrid architecture, extending the LSTM with a mean stochastic model (MSM-LSTM), is proposed to ensure convergence to the invariant measure. This novel hybrid method is fully data-driven and extends the forecasting capabilities of LSTM networks.

A variational approach to probing extreme events in turbulent dynamical systems.

Extreme events are ubiquitous in a wide range of dynamical systems, including turbulent fluid flows, nonlinear waves, large-scale networks, and biological systems. We propose a variational framework for probing conditions that trigger intermittent extreme events in high-dimensional nonlinear dynamical systems. We seek the triggers as the probabilistically feasible solutions of an appropriately constrained optimization problem, where the function to be maximized is a system observable exhibiting intermittent extreme bursts. The constraints are imposed to ensure the physical admissibility of the optimal solutions, that is, significant probability for their occurrence under the natural flow of the dynamical system. We apply the method to a body-forced incompressible Navier-Stokes equation, known as the Kolmogorov flow. We find that the intermittent bursts of the energy dissipation are independent of the external forcing and are instead caused by the spontaneous transfer of energy from large scales to the mean flow via nonlinear triad interactions. The global maximizer of the corresponding variational problem identifies the responsible triad, hence providing a precursor for the occurrence of extreme dissipation events. Specifically, monitoring the energy transfers within this triad allows us to develop a data-driven short-term predictor for the intermittent bursts of energy dissipation. We assess the performance of this predictor through direct numerical simulations.

Reduced-order description of transient instabilities and computation of finite-time Lyapunov exponents.

High-dimensional chaotic dynamical systems can exhibit strongly transient features. These are often associated with instabilities that have a finite-time duration. Because of the finite-time character of these transient events, their detection through infinite-time methods, e.g., long term averages, Lyapunov exponents or information about the statistical steady-state, is not possible. Here, we utilize a recently developed framework, the Optimally Time-Dependent (OTD) modes, to extract a time-dependent subspace that spans the modes associated with transient features associated with finite-time instabilities. As the main result, we prove that the OTD modes, under appropriate conditions, converge exponentially fast to the eigendirections of the Cauchy-Green tensor associated with the most intense finite-time instabilities. Based on this observation, we develop a reduced-order method for the computation of finite-time Lyapunov exponents (FTLE) and vectors. In high-dimensional systems, the computational cost of the reduced-order method is orders of magnitude lower than the full FTLE computation. We demonstrate the validity of the theoretical findings on two numerical examples.

Dynamical indicators for the prediction of bursting phenomena in high-dimensional systems.

Drawing upon the bursting mechanism in slow-fast systems, we propose indicators for the prediction of such rare extreme events which do not require a priori known slow and fast coordinates. The indicators are associated with functionals defined in terms of optimally time-dependent (OTD) modes. One such functional has the form of the largest eigenvalue of the symmetric part of the linearized dynamics reduced to these modes. In contrast to other choices of subspaces, the proposed modes are flow invariant and therefore a projection onto them is dynamically meaningful. We illustrate the application of these indicators on three examples: a prototype low-dimensional model, a body-forced turbulent fluid flow, and a unidirectional model of nonlinear water waves. We use Bayesian statistics to quantify the predictive power of the proposed indicators.

Circadian Rhythms in Rho1 Activity Regulate Neuronal Plasticity and Network Hierarchy.

Neuronal plasticity helps animals learn from their environment. However, it is challenging to link specific changes in defined neurons to altered behavior. Here, we focus on circadian rhythms in the structure of the principal s-LNv clock neurons in Drosophila. By quantifying neuronal architecture, we observed that s-LNv structural plasticity changes the amount of axonal material in addition to cycles of fasciculation and defasciculation. We found that this is controlled by rhythmic Rho1 activity that retracts s-LNv axonal termini by increasing myosin phosphorylation and simultaneously changes the balance of pre-synaptic and dendritic markers. This plasticity is required to change clock network hierarchy and allow seasonal adaptation. Rhythms in Rho1 activity are controlled by clock-regulated transcription of Puratrophin-1-like (Pura), a Rho1 GEF. Since spinocerebellar ataxia is associated with mutations in human Puratrophin-1, our data support the idea that defective actin-related plasticity underlies this ataxia.

Unsteady evolution of localized unidirectional deep-water wave groups.

We study the evolution of localized wave groups in unidirectional water wave envelope equations [the nonlinear Schrödinger (NLSE) and the modified NLSE (MNLSE)]. These localizations of energy can lead to disastrous extreme responses (rogue waves). We analytically quantify the role of such spatial localization, introducing a technique to reduce the underlying partial differential equation dynamics to a simple ordinary differential equation for the wave packet amplitude. We use this reduced model to show how the scale-invariant symmetries of the NLSE break down when the additional terms in the MNLSE are included, inducing a critical scale for the occurrence of extreme waves.

Blended particle filters for large-dimensional chaotic dynamical systems.

A major challenge in contemporary data science is the development of statistically accurate particle filters to capture non-Gaussian features in large-dimensional chaotic dynamical systems. Blended particle filters that capture non-Gaussian features in an adaptively evolving low-dimensional subspace through particles interacting with evolving Gaussian statistics on the remaining portion of phase space are introduced here. These blended particle filters are constructed in this paper through a mathematical formalism involving conditional Gaussian mixtures combined with statistically nonlinear forecast models compatible with this structure developed recently with high skill for uncertainty quantification. Stringent test cases for filtering involving the 40-dimensional Lorenz 96 model with a 5-dimensional adaptive subspace for nonlinear blended filtering in various turbulent regimes with at least nine positive Lyapunov exponents are used here. These cases demonstrate the high skill of the blended particle filter algorithms in capturing both highly non-Gaussian dynamical features as well as crucial nonlinear statistics for accurate filtering in extreme filtering regimes with sparse infrequent high-quality observations. The formalism developed here is also useful for multiscale filtering of turbulent systems and a simple application is sketched below.

Statistically accurate low-order models for uncertainty quantification in turbulent dynamical systems.

A framework for low-order predictive statistical modeling and uncertainty quantification in turbulent dynamical systems is developed here. These reduced-order, modified quasilinear Gaussian (ROMQG) algorithms apply to turbulent dynamical systems in which there is significant linear instability or linear nonnormal dynamics in the unperturbed system and energy-conserving nonlinear interactions that transfer energy from the unstable modes to the stable modes where dissipation occurs, resulting in a statistical steady state; such turbulent dynamical systems are ubiquitous in geophysical and engineering turbulence. The ROMQG method involves constructing a low-order, nonlinear, dynamical system for the mean and covariance statistics in the reduced subspace that has the unperturbed statistics as a stable fixed point and optimally incorporates the indirect effect of non-Gaussian third-order statistics for the unperturbed system in a systematic calibration stage. This calibration procedure is achieved through information involving only the mean and covariance statistics for the unperturbed equilibrium. The performance of the ROMQG algorithm is assessed on two stringent test cases: the 40-mode Lorenz 96 model mimicking midlatitude atmospheric turbulence and two-layer baroclinic models for high-latitude ocean turbulence with over 125,000 degrees of freedom. In the Lorenz 96 model, the ROMQG algorithm with just a single mode captures the transient response to random or deterministic forcing. For the baroclinic ocean turbulence models, the inexpensive ROMQG algorithm with 252 modes, less than 0.2% of the total, captures the nonlinear response of the energy, the heat flux, and even the one-dimensional energy and heat flux spectra.