Cumulant GAN.

In this article, we propose a novel loss function for training generative adversarial networks (GANs) aiming toward deeper theoretical understanding as well as improved stability and performance for the underlying optimization problem. The new loss function is based on cumulant generating functions (CGFs) giving rise to Cumulant GAN. Relying on a recently derived variational formula, we show that the corresponding optimization problem is equivalent to Rényi divergence minimization, thus offering a (partially) unified perspective of GAN losses: the Rényi family encompasses Kullback-Leibler divergence (KLD), reverse KLD, Hellinger distance, and χ2 -divergence. Wasserstein GAN is also a member of cumulant GAN. In terms of stability, we rigorously prove the linear convergence of cumulant GAN to the Nash equilibrium for a linear discriminator, Gaussian distributions, and the standard gradient descent ascent algorithm. Finally, we experimentally demonstrate that image generation is more robust relative to Wasserstein GAN and it is substantially improved in terms of both inception score (IS) and Fréchet inception distance (FID) when both weaker and stronger discriminators are considered.

Explainable and trustworthy artificial intelligence for correctable modeling in chemical sciences.

Data science has primarily focused on big data, but for many physics, chemistry, and engineering applications, data are often small, correlated and, thus, low dimensional, and sourced from both computations and experiments with various levels of noise. Typical statistics and machine learning methods do not work for these cases. Expert knowledge is essential, but a systematic framework for incorporating it into physics-based models under uncertainty is lacking. Here, we develop a mathematical and computational framework for probabilistic artificial intelligence (AI)-based predictive modeling combining data, expert knowledge, multiscale models, and information theory through uncertainty quantification and probabilistic graphical models (PGMs). We apply PGMs to chemistry specifically and develop predictive guarantees for PGMs generally. Our proposed framework, combining AI and uncertainty quantification, provides explainable results leading to correctable and, eventually, trustworthy models. The proposed framework is demonstrated on a microkinetic model of the oxygen reduction reaction.

Uncertainty quantification for generalized Langevin dynamics.

We present efficient finite difference estimators for goal-oriented sensitivity indices with applications to the generalized Langevin equation (GLE). In particular, we apply these estimators to analyze an extended variable formulation of the GLE where other well known sensitivity analysis techniques such as the likelihood ratio method are not applicable to key parameters of interest. These easily implemented estimators are formed by coupling the nominal and perturbed dynamics appearing in the finite difference through a common driving noise or common random path. After developing a general framework for variance reduction via coupling, we demonstrate the optimality of the common random path coupling in the sense that it produces a minimal variance surrogate for the difference estimator relative to sampling dynamics driven by independent paths. In order to build intuition for the common random path coupling, we evaluate the efficiency of the proposed estimators for a comprehensive set of examples of interest in particle dynamics. These reduced variance difference estimators are also a useful tool for performing global sensitivity analysis and for investigating non-local perturbations of parameters, such as increasing the number of Prony modes active in an extended variable GLE.

Effects of correlated parameters and uncertainty in electronic-structure-based chemical kinetic modelling.

Kinetic models based on first principles are becoming common place in heterogeneous catalysis because of their ability to interpret experimental data, identify the rate-controlling step, guide experiments and predict novel materials. To overcome the tremendous computational cost of estimating parameters of complex networks on metal catalysts, approximate quantum mechanical calculations are employed that render models potentially inaccurate. Here, by introducing correlative global sensitivity analysis and uncertainty quantification, we show that neglecting correlations in the energies of species and reactions can lead to an incorrect identification of influential parameters and key reaction intermediates and reactions. We rationalize why models often underpredict reaction rates and show that, despite the uncertainty being large, the method can, in conjunction with experimental data, identify influential missing reaction pathways and provide insights into the catalyst active site and the kinetic reliability of a model. The method is demonstrated in ethanol steam reforming for hydrogen production for fuel cells.

Efficient estimators for likelihood ratio sensitivity indices of complex stochastic dynamics.

We demonstrate that centered likelihood ratio estimators for the sensitivity indices of complex stochastic dynamics are highly efficient with low, constant in time variance and consequently they are suitable for sensitivity analysis in long-time and steady-state regimes. These estimators rely on a new covariance formulation of the likelihood ratio that includes as a submatrix a Fisher information matrix for stochastic dynamics and can also be used for fast screening of insensitive parameters and parameter combinations. The proposed methods are applicable to broad classes of stochastic dynamics such as chemical reaction networks, Langevin-type equations and stochastic models in finance, including systems with a high dimensional parameter space and/or disparate decorrelation times between different observables. Furthermore, they are simple to implement as a standard observable in any existing simulation algorithm without additional modifications.

The geometry of generalized force matching and related information metrics in coarse-graining of molecular systems.

Using the probabilistic language of conditional expectations, we reformulate the force matching method for coarse-graining of molecular systems as a projection onto spaces of coarse observables. A practical outcome of this probabilistic description is the link of the force matching method with thermodynamic integration. This connection provides a way to systematically construct a local mean force and to optimally approximate the potential of mean force through force matching. We introduce a generalized force matching condition for the local mean force in the sense that allows the approximation of the potential of mean force under both linear and non-linear coarse graining mappings (e.g., reaction coordinates, end-to-end length of chains). Furthermore, we study the equivalence of force matching with relative entropy minimization which we derive for general non-linear coarse graining maps. We present in detail the generalized force matching condition through applications to specific examples in molecular systems.

Accelerated Sensitivity Analysis in High-Dimensional Stochastic Reaction Networks.

Existing sensitivity analysis approaches are not able to handle efficiently stochastic reaction networks with a large number of parameters and species, which are typical in the modeling and simulation of complex biochemical phenomena. In this paper, a two-step strategy for parametric sensitivity analysis for such systems is proposed, exploiting advantages and synergies between two recently proposed sensitivity analysis methodologies for stochastic dynamics. The first method performs sensitivity analysis of the stochastic dynamics by means of the Fisher Information Matrix on the underlying distribution of the trajectories; the second method is a reduced-variance, finite-difference, gradient-type sensitivity approach relying on stochastic coupling techniques for variance reduction. Here we demonstrate that these two methods can be combined and deployed together by means of a new sensitivity bound which incorporates the variance of the quantity of interest as well as the Fisher Information Matrix estimated from the first method. The first step of the proposed strategy labels sensitivities using the bound and screens out the insensitive parameters in a controlled manner. In the second step of the proposed strategy, a finite-difference method is applied only for the sensitivity estimation of the (potentially) sensitive parameters that have not been screened out in the first step. Results on an epidermal growth factor network with fifty parameters and on a protein homeostasis with eighty parameters demonstrate that the proposed strategy is able to quickly discover and discard the insensitive parameters and in the remaining potentially sensitive parameters it accurately estimates the sensitivities. The new sensitivity strategy can be several times faster than current state-of-the-art approaches that test all parameters, especially in "sloppy" systems. In particular, the computational acceleration is quantified by the ratio between the total number of parameters over the number of the sensitive parameters.

Parametric sensitivity analysis for stochastic molecular systems using information theoretic metrics.

In this paper, we present a parametric sensitivity analysis (SA) methodology for continuous time and continuous space Markov processes represented by stochastic differential equations. Particularly, we focus on stochastic molecular dynamics as described by the Langevin equation. The utilized SA method is based on the computation of the information-theoretic (and thermodynamic) quantity of relative entropy rate (RER) and the associated Fisher information matrix (FIM) between path distributions, and it is an extension of the work proposed by Y. Pantazis and M. A. Katsoulakis [J. Chem. Phys. 138, 054115 (2013)]. A major advantage of the pathwise SA method is that both RER and pathwise FIM depend only on averages of the force field; therefore, they are tractable and computable as ergodic averages from a single run of the molecular dynamics simulation both in equilibrium and in non-equilibrium steady state regimes. We validate the performance of the extended SA method to two different molecular stochastic systems, a standard Lennard-Jones fluid and an all-atom methane liquid, and compare the obtained parameter sensitivities with parameter sensitivities on three popular and well-studied observable functions, namely, the radial distribution function, the mean squared displacement, and the pressure. Results show that the RER-based sensitivities are highly correlated with the observable-based sensitivities.

Goal-oriented sensitivity analysis for lattice kinetic Monte Carlo simulations.

In this paper we propose a new class of coupling methods for the sensitivity analysis of high dimensional stochastic systems and in particular for lattice Kinetic Monte Carlo (KMC). Sensitivity analysis for stochastic systems is typically based on approximating continuous derivatives with respect to model parameters by the mean value of samples from a finite difference scheme. Instead of using independent samples the proposed algorithm reduces the variance of the estimator by developing a strongly correlated-"coupled"- stochastic process for both the perturbed and unperturbed stochastic processes, defined in a common state space. The novelty of our construction is that the new coupled process depends on the targeted observables, e.g., coverage, Hamiltonian, spatial correlations, surface roughness, etc., hence we refer to the proposed method as goal-oriented sensitivity analysis. In particular, the rates of the coupled Continuous Time Markov Chain are obtained as solutions to a goal-oriented optimization problem, depending on the observable of interest, by considering the minimization functional of the corresponding variance. We show that this functional can be used as a diagnostic tool for the design and evaluation of different classes of couplings. Furthermore, the resulting KMC sensitivity algorithm has an easy implementation that is based on the Bortz-Kalos-Lebowitz algorithm's philosophy, where events are divided in classes depending on level sets of the observable of interest. Finally, we demonstrate in several examples including adsorption, desorption, and diffusion Kinetic Monte Carlo that for the same confidence interval and observable, the proposed goal-oriented algorithm can be two orders of magnitude faster than existing coupling algorithms for spatial KMC such as the Common Random Number approach. We also provide a complete implementation of the proposed sensitivity analysis algorithms, including various spatial KMC examples, in a supplementary MATLAB source code.

Parametric sensitivity analysis for biochemical reaction networks based on pathwise information theory.

BACKGROUND: Stochastic modeling and simulation provide powerful predictive methods for the intrinsic understanding of fundamental mechanisms in complex biochemical networks. Typically, such mathematical models involve networks of coupled jump stochastic processes with a large number of parameters that need to be suitably calibrated against experimental data. In this direction, the parameter sensitivity analysis of reaction networks is an essential mathematical and computational tool, yielding information regarding the robustness and the identifiability of model parameters. However, existing sensitivity analysis approaches such as variants of the finite difference method can have an overwhelming computational cost in models with a high-dimensional parameter space. RESULTS: We develop a sensitivity analysis methodology suitable for complex stochastic reaction networks with a large number of parameters. The proposed approach is based on Information Theory methods and relies on the quantification of information loss due to parameter perturbations between time-series distributions. For this reason, we need to work on path-space, i.e., the set consisting of all stochastic trajectories, hence the proposed approach is referred to as "pathwise". The pathwise sensitivity analysis method is realized by employing the rigorously-derived Relative Entropy Rate, which is directly computable from the propensity functions. A key aspect of the method is that an associated pathwise Fisher Information Matrix (FIM) is defined, which in turn constitutes a gradient-free approach to quantifying parameter sensitivities. The structure of the FIM turns out to be block-diagonal, revealing hidden parameter dependencies and sensitivities in reaction networks. CONCLUSIONS: As a gradient-free method, the proposed sensitivity analysis provides a significant advantage when dealing with complex stochastic systems with a large number of parameters. In addition, the knowledge of the structure of the FIM can allow to efficiently address questions on parameter identifiability, estimation and robustness. The proposed method is tested and validated on three biochemical systems, namely: (a) a protein production/degradation model where explicit solutions are available, permitting a careful assessment of the method, (b) the p53 reaction network where quasi-steady stochastic oscillations of the concentrations are observed, and for which continuum approximations (e.g. mean field, stochastic Langevin, etc.) break down due to persistent oscillations between high and low populations, and (c) an Epidermal Growth Factor Receptor model which is an example of a high-dimensional stochastic reaction network with more than 200 reactions and a corresponding number of parameters.

Information-theoretic tools for parametrized coarse-graining of non-equilibrium extended systems.

In this paper, we focus on the development of new methods suitable for efficient and reliable coarse-graining of non-equilibrium molecular systems. In this context, we propose error estimation and controlled-fidelity model reduction methods based on Path-Space Information Theory, combined with statistical parametric estimation of rates for non-equilibrium stationary processes. The approach we propose extends the applicability of existing information-based methods for deriving parametrized coarse-grained models to Non-Equilibrium systems with Stationary States. In the context of coarse-graining it allows for constructing optimal parametrized Markovian coarse-grained dynamics within a parametric family, by minimizing information loss (due to coarse-graining) on the path space. Furthermore, we propose an asymptotically equivalent method-related to maximum likelihood estimators for stochastic processes-where the coarse-graining is obtained by optimizing the information content in path space of the coarse variables, with respect to the projected computational data from a fine-scale simulation. Finally, the associated path-space Fisher Information Matrix can provide confidence intervals for the corresponding parameter estimators. We demonstrate the proposed coarse-graining method in (a) non-equilibrium systems with diffusing interacting particles, driven by out-of-equilibrium boundary conditions, as well as (b) multi-scale diffusions and the corresponding stochastic averaging limits, comparing them to our proposed methodologies.

A relative entropy rate method for path space sensitivity analysis of stationary complex stochastic dynamics.

We propose a new sensitivity analysis methodology for complex stochastic dynamics based on the relative entropy rate. The method becomes computationally feasible at the stationary regime of the process and involves the calculation of suitable observables in path space for the relative entropy rate and the corresponding Fisher information matrix. The stationary regime is crucial for stochastic dynamics and here allows us to address the sensitivity analysis of complex systems, including examples of processes with complex landscapes that exhibit metastability, non-reversible systems from a statistical mechanics perspective, and high-dimensional, spatially distributed models. All these systems exhibit, typically non-Gaussian stationary probability distributions, while in the case of high-dimensionality, histograms are impossible to construct directly. Our proposed methods bypass these challenges relying on the direct Monte Carlo simulation of rigorously derived observables for the relative entropy rate and Fisher information in path space rather than on the stationary probability distribution itself. We demonstrate the capabilities of the proposed methodology by focusing here on two classes of problems: (a) Langevin particle systems with either reversible (gradient) or non-reversible (non-gradient) forcing, highlighting the ability of the method to carry out sensitivity analysis in non-equilibrium systems; and, (b) spatially extended kinetic Monte Carlo models, showing that the method can handle high-dimensional problems.

Mechanistic principles of nanoparticle evolution to zeolite crystals.

Precursor nanoparticles that form spontaneously on hydrolysis of tetraethylorthosilicate in aqueous solutions of tetrapropylammonium (TPA) hydroxide evolve to TPA-silicalite-1, a molecular-sieve crystal that serves as a model for the self-assembly of porous inorganic materials in the presence of organic structure-directing agents. The structure and role of these nanoparticles are of practical significance for the fabrication of hierarchically ordered porous materials and molecular-sieve films, but still remain elusive. Here we show experimental findings of nanoparticle and crystal evolution during room-temperature ageing of the aqueous suspensions that suggest growth by aggregation of nanoparticles. A kinetic mechanism suggests that the precursor nanoparticle population is distributed, and that the 5-nm building units contributing most to aggregation only exist as an intermediate small fraction. The proposed oriented-aggregation mechanism should lead to strategies for isolating or enhancing the concentration of crystal-like nanoparticles.

A mathematical model for crystal growth by aggregation of precursor metastable nanoparticles.

A mathematical model is developed to describe aggregative crystal growth, including oriented aggregation, from evolving pre-existing primary nanoparticles with composition and structure that are different from that of the final crystalline aggregate. The basic assumptions of the model are based on the ideas introduced in an earlier published report [Buyanov and Krivoruchko, Kinet. Katal. 1976, 17, 666-675] to describe the growth of low-solubility metal hydroxides (e.g., iron oxides) by oriented aggregation. It is assumed that primary particles can be described as pseudo-species A, B, and C, which have the following properties: (1) fresh primary particles (colloidally stable inert nanoparticles, denoted as A), (2) mature primary particles (partially transformed nanoparticles at an optimum stage of development for attachment to a growing crystal, denoted as B), and (3) nucleated primary particles (denoted as C1). The evolution of primary particles, A --> B --> C1, is treated as two first-order consecutive reactions. Crystal growth via crystal-crystal aggregation (Ci +) is described using the Smoluchowski equation. The new element of this model is the inclusion of an additional crystal growth mechanism via the addition of primary particles (B) to crystals (Ci): (B + ). Two distinct, but constant, kernels (K not equal K') are used. It is shown that, when K' = 0, a steplike crystal size distribution (CSD) is obtained. Within a range of K'/K values (e.g., K'/K = 10(3)), CSD with multiple peaks are obtained. Comparison with predictions of models that do not include the intermediate stage of primary particles (B) indicates pronounced differences. Despite its simplicity, the model is able to capture the qualitative features of CSD evolution that have been obtained from crystal growth experiments in hematite, which is a system that is believed to undergo oriented aggregation.

Binomial distribution based tau-leap accelerated stochastic simulation.

Recently, Gillespie introduced the tau-leap approximate, accelerated stochastic Monte Carlo method for well-mixed reacting systems [J. Chem. Phys. 115, 1716 (2001)]. In each time increment of that method, one executes a number of reaction events, selected randomly from a Poisson distribution, to enable simulation of long times. Here we introduce a binomial distribution tau-leap algorithm (abbreviated as BD-tau method). This method combines the bounded nature of the binomial distribution variable with the limiting reactant and constrained firing concepts to avoid negative populations encountered in the original tau-leap method of Gillespie for large time increments, and thus conserve mass. Simulations using prototype reaction networks show that the BD-tau method is more accurate than the original method for comparable coarse-graining in time.

Spatially adaptive lattice coarse-grained Monte Carlo simulations for diffusion of interacting molecules.

While lattice kinetic Monte Carlo (KMC) methods provide insight into numerous complex physical systems governed by interatomic interactions, they are limited to relatively short length and time scales. Recently introduced coarse-grained Monte Carlo (CGMC) simulations can reach much larger length and time scales at considerably lower computational cost. In this paper we extend the CGMC methods to spatially adaptive meshes for the case of surface diffusion (canonical ensemble). We introduce a systematic methodology to derive the transition probabilities for the coarse-grained diffusion process that ensure the correct dynamics and noise, give the correct continuum mesoscopic equations, and satisfy detailed balance. Substantial savings in CPU time are demonstrated compared to microscopic KMC while retaining high accuracy.

Coarse-grained stochastic models for tropical convection and climate.

Prototype coarse-grained stochastic parametrizations for the interaction with unresolved features of tropical convection are developed here. These coarse-grained stochastic parametrizations involve systematically derived birth/death processes with low computational overhead that allow for direct interaction of the coarse-grained dynamical variables with the smaller-scale unresolved fluctuations. It is established here for an idealized prototype climate scenario that, in suitable regimes, these coarse-grained stochastic parametrizations can significantly impact the climatology as well as strongly increase the wave fluctuations about an idealized climatology.

Coarse-grained stochastic processes for microscopic lattice systems.

Diverse scientific disciplines ranging from materials science to catalysis to biomolecular dynamics to climate modeling involve nonlinear interactions across a large range of physically significant length scales. Here a class of coarse-grained stochastic processes and corresponding Monte Carlo simulation methods, describing computationally feasible mesoscopic length scales, are derived directly from microscopic lattice systems. It is demonstrated below that the coarse-grained stochastic models can capture large-scale structures while retaining significant microscopic information. The requirement of detailed balance is used as a systematic design principle to guarantee correct noise fluctuations for the coarse-grained model. The coarse-grained stochastic algorithms provide large computational savings without increasing programming complexity or computer time per executive event compared to microscopic Monte Carlo simulations.