Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations.

We present a framework to integrate tensor network (TN) methods with reinforcement learning (RL) for solving dynamical optimization tasks. We consider the RL actor-critic method, a model-free approach for solving RL problems, and introduce TNs as the approximators for its policy and value functions. Our "actor-critic with tensor networks" (ACTeN) method is especially well suited to problems with large and factorizable state and action spaces. As an illustration of the applicability of ACTeN we solve the exponentially hard task of sampling rare trajectories in two paradigmatic stochastic models, the East model of glasses and the asymmetric simple exclusion process, the latter being particularly challenging to other methods due to the absence of detailed balance. With substantial potential for further integration with the vast array of existing RL methods, the approach introduced here is promising both for applications in physics and to multi-agent RL problems more generally.

Hierarchical classical metastability in an open quantum East model.

We study in detail an open quantum generalization of a classical kinetically constrained model-the East model-known to exhibit slow glassy dynamics stemming from a complex hierarchy of metastable states with distinct lifetimes. Using the recently introduced theory of classical metastability for open quantum systems, we show that the driven open quantum East model features a hierarchy of classical metastabilities at low temperature and weak driving field. We find that the effective long-time description of its dynamics not only is classical, but shares many properties with the classical East model, such as obeying an effective detailed balance condition and lacking static interactions between excitations, but with this occurring within a modified set of metastable phases which are coherent, and with an effective temperature that is dependent on the coherent drive.

Reinforcement learning of rare diffusive dynamics.

We present a method to probe rare molecular dynamics trajectories directly using reinforcement learning. We consider trajectories that are conditioned to transition between regions of configuration space in finite time, such as those relevant in the study of reactive events, and trajectories exhibiting rare fluctuations of time-integrated quantities in the long time limit, such as those relevant in the calculation of large deviation functions. In both cases, reinforcement learning techniques are used to optimize an added force that minimizes the Kullback-Leibler divergence between the conditioned trajectory ensemble and a driven one. Under the optimized added force, the system evolves the rare fluctuation as a typical one, affording a variational estimate of its likelihood in the original trajectory ensemble. Low variance gradients employing value functions are proposed to increase the convergence of the optimal force. The method we develop employing these gradients leads to efficient and accurate estimates of both the optimal force and the likelihood of the rare event for a variety of model systems.

Spectral properties of simple classical and quantum reset processes.

We study the spectral properties of classical and quantum Markovian processes that are reset at random times to a specific configuration or state with a reset rate that is independent of the current state of the system. We demonstrate that this simple reset dynamics causes a uniform shift in the eigenvalues of the Markov generator, excluding the zero mode corresponding to the stationary state, which has the effect of accelerating or even inducing relaxation to a stationary state. Based on this result, we provide expressions for the stationary state and probability current of the reset process in terms of weighted sums over dynamical modes of the reset-free process. We also discuss the effect of resets on processes that display metastability. We illustrate our results with two classical stochastic processes, the totally asymmetric random walk and the one-dimensional Brownian motion, as well as two quantum models: a particle coherently hopping on a chain and the dissipative transverse field Ising model, known to exhibit metastability.

Metastability in an open quantum Ising model.

We apply a recently developed theory for metastability in open quantum systems to a one-dimensional dissipative quantum Ising model. Earlier results suggest this model features either a nonequilibrium phase transition or a smooth but sharp crossover, where the stationary state changes from paramagnetic to ferromagnetic, accompanied by strongly intermittent emission dynamics characteristic of first-order coexistence between dynamical phases. We show that for a range of parameters close to this transition or crossover point the dynamics of the finite system displays pronounced metastability, i.e., the system relaxes first to long-lived metastable states before eventual relaxation to the true stationary state. From the spectral properties of the quantum master operator we characterize the low-dimensional manifold of metastable states, which are shown to be probability mixtures of two, paramagnetic and ferromagnetic, metastable phases. We also show that for long times the dynamics can be approximated by a classical stochastic dynamics between the metastable phases that is directly related to the intermittent dynamics observed in quantum trajectories and thus the dynamical phases.