Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations.

We present a framework to integrate tensor network (TN) methods with reinforcement learning (RL) for solving dynamical optimization tasks. We consider the RL actor-critic method, a model-free approach for solving RL problems, and introduce TNs as the approximators for its policy and value functions. Our "actor-critic with tensor networks" (ACTeN) method is especially well suited to problems with large and factorizable state and action spaces. As an illustration of the applicability of ACTeN we solve the exponentially hard task of sampling rare trajectories in two paradigmatic stochastic models, the East model of glasses and the asymmetric simple exclusion process, the latter being particularly challenging to other methods due to the absence of detailed balance. With substantial potential for further integration with the vast array of existing RL methods, the approach introduced here is promising both for applications in physics and to multi-agent RL problems more generally.

Universal and nonuniversal probability laws in Markovian open quantum dynamics subject to generalized reset processes.

We consider quantum-jump trajectories of Markovian open quantum systems subject to stochastic in time resets of their state to an initial configuration. The reset events provide a partitioning of quantum trajectories into consecutive time intervals, defining sequences of random variables from the values of a trajectory observable within each of the intervals. For observables related to functions of the quantum state, we show that the probability of certain orderings in the sequences obeys a universal law. This law does not depend on the chosen observable and, in the case of Poissonian reset processes, not even on the details of the dynamics. When considering (discrete) observables associated with the counting of quantum jumps, the probabilities in general lose their universal character. Universality is only recovered in cases when the probability of observing equal outcomes in the same sequence is vanishingly small, which we can achieve in a weak-reset-rate limit. Our results extend previous findings on classical stochastic processes [N. R. Smith et al., Europhys. Lett. 142, 51002 (2023)0295-507510.1209/0295-5075/acd79e] to the quantum domain and to state-dependent reset processes, shedding light on relevant aspects for the emergence of universal probability laws.

Topological phases in the dynamics of the simple exclusion process.

We study the dynamical large deviations of the classical stochastic symmetric simple exclusion process (SSEP) by means of numerical matrix product states. We show that for half filling, long-time trajectories with a large enough imbalance between the number hops in even and odd bonds of the lattice belong to distinct symmetry-protected topological (SPT) phases. Using tensor network techniques, we obtain the large deviation (LD) phase diagram in terms of counting fields conjugate to the dynamical activity and the total hop imbalance. We show the existence of high activity trivial and nontrivial SPT phases (classified according to string order parameters) separated by either a critical phase or a critical point. Using the leading eigenstate of the tilted generator, obtained from infinite-system density-matrix renormalization group simulations, we construct a near-optimal dynamics for sampling the LDs, and show that the SPT phases manifest at the level of rare stochastic trajectories. We also show how to extend these results to other filling fractions, and discuss generalizations to asymmetric SEPs.

Exact Quench Dynamics of the Floquet Quantum East Model at the Deterministic Point.

We study the nonequilibrium dynamics of the Floquet quantum East model (a Trotterized version of the kinetically constrained quantum East spin chain) at its "deterministic point," where evolution is defined in terms of CNOT permutation gates. We solve exactly the thermalization dynamics for a broad class of initial product states by means of "space evolution." We prove: (i) the entanglement of a block of spins grows at most at one-half the maximal speed allowed by locality (i.e., half the speed of dual-unitary circuits); (ii) if the block of spins is initially prepared in a classical configuration, speed of entanglement is a quarter of the maximum; (iii) thermalization to the infinite temperature state is reached exactly in a time that scales with the size of the block.

General Upper Bounds on Fluctuations of Trajectory Observables.

Thermodynamic uncertainty relations (TURs) are general lower bounds on the size of fluctuations of dynamical observables. They have important consequences, one being that the precision of estimation of a current is limited by the amount of entropy production. Here, we prove the existence of general upper bounds on the size of fluctuations of any linear combination of fluxes (including all time-integrated currents or dynamical activities) for continuous-time Markov chains. We obtain these general relations by means of concentration bound techniques. These "inverse TURs" are valid for all times and not only in the long time limit. We illustrate our analytical results with a simple model, and discuss wider implications of these new relations.

Slow dynamics and nonergodicity of the bosonic quantum East model in the semiclassical limit.

We study the unitary dynamics of the bosonic quantum East model, a kinetically constrained lattice model which generalizes the quantum East model to arbitrary occupation per site. We consider the semiclassical limit of large (but finite) site occupancy so that the dynamics is approximated by an evolution equation of the Gross-Pitaevskii kind. This allows us to numerically study in detail system sizes of hundreds of sites. Like in the spin-1/2 case, we find two dynamical phases, an active one of fast thermalization and an inactive one of slow relaxation and the absence of ergodicity on numerically accessible timescales. The location of this apparent ergodic to nonergodic transition coincides with the localization transition of the ground state. We further characterize states which are nonergodic on all timescales in the otherwise ergodic regime.

Reply to "Comment on 'Exact large deviation statistics and trajectory phase transition of a deterministic boundary driven cellular automaton' ".

We reply to Whitelam's Comment [Phys. Rev. E 108, 036105 (2023)2470-004510.1103/PhysRevE.108.036105] on our paper [Phys. Rev. E 100, 020103(R) (2019)2470-004510.1103/PhysRevE.100.020103] where we compute the exact large deviation (LD) statistics of a wide class of observables in the rule 54 cellular automaton. Using some heuristic arguments, Whitelam states that despite the fact that the LD functions we compute display singular behavior, this is not indicative of a LD phase transition or of dynamical phase coexistence. Here, we refute this observation and confirm that the (standard) interpretation of our exact results stands.

Reaction-Limited Quantum Reaction-Diffusion Dynamics.

We consider the quantum nonequilibrium dynamics of systems where fermionic particles coherently hop on a one-dimensional lattice and are subject to dissipative processes analogous to those of classical reaction-diffusion models. Particles can either annihilate in pairs, A+A→0, or coagulate upon contact, A+A→A, and possibly also branch, A→A+A. In classical settings, the interplay between these processes and particle diffusion leads to critical dynamics as well as to absorbing-state phase transitions. Here, we analyze the impact of coherent hopping and of quantum superposition, focusing on the so-called reaction-limited regime. Here, spatial density fluctuations are quickly smoothed out due to fast hopping, which for classical systems is described by a mean-field approach. By exploiting the time-dependent generalized Gibbs ensemble method, we demonstrate that quantum coherence and destructive interference play a crucial role in these systems and are responsible for the emergence of locally protected dark states and collective behavior beyond mean field. This can manifest both at stationarity and during the relaxation dynamics. Our analytical results highlight fundamental differences between classical nonequilibrium dynamics and their quantum counterpart and show that quantum effects indeed change collective universal behavior.

Stochastic strong zero modes and their dynamical manifestations.

Strong zero modes (SZMs) are conserved operators localized at the edges of certain quantum spin chains, which give rise to long coherence times of edge spins. Here we define and analyze analogous operators in one-dimensional classical stochastic systems. For concreteness, we focus on chains with single occupancy and nearest-neighbor transitions, in particular particle hopping and pair creation and annihilation. For integrable choices of parameters we find the exact form of the SZM operators. Being in general nondiagonal in the classical basis, the dynamical consequences of stochastic SZMs are very different from those of their quantum counterparts. We show that the presence of a stochastic SZM is manifested through a class of exact relations between time-correlation functions, absent in the same system with periodic boundaries.

Generalized continuous Maxwell demons.

We introduce a family of generalized continuous Maxwell demons (GCMDs) operating on idealized single-bit equilibrium devices that combine the single-measurement Szilard and the repeated measurements of the continuous Maxwell demon protocols. We derive the cycle distributions for extracted work, information content, and time and compute the power and information-to-work efficiency fluctuations for the different models. We show that the efficiency at maximum power is maximal for an opportunistic protocol of continuous type in the dynamical regime dominated by rare events. We also extend the analysis to finite-time work extracting protocols by mapping them to a three-state GCMD. We show that dynamical finite-time correlations in this model increase the information-to-work conversion efficiency, underlining the role of temporal correlations in optimizing information-to-energy conversion. The effect of finite-time work extraction and demon memory resetting is also analyzed. We conclude that GCMD models are thermodynamically more efficient than the single-measurement Szilard and preferred for describing biological processes in an information-redundant world.

Optimal Sampling of Dynamical Large Deviations in Two Dimensions via Tensor Networks.

We use projected entangled-pair states (PEPS) to calculate the large deviation statistics of the dynamical activity of the two-dimensional East model, and the two-dimensional symmetric simple exclusion process (SSEP) with open boundaries, in lattices of up to 40×40 sites. We show that at long times both models have phase transitions between active and inactive dynamical phases. For the 2D East model we find that this trajectory transition is of the first order, while for the SSEP we find indications of a second order transition. We then show how the PEPS can be used to implement a trajectory sampling scheme capable of directly accessing rare trajectories. We also discuss how the methods described here can be extended to study rare events at finite times.

Trajectory phase transitions in non-interacting systems: all-to-all dynamics and the random energy model.

We study the fluctuations of time-additive random observables in the stochastic dynamics of a system of [Formula: see text] non-interacting Ising spins. We mainly consider the case of all-to-all dynamics where transitions are possible between any two spin configurations with uniform rates. We show that the cumulant generating function of the time-integral of a normally distributed quenched random function of configurations, i.e. the energy function of the random energy model (REM), has a phase transition in the large [Formula: see text] limit for trajectories of any time extent. We prove this by determining the exact limit of the scaled cumulant generating function. This is accomplished by connecting the dynamical problem to a spectral analysis of the all-to-all quantum REM. We also discuss finite [Formula: see text] corrections as observed in numerical simulations. This article is part of the theme issue 'Quantum annealing and computation: challenges and perspectives'.

Quantum reaction-limited reaction-diffusion dynamics of annihilation processes.

We investigate the quantum reaction-diffusion dynamics of fermionic particles which coherently hop in a one-dimensional lattice and undergo annihilation reactions. The latter are modelled as dissipative processes which involve losses of pairs 2A→∅, triplets 3A→∅, and quadruplets 4A→∅ of neighboring particles. When considering classical particles, the corresponding decay of their density in time follows an asymptotic power-law behavior. The associated exponent in one dimension is different from the mean-field prediction whenever diffusive mixing is not too strong and spatial correlations are relevant. This specifically applies to 2A→∅, while the mean-field power-law prediction just acquires a logarithmic correction for 3A→∅ and is exact for 4A→∅. A mean-field approach is also valid, for all the three processes, when the diffusive mixing is strong, i.e., in the so-called reaction-limited regime. Here we show that the picture is different for quantum systems. We consider the quantum reaction-limited regime and we show that for all the three processes power-law behavior beyond mean field is present as a consequence of quantum coherences, which are not related to space dimensionality. The decay in 3A→∅ is further, highly intricate, since the power-law behavior therein only appears within an intermediate time window, while at long times the density decay is not power law. Our results show that emergent critical behavior in quantum dynamics has a markedly different origin, based on quantum coherences, to that applying to classical critical phenomena, which is, instead, solely determined by the relevance of spatial correlations.

Slow dynamics and large deviations in classical stochastic Fredkin chains.

The Fredkin spin chain serves as an interesting theoretical example of a quantum Hamiltonian whose ground state exhibits a phase transition between three distinct phases, one of which violates the area law. Here we consider a classical stochastic version of the Fredkin model, which can be thought of as a simple exclusion process subject to additional kinetic constraints, and study its classical stochastic dynamics. The ground-state phase transition of the quantum chain implies an equilibrium phase transition in the stochastic problem, whose properties we quantify in terms of numerical matrix product states (MPSs). The stochastic model displays slow dynamics, including power-law decaying autocorrelation functions and hierarchical relaxation processes due to exponential localization. Like in other kinetically constrained models, the Fredkin chain has a rich structure in its dynamical large deviations-which we compute accurately via numerical MPSs-including an active-inactive phase transition and a hierarchy of trajectory phases connected to particular equilibrium states of the model. We also propose, via its height field representation, a generalization of the Fredkin model to two dimensions in terms of constrained dimer coverings of the honeycomb lattice.

Hierarchical classical metastability in an open quantum East model.

We study in detail an open quantum generalization of a classical kinetically constrained model-the East model-known to exhibit slow glassy dynamics stemming from a complex hierarchy of metastable states with distinct lifetimes. Using the recently introduced theory of classical metastability for open quantum systems, we show that the driven open quantum East model features a hierarchy of classical metastabilities at low temperature and weak driving field. We find that the effective long-time description of its dynamics not only is classical, but shares many properties with the classical East model, such as obeying an effective detailed balance condition and lacking static interactions between excitations, but with this occurring within a modified set of metastable phases which are coherent, and with an effective temperature that is dependent on the coherent drive.

Exact solution of the "Rule 150" reversible cellular automaton.

We study the dynamics and statistics of the Rule 150 reversible cellular automaton (RCA). This is a one-dimensional lattice system of binary variables with synchronous (Floquet) dynamics that corresponds to a bulk deterministic and reversible discretized version of the kinetically constrained "exclusive one-spin facilitated" (XOR) Fredrickson-Andersen (FA) model, where the local dynamics is restricted: A site flips if and only if its adjacent sites are in different states from each other. Similar to other RCA that have been recently studied, such as Rule 54 and Rule 201, the Rule 150 RCA is integrable, however, in contrast is noninteracting: The emergent quasiparticles, which are identified by the domain walls, behave as free fermions. This property allows us to solve the model by means of matrix product ansatz. In particular, we find the exact equilibrium and nonequilibrium stationary states for systems with closed (periodic) and open (stochastic) boundaries, respectively, resolve the full spectrum of the time evolution operator and, therefore, gain access to the relaxation dynamics, and obtain the exact large deviation statistics of dynamical observables in the long-time limit.

Finite Time Large Deviations via Matrix Product States.

Recent work has shown the effectiveness of tensor network methods for computing large deviation functions in constrained stochastic models in the infinite time limit. Here we show that these methods can also be used to study the statistics of dynamical observables at arbitrary finite time. This is a harder problem because, in contrast to the infinite time case, where only the extremal eigenstate of a tilted Markov generator is relevant, for finite time the whole spectrum plays a role. We show that finite time dynamical partition sums can be computed efficiently and accurately in one dimension using matrix product states and describe how to use such results to generate rare event trajectories on demand. We apply our methods to the Fredrickson-Andersen and East kinetically constrained models and to the symmetric simple exclusion process, unveiling dynamical phase diagrams in terms of counting field and trajectory time. We also discuss extensions of this method to higher dimensions.

Reinforcement learning of rare diffusive dynamics.

We present a method to probe rare molecular dynamics trajectories directly using reinforcement learning. We consider trajectories that are conditioned to transition between regions of configuration space in finite time, such as those relevant in the study of reactive events, and trajectories exhibiting rare fluctuations of time-integrated quantities in the long time limit, such as those relevant in the calculation of large deviation functions. In both cases, reinforcement learning techniques are used to optimize an added force that minimizes the Kullback-Leibler divergence between the conditioned trajectory ensemble and a driven one. Under the optimized added force, the system evolves the rare fluctuation as a typical one, affording a variational estimate of its likelihood in the original trajectory ensemble. Low variance gradients employing value functions are proposed to increase the convergence of the optimal force. The method we develop employing these gradients leads to efficient and accurate estimates of both the optimal force and the likelihood of the rare event for a variety of model systems.

Symmetry-induced fluctuation relations in open quantum systems.

We derive a general scheme to obtain quantum fluctuation relations for dynamical observables in open quantum systems. For concreteness, we consider Markovian nonunitary dynamics that is unraveled in terms of quantum jump trajectories and exploit techniques from the theory of large deviations like the tilted ensemble and the Doob transform. Our results here generalize to open quantum system fluctuation relations previously obtained for classical Markovian systems and add to the vast literature on fluctuation relations in the quantum domain, but without resorting to the standard two-point measurement scheme. We illustrate our findings with three examples to highlight and discuss the main features of our general result.

Optimal sampling of dynamical large deviations via matrix product states.

The large deviation statistics of dynamical observables is encoded in the spectral properties of deformed Markov generators. Recent works have shown that tensor network methods are well suited to compute accurately the relevant leading eigenvalues and eigenvectors. However, the efficient generation of the corresponding rare trajectories is a harder task. Here, we show how to exploit the matrix product state approximation of the dominant eigenvector to implement an efficient sampling scheme which closely resembles the optimal (so-called "Doob") dynamics that realizes the rare events. We demonstrate our approach on three well-studied lattice models, the Fredrickson-Andersen and East kinetically constrained models, and the symmetric simple exclusion process. We discuss how to generalize our approach to higher dimensions.