ABSTRACT
Modern deep neural networks (DNNs) represent a formidable challenge for theorists: according to the commonly accepted probabilistic framework that describes their performance, these architectures should overfit due to the huge number of parameters to train, but in practice they do not. Here we employ results from replica mean field theory to compute the generalization gap of machine learning models with quenched features, in the teacher-student scenario and for regression problems with quadratic loss function. Notably, this framework includes the case of DNNs where the last layer is optimized given a specific realization of the remaining weights. We show how these results-combined with ideas from statistical learning theory-provide a stringent asymptotic upper bound on the generalization gap of fully trained DNN as a function of the size of the dataset P. In particular, in the limit of large P and N_{out} (where N_{out} is the size of the last layer) and N_{out}âªP, the generalization gap approaches zero faster than 2N_{out}/P, for any choice of both architecture and teacher function. Notably, this result greatly improves existing bounds from statistical learning theory. We test our predictions on a broad range of architectures, from toy fully connected neural networks with few hidden layers to state-of-the-art deep convolutional neural networks.
ABSTRACT
A general method to determine covariant Lyapunov vectors in both discrete- and continuous-time dynamical systems is introduced. This allows us to address fundamental questions such as the degree of hyperbolicity, which can be quantified in terms of the transversality of these intrinsic vectors. For spatially extended systems, the covariant Lyapunov vectors have localization properties and spatial Fourier spectra qualitatively different from those composing the orthonormalized basis obtained in the standard procedure used to calculate the Lyapunov exponents.
ABSTRACT
It is argued that some phase transitions observed in models of nonequilibrium wetting phenomena are related to contact processes with long-range interactions. This is investigated by introducing a model where the activation rate of a site at the edge of an inactive island of length l is 1+a l(-sigma) . Mean-field analysis and numerical simulations indicate that for sigma>1 the transition is continuous and belongs to the universality class of directed percolation, while for 0
ABSTRACT
We study the nature of the synchronization transition in spatially extended systems by discussing a simple stochastic model. An analytic argument is put forward showing that, in the limit of discontinuous processes, the transition belongs to the directed percolation (DP) universality class. The analysis is complemented by a detailed investigation of the dependence of the first passage time for the amplitude of the difference field on the adopted threshold. We find the existence of a critical threshold separating the regime controlled by linear mechanisms from that controlled by collective phenomena. As a result of this analysis, we conclude that the synchronization transition belongs to the DP class also in continuous models. The conclusions are supported by numerical checks on coupled map lattices too.
ABSTRACT
A simple one-dimensional microscopic model of the depinning transition of an interface from an attractive hard wall is introduced and investigated. Upon varying a control parameter, the critical behavior observed along the transition line changes from a directed-percolation type to a multiplicative-noise type. Numerical simulations allow for a quantitative study of the multicritical point separating the two regions. Mean-field arguments and the mapping on yet a simpler model provide some further insight on the overall scenario.
