Hebbian dreaming for small datasets.

The dreaming Hopfield model constitutes a generalization of the Hebbian paradigm for neural networks, that is able to perform on-line learning when "awake" and also to account for off-line "sleeping" mechanisms. The latter have been shown to enhance storing in such a way that, in the long sleep-time limit, this model can reach the maximal storage capacity achievable by networks equipped with symmetric pairwise interactions. In this paper, we inspect the minimal amount of information that must be supplied to such a network to guarantee a successful generalization, and we test it both on random synthetic and on standard structured datasets (i.e., MNIST, Fashion-MNIST and Olivetti). By comparing these minimal thresholds of information with those required by the standard (i.e., always "awake") Hopfield model, we prove that the present network can save up to ∼90% of the dataset size, yet preserving the same performance of the standard counterpart. This suggests that sleep may play a pivotal role in explaining the gap between the large volumes of data required to train artificial neural networks and the relatively small volumes needed by their biological counterparts. Further, we prove that the model Cost function (typically used in statistical mechanics) admits a representation in terms of a standard Loss function (typically used in machine learning) and this allows us to analyze its emergent computational skills both theoretically and computationally: a quantitative picture of its capabilities as a function of its control parameters is achieved and consistency between the two approaches is highlighted. The resulting network is an associative memory for pattern recognition tasks that learns from examples on-line, generalizes correctly (in suitable regions of its control parameters) and optimizes its storage capacity by off-line sleeping: such a reduction of the training cost can be inspiring toward sustainable AI and in situations where data are relatively sparse.

From Pavlov Conditioning to Hebb Learning.

Hebb's learning traces its origin in Pavlov's classical conditioning; however, while the former has been extensively modeled in the past decades (e.g., by the Hopfield model and countless variations on theme), as for the latter, modeling has remained largely unaddressed so far. Furthermore, a mathematical bridge connecting these two pillars is totally lacking. The main difficulty toward this goal lies in the intrinsically different scales of the information involved: Pavlov's theory is about correlations between concepts that are (dynamically) stored in the synaptic matrix as exemplified by the celebrated experiment starring a dog and a ringing bell; conversely, Hebb's theory is about correlations between pairs of neurons as summarized by the famous statement that neurons that fire together wire together. In this letter, we rely on stochastic process theory to prove that as long as we keep neurons' and synapses' timescales largely split, Pavlov's mechanism spontaneously takes place and ultimately gives rise to synaptic weights that recover the Hebbian kernel.

Outperforming RBM Feature-Extraction Capabilities by "Dreaming" Mechanism.

Inspired by a formal equivalence between the Hopfield model and restricted Boltzmann machines (RBMs), we design a Boltzmann machine, referred to as the dreaming Boltzmann machine (DBM), which achieves better performances than the standard one. The novelty in our model lies in a precise prescription for intralayer connections among hidden neurons whose strengths depend on features correlations. We analyze learning and retrieving capabilities in DBMs, both theoretically and numerically, and compare them to the RBM reference. We find that, in a supervised scenario, the former significantly outperforms the latter. Furthermore, in the unsupervised case, the DBM achieves better performances both in features extraction and representation learning, especially when the network is properly pretrained. Finally, we compare both models in simple classification tasks and find that the DBM again outperforms the RBM reference.

The emergence of a concept in shallow neural networks.

We consider restricted Boltzmann machine (RBMs) trained over an unstructured dataset made of blurred copies of definite but unavailable "archetypes" and we show that there exists a critical sample size beyond which the RBM can learn archetypes, namely the machine can successfully play as a generative model or as a classifier, according to the operational routine. In general, assessing a critical sample size (possibly in relation to the quality of the dataset) is still an open problem in machine learning. Here, restricting to the random theory, where shallow networks suffice and the "grandmother-cell" scenario is correct, we leverage the formal equivalence between RBMs and Hopfield networks, to obtain a phase diagram for both the neural architectures which highlights regions, in the space of the control parameters (i.e., number of archetypes, number of neurons, size and quality of the training set), where learning can be accomplished. Our investigations are led by analytical methods based on the statistical-mechanics of disordered systems and results are further corroborated by extensive Monte Carlo simulations.

On the effective initialisation for restricted Boltzmann machines via duality with Hopfield model.

Restricted Boltzmann machines (RBMs) with a binary visible layer of size N and a Gaussian hidden layer of size P have been proved to be equivalent to a Hopfield neural network (HNN) made of N binary neurons and storing P patterns ξ, as long as the weights w in the former are identified with the patterns. Here we aim to leverage this equivalence to find effective initialisations for weights in the RBM when what is available is a set of noisy examples of each pattern, aiming to translate statistical mechanics background available for HNN to the study of RBM's learning and retrieval abilities. In particular, given a set of definite, structureless patterns we build a sample of blurred examples and prove that the initialisation where w corresponds to the empirical average ξ¯ over the sample is a fixed point under stochastic gradient descent. Further, as a toy application of the duality between HNN and RBM, we consider the simplest random auto-encoder (a three layer network made of two RBMs coupled by their hidden layer) and evidence that, as long as the parameter setting corresponds to the retrieval region of the dual HNN, reconstruction and denoising can be accomplished trivially, while when the system is in the spin-glass phase inference algorithms are necessary. This questions the need for larger retrieval regions which we obtain by applying a Gram-Schmidt orthogonalisation to the patterns: in fact, this procedure yields to a set of patterns devoid of correlations and for which the largest retrieval region can be accomplished. Finally we consider an application of duality also in a structured case: we test this approach on the MNIST dataset, and obtain that the network performs already ∼67% of successful classifications, suggesting it can be exploited as a computationally-cheap pre-training.

Analysis of temporal correlation in heart rate variability through maximum entropy principle in a minimal pairwise glassy model.

In this work we apply statistical mechanics tools to infer cardiac pathologies over a sample of M patients whose heart rate variability has been recorded via 24 h Holter device and that are divided in different classes according to their clinical status (providing a repository of labelled data). Considering the set of inter-beat interval sequences [Formula: see text], with [Formula: see text], we estimate their probability distribution [Formula: see text] exploiting the maximum entropy principle. By setting constraints on the first and on the second moment we obtain an effective pairwise [Formula: see text] model, whose parameters are shown to depend on the clinical status of the patient. In order to check this framework, we generate synthetic data from our model and we show that their distribution is in excellent agreement with the one obtained from experimental data. Further, our model can be related to a one-dimensional spin-glass with quenched long-range couplings decaying with the spin-spin distance as a power-law. This allows us to speculate that the 1/f noise typical of heart-rate variability may stem from the interplay between the parasympathetic and orthosympathetic systems.

Detecting cardiac pathologies via machine learning on heart-rate variability time series and related markers.

In this paper we develop statistical algorithms to infer possible cardiac pathologies, based on data collected from 24 h Holter recording over a sample of 2829 labelled patients; labels highlight whether a patient is suffering from cardiac pathologies. In the first part of the work we analyze statistically the heart-beat series associated to each patient and we work them out to get a coarse-grained description of heart variability in terms of 49 markers well established in the reference community. These markers are then used as inputs for a multi-layer feed-forward neural network that we train in order to make it able to classify patients. However, before training the network, preliminary operations are in order to check the effective number of markers (via principal component analysis) and to achieve data augmentation (because of the broadness of the input data). With such groundwork, we finally train the network and show that it can classify with high accuracy (at most ~85% successful identifications) patients that are healthy from those displaying atrial fibrillation or congestive heart failure. In the second part of the work, we still start from raw data and we get a classification of pathologies in terms of their related networks: patients are associated to nodes and links are drawn according to a similarity measure between the related heart-beat series. We study the emergent properties of these networks looking for features (e.g., degree, clustering, clique proliferation) able to robustly discriminate between networks built over healthy patients or over patients suffering from cardiac pathologies. We find overall very good agreement among the two paved routes.

Generalized Guerra's interpolation schemes for dense associative neural networks.

In this work we develop analytical techniques to investigate a broad class of associative neural networks set in the high-storage regime. These techniques translate the original statistical-mechanical problem into an analytical-mechanical one which implies solving a set of partial differential equations, rather than tackling the canonical probabilistic route. We test the method on the classical Hopfield model - where the cost function includes only two-body interactions (i.e., quadratic terms) - and on the "relativistic" Hopfield model - where the (expansion of the) cost function includes p-body (i.e., of degree p) contributions. Under the replica symmetric assumption, we paint the phase diagrams of these models by obtaining the explicit expression of their free energy as a function of the model parameters (i.e., noise level and memory storage). Further, since for non-pairwise models ergodicity breaking is non necessarily a critical phenomenon, we develop a fluctuation analysis and find that criticality is preserved in the relativistic model.

A statistical inference approach to reconstruct intercellular interactions in cell migration experiments.

Migration of cells can be characterized by two prototypical types of motion: individual and collective migration. We propose a statistical inference approach designed to detect the presence of cell-cell interactions that give rise to collective behaviors in cell motility experiments. This inference method has been first successfully tested on synthetic motional data and then applied to two experiments. In the first experiment, cells migrate in a wound-healing model: When applied to this experiment, the inference method predicts the existence of cell-cell interactions, correctly mirroring the strong intercellular contacts that are present in the experiment. In the second experiment, dendritic cells migrate in a chemokine gradient. Our inference analysis does not provide evidence for interactions, indicating that cells migrate by sensing independently the chemokine source. According to this prediction, we speculate that mature dendritic cells disregard intercellular signals that could otherwise delay their arrival to lymph vessels.

Neural Networks with a Redundant Representation: Detecting the Undetectable.

We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order P=4. The latter is known to be able to Hebbian store an amount of patterns scaling as N^{P-1}, where N denotes the number of constituting binary neurons interacting P wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with N, while P>2) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with P=4 is able to retrieve information whose intensity is O(1) even in the presence of a noise O(sqrt[N]) in the large N limit. This striking skill stems from a redundancy representation of patterns-which is afforded given the (relatively) low-load information storage-and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations.

Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks.

The Hopfield model and the Boltzmann machine are among the most popular examples of neural networks. The latter, widely used for classification and feature detection, is able to efficiently learn a generative model from observed data and constitutes the benchmark for statistical learning. The former, designed to mimic the retrieval phase of an artificial associative memory lays in between two paradigmatic statistical mechanics models, namely the Curie-Weiss and the Sherrington-Kirkpatrick, which are recovered as the limiting cases of, respectively, one and many stored memories. Interestingly, the Boltzmann machine and the Hopfield network, if considered to be two cognitive processes (learning and information retrieval), are nothing more than two sides of the same coin. In fact, it is possible to exactly map the one into the other. We will inspect such an equivalence retracing the most representative steps of the research in this field.

Exact results for the first-passage properties in a class of fractal networks.

In this work, we consider a class of recursively grown fractal networks Gn(t) whose topology is controlled by two integer parameters, t and n. We first analyse the structural properties of Gn(t) (including fractal dimension, modularity, and clustering coefficient), and then we move to its transport properties. The latter are studied in terms of first-passage quantities (including the mean trapping time, the global mean first-passage time, and Kemeny's constant), and we highlight that their asymptotic behavior is controlled by the network's size and diameter. Remarkably, if we tune n (or, analogously, t) while keeping the network size fixed, as n increases (t decreases) the network gets more and more clustered and modular while its diameter is reduced, implying, ultimately, a better transport performance. The connection between this class of networks and models for polymer architectures is also discussed.

Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones.

The standard Hopfield model for associative neural networks accounts for biological Hebbian learning and acts as the harmonic oscillator for pattern recognition, however its maximal storage capacity is α∼0.14, far from the theoretical bound for symmetric networks, i.e. α=1. Inspired by sleeping and dreaming mechanisms in mammal brains, we propose an extension of this model displaying the standard on-line (awake) learning mechanism (that allows the storage of external information in terms of patterns) and an off-line (sleep) unlearning&consolidating mechanism (that allows spurious-pattern removal and pure-pattern reinforcement): this obtained daily prescription is able to saturate the theoretical bound α=1, remaining also extremely robust against thermal noise. The emergent neural and synaptic features are analyzed both analytically and numerically. In particular, beyond obtaining a phase diagram for neural dynamics, we focus on synaptic plasticity and we give explicit prescriptions on the temporal evolution of the synaptic matrix. We analytically prove that our algorithm makes the Hebbian kernel converge with high probability to the projection matrix built over the pure stored patterns. Furthermore, we obtain a sharp and explicit estimate for the "sleep rate" in order to ensure such a convergence. Finally, we run extensive numerical simulations (mainly Monte Carlo sampling) to check the approximations underlying the analytical investigations (e.g., we developed the whole theory at the so called replica-symmetric level, as standard in the Amit-Gutfreund-Sompolinsky reference framework) and possible finite-size effects, finding overall full agreement with the theory.

First encounters on combs.

We consider two random walkers embedded in a finite, two-dimension comb and we study the mean first-encounter time (MFET) evidencing (mainly numerically) different scalings with the linear size of the underlying network according to the initial position of the walkers. If one of the two players is not allowed to move, then the first-encounter problem can be recast into a first-passage problem (MFPT) for which we also obtain exact results for different initial configurations. By comparing MFET and MFPT, we are able to figure out possible search strategies and, in particular, we show that letting one player be fixed can be convenient to speed up the search as long as we can finely control the initial setting, while, for a random setting, on average, letting one player rest would slow down the search.

Modeling ozone uptake by urban and peri-urban forest: a case study in the Metropolitan City of Rome.

Urban and peri-urban forests are green infrastructures (GI) that play a substantial role in delivering ecosystem services such as the amelioration of air quality by the removal of air pollutants, among which is ozone (O3), which is the most harmful pollutant in Mediterranean metropolitan areas. Models may provide a reliable estimate of gas exchanges between vegetation and atmosphere and are thus a powerful tool to quantify and compare O3 removal in different contexts. The present study modeled the O3 stomatal uptake at canopy level of an urban and a peri-urban forest in the Metropolitan City of Rome in two different years. Results show different rates of O3 fluxes between the two forests, due to different exposure to the pollutant, management practice effects on forest structure and functionality, and environmental conditions, namely, different stressors affecting the gas exchange rates of the two GIs. The periodic components of the time series calculated by means of the spectral analysis show that seasonal variation of modeled canopy transpiration is driven by precipitation in peri-urban forests, whereas in the urban forest seasonal variations are driven by vapor pressure deficit of ambient air. Moreover, in the urban forest high water availability during summer months, owing to irrigation practice, leads to an increase in O3 uptake, thus suggesting that irrigation may enhance air phytoremediation in urban areas.

Organs on chip approach: a tool to evaluate cancer -immune cells interactions.

In this paper we discuss the applicability of numerical descriptors and statistical physics concepts to characterize complex biological systems observed at microscopic level through organ on chip approach. To this end, we employ data collected on a microfluidic platform in which leukocytes can move through suitably built channels toward their target. Leukocyte behavior is recorded by standard time lapse imaging. In particular, we analyze three groups of human peripheral blood mononuclear cells (PBMC): heterozygous mutants (in which only one copy of the FPR1 gene is normal), homozygous mutants (in which both alleles encoding FPR1 are loss-of-function variants) and cells from 'wild type' donors (with normal expression of FPR1). We characterize the migration of these cells providing a quantitative confirmation of the essential role of FPR1 in cancer chemotherapy response. Indeed wild type PBMC perform biased random walks toward chemotherapy-treated cancer cells establishing persistent interactions with them. Conversely, heterozygous mutants present a weaker bias in their motion and homozygous mutants perform rather uncorrelated random walks, both failing to engage with their targets. We next focus on wild type cells and study the interactions of leukocytes with cancerous cells developing a novel heuristic procedure, inspired by Lyapunov stability in dynamical systems.

Scaling laws for diffusion on (trans)fractal scale-free networks.

Fractal (or transfractal) features are common in real-life networks and are known to influence the dynamic processes taking place in the network itself. Here, we consider a class of scale-free deterministic networks, called (u, v)-flowers, whose topological properties can be controlled by tuning the parameters u and v; in particular, for u > 1, they are fractals endowed with a fractal dimension df, while for u = 1, they are transfractal endowed with a transfractal dimension d̃f. In this work, we investigate dynamic processes (i.e., random walks) and topological properties (i.e., the Laplacian spectrum) and we show that, under proper conditions, the same scalings (ruled by the related dimensions) emerge for both fractal and transfractal dimensions.

The exact Laplacian spectrum for the Dyson hierarchical network.

We consider the Dyson hierarchical graph , that is a weighted fully-connected graph, where the pattern of weights is ruled by the parameter σ ∈ (1/2, 1]. Exploiting the deterministic recursivity through which is built, we are able to derive explicitly the whole set of the eigenvalues and the eigenvectors for its Laplacian matrix. Given that the Laplacian operator is intrinsically implied in the analysis of dynamic processes (e.g., random walks) occurring on the graph, as well as in the investigation of the dynamical properties of connected structures themselves (e.g., vibrational structures and relaxation modes), this result allows addressing analytically a large class of problems. In particular, as examples of applications, we study the random walk and the continuous-time quantum walk embedded in , the relaxation times of a polymer whose structure is described by , and the community structure of in terms of modularity measures.

Complete integrability of information processing by biochemical reactions.

Statistical mechanics provides an effective framework to investigate information processing in biochemical reactions. Within such framework far-reaching analogies are established among (anti-) cooperative collective behaviors in chemical kinetics, (anti-)ferromagnetic spin models in statistical mechanics and operational amplifiers/flip-flops in cybernetics. The underlying modeling - based on spin systems - has been proved to be accurate for a wide class of systems matching classical (e.g. Michaelis-Menten, Hill, Adair) scenarios in the infinite-size approximation. However, the current research in biochemical information processing has been focusing on systems involving a relatively small number of units, where this approximation is no longer valid. Here we show that the whole statistical mechanical description of reaction kinetics can be re-formulated via a mechanical analogy - based on completely integrable hydrodynamic-type systems of PDEs - which provides explicit finite-size solutions, matching recently investigated phenomena (e.g. noise-induced cooperativity, stochastic bi-stability, quorum sensing). The resulting picture, successfully tested against a broad spectrum of data, constitutes a neat rationale for a numerically effective and theoretically consistent description of collective behaviors in biochemical reactions.

Two-particle problem in comblike structures.

Encounters between walkers performing a random motion on an appropriate structure can describe a wide variety of natural phenomena ranging from pharmacokinetics to foraging. On homogeneous structures the asymptotic encounter probability between two walkers is (qualitatively) independent of whether both walkers are moving or one is kept fixed. On infinite comblike structures this is no longer the case and here we deepen the mechanisms underlying the emergence of a finite probability that two random walkers will never meet, while one single random walker is certain to visit any site. In particular, we introduce an analytical approach to address this problem and even more general problems such as the case of two walkers with different diffusivity, particles walking on a finite comb and on arbitrary bundled structures, possibly in the presence of loops. Our investigations are both analytical and numerical and highlight that, in general, the outcome of a reaction involving two reactants on a comblike architecture can strongly differ according to whether both reactants are moving (no matter their relative diffusivities) or only one is moving and according to the density of shortcuts among the branches.