Emergence of Compositional Representations in Restricted Boltzmann Machines.

Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine-learning tasks. Restricted Boltzmann machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits data set MNIST.

Neural assemblies revealed by inferred connectivity-based models of prefrontal cortex recordings.

We present two graphical model-based approaches to analyse the distribution of neural activities in the prefrontal cortex of behaving rats. The first method aims at identifying cell assemblies, groups of synchronously activating neurons possibly representing the units of neural coding and memory. A graphical (Ising) model distribution of snapshots of the neural activities, with an effective connectivity matrix reproducing the correlation statistics, is inferred from multi-electrode recordings, and then simulated in the presence of a virtual external drive, favoring high activity (multi-neuron) configurations. As the drive increases groups of neurons may activate together, and reveal the existence of cell assemblies. The identified groups are then showed to strongly coactivate in the neural spiking data and to be highly specific of the inferred connectivity network, which offers a sparse representation of the correlation pattern across neural cells. The second method relies on the inference of a Generalized Linear Model, in which spiking events are integrated over time by neurons through an effective connectivity matrix. The functional connectivity matrices inferred with the two approaches are compared. Sampling of the inferred GLM distribution allows us to study the spatio-temporal patterns of activation of neurons within the identified cell assemblies, particularly their activation order: the prevalence of one order with respect to the others is weak and reflects the neuron average firing rates and the strength of the largest effective connections. Other properties of the identified cell assemblies (spatial distribution of coactivation events and firing rates of coactivating neurons) are discussed.

Transitions between Spatial Attractors in Place-Cell Models.

The spontaneous transitions between D-dimensional spatial maps in an attractor neural network are studied. Two scenarios for the transition from one map to another are found, depending on the level of noise: (i) through a mixed state, partly localized in both maps around positions where the maps are most similar, and (ii) through a weakly localized state in one of the two maps, followed by a condensation in the arrival map. Our predictions are confirmed by numerical simulations and qualitatively compared to recent recordings of hippocampal place cells during quick-environment-changing experiments in rats.

Large pseudocounts and L2-norm penalties are necessary for the mean-field inference of Ising and Potts models.

The mean-field (MF) approximation offers a simple, fast way to infer direct interactions between elements in a network of correlated variables, a common, computationally challenging problem with practical applications in fields ranging from physics and biology to the social sciences. However, MF methods achieve their best performance with strong regularization, well beyond Bayesian expectations, an empirical fact that is poorly understood. In this work, we study the influence of pseudocount and L(2)-norm regularization schemes on the quality of inferred Ising or Potts interaction networks from correlation data within the MF approximation. We argue, based on the analysis of small systems, that the optimal value of the regularization strength remains finite even if the sampling noise tends to zero, in order to correct for systematic biases introduced by the MF approximation. Our claim is corroborated by extensive numerical studies of diverse model systems and by the analytical study of the m-component spin model for large but finite m. Additionally, we find that pseudocount regularization is robust against sampling noise and often outperforms L(2)-norm regularization, particularly when the underlying network of interactions is strongly heterogeneous. Much better performances are generally obtained for the Ising model than for the Potts model, for which only couplings incoming onto medium-frequency symbols are reliably inferred.

Stochastic ratchet mechanisms for replacement of proteins bound to DNA.

Experiments indicate that unbinding rates of proteins from DNA can depend on the concentration of proteins in nearby solution. Here we present a theory of multistep replacement of DNA-bound proteins by solution-phase proteins. For four different kinetic scenarios we calculate the dependence of protein unbinding and replacement rates on solution protein concentration. We find (1) strong effects of progressive "rezipping" of the solution-phase protein onto DNA sites liberated by "unzipping" of the originally bound protein, (2) that a model in which solution-phase proteins bind nonspecifically to DNA can describe experiments on exchanges between the nonspecific DNA-binding proteins Fis-Fis and Fis-HU, and (3) that a binding specific model describes experiments on the exchange of CueR proteins on specific binding sites.

Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: collective motion of the activity.

The dynamics of a neural model for hippocampal place cells storing spatial maps is studied. In the absence of external input, depending on the number of cells and on the values of control parameters (number of environments stored, level of neural noise, average level of activity, connectivity of place cells), a "clump" of spatially localized activity can diffuse or remains pinned due to crosstalk between the environments. In the single-environment case, the macroscopic coefficient of diffusion of the clump and its effective mobility are calculated analytically from first principles and corroborated by numerical simulations. In the multienvironment case the heights and the widths of the pinning barriers are analytically characterized with the replica method; diffusion within one map is then in competition with transitions between different maps. Possible mechanisms enhancing mobility are proposed and tested.

Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: phase diagram.

We study the stable phases of an attractor neural network model, with binary units, for hippocampal place cells encoding one-dimensional (1D) or 2D spatial maps or environments. Different maps correspond to random allocations (permutations) of the place fields. Based on replica calculations we show that, below critical levels for the noise in the neural response and for the number of environments, the network activity is spatially localized in one environment. For high noise and loads the network activity extends over space, either uniformly or with spatial heterogeneities due to the crosstalk between the maps, and memory of environments is lost. Remarkably the spatially localized regime is very robust against the neural noise until it reaches its critical level. Numerical simulations are in excellent quantitative agreement with our theoretical predictions.

High-dimensional inference with the generalized Hopfield model: principal component analysis and corrections.

We consider the problem of inferring the interactions between a set of N binary variables from the knowledge of their frequencies and pairwise correlations. The inference framework is based on the Hopfield model, a special case of the Ising model where the interaction matrix is defined through a set of patterns in the variable space, and is of rank much smaller than N. We show that maximum likelihood inference is deeply related to principal component analysis when the amplitude of the pattern components ξ is negligible compared to √N. Using techniques from statistical mechanics, we calculate the corrections to the patterns to the first order in ξ/√N. We stress the need to generalize the Hopfield model and include both attractive and repulsive patterns in order to correctly infer networks with sparse and strong interactions. We present a simple geometrical criterion to decide how many attractive and repulsive patterns should be considered as a function of the sampling noise. We moreover discuss how many sampled configurations are required for a good inference, as a function of the system size N and of the amplitude ξ. The inference approach is illustrated on synthetic and biological data.

Adaptive cluster expansion for inferring boltzmann machines with noisy data.

We introduce a procedure to infer the interactions among a set of binary variables, based on their sampled frequencies and pairwise correlations. The algorithm builds the clusters of variables contributing most to the entropy of the inferred Ising model and rejects the small contributions due to the sampling noise. Our procedure successfully recovers benchmark Ising models even at criticality and in the low temperature phase, and is applied to neurobiological data.

Inferring DNA sequences from mechanical unzipping data: the large-bandwidth case.

The complementary strands of DNA molecules can be separated when stretched apart by a force; the unzipping signal is correlated to the base content of the sequence but is affected by thermal and instrumental noise. We consider here the ideal case where opening events are known to a very good time resolution (very large bandwidth), and study how the sequence can be reconstructed from the unzipping data. Our approach relies on the use of statistical Bayesian inference and of Viterbi decoding algorithm. Performances are studied numerically on Monte Carlo generated data, and analytically. We show how multiple unzippings of the same molecule may be exploited to improve the quality of the prediction, and calculate analytically the number of required unzippings as a function of the bandwidth, the sequence content, and the elasticity parameters of the unzipped strands.

Inference of DNA sequences from mechanical unzipping: an ideal-case study.

The performances of Bayesian inference to predict the sequence of DNA molecules from fixed-force unzipping experiments are investigated. We show that the probability of misprediction decreases exponentially with the amount of collected data. The decay rate is calculated as a function of biochemical parameters (binding free energies), the sequence content, the applied force, the elastic properties of a DNA single strand, and time resolution.

Rigorous decimation-based construction of ground pure states for spin-glass models on random lattices.

A constructive scheme for determining pure states at very low temperature in the 3-spins glass model on a random lattice is provided, in full agreement with Parisi's one step replica symmetry breaking (RSB) scheme. Proof is based on the analysis of a partial decimation procedure and of the statistical properties of its output, i.e., a reduced Hamiltonian acting on a subset of the initial spins. The number of ground states (GS) in each state, the number of states, and the distances between GS are calculated and correspond to RSB predictions.

Slow nucleic acid unzipping kinetics from sequence-defined barriers.

Recent experiments on unzipping of RNA helix-loop structures by force have shown that approximately 40-base molecules can undergo kinetic transitions between two well-defined "open" and "closed" states, on a timescale approximately 1 sec [Liphardt et al., Science 297, 733-737 (2001)]. Using a simple dynamical model, we show that these phenomena result from the slow kinetics of crossing large free energy barriers which separate the open and closed conformations. The dependence of barriers on sequence along the helix, and on the size of the loop(s) is analyzed. Some DNA and RNA sequences that could show dynamics on different time scales, or three(or more)-state unzipping, are proposed. Our dynamical model is also applied to the unzipping of long (kilo-basepair) DNA molecules at constant force.

Force-extension behavior of folding polymers.

The elastic response of flexible polymers made of elements which can be either folded or unfolded, having different lengths in these two states, is discussed. These situations are common for biopolymers as a result of folding interactions intrinsic to the monomers, or as a result of binding of other smaller molecules along the polymer length. Using simple flexible-chain models, we show that even when the energy epsilon associated with maintaining the folded state is comparable to k(B) T, the elastic response of such a chain can mimic usual polymer linear elasticity, but with a force scale enhanced above that expected from the flexibility of the chain backbone. We discuss recent experiments on single-stranded DNA, chromatin fiber and double-stranded DNA with proteins weakly absorbed along its length which show this effect. Effects of polymer semiflexiblity and torsional stiffness relevant to experiments on proteins binding to dsDNA are analyzed. We finally discuss the competition between electrostatic self-repulsion and folding interactions responsible for the complex elastic response of single-stranded DNA.

Force and kinetic barriers to unzipping of the DNA double helix.

A theory of the unzipping of double-stranded DNA is presented and is compared to recent micromanipulation experiments. It is shown that the interactions that stabilize the double helix and the elastic rigidity of single strands simply determine the sequence-dependent approximately 12-pN force threshold for DNA strand separation. Using a semimicroscopic model of the binding between nucleotide strands, we show that the greater rigidity of the strands when formed into double-stranded DNA, relative to that of isolated strands, gives rise to a potential barrier to unzipping. The effects of this barrier are derived analytically. The force to keep the extremities of the molecule at a fixed distance, the kinetic rates for strand unpairing at fixed applied force, and the rupture force as a function of loading rate are calculated. The dependence of the kinetics and of the rupture force on molecule length is also analyzed.

Trajectories in phase diagrams, growth processes, and computational complexity: how search algorithms solve the 3-satisfiability problem.

Decision and optimization problems typically fall into one of two categories for any particular solving algorithm. The problem is either solved quickly (easy) or demands an impractically long computational effort (hard). Here we show that some characteristic parameters of problems can be tracked during a run of the algorithm defining a trajectory through the parameter space. Focusing on 3-satisfiability, a recognized representative of hard problems, we analyze trajectories generated by search algorithms. These trajectories can cross well-defined phases, corresponding to domains of easy or hard instances, and allow one to successfully predict the times of resolution.