Frequency patterns of semantic change: corpus-based evidence of a near-critical dynamics in language change.

It is generally believed that when a linguistic item acquires a new meaning, its overall frequency of use rises with time with an S-shaped growth curve. Yet, this claim has only been supported by a limited number of case studies. In this paper, we provide the first corpus-based large-scale confirmation of the S-curve in language change. Moreover, we uncover another generic pattern, a latency phase preceding the S-growth, during which the frequency remains close to constant. We propose a usage-based model which predicts both phases, the latency and the S-growth. The driving mechanism is a random walk in the space of frequency of use. The underlying deterministic dynamics highlights the role of a control parameter which tunes the system at the vicinity of a saddle-node bifurcation. In the neighbourhood of the critical point, the latency phase corresponds to the diffusion time over the critical region, and the S-growth to the fast convergence that follows. The durations of the two phases are computed as specific first-passage times, leading to distributions that fit well the ones extracted from our dataset. We argue that our results are not specific to the studied corpus, but apply to semantic change in general.

Is there something out there? Inferring space from sensorimotor dependencies.

This letter suggests that in biological organisms, the perceived structure of reality, in particular the notions of body, environment, space, object, and attribute, could be a consequence of an effort on the part of brains to account for the dependency between their inputs and their outputs in terms of a small number of parameters. To validate this idea, a procedure is demonstrated whereby the brain of a (simulated) organism with arbitrary input and output connectivity can deduce the dimensionality of the rigid group of the space underlying its input-output relationship, that is, the dimension of what the organism will call physical space.

Blind source separation in the presence of weak sources.

We investigate the information processing of a linear mixture of independent sources of different magnitudes. In particular we consider the case where a number m of the sources can be considered as "strong" as compared to the other ones, the "weak" sources. We find that it is preferable to perform blind source separation in the space spanned by the strong sources, and that this can be easily done by first projecting the signal onto the m largest principal components. We illustrate the analytical results with numerical simulations.

Nonlinear feedforward networks with stochastic outputs: infomax implies redundancy reduction.

We prove that maximization of mutual information between the output and the input of a feedforward neural network leads to full redundancy reduction under the following sufficient conditions: (i) the input signal is a (possibly nonlinear) invertible mixture of independent components; (ii) there is no input noise; (iii) the activity of each output neuron is a (possibly) stochastic variable with a probability distribution depending on the stimulus through a deterministic function of the inputs (where both the probability distributions and the functions can be different from neuron to neuron); (iv) optimization of the mutual information is performed over all these deterministic functions. This result extends that obtained by Nadal and Parga (1994) who considered the case of deterministic outputs.

Modeling memory: what do we learn from attractor neural networks?

In this paper we summarize some of the main contributions of models of recurrent neural networks with associative memory properties. We compare the behavior of these attractor neural networks with empirical data from both physiology and psychology. This type of network could be used in models with more complex functions.

Mutual information, Fisher information, and population coding.

In the context of parameter estimation and model selection, it is only quite recently that a direct link between the Fisher information and information-theoretic quantities has been exhibited. We give an interpretation of this link within the standard framework of information theory. We show that in the context of population coding, the mutual information between the activity of a large array of neurons and a stimulus to which the neurons are tuned is naturally related to the Fisher information. In the light of this result, we consider the optimization of the tuning curves parameters in the case of neurons responding to a stimulus represented by an angular variable.

Trio learning: a new strategy for building hybrid neural trees.

Neural trees are constructive algorithms which build decision trees whose nodes are binary neurons. We propose a new learning scheme, "trio-learning," which leads to a significant reduction in the tree complexity. In this strategy, each node of the tree is optimized by taking into account the knowledge that it will be followed by two son nodes. Moreover, trio-learning can be used to build hybrid trees, with internal nodes and terminal nodes of different nature, for solving any standard tasks (e.g. classification, regression, density estimation). Significant results on a handwritten character classification are presented.

Rule extraction with fuzzy neural network.

This paper deals with the learning of understandable decision rules with connectionist systems. Our approach consists of extracting fuzzy control rules with a new fuzzy neural network. Whereas many other works on this area propose to use combinations of nonlinear neurons to approximate fuzzy operations, we use a fuzzy neuron that computes max-min operations. Thus, this neuron can be interpreted as a possibility estimator, just as sigma-pi neurons can support a probabilistic interpretation. Within this context, possibilistic inferences can be drawn through the multi-layered network, using a distributed representation of the information. A new learning procedure has been developed in order that each part of the network can be learnt sequentially, while other parts are frozen. Each step of the procedure is based on the same kind of learning scheme: the backpropagation of a well-chosen cost function with appropriate derivatives of max-min function. An appealing result of the learning phase is the ability of the network to automatically reduce the number of the condition-parts of the rules, if needed. The network has been successfully tested on the learning of a control rule base for an inverted pendulum.

Neural networks that learn temporal sequences by selection.

A model for formal neural networks that learn temporal sequences by selection is proposed on the basis of observations on the acquisition of song by birds, on sequence-detecting neurons, and on allosteric receptors. The model relies on hypothetical elementary devices made up of three neurons, the synaptic triads, which yield short-term modification of synaptic efficacy through heterosynaptic interactions, and on a local Hebbian learning rule. The functional units postulated are mutually inhibiting clusters of synergic neurons and bundles of synapses. Networks formalized on this basis display capacities for passive recognition and for production of temporal sequences that may include repetitions. Introduction of the learning rule leads to the differentiation of sequence-detecting neurons and to the stabilization of ongoing temporal sequences. A network architecture composed of three layers of neuronal clusters is shown to exhibit active recognition and learning of time sequences by selection: the network spontaneously produces prerepresentations that are selected according to their resonance with the input percepts. Predictions of the model are discussed.