Mirror Descent of Hopfield Model.

Mirror descent is an elegant optimization technique that leverages a dual space of parametric models to perform gradient descent. While originally developed for convex optimization, it has increasingly been applied in the field of machine learning. In this study, we propose a novel approach for using mirror descent to initialize the parameters of neural networks. Specifically, we demonstrate that by using the Hopfield model as a prototype for neural networks, mirror descent can effectively train the model with significantly improved performance compared to traditional gradient descent methods that rely on random parameter initialization. Our findings highlight the potential of mirror descent as a promising initialization technique for enhancing the optimization of machine learning models.

Scale-invariant representation of machine learning.

The success of machine learning has resulted from its structured representation of data. Similar data have close internal representations as compressed codes for classification or emerged labels for clustering. We observe that the frequency of internal codes or labels follows power laws in both supervised and unsupervised learning models. This scale-invariant distribution implies that machine learning largely compresses frequent typical data, and simultaneously, differentiates many atypical data as outliers. In this study, we derive the process by which these power laws can naturally arise in machine learning. In terms of information theory, the scale-invariant representation corresponds to a maximally uncertain data grouping among possible representations that guarantee a given learning accuracy.

Inference of stochastic time series with missing data.

Inferring dynamics from time series is an important objective in data analysis. In particular, it is challenging to infer stochastic dynamics given incomplete data. We propose an expectation maximization (EM) algorithm that iterates between alternating two steps: E-step restores missing data points, while M-step infers an underlying network model from the restored data. Using synthetic data of a kinetic Ising model, we confirm that the algorithm works for restoring missing data points as well as inferring the underlying model. At the initial iteration of the EM algorithm, the model inference shows better model-data consistency with observed data points than with missing data points. As we keep iterating, however, missing data points show better model-data consistency. We find that demanding equal consistency of observed and missing data points provides an effective stopping criterion for the iteration to prevent going beyond the most accurate model inference. Using the EM algorithm and the stopping criterion together, we infer missing data points from a time-series data of real neuronal activities. Our method reproduces collective properties of neuronal activities such as correlations and firing statistics even when 70% of data points are masked as missing points.

Information Flows of Diverse Autoencoders.

Deep learning methods have had outstanding performances in various fields. A fundamental query is why they are so effective. Information theory provides a potential answer by interpreting the learning process as the information transmission and compression of data. The information flows can be visualized on the information plane of the mutual information among the input, hidden, and output layers. In this study, we examine how the information flows are shaped by the network parameters, such as depth, sparsity, weight constraints, and hidden representations. Here, we adopt autoencoders as models of deep learning, because (i) they have clear guidelines for their information flows, and (ii) they have various species, such as vanilla, sparse, tied, variational, and label autoencoders. We measured their information flows using Rényi's matrix-based α-order entropy functional. As learning progresses, they show a typical fitting phase where the amounts of input-to-hidden and hidden-to-output mutual information both increase. In the last stage of learning, however, some autoencoders show a simplifying phase, previously called the "compression phase", where input-to-hidden mutual information diminishes. In particular, the sparsity regularization of hidden activities amplifies the simplifying phase. However, tied, variational, and label autoencoders do not have a simplifying phase. Nevertheless, all autoencoders have similar reconstruction errors for training and test data. Thus, the simplifying phase does not seem to be necessary for the generalization of learning.

Erasure machine: Inverse Ising inference from reweighting of observation frequencies.

Maximum likelihood estimation (MLE) is fundamental to system inference for stochastic systems. In some generality, MLE will converge to the correct model in the infinite data limit. In the context of physical approaches to system inference, such as Boltzmann machines, MLE requires the arduous computation of partition functions summing over all configurations, both observed and unobserved. We present a conceptually transparent data-driven inference computation based on a reweighting of observed configuration frequencies that allows us to recast the inference problem as a simpler calculation. Modeling our approach on the high-temperature limit of statistical physics, we reweight the frequencies of observed configurations by multiplying with reciprocals of Boltzmann weights and update the Boltzmann weights iteratively to make these products close to the high-temperature limit of the Boltzmann weights. This converts the required partition function computation in the reweighted MLE to a tractable leading-order high-temperature term. We show that this is a convex optimization at each step. Then, for systems with a large number of degrees of freedom where other approaches are intractable, we demonstrate that this data-driven algorithm gives accurate inference with both synthetic data and two real-world examples.

Synchronization of active rotators interacting with environment.

Multiple organs in a living system respond to environmental changes, and the signals from the organs regulate the physiological environment. Inspired by this biological feedback, we propose a simple autonomous system of active rotators to explain how multiple units are synchronized under a fluctuating environment. We find that the feedback via an environment can entrain rotators to have synchronous phases for specific conditions. This mechanism is markedly different from the simple entrainment by a common oscillatory external stimulus that is not interacting with systems. We theoretically examine how the phase synchronization depends on the interaction strength between rotators and environment. Furthermore, we successfully demonstrate the proposed model by realizing an analog electric circuit with microelectronic devices. This bioinspired platform can be used as a sensor for monitoring varying environments and as a controller for amplifying signals by their feedback-induced synchronization.

Tripartite cell networks for glucose homeostasis.

Controlling the excess and shortage of energy is a fundamental task for living organisms. Diabetes is a representative metabolic disease caused by the malfunction of energy homeostasis. The islets of Langerhans in the pancreas release long-range messengers, hormones, into the blood to regulate the homeostasis of the primary energy fuel, glucose. The hormone and glucose levels in the blood show rhythmic oscillations with a characteristic period of 5-10 min, and the functional roles of the oscillations are not clear. Each islet has [Formula: see text] and [Formula: see text] cells that secrete glucagon and insulin, respectively. These two counter-regulatory hormones appear sufficient to increase and decrease glucose levels. However, pancreatic islets have a third cell type, [Formula: see text] cells, which secrete somatostatin. The three cell populations have a unique spatial organization in islets, and they interact to perturb their hormone secretions. The mini-organs of islets are scattered throughout the exocrine pancreas. Considering that the human pancreas contains approximately a million islets, the coordination of hormone secretion from the multiple sources of islets and cells within the islets should have a significant effect on human physiology. In this review, we introduce the hierarchical organization of tripartite cell networks, and recent biophysical modeling to systematically understand the oscillations and interactions of [Formula: see text], [Formula: see text], and [Formula: see text] cells. Furthermore, we discuss the functional roles and clinical implications of hormonal oscillations and their phase coordination for the diagnosis of type II diabetes.

Data-driven inference of hidden nodes in networks.

The explosion of activity in finding interactions in complex systems is driven by availability of copious observations of complex natural systems. However, such systems, e.g., the human brain, are rarely completely observable. Interaction network inference must then contend with hidden variables affecting the behavior of the observed parts of the system. We present an effective approach for model inference with hidden variables. From configurations of observed variables, we identify the observed-to-observed, hidden-to-observed, observed-to-hidden, and hidden-to-hidden interactions, the configurations of hidden variables, and the number of hidden variables. We demonstrate the performance of our method by simulating a kinetic Ising model, and show that our method outperforms existing methods. Turning to real data, we infer the hidden nodes in a neuronal network in the salamander retina and a stock market network. We show that predictive modeling with hidden variables is significantly more accurate than that without hidden variables. Finally, an important hidden variable problem is to find the number of clusters in a dataset. We apply our method to classify MNIST handwritten digits. We find that there are about 60 clusters which are roughly equally distributed among the digits.

Network inference in stochastic systems from neurons to currencies: Improved performance at small sample size.

The fundamental problem in modeling complex phenomena such as human perception using probabilistic methods is that of deducing a stochastic model of interactions between the constituents of a system from observed configurations. Even in this era of big data, the complexity of the systems being modeled implies that inference methods must be effective in the difficult regimes of small sample sizes and large coupling variability. Thus, model inference by means of minimization of a cost function requires additional assumptions such as sparsity of interactions to avoid overfitting. In this paper, we completely divorce iterative model updates from the value of a cost function quantifying goodness of fit. This separation enables the use of goodness of fit as a natural rationale for terminating model updates, thereby avoiding overfitting. We do this within the mathematical formalism of statistical physics by defining a formal free energy of observations from a partition function with an energy function chosen precisely to enable an iterative model update. Minimizing this free energy, we demonstrate coupling strength inference in nonequilibrium kinetic Ising models, and show that our method outperforms other existing methods in the regimes of interest. Our method has no tunable learning rate, scales to large system sizes, and has a systematic expansion to obtain higher-order interactions. As applications, we infer a functional connectivity network in the salamander retina and a currency exchange rate network from time-series data of neuronal spiking and currency exchange rates, respectively. Accurate small sample size inference is critical for devising a profitable currency hedging strategy.

Broad cross-reactivity of the T-cell repertoire achieves specific and sufficiently rapid target searching.

The molecular recognition of T-cell receptors is the hallmark of the adaptive immunity. Given the finiteness of the T-cell repertoire, individual T-cell receptors are necessary to be cross-reactive to multiple antigenic peptides. In this study, we quantify the variability of the cross-reactivity by using a string model that estimates the binding affinity between two sequences of amino acids. We examine sequences of 10,000 human T-cell receptors and 10,000 antigenic peptides, and obtain a full spectrum of cross-reactivity of the receptor-peptide binding. Then, we find that the cross-reactivity spectrum is broad. Some T-cells are reactive to 1000 peptides, but some T-cells are reactive to only one or two peptides. Since the degree of cross-reactivity has a correlation with the (un)binding affinity of receptors, we further investigate how the broad cross-reactivity affects the target searching of T-cells. High cross-reactive T-cells may not require many trials for searching correct targets, but they may spend long time to unbind from incorrect targets. In contrast, low cross-reactive T-cells may not spend long time to ignore incorrect targets, but they require many trials for screening correct targets. We evaluate this hypothesis, and show that the broad cross-reactivity of the natural T-cell repertoire can balance the trade-off between the rapid screening and unbinding penalty.

Local complexity predicts global synchronization of hierarchically networked oscillators.

We study the global synchronization of hierarchically-organized Stuart-Landau oscillators, where each subsystem consists of three oscillators with activity-dependent couplings. We considered all possible coupling signs between the three oscillators, and found that they can generate different numbers of phase attractors depending on the network motif. Here, the subsystems are coupled through mean activities of total oscillators. Under weak inter-subsystem couplings, we demonstrate that the synchronization between subsystems is highly correlated with the number of attractors in uncoupled subsystems. Among the network motifs, perfect anti-symmetric ones are unique to generate both single and multiple attractors depending on the activities of oscillators. The flexible local complexity can make global synchronization controllable.

Insulin modulates the frequency of Ca2+ oscillations in mouse pancreatic islets.

Pancreatic islets can adapt to oscillatory glucose to produce synchronous insulin pulses. Can islets adapt to other oscillatory stimuli, specifically insulin? To answer this question, we stimulated islets with pulses of exogenous insulin and measured their Ca2+ oscillations. We observed that sufficiently high insulin (> 500 nM) with an optimal pulse period (~ 4 min) could make islets to produce synchronous Ca2+ oscillations. Glucose and insulin, which are key stimulatory factors of islets, modulate islet Ca2+ oscillations differently. Glucose increases the active-to-silent ratio of phases, whereas insulin increases the period of the oscillation. To examine the dual modulation, we adopted a phase oscillator model that incorporated the phase and frequency modulations. This mathematical model showed that out-of-phase oscillations of glucose and insulin were more effective at synchronizing islet Ca2+ oscillations than in-phase stimuli. This finding suggests that a phase shift in glucose and insulin oscillations can enhance inter-islet synchronization.

A Local Counter-Regulatory Motif Modulates the Global Phase of Hormonal Oscillations.

Counter-regulatory elements maintain dynamic equilibrium ubiquitously in living systems. The most prominent example, which is critical to mammalian survival, is that of pancreatic α and ß cells producing glucagon and insulin for glucose homeostasis. These cells are not found in a single gland but are dispersed in multiple micro-organs known as the islets of Langerhans. Within an islet, these two reciprocal cell types interact with each other and with an additional cell type: the δ cell. By testing all possible motifs governing the interactions of these three cell types, we found that a unique set of positive/negative intra-islet interactions between different islet cell types functions not only to reduce the superficially wasteful zero-sum action of glucagon and insulin but also to enhance/suppress the synchronization of hormone secretions between islets under high/normal glucose conditions. This anti-symmetric interaction motif confers effective controllability for network (de)synchronization.

Phase modulation of insulin pulses enhances glucose regulation and enables inter-islet synchronization.

Insulin is secreted in a pulsatile manner from multiple micro-organs called the islets of Langerhans. The amplitude and phase (shape) of insulin secretion are modulated by numerous factors including glucose. The role of phase modulation in glucose homeostasis is not well understood compared to the obvious contribution of amplitude modulation. In the present study, we measured Ca2+ oscillations in islets as a proxy for insulin pulses, and we observed their frequency and shape changes under constant/alternating glucose stimuli. Here we asked how the phase modulation of insulin pulses contributes to glucose regulation. To directly answer this question, we developed a phenomenological oscillator model that drastically simplifies insulin secretion, but precisely incorporates the observed phase modulation of insulin pulses in response to glucose stimuli. Then, we mathematically modeled how insulin pulses regulate the glucose concentration in the body. The model of insulin oscillation and glucose regulation describes the glucose-insulin feedback loop. The data-based model demonstrates that the existence of phase modulation narrows the range within which the glucose concentration is maintained through the suppression/enhancement of insulin secretion in conjunction with the amplitude modulation of this secretion. The phase modulation is the response of islets to glucose perturbations. When multiple islets are exposed to the same glucose stimuli, they can be entrained to generate synchronous insulin pulses. Thus, we conclude that the phase modulation of insulin pulses is essential for glucose regulation and inter-islet synchronization.

Scaling Law for Irreversible Entropy Production in Critical Systems.

We examine the Jarzynski equality for a quenching process across the critical point of second-order phase transitions, where absolute irreversibility and the effect of finite-sampling of the initial equilibrium distribution arise in a single setup with equal significance. We consider the Ising model as a prototypical example for spontaneous symmetry breaking and take into account the finite sampling issue by introducing a tolerance parameter. The initially ordered spins become disordered by quenching the ferromagnetic coupling constant. For a sudden quench, the deviation from the Jarzynski equality evaluated from the ideal ensemble average could, in principle, depend on the reduced coupling constant ε0 of the initial state and the system size L. We find that, instead of depending on ε0 and L separately, this deviation exhibits a scaling behavior through a universal combination of ε0 and L for a given tolerance parameter, inherited from the critical scaling laws of second-order phase transitions. A similar scaling law can be obtained for the finite-speed quench as well within the Kibble-Zurek mechanism.

Design Principles of Pancreatic Islets: Glucose-Dependent Coordination of Hormone Pulses.

Pancreatic islets are functional units involved in glucose homeostasis. The multicellular system comprises three main cell types; ß and α cells reciprocally decrease and increase blood glucose by producing insulin and glucagon pulses, while the role of δ cells is less clear. Although their spatial organization and the paracrine/autocrine interactions between them have been extensively studied, the functional implications of the design principles are still lacking. In this study, we formulated a mathematical model that integrates the pulsatility of hormone secretion and the interactions and organization of islet cells and examined the effects of different cellular compositions and organizations in mouse and human islets. A common feature of both species was that islet cells produced synchronous hormone pulses under low- and high-glucose conditions, while they produced asynchronous hormone pulses under normal glucose conditions. However, the synchronous coordination of insulin and glucagon pulses at low glucose was more pronounced in human islets that had more α cells. When ß cells were selectively removed to mimic diabetic conditions, the anti-synchronicity of insulin and glucagon pulses was deteriorated at high glucose, but it could be partially recovered when the re-aggregation of remaining cells was considered. Finally, the third cell type, δ cells, which introduced additional complexity in the multicellular system, prevented the excessive synchronization of hormone pulses. Our computational study suggests that controllable synchronization is a design principle of pancreatic islets.

Traveling wave in a three-dimensional array of conformist and contrarian oscillators.

We consider a system of conformist and contrarian oscillators coupled locally in a three-dimensional cubic lattice and explore collective behavior of the system. The conformist oscillators attractively interact with the neighbor oscillators and therefore tend to be aligned with the neighbors' phase. The contrarian oscillators interact repulsively with the neighbors and therefore tend to be out of phase with them. In this paper, we investigate whether many peculiar dynamics that have been observed in the mean-field system with global coupling can emerge even with local coupling. In particular, we pay attention to the possibility that a traveling wave may arise. We find that the traveling wave occurs due to coupling asymmetry and not by global coupling; this observation confirms that the global coupling is not essential to the occurrence of a traveling wave in the system. The traveling wave can be a mechanism for the coherent rhythm generation of the circadian clock or of hormone secretion in biological systems under local coupling.

Diabetes Associated Metabolomic Perturbations in NOD Mice.

Non-obese diabetic (NOD) mice are a widely-used model oftype1 diabetes (T1D). However, not all animals develop overt diabetes. This study examined the circulating metabolomic profiles of NOD mice progressing or not progressing to T1D. Total beta-cell mass was quantified in the intact pancreas using transgenic NOD mice expressinggreen fluorescent protein under the control of mouse insulin I promoter.While both progressor and non-progressor animals displayed lymphocyte infiltration and endoplasmic reticulum stress in the pancreas tissue;overt T1D did not develop until animals lost ~70% of the total beta-cell mass.Gas chromatography time of flight mass spectrometry (GC-TOF) was used to measure >470 circulating metabolites in male and female progressor and non-progressor animals (n=76) across a wide range of ages (neonates to >40-wk).Statistical and multivariate analyses were used to identify age and sex independent metabolic markers which best differentiated progressor and non-progressor animals' metabolic profiles. Key T1D-associated perturbations were related with: (1) increased plasma glucose and reduced 1,5-anhydroglucitol markers of glycemic control; (2) increased allantoin, gluconic acid and nitric oxide-derived saccharic acid markers of oxidative stress; (3) reduced lysine, an insulin secretagogue; (4) increased branched-chain amino acids, isoleucine and valine; (5) reduced unsaturated fatty acids including arachidonic acid; and (6)perturbations in urea cycle intermediates suggesting increased arginine-dependent NO synthesis. Together these findings highlight the strength of the unique approach of comparing progressor and non-progressor NOD mice to identify metabolic perturbations involved in T1D progression.

A conserved rule for pancreatic islet organization.

Morphogenesis, spontaneous formation of organism structure, is essential for life. In the pancreas, endocrine α, ß, and δ cells are clustered to form islets of Langerhans, the critical micro-organ for glucose homeostasis. The spatial organization of endocrine cells in islets looks different between species. Based on the three-dimensional positions of individual cells in islets, we computationally inferred the relative attractions between cell types, and found that the attractions between homotypic cells were slightly, but significantly, stronger than the attractions between heterotypic cells commonly in mouse, pig, and human islets. The difference between α-ß cell attraction and ß-ß cell attraction was minimal in human islets, maximizing the plasticity of islet structures. Our result suggests that although the cellular composition and attractions of pancreatic endocrine cells are quantitatively different between species, the physical mechanism of islet morphogenesis may be evolutionarily conserved.

Stable and flexible system for glucose homeostasis.

Pancreatic islets, controlling glucose homeostasis, consist of α, ß, and δ cells. It has been observed that α and ß cells generate out-of-phase synchronization in the release of glucagon and insulin, counter-regulatory hormones for increasing and decreasing glucose levels, while ß and δ cells produce in-phase synchronization in the release of the insulin and somatostatin. Pieces of interactions between the islet cells have been observed for a long time, although their physiological role as a whole has not been explored yet. We model the synchronized hormone pulses of islets with coupled phase oscillators that incorporate the observed cellular interactions. The integrated model shows that the interaction from ß to δ cells, of which sign is a subject of controversy, should be positive to reproduce the in-phase synchronization between ß and δ cells. The model also suggests that δ cells help the islet system flexibly respond to changes of glucose environment.