Sticky issues in turbulent transport.

We investigate the role of partial stickiness among particles or with a surface for turbulent transport. For the former case, we re-derive known results for the case of the compressible Kraichnan model by using a method based on bi-orthogonality for the expansion of the propagator in terms of left and right eigenvectors. In particular, we show that enforcing the constraints of orthogonality and normalization yields results that were previously obtained by a rigorous, yet possibly less intuitive method. For the latter case, we introduce a general model of transport within the atmospheric boundary layer. As suggested by experimental observations on the transport of atmospheric tracers, both drift and diffusivity scale with the height to the ground. The strength of the drift is parameterized by a velocity V. We use the bi-orthogonality method to show that for V in the range -1 < V < 0 and 0 < V < 1 there is a one-parameter family of boundary conditions that are a priori admissible. Outside of that range, there is a single boundary condition that is admissible. In physical terms, the one-parameter family is parametrized by the degree to which particles stick to the ground.

Seeking and sharing information in collective olfactory search.

Searching for a target is a task of fundamental importance for many living organisms. Long-distance search guided by olfactory cues is a prototypical example of such a process. The searcher receives signals that are sparse and very noisy, making the task extremely difficult. Information-seeking strategies have thus been proven to be effective for individual olfactory search and their extension to collective search has been the subject of some exploratory studies. Here, we study in detail how sharing information among members of a group affects the search behavior when agents adopt information-seeking strategies as Infotaxis and its recently introduced variant, Space-Aware Infotaxis. Our results show that even in absence of explicit coordination, sharing information leads to an effective partitioning of the search space among agents that results in a significant decrease of mean search times.

Olfactory search with finite-state controllers.

Long-range olfactory search is an extremely difficult task in view of the sparsity of odor signals that are available to the searcher and the complex encoding of the information about the source location. Current algorithmic approaches typically require a continuous memory space, sometimes of large dimensionality, which may hamper their optimization and often obscure their interpretation. Here, we show how finite-state controllers with a small set of discrete memory states are expressive enough to display rich, time-extended behavioral modules that resemble the ones observed in living organisms. Finite-state controllers optimized for olfactory search have an immediate interpretation in terms of approximate clocks and coarse-grained spatial maps, suggesting connections with neural models of search behavior.

Taming Lagrangian chaos with multi-objective reinforcement learning.

We consider the problem of two active particles in 2D complex flows with the multi-objective goals of minimizing both the dispersion rate and the control activation cost of the pair. We approach the problem by means of multi-objective reinforcement learning (MORL), combining scalarization techniques together with a Q-learning algorithm, for Lagrangian drifters that have variable swimming velocity. We show that MORL is able to find a set of trade-off solutions forming an optimal Pareto frontier. As a benchmark, we show that a set of heuristic strategies are dominated by the MORL solutions. We consider the situation in which the agents cannot update their control variables continuously, but only after a discrete (decision) time, [Formula: see text]. We show that there is a range of decision times, in between the Lyapunov time and the continuous updating limit, where reinforcement learning finds strategies that significantly improve over heuristics. In particular, we discuss how large decision times require enhanced knowledge of the flow, whereas for smaller [Formula: see text] all a priori heuristic strategies become Pareto optimal.

Erratum: Flow Navigation by Smart Microswimmers via Reinforcement Learning [Phys. Rev. Lett. 118, 158004 (2017)].

This corrects the article DOI: 10.1103/PhysRevLett.118.158004.

Tempo is the key.

Walking flies find the source of attractive odors by changing how frequently they stop and turn in response to the smell.

Learning to flock through reinforcement.

Flocks of birds, schools of fish, and insect swarms are examples of the coordinated motion of a group that arises spontaneously from the action of many individuals. Here, we study flocking behavior from the viewpoint of multiagent reinforcement learning. In this setting, a learning agent tries to keep contact with the group using as sensory input the velocity of its neighbors. This goal is pursued by each learning individual by exerting a limited control on its own direction of motion. By means of standard reinforcement learning algorithms we show that (i) a learning agent exposed to a group of teachers, i.e., hard-wired flocking agents, learns to follow them, and (ii) in the absence of teachers, a group of independently learning agents evolves towards a state where each agent knows how to flock. In both scenarios, the emergent policy (or navigation strategy) corresponds to the polar velocity alignment mechanism of the well-known Vicsek model. These results (a) show that such a velocity alignment may have naturally evolved as an adaptive behavior that aims at minimizing the rate of neighbor loss, and (b) prove that this alignment does not only favor (local) polar order, but it corresponds to the best policy or strategy to keep group cohesion when the sensory input is limited to the velocity of neighboring agents. In short, to stay together, steer together.

Collective olfactory search in a turbulent environment.

Finding the source of an odor dispersed by a turbulent flow is a vital task for many organisms. When many individuals concurrently perform the same olfactory search task, sharing information about other members' decisions can potentially boost the performance. But how much of this information is actually exploitable for the collective task? Here we show, in a model of a swarm of agents inspired by moth behavior, that there is an optimal way to blend the private information about odor and wind detections with the public information about other agents' heading direction. Our results suggest an efficient multiagent olfactory search algorithm that could prove useful in robotics, e.g., in the identification of sources of harmful volatile compounds.

Generosity, selfishness and exploitation as optimal greedy strategies for resource sharing.

Resource sharing outside the kinship bonds is rare. Besides humans, it occurs in chimpanzee, wild dogs and hyenas, as well as in vampire bats. Resource sharing is an instance of animal cooperation, where an animal gives away part of the resources that it owns for the benefit of a recipient. Taking inspiration from blood-sharing in vampire bats, here we show the emergence of generosity in a Markov game, which couples the resource sharing between two players with the gathering task of that resource. At variance with the classical evolutionary models for cooperation, the optimal strategies of this game can be potentially learned by animals during their life-time. The players act greedily, that is, they try to individually maximize only their personal income. Nonetheless, the analytical solution of the model shows that three non trivial optimal behaviours emerge depending on conditions. Besides the obvious case when players are selfish in their choice of resource division, there are conditions under which both players are generous. Moreover, we also found a range of situations in which one selfish player exploits another generous individual, for the satisfaction of both players. Our results show that resource sharing is favoured by three factors: a long time horizon over which the players try to optimize their own game, the similarity among players in their ability of performing the resource-gathering task, as well as by the availability of resources in the environment. These concurrent requirements lead to identify necessary conditions for the emergence of generosity.

Glider soaring via reinforcement learning in the field.

Soaring birds often rely on ascending thermal plumes (thermals) in the atmosphere as they search for prey or migrate across large distances1-4. The landscape of convective currents is rugged and shifts on timescales of a few minutes as thermals constantly form, disintegrate or are transported away by the wind5,6. How soaring birds find and navigate thermals within this complex landscape is unknown. Reinforcement learning7 provides an appropriate framework in which to identify an effective navigational strategy as a sequence of decisions made in response to environmental cues. Here we use reinforcement learning to train a glider in the field to navigate atmospheric thermals autonomously. We equipped a glider of two-metre wingspan with a flight controller that precisely controlled the bank angle and pitch, modulating these at intervals with the aim of gaining as much lift as possible. A navigational strategy was determined solely from the glider's pooled experiences, collected over several days in the field. The strategy relies on on-board methods to accurately estimate the local vertical wind accelerations and the roll-wise torques on the glider, which serve as navigational cues. We establish the validity of our learned flight policy through field experiments, numerical simulations and estimates of the noise in measurements caused by atmospheric turbulence. Our results highlight the role of vertical wind accelerations and roll-wise torques as effective mechanosensory cues for soaring birds and provide a navigational strategy that is directly applicable to the development of autonomous soaring vehicles.

Exploring the function of bacterial chemotaxis.

Bacterial chemotaxis is a classical subject: our knowledge of its molecular pathway has grown very detailed, and experimental observations, as well as mathematical models of the dynamics of chemotactic populations, have a history of several decades. This should not lead to the conclusion that only minor details are left to be understood. Indeed, it is believed that bacterial chemotaxis is under selection for efficiency, yet the underlying functional forces remain largely unknown. These aspects are discussed here by the presentation of illustrative examples related to the role of adaptation and signal integration. Both are expected to be important in ecologically relevant conditions, where chemotaxis should be strongly coupled with metabolism and growth, due to the presence of diverse chemoattractant cues and their active consumption by multiple types of bacteria competing for growth.

Flow Navigation by Smart Microswimmers via Reinforcement Learning.

Smart active particles can acquire some limited knowledge of the fluid environment from simple mechanical cues and exert a control on their preferred steering direction. Their goal is to learn the best way to navigate by exploiting the underlying flow whenever possible. As an example, we focus our attention on smart gravitactic swimmers. These are active particles whose task is to reach the highest altitude within some time horizon, given the constraints enforced by fluid mechanics. By means of numerical experiments, we show that swimmers indeed learn nearly optimal strategies just by experience. A reinforcement learning algorithm allows particles to learn effective strategies even in difficult situations when, in the absence of control, they would end up being trapped by flow structures. These strategies are highly nontrivial and cannot be easily guessed in advance. This Letter illustrates the potential of reinforcement learning algorithms to model adaptive behavior in complex flows and paves the way towards the engineering of smart microswimmers that solve difficult navigation problems.

Cell-size distribution in epithelial tissue formation and homeostasis.

How cell growth and proliferation are orchestrated in living tissues to achieve a given biological function is a central problem in biology. During development, tissue regeneration and homeostasis, cell proliferation must be coordinated by spatial cues in order for cells to attain the correct size and shape. Biological tissues also feature a notable homogeneity of cell size, which, in specific cases, represents a physiological need. Here, we study the temporal evolution of the cell-size distribution by applying the theory of kinetic fragmentation to tissue development and homeostasis. Our theory predicts self-similar probability density function (PDF) of cell size and explains how division times and redistribution ensure cell size homogeneity across the tissue. Theoretical predictions and numerical simulations of confluent non-homeostatic tissue cultures show that cell size distribution is self-similar. Our experimental data confirm predictions and reveal that, as assumed in the theory, cell division times scale like a power-law of the cell size. We find that in homeostatic conditions there is a stationary distribution with lognormal tails, consistently with our experimental data. Our theoretical predictions and numerical simulations show that the shape of the PDF depends on how the space inherited by apoptotic cells is redistributed and that apoptotic cell rates might also depend on size.

Learning to soar in turbulent environments.

Birds and gliders exploit warm, rising atmospheric currents (thermals) to reach heights comparable to low-lying clouds with a reduced expenditure of energy. This strategy of flight (thermal soaring) is frequently used by migratory birds. Soaring provides a remarkable instance of complex decision making in biology and requires a long-term strategy to effectively use the ascending thermals. Furthermore, the problem is technologically relevant to extend the flying range of autonomous gliders. Thermal soaring is commonly observed in the atmospheric convective boundary layer on warm, sunny days. The formation of thermals unavoidably generates strong turbulent fluctuations, which constitute an essential element of soaring. Here, we approach soaring flight as a problem of learning to navigate complex, highly fluctuating turbulent environments. We simulate the atmospheric boundary layer by numerical models of turbulent convective flow and combine them with model-free, experience-based, reinforcement learning algorithms to train the gliders. For the learned policies in the regimes of moderate and strong turbulence levels, the glider adopts an increasingly conservative policy as turbulence levels increase, quantifying the degree of risk affordable in turbulent environments. Reinforcement learning uncovers those sensorimotor cues that permit effective control over soaring in turbulent environments.

The Role of Adaptation in Bacterial Speed Races.

Evolution of biological sensory systems is driven by the need for efficient responses to environmental stimuli. A paradigm among prokaryotes is the chemotaxis system, which allows bacteria to navigate gradients of chemoattractants by biasing their run-and-tumble motion. A notable feature of chemotaxis is adaptation: after the application of a step stimulus, the bacterial running time relaxes to its pre-stimulus level. The response to the amino acid aspartate is precisely adapted whilst the response to serine is not, in spite of the same pathway processing the signals preferentially sensed by the two receptors Tar and Tsr, respectively. While the chemotaxis pathway in E. coli is well characterized, the role of adaptation, its functional significance and the ecological conditions where chemotaxis is selected, are largely unknown. Here, we investigate the role of adaptation in the climbing of gradients by E. coli. We first present theoretical arguments that highlight the mechanisms that control the efficiency of the chemotactic up-gradient motion. We discuss then the limitations of linear response theory, which motivate our subsequent experimental investigation of E. coli speed races in gradients of aspartate, serine and combinations thereof. By using microfluidic techniques, we engineer controlled gradients and demonstrate that bacterial fronts progress faster in equal-magnitude gradients of serine than aspartate. The effect is observed over an extended range of concentrations and is not due to differences in swimming velocities. We then show that adding a constant background of serine to gradients of aspartate breaks the adaptation to aspartate, which results in a sped-up progression of the fronts and directly illustrate the role of adaptation in chemotactic gradient-climbing.

Three-dimensional chemotaxis-driven aggregation of tumor cells.

One of the most important steps in tumor progression involves the transformation from a differentiated epithelial phenotype to an aggressive, highly motile phenotype, where tumor cells invade neighboring tissues. Invasion can occur either by isolated mesenchymal cells or by aggregates that migrate collectively and do not lose completely the epithelial phenotype. Here, we show that, in a three-dimensional cancer cell culture, collective migration of cells eventually leads to aggregation in large clusters. We present quantitative measurements of cluster velocity, coalescence rates, and proliferation rates. These results cannot be explained in terms of random aggregation. Instead, a model of chemotaxis-driven aggregation - mediated by a diffusible attractant - is able to capture several quantitative aspects of our results. Experimental assays of chemotaxis towards culture conditioned media confirm this hypothesis. Theoretical and numerical results further suggest an important role for chemotactic-driven aggregation in spreading and survival of tumor cells.

Estimating cellular parameters through optimization procedures: elementary principles and applications.

Construction of quantitative models is a primary goal of quantitative biology, which aims to understand cellular and organismal phenomena in a quantitative manner. In this article, we introduce optimization procedures to search for parameters in a quantitative model that can reproduce experimental data. The aim of optimization is to minimize the sum of squared errors (SSE) in a prediction or to maximize likelihood. A (local) maximum of likelihood or (local) minimum of the SSE can efficiently be identified using gradient approaches. Addition of a stochastic process enables us to identify the global maximum/minimum without becoming trapped in local maxima/minima. Sampling approaches take advantage of increasing computational power to test numerous sets of parameters in order to determine the optimum set. By combining Bayesian inference with gradient or sampling approaches, we can estimate both the optimum parameters and the form of the likelihood function related to the parameters. Finally, we introduce four examples of research that utilize parameter optimization to obtain biological insights from quantified data: transcriptional regulation, bacterial chemotaxis, morphogenesis, and cell cycle regulation. With practical knowledge of parameter optimization, cell and developmental biologists can develop realistic models that reproduce their observations and thus, obtain mechanistic insights into phenomena of interest.

Polarity, cell division, and out-of-equilibrium dynamics control the growth of epithelial structures.

The growth of a well-formed epithelial structure is governed by mechanical constraints, cellular apico-basal polarity, and spatially controlled cell division. Here we compared the predictions of a mathematical model of epithelial growth with the morphological analysis of 3D epithelial structures. In both in vitro cyst models and in developing epithelial structures in vivo, epithelial growth could take place close to or far from mechanical equilibrium, and was determined by the hierarchy of time-scales of cell division, cell-cell rearrangements, and lumen dynamics. Equilibrium properties could be inferred by the analysis of cell-cell contact topologies, and the nonequilibrium phenotype was altered by inhibiting ROCK activity. The occurrence of an aberrant multilumen phenotype was linked to fast nonequilibrium growth, even when geometric control of cell division was correctly enforced. We predicted and verified experimentally that slowing down cell division partially rescued a multilumen phenotype induced by altered polarity. These results improve our understanding of the development of epithelial organs and, ultimately, of carcinogenesis.

Entropic anomaly and maximal efficiency of microscopic heat engines.

The efficiency of microscopic heat engines in a thermally heterogenous environment is considered. We show that-as a consequence of the recently discovered entropic anomaly-quasistatic engines, whose efficiency is maximal in a fluid at uniform temperature, have in fact vanishing efficiency in the presence of temperature gradients. For slow cycles the efficiency falls off as the inverse of the period. The maximum efficiency is reached at a finite value of the cycle period that is inversely proportional to the square root of the gradient intensity. The relative loss in maximal efficiency with respect to the thermally homogeneous case grows as the square root of the gradient. As an illustration of these general results, we construct an explicit, analytically solvable example of a Carnot stochastic engine. In this thought experiment, a Brownian particle is confined by a harmonic trap and immersed in a fluid with a linear temperature profile. This example may serve as a template for the design of real experiments in which the effect of the entropic anomaly can be measured.

White-noise limit of nonwhite nonequilibrium processes.

The asymptotic behavior of a stochastic process subject to a colored noise is considered in the limit of vanishing correlation time of the noise. The interpretation of the multiplicative noise of the effective equation is investigated. The mathematically consistent formulation of the stochastic calculus for the limiting process is given. It differs in general from the Stratonovich one which is recovered when the colored noise obeys detailed balance or is a one-dimensional process.