Morphological computation and decentralized learning in a swarm of sterically interacting robots.

Whereas naturally occurring swarms thrive when crowded, physical interactions in robotic swarms are either avoided or carefully controlled, thus limiting their operational density. Here, we present a mechanical design rule that allows robots to act in a collision-dominated environment. We introduce Morphobots, a robotic swarm platform developed to implement embodied computation through a morpho-functional design. By engineering a three-dimensional printed exoskeleton, we encode a reorientation response to an external body force (such as gravity) or a surface force (such as a collision). We show that the force orientation response is generic and can augment existing swarm robotic platforms (e.g., Kilobots) as well as custom robots even 10 times larger. At the individual level, the exoskeleton improves motility and stability and also allows encoding of two contrasting dynamical behaviors in response to an external force or a collision (including collision with a wall or a movable obstacle and on a dynamically tilting plane). This force orientation response adds a mechanical layer to the robot's sense-act cycle at the swarm level, leveraging steric interactions for collective phototaxis when crowded. Enabling collisions also promotes information flow, facilitating online distributed learning. Each robot runs an embedded algorithm that ultimately optimizes collective performance. We identify an effective parameter that controls the force orientation response and explore its implications in swarms that transition from dilute to crowded. Experimenting with physical swarms (of up to 64 robots) and simulated swarms (of up to 8192 agents) shows that the effect of morphological computation increases with growing swarm size.

Automated optimization of multilevel models of collective behaviour: application to mixed society of animals and robots.

Animal societies exhibit complex dynamics that require multi-level descriptions. They are difficult to model, as they encompass information at different levels of description, such as individual physiology, individual behaviour, group behaviour and features of the environment. The collective behaviour of a group of animals can be modelled as a dynamical system. Typically, models of behaviour are either macroscopic (differential equations of population dynamics) or microscopic (such as Markov chains, explicitly specifying the spatio-temporal state of each individual). These two kind of models offer distinct and complementary descriptions of the observed behaviour. Macroscopic models offer mean field description of the collective dynamics, where collective choices are considered as the stable steady states of a nonlinear system governed by control parameters leading to bifurcation diagrams. Microscopic models can be used to perform computer simulations or as building blocks for robot controllers, at the individual level, of the observed spatial behaviour of animals. Here, we present a methodology to translate a macroscopic model into different microscopic models. We automatically calibrate the microscopic models so that the resulting simulated collective dynamics fit the solutions of the reference macroscopic model for a set of parameter values corresponding to a bifurcation diagram leading to multiple steady states. We apply evolutionary algorithms to simultaneously optimize the parameters of the models at different levels of description. This methodology is applied, in simulation, to an experimentally validated shelter-selection problem solved by gregarious insects and robots. Our framework can be used for multi-level modelling of collective behaviour in animals and robots.

Policy search with rare significant events: Choosing the right partner to cooperate with.

This paper focuses on a class of reinforcement learning problems where significant events are rare and limited to a single positive reward per episode. A typical example is that of an agent who has to choose a partner to cooperate with, while a large number of partners are simply not interested in cooperating, regardless of what the agent has to offer. We address this problem in a continuous state and action space with two different kinds of search methods: a gradient policy search method and a direct policy search method using an evolution strategy. We show that when significant events are rare, gradient information is also scarce, making it difficult for policy gradient search methods to find an optimal policy, with or without a deep neural architecture. On the other hand, we show that direct policy search methods are invariant to the rarity of significant events, which is yet another confirmation of the unique role evolutionary algorithms has to play as a reinforcement learning method.

Meta-control of social learning strategies.

Social learning, copying other's behavior without actual experience, offers a cost-effective means of knowledge acquisition. However, it raises the fundamental question of which individuals have reliable information: successful individuals versus the majority. The former and the latter are known respectively as success-based and conformist social learning strategies. We show here that while the success-based strategy fully exploits the benign environment of low uncertainly, it fails in uncertain environments. On the other hand, the conformist strategy can effectively mitigate this adverse effect. Based on these findings, we hypothesized that meta-control of individual and social learning strategies provides effective and sample-efficient learning in volatile and uncertain environments. Simulations on a set of environments with various levels of volatility and uncertainty confirmed our hypothesis. The results imply that meta-control of social learning affords agents the leverage to resolve environmental uncertainty with minimal exploration cost, by exploiting others' learning as an external knowledge base.

Social learning in swarm robotics.

In this paper, we present an implementation of social learning for swarm robotics. We consider social learning as a distributed online reinforcement learning method applied to a collective of robots where sensing, acting and coordination are performed on a local basis. While some issues are specific to artificial systems, such as the general objective of learning efficient (and ideally, optimal) behavioural strategies to fulfill a task defined by a supervisor, some other issues are shared with social learning in natural systems. We discuss some of these issues, paving the way towards cumulative cultural evolution in robot swarms, which could enable complex social organization necessary to achieve challenging robotic tasks. This article is part of a discussion meeting issue 'The emergence of collective knowledge and cumulative culture in animals, humans and machines'.

Nothing better to do? Environment quality and the evolution of cooperation by partner choice.

The effects of partner choice have been documented in a large number of biological systems such as sexual markets, interspecific mutualisms, or human cooperation. There are, however, a number of situations in which one would expect this mechanism to play a role, but where no such effect has ever been demonstrated. This is the case in particular in many intraspecific interactions, such as collective hunts, in non-human animals. Here we use individual-based simulations to solve this apparent paradox. We show that the conditions for partner choice to operate are in fact restrictive. They entail that individuals can compare social opportunities and choose the best. The challenge is that social opportunities are often rare because they necessitate the co-occurrence of (i) at least one available partner, and (ii) a resource to exploit together with this partner. This has three consequences. First, partner choice cannot lead to the evolution of cooperation when resources are scarce, which explains that this mechanism could never be observed in many cases of intraspecific cooperation in animals. Second, partner choice can operate when partners constitute in themselves a resource, which is the case in sexual interactions and interspecific mutualisms. Third, partner choice can lead to the evolution of cooperation when individuals live in a rich environment, and/or when they are highly efficient at extracting resources from their environment.

Indirect genetic effects allow escape from the inefficient equilibrium in a coordination game.

Social interactions involving coordination between individuals are subject to an "evolutionary trap." Once a suboptimal strategy has evolved, mutants playing an alternative strategy are counterselected because they fail to coordinate with the majority. This creates a detrimental situation from which evolution cannot escape, preventing the evolution of efficient collective behaviors. Here, we study this problem using evolutionary robotics simulations. We first confirm the existence of an evolutionary trap in a simple setting. We then, however, reveal that evolution can solve this problem in a more realistic setting where individuals need to coordinate with one another. In this setting, simulated robots evolve an ability to adapt plastically their behavior to one another, as this improves the efficiency of their interaction. This ability has an unintended evolutionary consequence: a genetic mutation affecting one individual's behavior also indirectly alters their partner's behavior because the two individuals influence one another. As a consequence of this indirect genetic effect, pairs of partners can virtually change strategy together with a single mutation, and the evolutionary barrier between alternative strategies disappears. This finding reveals a general principle that could play a role in nature to smoothen the transition to efficient collective behaviors in all games with multiple equilibriums.

Embodied Evolution in Collective Robotics: A Review.

This article provides an overview of evolutionary robotics techniques applied to online distributed evolution for robot collectives, namely, embodied evolution. It provides a definition of embodied evolution as well as a thorough description of the underlying concepts and mechanisms. This article also presents a comprehensive summary of research published in the field since its inception around the year 2000, providing various perspectives to identify the major trends. In particular, we identify a shift from considering embodied evolution as a parallel search method within small robot collectives (fewer than 10 robots) to embodied evolution as an online distributed learning method for designing collective behaviors in swarm-like collectives. This article concludes with a discussion of applications and open questions, providing a milestone for past and an inspiration for future research.

How mimetic should a robotic fish be to socially integrate into zebrafish groups?

Biomimetic robots are promising tools in animal behavioural studies. If they are socially integrated in a group of animals, they can produce calibrated social stimuli to test the animal responses. However, the design of such social robots is challenging as it involves both a luring capability including appropriate robot behaviours, and the acceptation of the robots by the animals as social companions. Here, we investigate the integration of a biomimetic robot driven by biomimetic behavioural models into a group of zebrafish (Danio rerio). The robot behaviours are based on a stochastic model linking zebrafish visual perception to individual behaviour and calibrated experimentally to correspond to the behaviour of zebrafish. We show that our robot can be integrated into a group of zebrafish, mimic their behaviour and exhibit similar collective dynamics compared to fish-only groups. This study shows that an autonomous biomimetic robot was enhanced by a biomimetic behavioural model so that it can socially integrate into groups of fish.

Massively parallel and multiparameter titration of biochemical assays with droplet microfluidics.

Biochemical systems in which multiple components take part in a given reaction are of increasing interest. Because the interactions between these different components are complex and difficult to predict from basic reaction kinetics, it is important to test for the effect of variations in the concentration for each reagent in a combinatorial manner. For example, in PCR, an increase in the concentration of primers initially increases template amplification, but large amounts of primers result in primer-dimer by-products that inhibit the amplification of the template. Manual titration of biochemical mixtures rapidly becomes costly and laborious, forcing scientists to settle for suboptimal concentrations. Here we present a droplet-based microfluidics platform for mapping of the concentration space of up to three reaction components followed by detection with a fluorescent readout. The concentration of each reaction component is read through its internal standard (barcode), which is fluorescent but chemically orthogonal. We describe in detail the workflow, which comprises the following: (i) production of the microfluidics chips, (ii) preparation of the biochemical mixes, (iii) their mixing and compartmentalization into water-in-oil emulsion droplets via microfluidics, (iv) incubation and imaging of the fluorescent barcode and reporter signals by fluorescence microscopy and (v) image processing and data analysis. We also provide recommendations for choosing the appropriate fluorescent markers, programming the pressure profiles and analyzing the generated data. Overall, this platform allows a researcher with a few weeks of training to acquire ∼10,000 data points (in a 1D, 2D or 3D concentration space) over the course of a day from as little as 100-1,000 µl of reaction mix.

To Cooperate or Not to Cooperate: Why Behavioural Mechanisms Matter.

Mutualistic cooperation often requires multiple individuals to behave in a coordinated fashion. Hence, while the evolutionary stability of mutualistic cooperation poses no particular theoretical difficulty, its evolutionary emergence faces a chicken and egg problem: an individual cannot benefit from cooperating unless other individuals already do so. Here, we use evolutionary robotic simulations to study the consequences of this problem for the evolution of cooperation. In contrast with standard game-theoretic results, we find that the transition from solitary to cooperative strategies is very unlikely, whether interacting individuals are genetically related (cooperation evolves in 20% of all simulations) or unrelated (only 3% of all simulations). We also observe that successful cooperation between individuals requires the evolution of a specific and rather complex behaviour. This behavioural complexity creates a large fitness valley between solitary and cooperative strategies, making the evolutionary transition difficult. These results reveal the need for research on biological mechanisms which may facilitate this transition.

Combining environment-driven adaptation and task-driven optimisation in evolutionary robotics.

Embodied evolutionary robotics is a sub-field of evolutionary robotics that employs evolutionary algorithms on the robotic hardware itself, during the operational period, i.e., in an on-line fashion. This enables robotic systems that continuously adapt, and are therefore capable of (re-)adjusting themselves to previously unknown or dynamically changing conditions autonomously, without human oversight. This paper addresses one of the major challenges that such systems face, viz. that the robots must satisfy two sets of requirements. Firstly, they must continue to operate reliably in their environment (viability), and secondly they must competently perform user-specified tasks (usefulness). The solution we propose exploits the fact that evolutionary methods have two basic selection mechanisms-survivor selection and parent selection. This allows evolution to tackle the two sets of requirements separately: survivor selection is driven by the environment and parent selection is based on task-performance. This idea is elaborated in the Multi-Objective aNd open-Ended Evolution (monee) framework, which we experimentally validate. Experiments with robotic swarms of 100 simulated e-pucks show that monee does indeed promote task-driven behaviour without compromising environmental adaptation. We also investigate an extension of the parent selection process with a 'market mechanism' that can ensure equitable distribution of effort over multiple tasks, a particularly pressing issue if the environment promotes specialisation in single tasks.