Placing Approach-Avoidance Conflict Within the Framework of Multi-objective Reinforcement Learning.

Many psychiatric disorders are marked by impaired decision-making during an approach-avoidance conflict. Current experiments elicit approach-avoidance conflicts in bandit tasks by pairing an individual's actions with consequences that are simultaneously desirable (reward) and undesirable (harm). We frame approach-avoidance conflict tasks as a multi-objective multi-armed bandit. By defining a general decision-maker as a limiting sequence of actions, we disentangle the decision process from learning. Each decision maker can then be identified as a multi-dimensional point representing its long-term average expected outcomes, while different decision making models can be associated by the geometry of their 'feasible region', the set of all possible long term performances on a fixed task. We introduce three example decision-makers based on popular reinforcement learning models and characterize their feasible regions, including whether they can be Pareto optimal. From this perspective, we find that existing tasks are unable to distinguish between the three examples of decision-makers. We show how to design new tasks whose geometric structure can be used to better distinguish between decision-makers. These findings are expected to guide the design of approach-avoidance conflict tasks and the modeling of resulting decision-making behavior.

A Competition of Critics in Human Decision-Making.

Recent experiments and theories of human decision-making suggest positive and negative errors are processed and encoded differently by serotonin and dopamine, with serotonin possibly serving to oppose dopamine and protect against risky decisions. We introduce a temporal difference (TD) model of human decision-making to account for these features. Our model involves two critics, an optimistic learning system and a pessimistic learning system, whose predictions are integrated in time to control how potential decisions compete to be selected. Our model predicts that human decision-making can be decomposed along two dimensions: the degree to which the individual is sensitive to (1) risk and (2) uncertainty. In addition, we demonstrate that the model can learn about the mean and standard deviation of rewards, and provide information about reaction time despite not modeling these variables directly. Lastly, we simulate a recent experiment to show how updates of the two learning systems could relate to dopamine and serotonin transients, thereby providing a mathematical formalism to serotonin's hypothesized role as an opponent to dopamine. This new model should be useful for future experiments on human decision-making.

Rescorla-Wagner Models with Sparse Dynamic Attention.

The Rescorla-Wagner (R-W) model describes human associative learning by proposing that an agent updates associations between stimuli, such as events in their environment or predictive cues, proportionally to a prediction error. While this model has proven informative in experiments, it has been posited that humans selectively attend to certain cues to overcome a problem with the R-W model scaling to large cue dimensions. We formally characterize this scaling problem and provide a solution that involves limiting attention in a R-W model to a sparse set of cues. Given the universal difficulty in selecting features for prediction, sparse attention faces challenges beyond those faced by the R-W model. We demonstrate several ways in which a naive attention model can fail explain those failures and leverage that understanding to produce a Sparse Attention R-W with Inference framework (SAR-WI). The SAR-WI framework not only satisfies a constraint on the number of attended cues, it also performs as well as the R-W model on a number of natural learning tasks, can correctly infer associative strengths, and focuses attention on predictive cues while ignoring uninformative cues. Given the simplicity of proposed alterations, we hope this work informs future development and empirical validation of associative learning models that seek to incorporate sparse attention.

Critically slow learning in flashcard learning models.

Algorithmic education theory examines, among other things, the trade-off between reviewing old material and studying new material: time spent learning the new comes at the expense of reviewing and solidifying one's understanding of the old. This trade-off is captured in the "Slow Flashcard System" (SFS)-a system that has been studied not only for its applications in educational software but also for its critical properties; it is a simple discrete deterministic system capable of remarkable complexity, with standing conjectures regarding its longterm behavior. Here, we introduce a probabilistic model of SFS and further derive a continuous time, continuous space partial differential equation model. These two models of SFS shed light on the longterm behavior of SFS and open new avenues of research.

Estimating Intermittent Individual Spawning Behavior via Disaggregating Group Data.

In order to understand fish biology and reproduction, it is important to know the fecundity patterns of individual fish, as frequently established by recording the output of mixed-sex groups of fish in a laboratory setting. However, for understanding individual reproductive health and modeling purposes it is important to estimate individual fecundity from group fecundity. We created a multistage method that disaggregates group-level data into estimates for individual-level clutch size and spawning interval distributions. The first stage of the method develops estimates of the daily spawning probability of fish. Daily spawning probabilities are then used to calculate the log likelihood of candidate distributions of clutch size. Selecting the best candidate distribution for clutch size allows for a Monte Carlo resampling of annotations of the original data which state how many fish spawned on which day. We verify this disaggregation technique by combining data from fathead minnow pairs, and checking that the disaggregation method reproduced the original clutch sizes and spawning intervals. This method will allow scientists to estimate individual clutch size and spawning interval distributions from group spawning data without specialized or elaborate experimental designs.

Altruistic aging: The evolutionary dynamics balancing longevity and evolvability.

Altruism is typically associated with traits or behaviors that benefit the population as a whole, but are costly to the individual. We propose that, when the environment is rapidly changing, senescence (age-related deterioration) can be altruistic. According to numerical simulations of an agent-based model, while long-lived individuals can outcompete their short lived peers, populations composed of long-lived individuals are more likely to go extinct during periods of rapid environmental change. Moreover, as in many situations where other cooperative behavior arises, senescence can be stabilized in a structured population.

Frequency adjustment and synchrony in networks of delayed pulse-coupled oscillators.

We introduce a system of pulse-coupled oscillators that can change both their phases and frequencies and prove that when there is a separation of time scales between phase and frequency adjustment the system converges to exact synchrony on strongly connected graphs with time delays. The analysis involves decomposing the network into a forest of tree-like structures that capture causality. These results provide a robust method of sensor net synchronization as well as demonstrate a new avenue of possible pulse-coupled oscillator research.

Probabilistic convergence guarantees for type-II pulse-coupled oscillators.

We show that a large class of pulse-coupled oscillators converge with high probability from random initial conditions on a large class of graphs with time delays. Our analysis combines previous local convergence results, probabilistic network analysis, and a classification scheme for type-II phase response curves to produce rigorous lower bounds for convergence probabilities based on network density. These results suggest methods for the analysis of pulse-coupled oscillators, and provide insights into the balance of excitation and inhibition in the operation of biological type-II phase response curves and also the design of decentralized and minimal clock synchronization schemes in sensor nets.

Robust convergence in pulse-coupled oscillators with delays.

We show that for pulse-coupled oscillators a class of phase response curves with both excitation and inhibition exhibit robust convergence to synchrony on arbitrary aperiodic connected graphs with delays. We describe the basins of convergence and give explicit bounds on the convergence times. These results provide new and more robust methods for synchronization of sensor nets and also have biological implications.