A rational model of the Dunning-Kruger effect supports insensitivity to evidence in low performers.

Evaluating one's own performance on a task, typically known as 'self-assessment', is perceived as a fundamental skill, but people appear poorly calibrated to their abilities. Studies seem to show poorer calibration for low performers than for high performers, which could indicate worse metacognitive ability among low performers relative to others (the Dunning-Kruger effect). By developing a rational model of self-assessment, we show that such an effect could be produced by two psychological mechanisms, in either isolation or conjunction: influence of prior beliefs about ability or a relation between performance and skill at determining correctness on each problem. To disentangle these explanations, we conducted a large-scale replication of a seminal paper with approximately 4,000 participants in each of two studies. Comparing the predictions of two variants of our rational model provides support for low performers being less able to estimate whether they are correct in the domains of grammar and logical reasoning.

Assessing Mathematics Misunderstandings via Bayesian Inverse Planning.

Online educational technologies offer opportunities for providing individualized feedback and detailed profiles of students' skills. Yet many technologies for mathematics education assess students based only on the correctness of either their final answers or responses to individual steps. In contrast, examining the choices students make for how to solve the equation and the ways in which they might answer incorrectly offers the opportunity to obtain a more nuanced perspective of their algebra skills. To automatically make sense of step-by-step solutions, we propose a Bayesian inverse planning model for equation solving that computes an assessment of a learner's skills based on her pattern of errors in individual steps and her choices about what sequence of problem-solving steps to take. Bayesian inverse planning builds on existing machine learning tools to create a generative model relating (mis)-understandings to equation solving choices. Two behavioral experiments demonstrate that the model can interpret people's equation solving and that its assessments are consistent with those of experienced teachers. A third experiment uses this model to tailor guidance for learners based on individual differences in misunderstandings, closing the loop between assessing understanding, and using that assessment within an educational technology. Finally, because the bottleneck in applying inverse planning to a new domain is in creating the model of possible student misunderstandings, we show how to combine inverse planning with an existing production rule model to make inferences about student misunderstandings of fraction arithmetic.

Explaining Constrains Causal Learning in Childhood.

Three experiments investigate how self-generated explanation influences children's causal learning. Five-year-olds (N = 114) observed data consistent with two hypotheses and were prompted to explain or to report each observation. In Study 1, when making novel generalizations, explainers were more likely to favor the hypothesis that accounted for more observations. In Study 2, explainers favored a hypothesis that was consistent with prior knowledge. Study 3 pitted a hypothesis that accounted for more observations against a hypothesis consistent with prior knowledge. Explainers were more likely to base generalizations on prior knowledge. Findings suggest that attempts to explain drive children to evaluate hypotheses using features of "good" explanations, or those supporting generalizations with broad scope, as informed by children's prior knowledge and observations.

Faster Teaching via POMDP Planning.

Human and automated tutors attempt to choose pedagogical activities that will maximize student learning, informed by their estimates of the student's current knowledge. There has been substantial research on tracking and modeling student learning, but significantly less attention on how to plan teaching actions and how the assumed student model impacts the resulting plans. We frame the problem of optimally selecting teaching actions using a decision-theoretic approach and show how to formulate teaching as a partially observable Markov decision process planning problem. This framework makes it possible to explore how different assumptions about student learning and behavior should affect the selection of teaching actions. We consider how to apply this framework to concept learning problems, and we present approximate methods for finding optimal teaching actions, given the large state and action spaces that arise in teaching. Through simulations and behavioral experiments, we explore the consequences of choosing teacher actions under different assumed student models. In two concept-learning tasks, we show that this technique can accelerate learning relative to baseline performance.

Inferring learners' knowledge from their actions.

Watching another person take actions to complete a goal and making inferences about that person's knowledge is a relatively natural task for people. This ability can be especially important in educational settings, where the inferences can be used for assessment, diagnosing misconceptions, and providing informative feedback. In this paper, we develop a general framework for automatically making such inferences based on observed actions; this framework is particularly relevant for inferring student knowledge in educational games and other interactive virtual environments. Our approach relies on modeling action planning: We formalize the problem as a Markov decision process in which one must choose what actions to take to complete a goal, where choices will be dependent on one's beliefs about how actions affect the environment. We use a variation of inverse reinforcement learning to infer these beliefs. Through two lab experiments, we show that this model can recover people's beliefs in a simple environment, with accuracy comparable to that of human observers. We then demonstrate that the model can be used to provide real-time feedback and to model data from an existing educational game.

Optimally designing games for behavioural research.

Computer games can be motivating and engaging experiences that facilitate learning, leading to their increasing use in education and behavioural experiments. For these applications, it is often important to make inferences about the knowledge and cognitive processes of players based on their behaviour. However, designing games that provide useful behavioural data are a difficult task that typically requires significant trial and error. We address this issue by creating a new formal framework that extends optimal experiment design, used in statistics, to apply to game design. In this framework, we use Markov decision processes to model players' actions within a game, and then make inferences about the parameters of a cognitive model from these actions. Using a variety of concept learning games, we show that in practice, this method can predict which games will result in better estimates of the parameters of interest. The best games require only half as many players to attain the same level of precision.

Analyzing the rate at which languages lose the influence of a common ancestor.

Analyzing the rate at which languages change can clarify whether similarities across languages are solely the result of cognitive biases or might be partially due to descent from a common ancestor. To demonstrate this approach, we use a simple model of language evolution to mathematically determine how long it should take for the distribution over languages to lose the influence of a common ancestor and converge to a form that is determined by constraints on language learning. We show that modeling language learning as Bayesian inference of n binary parameters or the ordering of n constraints results in convergence in a number of generations that is on the order of n log n. We relax some of the simplifying assumptions of this model to explore how different assumptions about language evolution affect predictions about the time to convergence; in general, convergence time increases as the model becomes more realistic. This allows us to characterize the assumptions about language learning (given the models that we consider) that are sufficient for convergence to have taken place on a timescale that is consistent with the origin of human languages. These results clearly identify the consequences of a set of simple models of language evolution and show how analysis of convergence rates provides a tool that can be used to explore questions about the relationship between accounts of language learning and the origins of similarities across languages.

Greater learnability is not sufficient to produce cultural universals.

Looking across human societies reveals regularities in the languages that people speak and the concepts that they use. One explanation that has been proposed for these "cultural universals" is differences in the ease with which people learn particular languages and concepts. A difference in learnability means that languages and concepts possessing a particular property are more likely to be accurately transmitted from one generation of learners to the next. Intuitively, this difference could allow languages and concepts that are more learnable to become more prevalent after multiple generations of cultural transmission. If this is the case, the prevalence of languages and concepts with particular properties can be explained simply by demonstrating empirically that they are more learnable. We evaluate this argument using mathematical analysis and behavioral experiments. Specifically, we provide two counter-examples that show how greater learnability need not result in a property becoming prevalent. First, more learnable languages and concepts can nonetheless be less likely to be produced spontaneously as a result of transmission failures. We simulated cultural transmission in the laboratory to show that this can occur for memory of distinctive items: these items are more likely to be remembered, but not generated spontaneously once they have been forgotten. Second, when there are many languages or concepts that lack the more learnable property, sheer numbers can swamp the benefit produced by greater learnability. We demonstrate this using a second series of experiments involving artificial language learning. Both of these counter-examples show that simply finding a learnability bias experimentally is not sufficient to explain why a particular property is prevalent in the languages or concepts used in human societies: explanations for cultural universals based on cultural transmission need to consider the full set of hypotheses a learner could entertain and all of the kinds of errors that can occur in transmission.