BARD: A Structured Technique for Group Elicitation of Bayesian Networks to Support Analytic Reasoning.

In many complex, real-world situations, problem solving and decision making require effective reasoning about causation and uncertainty. However, human reasoning in these cases is prone to confusion and error. Bayesian networks (BNs) are an artificial intelligence technology that models uncertain situations, supporting better probabilistic and causal reasoning and decision making. However, to date, BN methodologies and software require (but do not include) substantial upfront training, do not provide much guidance on either the model building process or on using the model for reasoning and reporting, and provide no support for building BNs collaboratively. Here, we contribute a detailed description and motivation for our new methodology and application, Bayesian ARgumentation via Delphi (BARD). BARD utilizes BNs and addresses these shortcomings by integrating (1) short, high-quality e-courses, tips, and help on demand; (2) a stepwise, iterative, and incremental BN construction process; (3) report templates and an automated explanation tool; and (4) a multiuser web-based software platform and Delphi-style social processes. The result is an end-to-end online platform, with associated online training, for groups without prior BN expertise to understand and analyze a problem, build a model of its underlying probabilistic causal structure, validate and reason with the causal model, and (optionally) use it to produce a written analytic report. Initial experiments demonstrate that, for suitable problems, BARD aids in reasoning and reporting. Comparing their effect sizes also suggests BARD's BN-building and collaboration combine beneficially and cumulatively.

An investigation of the role of some person and situation variables in multiple cue probability learning.

Making decisions using judgements of multiple non-deterministic indicators is an important task, both in everyday and professional life. Learning of such decision making has often been studied as the mapping of stimuli (cues) to an environmental variable (criterion); however, little attention has been paid to the effects of situation-by-person interactions on this learning. Accordingly, we manipulated cue and feedback presentation mode (graphic or numeric) and task difficulty, and measured individual differences in working memory capacity (WMC). We predicted that graphic presentation, fewer cues, and elevated WMC would facilitate learning, and that person and task characteristics would interact such that presentation mode compatible with the decision maker's cognitive capability (enhanced visual or verbal WMC) would assist learning, particularly for more difficult tasks. We found our predicted main effects, but no significant interactions, except that those with greater WMC benefited to a larger extent with graphic than with numeric presentation, regardless of which type of working memory was enhanced or number of cues. Our findings suggest that the conclusions of past research based predominantly on tasks using numeric presentation need to be reevaluated and cast light on how working memory helps us learn multiple cue-criterion relationships, with implications for dual-process theories of cognition.

The aggregation of expert judgment: do good things come to those who weight?

Good policy making should be based on available scientific knowledge. Sometimes this knowledge is well established through research, but often scientists must simply express their judgment, and this is particularly so in risk scenarios that are characterized by high levels of uncertainty. Usually in such cases, the opinions of several experts will be sought in order to pool knowledge and reduce error, raising the question of whether individual expert judgments should be given different weights. We argue--against the commonly advocated "classical method"--that no significant benefits are likely to accrue from unequal weighting in mathematical aggregation. Our argument hinges on the difficulty of constructing reliable and valid measures of substantive expertise upon which to base weights. Practical problems associated with attempts to evaluate experts are also addressed. While our discussion focuses on one specific weighting scheme that is currently gaining in popularity for expert knowledge elicitation, our general thesis applies to externally imposed unequal weighting schemes more generally.

Market entry decisions: effects of absolute and relative confidence.

In a market entry game, the number of entrants usually approaches game-theoretic equilibrium quickly, but in real-world markets business start-ups typically exceed market capacity, resulting in chronically high failure rates and suboptimal industry profits. Excessive entry has been attributed to overconfidence arising when expected payoffs depend partly on skill. In an experimental test of this hypothesis, 96 participants played 24 rounds of a market entry game, with expected payoffs dependent partly on skill on half the rounds, after their confidence was manipulated and measured. The results provide direct support for the hypothesis that high levels of confidence are largely responsible for excessive entry, and they suggest that absolute confidence, independent of interpersonal comparison, rather than confidence about one's abilities relative to others, drives excessive entry decisions when skill is involved.

An empirical test of the relative validity of expert and lay judgments of risk.

This article investigates how accurately experts (underwriters) and lay persons (university students) judge the risks posed by life-threatening events Only one prior study (Slovic, Fischhoff, & Lichtenstein, 1985) has previously investigated the veracity of expert versus lay judgments of the magnitude of risk. In that study, a heterogeneous grouping of 15 experts was found to judge, using marginal estimations, a variety of risks as closer to the true annual frequencies of death than convenience samples of the lay population. In this study, we use a larger, homogenous sample of experts performing an ecologically valid task. We also ask our respondents to assess frequencies and relative frequencies directly, rather than ask for a "risk" estimate--a response mode subject to possible qualitative attributions-as was done in the Slovic et al. study. Although we find that the experts outperformed lay persons on a number of measures, the differences are small, and both groups showed similar global biases in terms of: (1) overestimating the likelihood of dying from a condition (marginal probability) and of dying from a condition given that it happens to you (conditional probability), and (2) underestimating the ratios of marginal and conditional likelihoods between pairs of potentially lethal events. In spite of these scaling problems, both groups showed quite good performance in ordering the lethal events in terms of marginal and conditional likelihoods. We discuss the nature of expertise using a framework developed by Bolger and Wright (1994), and consider whether the commonsense assumption of the superiority of expert risk assessors in making magnitude judgments of risk is, in fact, sensible.