The wisdom of many in few: Finding individuals who are as wise as the crowd.

Is forecasting ability a stable trait? While domain knowledge and reasoning abilities are necessary for making accurate forecasts, research shows that knowing how accurate forecasters have been in the past is the best predictor of future accuracy. However, unlike the measurement of other traits, evaluating forecasting skill requires substantial time investment. Forecasters must make predictions about events that may not resolve for many days, weeks, months, or even years into the future before their accuracy can be estimated. Our work builds upon methods such as cultural consensus theory and proxy scoring rules to show talented forecasters can be discriminated in real time, without requiring any event resolutions. We define a peer similarity-based intersubjective evaluation method and test its utility in a unique longitudinal forecasting experiment. Because forecasters predicted all events at the same points in time, many of the confounds common to forecasting tournaments or observational data were eliminated. This allowed us to demonstrate the effectiveness of our method in real time, as time progressed and more information about forecasters became available. Intersubjective accuracy scores, which can be obtained immediately after the forecasts are made, were both valid and reliable estimators of forecasting talent. We also found that asking forecasters to make meta-predictions about what they expect others to believe can serve as an incentive-compatible method of intersubjective evaluation. Our results indicate that selecting small groups of, or even single forecasters, based on intersubjective accuracy can yield subsequent forecasts that approximate the actual accuracy of much larger crowd aggregates. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

To deliberate or not? The role of intuition and deliberation when controlling for irrelevant information in selection decisions.

In selection decisions, decision makers often struggle to ignore irrelevant information, such as candidates' age, gender and attractiveness, which can lead to suboptimal decisions. One way to correct the effects of these irrelevant attributes is to consider them as suppressor variables, and penalize individuals who unjustifiably benefit from them. Previous research demonstrated that people have difficulties doing so. In five experiments (N = 1325), we examined the mechanism at the core of people's ability to do so. We found that triggering System 2 did not improve participants' ability to correct for this bias. The majority of those who were successful did so even when denied the opportunity to deliberate. We suggest that logic intuition-not deliberation-is the basis for successfully considering irrelevant information as suppressors. Our results are in line with a revised dual-process approach, in which solving reasoning problems can occur directly through System 1 and does not require an override by a System 2's-based process.

A survey of human judgement and quantitative forecasting methods.

This paper's top-level goal is to provide an overview of research conducted in the many academic domains concerned with forecasting. By providing a summary encompassing these domains, this survey connects them, establishing a common ground for future discussions. To this end, we survey literature on human judgement and quantitative forecasting as well as hybrid methods that involve both humans and algorithmic approaches. The survey starts with key search terms that identified more than 280 publications in the fields of computer science, operations research, risk analysis, decision science, psychology and forecasting. Results show an almost 10-fold increase in the application-focused forecasting literature between the 1990s and the current decade, with a clear rise of quantitative, data-driven forecasting models. Comparative studies of quantitative methods and human judgement show that (1) neither method is universally superior, and (2) the better method varies as a function of factors such as availability, quality, extent and format of data, suggesting that (3) the two approaches can complement each other to yield more accurate and resilient models. We also identify four research thrusts in the human/machine-forecasting literature: (i) the choice of the appropriate quantitative model, (ii) the nature of the interaction between quantitative models and human judgement, (iii) the training and incentivization of human forecasters, and (iv) the combination of multiple forecasts (both algorithmic and human) into one. This review surveys current research in all four areas and argues that future research in the field of human/machine forecasting needs to consider all of them when investigating predictive performance. We also address some of the ethical dilemmas that might arise due to the combination of quantitative models with human judgement.

Achieving More With Less: Intuitive Correction in Selection.

Choosing between candidates for a position can be tricky, especially when the selection test is affected by irrelevant characteristics (e.g., reading speed). One can correct for this irrelevant attribute by penalizing individuals who have unjustifiably benefited from it. Statistical models do so by including the irrelevant attribute as a suppressor variable, but can people do the same without the help of a model? In three experiments (total N = 357), participants had to choose between two candidates, one of whom had higher levels of an irrelevant attribute and thus enjoyed an unfair advantage. Participants showed a substantial preference for the candidate with high levels of the irrelevant attribute, thus choosing the less suitable candidate. This bias was attenuated when the irrelevant attribute was a situational factor, probably by making the correction process more intuitive. Understanding the intuitive judgment of suppressor variables can help candidates from underprivileged groups boost their chances to succeed.

Does the perceived accuracy of urine drug testing impact clinical decision-making?

Background: Urine drug testing techniques have different rates of false-positive and false-negative test results. However, clinicians may have highly varying perceptions of test accuracy and may compensate for perceived inaccuracy by incorporating other factors into their interpretation of observed test results. Thus, there is the potential for adverse consequences from decisions based on inaccurate test results or interpretation. Methods: We surveyed 466 members of the American Society of Addiction Medicine to examine clinicians' perceptions of the accuracy of 2 types of urine drug tests, immunoassay (IA) and liquid chromatography-tandem mass spectrometry (LC-MS/MS), and the extent to which behavioral and demographic factors influence the interpretation of test results. Participants read 4 brief vignettes describing positive and negative test results in hypothetical patients who differed along several dimensions (gender, age, race/ethnicity, comorbid mental disorder, court-ordered versus voluntary status, treatment compliance). Outcome variables include likelihood of renewed drug use, likelihood of test error, whether to request additional testing, and whether to report the violation to a probation officer. Results: The strongest predictor of study outcomes was treatment compliance (consistent versus inconsistent attendance), as this was the only independent variable to generate effect sizes of medium strength. Significant effect sizes were also found for type of test used (IA versus LC-MS/MS), legal status (court-mandated versus voluntary), presence of a comorbid mental disorder, treatment history, and race, although effect sizes for these variables were small and less consistently observed. Conclusions: These results highlight the potential for error in clinician judgments about urine drug testing. Not only were participants likely to underestimate the accuracy of "confirmatory" LC-MS/MS testing, but vignettes suggested that a number of historical and demographic factors may influence interpretation of test results.

The Role of Type and Source of Uncertainty on the Processing of Climate Models Projections.

Scientists agree that the climate is changing due to human activities, but there is less agreement about the specific consequences and their timeline. Disagreement among climate projections is attributable to the complexity of climate models that differ in their structure, parameters, initial conditions, etc. We examine how different sources of uncertainty affect people's interpretation of, and reaction to, information about climate change by presenting participants forecasts from multiple experts. Participants viewed three types of sets of sea-level rise projections: (1) precise, but conflicting; (2) imprecise, but agreeing, and (3) hybrid that were both conflicting and imprecise. They estimated the most likely sea-level rise, provided a range of possible values and rated the sets on several features - ambiguity, credibility, completeness, etc. In Study 1, everyone saw the same hybrid set. We found that participants were sensitive to uncertainty between sources, but not to uncertainty about which model was used. The impacts of conflict and imprecision were combined for estimation tasks and compromised for feature ratings. Estimates were closer to the experts' original projections, and sets were rated more favorably under imprecision. Estimates were least consistent with (narrower than) the experts in the hybrid condition, but participants rated the conflicting set least favorably. In Study 2, we investigated the hybrid case in more detail by creating several distinct interval sets that combine conflict and imprecision. Two factors drive perceptual differences: overlap - the structure of the forecast set (whether intersecting, nested, tangent, or disjoint) - and asymmetry - the balance of the set. Estimates were primarily driven by asymmetry, and preferences were primarily driven by overlap. Asymmetric sets were least consistent with the experts: estimated ranges were narrower, and estimates of the most likely value were shifted further below the set mean. Intersecting and nested sets were rated similarly to imprecision, and ratings of disjoint and tangent sets were rated like conflict. Our goal was to determine which underlying factors of information sets drive perceptions of uncertainty in consistent, predictable ways. The two studies lead us to conclude that perceptions of agreement require intersection and balance, and overly precise forecasts lead to greater perceptions of disagreement and a greater likelihood of the public discrediting and misinterpreting information.

Analyzing Test-Taking Behavior: Decision Theory Meets Psychometric Theory.

We investigate the implications of penalizing incorrect answers to multiple-choice tests, from the perspective of both test-takers and test-makers. To do so, we use a model that combines a well-known item response theory model with prospect theory (Kahneman and Tversky, Prospect theory: An analysis of decision under risk, Econometrica 47:263-91, 1979). Our results reveal that when test-takers are fully informed of the scoring rule, the use of any penalty has detrimental effects for both test-takers (they are always penalized in excess, particularly those who are risk averse and loss averse) and test-makers (the bias of the estimated scores, as well as the variance and skewness of their distribution, increase as a function of the severity of the penalty).

Verbal and numerical consumer recommendations: switching between recommendation formats leads to preference inconsistencies.

Many Web sites provide consumers with product recommendations, which are typically presented by a sequence of verbal reviews and numerical ratings. In three experiments, we demonstrate that when participants switch between formats (e.g., from verbal to numerical), they are more prone to preference inconsistencies than when they aggregate the recommendations within the same format (e.g., verbal). When evaluating recommendations, participants rely primarily on central-location measures (e.g., mean) and less on other distribution characteristics (e.g., variance). We explain our findings within the theoretical framework of stimulus-response compatibility and we make practical recommendations for the design of recommendation systems and Web portals.

The Effect of the Raters' Marginal Distributions on Their Matched Agreement: A Rescaling Framework for Interpreting Kappa.

Cohen's κ measures the improvement in classification above chance level and it is the most popular measure of interjudge agreement. Yet, there is considerable confusion about its interpretation. Specifically, researchers often ignore the fact that the observed level of matched agreement is bounded from above and below and the bounds are a function of the particular marginal distributions of the table. We propose that these bounds should be used to rescale the components of κ (observed and expected agreement). Rescaling κ in this manner results in κ', a measure that was originally proposed by Cohen (1960) and was largely ignored in both research and practice. This measure provides a common scale for agreement measures of tables with different marginal distributions. It reaches the maximal value of 1 when the judges show the highest level of agreement possible, given their marginal disagreements. We conclude that κ' should be used to measure the level of matched agreement contingent on a particular set of marginal distributions. The article provides a framework and a set of guidelines that facilitate comparisons between various types of agreement tables. We illustrate our points with simulations and real data from two studies-one involving judges' ratings of baseball players and one involving ratings of essays in high-stakes tests.

How to measure diversity when you must.

Racial/ethnic diversity has become an increasingly important variable in the social sciences. Research from multiple disciplines consistently demonstrates the tremendous impact of ethnic diversity on individuals and organizations. Investigators use a variety of measures, and their choices can affect the conclusions that can be drawn and limit the ability to compare and generalize results across studies effectively. The current article reviews 3 popular approaches to the measurement of diversity: the simplistic majority-minority approach and 2 multiple categories variants, the generalized variance and the lesser used entropy statistic. We discuss the properties of each approach and reject the majority-minority approach. We provide 5 examples using the generalized variance and entropy statistics and illustrate their versatility and flexibility. We urge investigators to adopt these multicategory measures and to use our discussion to determine which measure of diversity is most appropriate given the nature of one's data set and research question.

Improving communication of uncertainty in the reports of the intergovernmental panel on climate change.

The Intergovernmental Panel on Climate Change (IPCC) assesses information relevant to the understanding of climate change and explores options for adaptation and mitigation. The IPCC reports communicate uncertainty by using a set of probability terms accompanied by global interpretational guidelines. The judgment literature indicates that there are large differences in the way people understand such phrases, and that their use may lead to confusion and errors in communication. We conducted an experiment in which subjects read sentences from the 2007 IPCC report and assigned numerical values to the probability terms. The respondents' judgments deviated significantly from the IPCC guidelines, even when the respondents had access to these guidelines. These results suggest that the method used by the IPCC is likely to convey levels of imprecision that are too high. We propose an alternative form of communicating uncertainty, illustrate its effectiveness, and suggest several additional ways to improve the communication of uncertainty.

Predicting World Cup results: do goals seem more likely when they pay off?

Bar-Hillel and Budescu (1995) failed to find a desirability bias in probability estimation. The World Cup soccer tournament provided an opportunity to revisit the phenomenon in a context in which desirability biases are notoriously rampant. Participants estimated the probabilities of various teams' winning their upcoming games. They were promised money if one team-randomly designated by the experimenter-won its upcoming game. Participants assigned a higher probability to a victory by their target team than did other participants, whose promised monetary reward was contingent on the victory of its opponent. Prima facie, this seems to be a desirability bias. However, in a follow-up study that made one team salient, without promising monetary rewards, participants also judged their target team to be more likely to win. On grounds of parsimony, we conclude that what appears to be a desirability bias may just be a salience/marking effect, and-although optimism is a robust and ubiquitous human phenomenon-that wishful thinking still remains elusive.

Collective induction without cooperation? Learning and knowledge transfer in cooperative groups and competitive auctions.

There is strong evidence that groups perform better than individuals do on intellective tasks with demonstrably correct solutions. Typically, these studies assume that group members share common goals. The authors extend this line of research by replacing standard face-to-face group interactions with competitive auctions, allowing for conflicting individual incentives. In a series of studies involving the well-known Wason selection task, they demonstrate that competitive auctions induce learning effects equally impressive as those of standard group interactions, and they uncover specific and general knowledge transfers from these institutions to new reasoning problems. The authors identify payoff feedback and information pooling as the driving factors underlying these findings, and they explain these factors within the theoretical framework of collective induction.

You say "probable" and I say "likely": improving interpersonal communication with verbal probability phrases.

When forecasters and decision makers describe uncertain events using verbal probability terms, there is a risk of miscommunication because people use different probability phrases and interpret them in different ways. In an effort to facilitate the communication process, the authors investigated various ways of converting the forecasters' verbal probabilities to the decision maker's terms. The authors present 3 studies in which participants judged the probabilities of distinct events using both numerical and verbal probabilities. They define several indexes of interindividual coassignment of phrases to the same events and evaluate the conversion methods by comparing the values of these indexes for the converted and the unconverted judgments. In all the cases studied, the conversion methods significantly reduced the error rates in communicating uncertainties.

Evidence that androgenic and estrogenic metabolites contribute to the effects of dehydroepiandrosterone on cognition in postmenopausal women.

Prior studies of the effects of dehydroepiandrosterone (DHEA) on cognition have produced complex and inconsistent results. We hypothesize that these results may arise, in part, because of DHEA's metabolism into estrogens and androgens that produce opposing effects on cognition. Our study administered 50 mg of oral DHEA daily for 4 weeks in a placebo-controlled crossover design to six postmenopausal women. We measured blood levels of androgens (total testosterone, free testosterone, DHEA, DHEAS), estrogens (estradiol, estrone), and cognitive performance on recognition memory, perceptual identification, digit span memory, and visual attentional vigilance under both drug and placebo conditions. Multiple regression models incorporating the factors of age and body mass index (BMI) were used to ascertain the relation between sex steroids and cognitive performance. Our results demonstrated that estrogens produced a positive effect on recognition memory, while androgens produced a negative effect. This pattern reversed in perceptual identification with estrogens producing a negative effect and androgens producing a positive effect. In addition, BMI produced a negative effect on digit span memory, age produced a negative effect on perceptual identification, and androgens produced a negative effect on visual attentional vigilance. These results help, in part, to explain DHEA's complex effects on cognition. The diverse effects of sex steroids across tasks underscore the importance of identifying the specific cognitive mechanisms influenced by sex steroids and emphasizes that one should not expect sex steroids to produce homogeneous effects across cognitive tasks.

The dominance analysis approach for comparing predictors in multiple regression.

A general method is presented for comparing the relative importance of predictors in multiple regression. Dominance analysis (D. V. Budescu, 1993), a procedure that is based on an examination of the R2 values for all possible subset models, is refined and extended by introducing several quantitative measures of dominance that differ in the strictness of the dominance definition. These are shown to be intuitive, meaningful, and informative measures that can address a variety of research questions pertaining to predictor importance. The bootstrap is used to assess the stability of dominance results across repeated sampling, and it is shown that these methods provide the researcher with more insights into the pattern of importance in a set of predictors than were previously available.