Taboos and Self-Censorship Among U.S. Psychology Professors.

We identify points of conflict and consensus regarding (a) controversial empirical claims and (b) normative preferences for how controversial scholarship-and scholars-should be treated. In 2021, we conducted qualitative interviews (n = 41) to generate a quantitative survey (N = 470) of U.S. psychology professors' beliefs and values. Professors strongly disagreed on the truth status of 10 candidate taboo conclusions: For each conclusion, some professors reported 100% certainty in its veracity and others 100% certainty in its falsehood. Professors more confident in the truth of the taboo conclusions reported more self-censorship, a pattern that could bias perceived scientific consensus regarding the inaccuracy of controversial conclusions. Almost all professors worried about social sanctions if they were to express their own empirical beliefs. Tenured professors reported as much self-censorship and as much fear of consequences as untenured professors, including fear of getting fired. Most professors opposed suppressing scholarship and punishing peers on the basis of moral concerns about research conclusions and reported contempt for peers who petition to retract papers on moral grounds. Younger, more left-leaning, and female faculty were generally more opposed to controversial scholarship. These results do not resolve empirical or normative disagreements among psychology professors, but they may provide an empirical context for their discussion.

Assume a can opener.

We propose a friendly amendment to integrative experiment design (IED), adversarial-collaboration IED, that incentivizes research teams from competing theoretical perspectives to identify zones of the design space where they possess an explanatory edge. This amendment is especially critical in debates that have high policy stakes and carry a strong normative-political charge that might otherwise prevent free exchange of ideas.

Prosocial motives underlie scientific censorship by scientists: A perspective and research agenda.

Science is among humanity's greatest achievements, yet scientific censorship is rarely studied empirically. We explore the social, psychological, and institutional causes and consequences of scientific censorship (defined as actions aimed at obstructing particular scientific ideas from reaching an audience for reasons other than low scientific quality). Popular narratives suggest that scientific censorship is driven by authoritarian officials with dark motives, such as dogmatism and intolerance. Our analysis suggests that scientific censorship is often driven by scientists, who are primarily motivated by self-protection, benevolence toward peer scholars, and prosocial concerns for the well-being of human social groups. This perspective helps explain both recent findings on scientific censorship and recent changes to scientific institutions, such as the use of harm-based criteria to evaluate research. We discuss unknowns surrounding the consequences of censorship and provide recommendations for improving transparency and accountability in scientific decision-making to enable the exploration of these unknowns. The benefits of censorship may sometimes outweigh costs. However, until costs and benefits are examined empirically, scholars on opposing sides of ongoing debates are left to quarrel based on competing values, assumptions, and intuitions.

Validating a forced-choice method for eliciting quality-of-reasoning judgments.

In this paper we investigate the criterion validity of forced-choice comparisons of the quality of written arguments with normative solutions. Across two studies, novices and experts assessing quality of reasoning through a forced-choice design were both able to choose arguments supporting more accurate solutions-62.2% (SE = 1%) of the time for novices and 74.4% (SE = 1%) for experts-and arguments produced by larger teams-up to 82% of the time for novices and 85% for experts-with high inter-rater reliability, namely 70.58% (95% CI = 1.18) agreement for novices and 80.98% (95% CI = 2.26) for experts. We also explored two methods for increasing efficiency. We found that the number of comparative judgments needed could be substantially reduced with little accuracy loss by leveraging transitivity and producing quality-of-reasoning assessments using an AVL tree method. Moreover, a regression model trained to predict scores based on automatically derived linguistic features of participants' judgments achieved a high correlation with the objective accuracy scores of the arguments in our dataset. Despite the inherent subjectivity involved in evaluating differing quality of reasoning, the forced-choice paradigm allows even novice raters to perform beyond chance and can provide a valid, reliable, and efficient method for producing quality-of-reasoning assessments at scale.

Human and Algorithmic Predictions in Geopolitical Forecasting: Quantifying Uncertainty in Hard-to-Quantify Domains.

Research on clinical versus statistical prediction has demonstrated that algorithms make more accurate predictions than humans in many domains. Geopolitical forecasting is an algorithm-unfriendly domain, with hard-to-quantify data and elusive reference classes that make predictive model-building difficult. Furthermore, the stakes can be high, with missed forecasts leading to mass-casualty consequences. For these reasons, geopolitical forecasting is typically done by humans, even though algorithms play important roles. They are essential as aggregators of crowd wisdom, as frameworks to partition human forecasting variance, and as inputs to hybrid forecasting models. Algorithms are extremely important in this domain. We doubt that humans will relinquish control to algorithms anytime soon-nor do we think they should. However, the accuracy of forecasts will greatly improve if humans are aided by algorithms.

AI and the transformation of social science research.

Careful bias management and data fidelity are key.

Harm Hypervigilance in Public Reactions to Scientific Evidence.

Two preregistered studies from two different platforms with representative U.S. adult samples (N = 1,865) tested the harm-hypervigilance hypothesis in risk assessments of controversial behavioral science. As expected, across six sets of scientific findings, people consistently overestimated others' harmful reactions (medium to large average effect sizes) and underestimated helpful ones, even when incentivized for accuracy. Additional analyses found that (a) harm overestimations were associated with support for censoring science, (b) people who were more offended by scientific findings reported greater difficulty understanding them, and (c) evidence was moderately consistent for an association between more conservative ideology and harm overestimations. These findings are particularly relevant because journals have begun evaluating potential downstream harms of scientific findings. We discuss implications of our work and invite scholars to develop rigorous tests of (a) the social pressures that lead science astray and (b) the actual costs and benefits of publishing or not publishing potentially controversial conclusions.

The internal validity obsession.

Until social psychology devotes as much attention to construct and external validity as it does to internal validity, the field will continue to produce theories that fail to replicate in the field and cannot be used to meliorate social problems.

From discipline-centered rivalries to solution-centered science: Producing better probability estimates for policy makers.

From 2011 to 2015, the U.S. intelligence community sponsored a series of forecasting tournaments that challenged university-based researchers to invent measurably better methods of forecasting political events. Our group, the Good Judgment Project, won these tournaments by balancing the collaboration and competition of members across disciplines. At the outset, psychologists were ahead of economists in identifying individual differences in forecasting skill and developing methods of debiasing forecasts, whereas economists were ahead of psychologists in designing simple market mechanisms that distilled predictive signals from noisy individual-level data. Working closely with statisticians, psychologists eventually beat the markets by producing better probability estimates that funneled top forecasters into elite teams and aggregated their judgments using a log-odds formula tuned to the diversity of the forecasters. Our research group performed best when team members strove to get as much as possible from their home disciplines, but acknowledged their limitations and welcomed help from outsiders. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Forecasting tournaments, epistemic humility and attitude depolarization.

People often express political opinions in starkly dichotomous terms, such as "Trump will either trigger a ruinous trade war or save U.S. factory workers from disaster." This mode of communication promotes polarization into ideological in-groups and out-groups. We explore the power of an emerging methodology, forecasting tournaments, to encourage clashing factions to do something odd: to translate their beliefs into nuanced probability judgments and track accuracy over time and questions. In theory, tournaments advance the goals of "deliberative democracy" by incentivizing people to be flexible belief updaters whose views converge in response to facts, thus depolarizing unnecessarily polarized debates. We examine the hypothesis that, in the process of thinking critically about their beliefs, tournament participants become more moderate in their own political attitudes and those they attribute to the other side. We view tournaments as belonging to a broader class of psychological inductions that increase epistemic humility and that include asking people to explore alternative perspectives, probing the depth of their cause-effect understanding and holding them accountable to audiences with difficult-to-guess views.

Correcting Judgment Correctives in National Security Intelligence.

Intelligence analysts, like other professionals, form norms that define standards of tradecraft excellence. These norms, however, have evolved in an idiosyncratic manner that reflects the influence of prominent insiders who had keen psychological insights but little appreciation for how to translate those insights into testable hypotheses. The net result is that the prevailing tradecraft norms of best practice are only loosely grounded in the science of judgment and decision-making. The "common sense" of prestigious opinion leaders inside the intelligence community has pre-empted systematic validity testing of the training techniques and judgment aids endorsed by those opinion leaders. Drawing on the scientific literature, we advance hypotheses about how current best practices could well be reducing rather than increasing the quality of analytic products. One set of hypotheses pertain to the failure of tradecraft training to recognize the most basic threat to accuracy: measurement error in the interpretation of the same data and in the communication of interpretations. Another set of hypotheses focuses on the insensitivity of tradecraft training to the risk that issuing broad-brush, one-directional warnings against bias (e.g., over-confidence) will be less likely to encourage self-critical, deliberative cognition than simple response-threshold shifting that yields the mirror-image bias (e.g., under-confidence). Given the magnitude of the consequences of better and worse intelligence analysis flowing to policy-makers, we see a compelling case for greater funding of efforts to test what actually works.

Bringing probability judgments into policy debates via forecasting tournaments.

Political debates often suffer from vague-verbiage predictions that make it difficult to assess accuracy and improve policy. A tournament sponsored by the U.S. intelligence community revealed ways in which forecasters can better use probability estimates to make predictions-even for seemingly "unique" events-and showed that tournaments are a useful tool for generating knowledge. Drawing on the literature about the effects of accountability, the authors suggest that tournaments may hold even greater potential as tools for depolarizing political debates and resolving policy disputes.

Debunking the Myth of Value-Neutral Virginity: Toward Truth in Scientific Advertising.

The scientific community often portrays science as a value-neutral enterprise that crisply demarcates facts from personal value judgments. We argue that this depiction is unrealistic and important to correct because science serves an important knowledge generation function in all modern societies. Policymakers often turn to scientists for sound advice, and it is important for the wellbeing of societies that science delivers. Nevertheless, scientists are human beings and human beings find it difficult to separate the epistemic functions of their judgments (accuracy) from the social-economic functions (from career advancement to promoting moral-political causes that "feel self-evidently right"). Drawing on a pluralistic social functionalist framework that identifies five functionalist mindsets-people as intuitive scientists, economists, politicians, prosecutors, and theologians-we consider how these mindsets are likely to be expressed in the conduct of scientists. We also explore how the context of policymaker advising is likely to activate or de-activate scientists' social functionalist mindsets. For instance, opportunities to advise policymakers can tempt scientists to promote their ideological beliefs and values, even if advising also brings with it additional accountability pressures. We end prescriptively with an appeal to scientists to be more circumspect in characterizing their objectivity and honesty and to reject idealized representations of scientific behavior that inaccurately portray scientists as value-neutral virgins.

Perceptual dehumanization of faces is activated by norm violations and facilitates norm enforcement.

This article uses methods drawn from perceptual psychology to answer a basic social psychological question: Do people process the faces of norm violators differently from those of others--and, if so, what is the functional significance? Seven studies suggest that people process these faces different and the differential processing makes it easier to punish norm violators. Studies 1 and 2 use a recognition-recall paradigm that manipulated facial-inversion and spatial frequency to show that people rely upon face-typical processing less when they perceive norm violators' faces. Study 3 uses a facial composite task to demonstrate that the effect is actor dependent, not action dependent, and to suggest that configural processing is the mechanism of perceptual change. Studies 4 and 5 use offset faces to show that configural processing is only attenuated when they belong to perpetrators who are culpable. Studies 6 and 7 show that people find it easier to punish inverted faces and harder to punish faces displayed in low spatial frequency. Taken together, these data suggest a bidirectional flow of causality between lower-order perceptual and higher-order cognitive processes in norm enforcement.

Balancing Fairness and Efficiency: The Impact of Identity-Blind and Identity-Conscious Accountability on Applicant Screening.

This study compared two forms of accountability that can be used to promote diversity and fairness in personnel selections: identity-conscious accountability (holding decision makers accountable for which groups are selected) versus identity-blind accountability (holding decision makers accountable for making fair selections). In a simulated application screening process, undergraduate participants (majority female) sorted applicants under conditions of identity-conscious accountability, identity-blind accountability, or no accountability for an applicant pool in which white males either did or did not have a human capital advantage. Under identity-conscious accountability, participants exhibited pro-female and pro-minority bias, particularly in the white-male-advantage applicant pool. Under identity-blind accountability, participants exhibited no biases and candidate qualifications dominated interview recommendations. Participants exhibited greater resentment toward management under identity-conscious accountability.

Improving Intelligence Analysis With Decision Science.

Intelligence analysis plays a vital role in policy decision making. Key functions of intelligence analysis include accurately forecasting significant events, appropriately characterizing the uncertainties inherent in such forecasts, and effectively communicating those probabilistic forecasts to stakeholders. We review decision research on probabilistic forecasting and uncertainty communication, drawing attention to findings that could be used to reform intelligence processes and contribute to more effective intelligence oversight. We recommend that the intelligence community (IC) regularly and quantitatively monitor its forecasting accuracy to better understand how well it is achieving its functions. We also recommend that the IC use decision science to improve these functions (namely, forecasting and communication of intelligence estimates made under conditions of uncertainty). In the case of forecasting, decision research offers suggestions for improvement that involve interventions on data (e.g., transforming forecasts to debias them) and behavior (e.g., via selection, training, and effective team structuring). In the case of uncertainty communication, the literature suggests that current intelligence procedures, which emphasize the use of verbal probabilities, are ineffective. The IC should, therefore, leverage research that points to ways in which verbal probability use may be improved as well as exploring the use of numerical probabilities wherever feasible.

Identifying and cultivating superforecasters as a method of improving probabilistic predictions.

Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created-and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.

Using the IAT to predict ethnic and racial discrimination: small effect sizes of unknown societal significance.

Greenwald, Banaji, and Nosek (2015) present a reanalysis of the meta-analysis by Oswald, Mitchell, Blanton, Jaccard, and Tetlock (2013) that examined the effect sizes of Implicit Association Tests (IATs) designed to predict racial and ethnic discrimination. We discuss points of agreement and disagreement with respect to methods used to synthesize the IAT studies, and we correct an error by Greenwald et al. that obscures a key contribution of our meta-analysis. In the end, all of the meta-analyses converge on the conclusion that, across diverse methods of coding and analyzing the data, IAT scores are not good predictors of ethnic or racial discrimination, and explain, at most, small fractions of the variance in discriminatory behavior in controlled laboratory settings. The thought experiments presented by Greenwald et al. go well beyond the lab to claim systematic IAT effects in noisy real-world settings, but these hypothetical exercises depend crucially on untested and, arguably, untenable assumptions.

Brutality under cover of ambiguity: activating, perpetuating, and deactivating covert retributivism.

Five studies tested four hypotheses on the drivers of punitive judgments. Study 1 showed that people imposed covertly retributivist physical punishments on extreme norm violators when they could plausibly deny that is what they were doing (attributional ambiguity). Studies 2 and 3 showed that covert retributivism could be suppressed by subtle accountability manipulations that cue people to the possibility that they might be under scrutiny. Studies 4 and 5 showed how covert retributivism can become self-sustaining by biasing the lessons people learn from experience. Covert retributivists did not scale back punitiveness in response to feedback that the justice system makes false-conviction errors but they did ramp up punitiveness in response to feedback that the system makes false-acquittal errors. Taken together, the results underscore the paradoxical nature of covert retributivism: It is easily activated by plausible deniability and persistent in the face of false-conviction feedback but also easily deactivated by minimalist forms of accountability.

The psychology of intelligence analysis: drivers of prediction accuracy in world politics.

This article extends psychological methods and concepts into a domain that is as profoundly consequential as it is poorly understood: intelligence analysis. We report findings from a geopolitical forecasting tournament that assessed the accuracy of more than 150,000 forecasts of 743 participants on 199 events occurring over 2 years. Participants were above average in intelligence and political knowledge relative to the general population. Individual differences in performance emerged, and forecasting skills were surprisingly consistent over time. Key predictors were (a) dispositional variables of cognitive ability, political knowledge, and open-mindedness; (b) situational variables of training in probabilistic reasoning and participation in collaborative teams that shared information and discussed rationales (Mellers, Ungar, et al., 2014); and (c) behavioral variables of deliberation time and frequency of belief updating. We developed a profile of the best forecasters; they were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness. They had greater understanding of geopolitics, training in probabilistic reasoning, and opportunities to succeed in cognitively enriched team environments. Last but not least, they viewed forecasting as a skill that required deliberate practice, sustained effort, and constant monitoring of current affairs.