Psychological proximity improves reasoning in academic aptitude tests.

Performance on standardized academic aptitude tests (AAT) can determine important life outcomes. However, it is not clear whether and which aspects of the content of test questions affect performance. We examined the effect of psychological distance embedded in test questions. In Study 1 (N = 41,209), we classified the content of existing AAT questions as invoking proximal versus distal details. We found better performance with proximal compared to distal questions, especially for low-achieving examinees. Studies 2 and 3 manipulated the distance of questions adapted from AATs and examined three moderators: overall AAT score, working-memory capacity, and presence of irrelevant information. In Study 2 (N = 129), proximity (versus distance) improved the performance of low-achieving participants. In Study 3 (N = 1744), a field study, among low-achieving examinees, proximity improved performance on questions that included irrelevant information. Together, these results suggest that the psychological distance that is invoked by the content of test questions has important consequences for performance in real-life high-stakes tests.

The Effect of the Raters' Marginal Distributions on Their Matched Agreement: A Rescaling Framework for Interpreting Kappa.

Cohen's κ measures the improvement in classification above chance level and it is the most popular measure of interjudge agreement. Yet, there is considerable confusion about its interpretation. Specifically, researchers often ignore the fact that the observed level of matched agreement is bounded from above and below and the bounds are a function of the particular marginal distributions of the table. We propose that these bounds should be used to rescale the components of κ (observed and expected agreement). Rescaling κ in this manner results in κ', a measure that was originally proposed by Cohen (1960) and was largely ignored in both research and practice. This measure provides a common scale for agreement measures of tables with different marginal distributions. It reaches the maximal value of 1 when the judges show the highest level of agreement possible, given their marginal disagreements. We conclude that κ' should be used to measure the level of matched agreement contingent on a particular set of marginal distributions. The article provides a framework and a set of guidelines that facilitate comparisons between various types of agreement tables. We illustrate our points with simulations and real data from two studies-one involving judges' ratings of baseball players and one involving ratings of essays in high-stakes tests.

You say "probable" and I say "likely": improving interpersonal communication with verbal probability phrases.

When forecasters and decision makers describe uncertain events using verbal probability terms, there is a risk of miscommunication because people use different probability phrases and interpret them in different ways. In an effort to facilitate the communication process, the authors investigated various ways of converting the forecasters' verbal probabilities to the decision maker's terms. The authors present 3 studies in which participants judged the probabilities of distinct events using both numerical and verbal probabilities. They define several indexes of interindividual coassignment of phrases to the same events and evaluate the conversion methods by comparing the values of these indexes for the converted and the unconverted judgments. In all the cases studied, the conversion methods significantly reduced the error rates in communicating uncertainties.