Validating a forced-choice method for eliciting quality-of-reasoning judgments.

In this paper we investigate the criterion validity of forced-choice comparisons of the quality of written arguments with normative solutions. Across two studies, novices and experts assessing quality of reasoning through a forced-choice design were both able to choose arguments supporting more accurate solutions-62.2% (SE = 1%) of the time for novices and 74.4% (SE = 1%) for experts-and arguments produced by larger teams-up to 82% of the time for novices and 85% for experts-with high inter-rater reliability, namely 70.58% (95% CI = 1.18) agreement for novices and 80.98% (95% CI = 2.26) for experts. We also explored two methods for increasing efficiency. We found that the number of comparative judgments needed could be substantially reduced with little accuracy loss by leveraging transitivity and producing quality-of-reasoning assessments using an AVL tree method. Moreover, a regression model trained to predict scores based on automatically derived linguistic features of participants' judgments achieved a high correlation with the objective accuracy scores of the arguments in our dataset. Despite the inherent subjectivity involved in evaluating differing quality of reasoning, the forced-choice paradigm allows even novice raters to perform beyond chance and can provide a valid, reliable, and efficient method for producing quality-of-reasoning assessments at scale.

The workload capacity of semantic search in convergent thinking.

The present study used Systems Factorial Technology (Townsend & Nozawa, 1995) to investigate how people combine dual cues in semantic memory search. Our aims were (a) to understand how cues interact during the process of semantic search in convergent thinking and (b) to determine how workload capacity (i.e. cue-processing efficiency) is related to search performance. In two experiments, participants completed a typical convergent thinking test and a word production task. The results revealed that: (a) collective evidence supports similar patterns in cue-combination strategy despite individual differences in workload capacity, and (b) there exists a negative correlation between workload capacity and performance on convergent thinking test. A potential explanation is that, for the creative individual, loading many candidate answers leads to consumption of substantial processing resources that obtains as low workload capacity but also allows creative individuals to switch more easily from one candidate to another so that there is a higher probability of successfully producing an answer within a limited time. Our results further imply that workload capacity is a significant factor for the semantic search process in convergent thinking and provides new insight on the model of semantic search and creativity. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Naming unrelated words predicts creativity.

Several theories posit that creative people are able to generate more divergent ideas. If this is correct, simply naming unrelated words and then measuring the semantic distance between them could serve as an objective measure of divergent thinking. To test this hypothesis, we asked 8,914 participants to name 10 words that are as different from each other as possible. A computational algorithm then estimated the average semantic distance between the words; related words (e.g., cat and dog) have shorter distances than unrelated ones (e.g., cat and thimble). We predicted that people producing greater semantic distances would also score higher on traditional creativity measures. In Study 1, we found moderate to strong correlations between semantic distance and two widely used creativity measures (the Alternative Uses Task and the Bridge-the-Associative-Gap Task). In Study 2, with participants from 98 countries, semantic distances varied only slightly by basic demographic variables. There was also a positive correlation between semantic distance and performance on a range of problems known to predict creativity. Overall, semantic distance correlated at least as strongly with established creativity measures as those measures did with each other. Naming unrelated words in what we call the Divergent Association Task can thus serve as a brief, reliable, and objective measure of divergent thinking.

Once more with feeling: Normative data for the aha experience in insight and noninsight problems.

Despite the presumed ability of insight problems to elicit the subjective feeling of insight, as well as the use of so-called insight problems to investigate this phenomenon for over 100 years, no research has collected normative data regarding the ability of insight problems to actually elicit the feeling of insight in a given individual. The work described in this article provides an overview of both classic and contemporary problems used to examine the construct of insight and presents normative data on the success rate, mean time to solution, and mean rating of aha experience for each problem and task type. We suggest using these data in future work as a reference for selecting problems on the basis of their ability to elicit an aha experience.

Insight Is Not in the Problem: Investigating Insight in Problem Solving across Task Types.

The feeling of insight in problem solving is typically associated with the sudden realization of a solution that appears obviously correct (Kounios et al., 2006). Salvi et al. (2016) found that a solution accompanied with sudden insight is more likely to be correct than a problem solved through conscious and incremental steps. However, Metcalfe (1986) indicated that participants would often present an inelegant but plausible (wrong) answer as correct with a high feeling of warmth (a subjective measure of closeness to solution). This discrepancy may be due to the use of different tasks or due to different methods in the measurement of insight (i.e., using a binary vs. continuous scale). In three experiments, we investigated both findings, using many different problem tasks (e.g., Compound Remote Associates, so-called classic insight problems, and non-insight problems). Participants rated insight-related affect (feelings of Aha-experience, confidence, surprise, impasse, and pleasure) on continuous scales. As expected we found that, for problems designed to elicit insight, correct solutions elicited higher proportions of reported insight in the solution compared to non-insight solutions; further, correct solutions elicited stronger feelings of insight compared to incorrect solutions.

Detecting unidentified changes.

Does becoming aware of a change to a purely visual stimulus necessarily cause the observer to be able to identify or localise the change or can change detection occur in the absence of identification or localisation? Several theories of visual awareness stress that we are aware of more than just the few objects to which we attend. In particular, it is clear that to some extent we are also aware of the global properties of the scene, such as the mean luminance or the distribution of spatial frequencies. It follows that we may be able to detect a change to a visual scene by detecting a change to one or more of these global properties. However, detecting a change to global property may not supply us with enough information to accurately identify or localise which object in the scene has been changed. Thus, it may be possible to reliably detect the occurrence of changes without being able to identify or localise what has changed. Previous attempts to show that this can occur with natural images have produced mixed results. Here we use a novel analysis technique to provide additional evidence that changes can be detected in natural images without also being identified or localised. It is likely that this occurs by the observers monitoring the global properties of the scene.