How experts and novices judge other people's knowledgeability from language use.

How accurate are people in judging someone else's knowledge based on their language use, and do more knowledgeable people use different cues to make these judgments? We address this by recruiting a group of participants ("informants") to answer general knowledge questions and describe various images belonging to different categories (e.g., cartoons, basketball). A second group of participants ("evaluators") also answer general knowledge questions and decide who is more knowledgeable within pairs of informants, based on these descriptions. Evaluators perform above chance at identifying the most knowledgeable informants (65% with only one description available). The less knowledgeable evaluators base their decisions on the number of specific statements, regardless of whether the statements are true or false. The more knowledgeable evaluators treat true and false statements differently and penalize the knowledge they attribute to informants who produce specific yet false statements. Our findings demonstrate the power of a few words when assessing others' knowledge and have implications for how misinformation is processed differently between experts and novices.

Differentiating mental models of self and others: A hierarchical framework for knowledge assessment.

Developing an accurate model of another agent's knowledge is central to communication and cooperation between agents. In this article, we propose a hierarchical framework of knowledge assessment that explains how people construct mental models of their own knowledge and the knowledge of others. Our framework posits that people integrate information about their own and others' knowledge via Bayesian inference. To evaluate this claim, we conduct an experiment in which participants repeatedly assess their own performance (a metacognitive task) and the performance of another person (a type of theory of mind task) on the same image classification tasks. We contrast the hierarchical framework with simpler alternatives that assume different degrees of differentiation between mental models of self and others. Our model accurately captures participants' assessment of their own performance and the performance of others in the task: Initially, people rely on their own self-assessment process to reason about the other person's performance, leading to similar self- and other-performance predictions. As more information about the other person's ability becomes available, the mental model for the other person becomes increasingly distinct from the mental model of self. Simulation studies also confirm that our framework explains a wide range of findings about human knowledge assessment of themselves and others. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Three Challenges for AI-Assisted Decision-Making.

Artificial intelligence (AI) has the potential to improve human decision-making by providing decision recommendations and problem-relevant information to assist human decision-makers. However, the full realization of the potential of human-AI collaboration continues to face several challenges. First, the conditions that support complementarity (i.e., situations in which the performance of a human with AI assistance exceeds the performance of an unassisted human or the AI in isolation) must be understood. This task requires humans to be able to recognize situations in which the AI should be leveraged and to develop new AI systems that can learn to complement the human decision-maker. Second, human mental models of the AI, which contain both expectations of the AI and reliance strategies, must be accurately assessed. Third, the effects of different design choices for human-AI interaction must be understood, including both the timing of AI assistance and the amount of model information that should be presented to the human decision-maker to avoid cognitive overload and ineffective reliance strategies. In response to each of these three challenges, we present an interdisciplinary perspective based on recent empirical and theoretical findings and discuss new research directions.

Teaching categories via examples and explanations.

People often learn categories through interaction with knowledgeable others who may use verbal explanations, visual exemplars, or both, to share their knowledge. Verbal and nonverbal means of pedagogical communication are commonly used in conjunction, but their respective roles are not fully understood. In this work, we studied how well these modes of communication work with different category structures. We conducted two experiments to investigate the effect of perceptual confusability and stimulus dimensionality on the effectiveness of verbal, exemplar-based, and mixed communication. One group of participants - teachers - learned a categorization rule and prepared learning materials for the students. Students studied the materials prepared for them and then demonstrated their knowledge on test stimuli. All communication modes were generally successful, but not equivalent, with mixed communication consistently showing best results. When teachers were free to generate as many visual exemplars or words as they wish, verbal and exemplar-based communication showed similar performance, although the verbal channel was slightly less reliable in situations requiring high perceptual precision. At the same time, verbal communication was better suited to handling high-dimensional stimuli when communication volume was restricted. We believe that our work serves as an important step towards studying language as a means for pedagogical category leaning.

Jointly modeling behavioral and EEG measures of proactive control in task switching.

In this study, we implement joint modeling of behavioral and single-trial electroencephalography (EEG) data derived from a cued-trials task-switching paradigm to test the hypothesis that trial-by-trial adjustment of response criterion can be linked to changes in the event-related potentials (ERPs) elicited during the cue-target interval (CTI). Specifically, we assess whether ERP components associated with preparation to switch task and preparation of the relevant task are linked to a response criterion parameter derived from a simple diffusion decision model (DDM). Joint modeling frameworks characterize the brain-behavior link by simultaneously modeling behavioral and neural data and implementing a linking function to bind these two submodels. We examined three joint models: The first characterized the core link between EEG and criterion, the second added a switch preparation input parameter and the third also added a task preparation input parameter. The criterion-EEG link was strongest just before target onset. Inclusion of switch and task preparation parameters did not improve the performance of the criterion-EEG link but was necessary to accurately model the ERP waveform morphology. While we successfully jointly modeled latent model parameters and EEG data from a task-switching paradigm, these findings show that customized cognitive models are needed that are tailored to the multiple cognitive control processes underlying task-switching performance. This is the first paper to implement joint modeling of behavioral measures and single-trial electroencephalography (EEG) data derived from the cue-target interval in a cued-trials task-switching paradigm. Model hyperparameters showed a strong link between response criterion and the pre-target negativity amplitude. Additional parameters (switch preparation, task preparation) were necessary to model the cue-locked ERP waveform morphology. This is consistent with multiple cognitive control processes underlying proactive control and points to the need for more nuanced models of task-switching performance.

Building and benchmarking the motivated deception corpus: Improving the quality of deceptive text through gaming.

When one studies fake news or false reviews, the first step to take is to find a corpus of text samples to work with. However, most deceptive corpora suffer from an intrinsic problem: there is little incentive for the providers of the deception to put their best effort, which risks lowering the quality and realism of the deception. The corpus described in this project, the Motivated Deception Corpus, aims to rectify this problem by gamifying the process of deceptive text collection. By having subjects play the game Two Truths and a Lie, and by rewarding those subjects that successfully fool their peers, we collect samples in such a way that the process itself improves the quality of the text. We have amassed a large corpus of deceptive text that is strongly incentivized to be convincing, and thus more reflective of real deceptive text. We provide results from several configurations of neural network prediction models to establish machine learning benchmarks on the data. This new corpus is demonstratively more challenging to classify with the current state of the art than previous corpora.

Linking computational models of two core tasks of cognitive control.

Cognitive control refers to the ability to maintain goal-relevant information in the face of distraction, making it a core construct for understanding human thought and behavior. There is great theoretical and practical value in building theories that can be used to explain or to predict variations in cognitive control as a function of experimental manipulations or individual differences. A critical step toward building such theories is determining which latent constructs are shared between laboratory tasks that are designed to measure cognitive control. In the current work, we examine this question in a novel way by formally linking computational models of two canonical cognitive control tasks, the Eriksen flanker and task-switching task. Specifically, we examine whether model parameters that capture cognitive control processes in one task can be swapped across models to make predictions about individual differences in performance on another task. We apply our modeling and analysis to a large scale data set from an online cognitive training platform, which optimizes our ability to detect individual differences in the data. Our results suggest that the flanker and task-switching tasks probe common control processes. This finding supports the view that higher level cognitive control processes as opposed to solely strategies in speed and accuracy tradeoffs, or perceptual processing and motor response speed are shared across the two tasks. We discuss how our computational modeling substitution approach addresses limitations of prior efforts to relate performance across different cognitive control tasks, and how our findings inform current theories of cognitive control. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Comparing models of learning and relearning in large-scale cognitive training data sets.

Practice in real-world settings exhibits many idiosyncracies of scheduling and duration that can only be roughly approximated by laboratory research. Here we investigate 39,157 individuals' performance on two cognitive games on the Lumosity platform over a span of 5 years. The large-scale nature of the data allows us to observe highly varied lengths of uncontrolled interruptions to practice and offers a unique view of learning in naturalistic settings. We enlist a suite of models that grow in the complexity of the mechanisms they postulate and conclude that long-term naturalistic learning is best described with a combination of long-term skill and task-set preparedness. We focus additionally on the nature and speed of relearning after breaks in practice and conclude that those components must operate interactively to produce the rapid relearning that is evident even at exceptionally long delays (over 2 years). Naturalistic learning over long time spans provides a strong test for the robustness of theoretical accounts of learning, and should be more broadly used in the learning sciences.

Bayesian modeling of human-AI complementarity.

SignificanceWith the increase in artificial intelligence in real-world applications, there is interest in building hybrid systems that take both human and machine predictions into account. Previous work has shown the benefits of separately combining the predictions of diverse machine classifiers or groups of people. Using a Bayesian modeling framework, we extend these results by systematically investigating the factors that influence the performance of hybrid combinations of human and machine classifiers while taking into account the unique ways human and algorithmic confidence is expressed.

A Critical Review of Network-Based and Distributional Approaches to Semantic Memory Structure and Processes.

Some of the earliest work on understanding how concepts are organized in memory used a network-based approach, where words or concepts are represented as nodes, and relationships between words are represented by links between nodes. Over the past two decades, advances in network science and graph theoretical methods have led to the development of computational semantic networks. This review provides a modern perspective on how computational semantic networks have proven to be useful tools to investigate the structure of semantic memory as well as search and retrieval processes within semantic memory, to ultimately model performance in a wide variety of cognitive tasks. Regarding representation, the review focuses on the distinctions and similarities between network-based (based on behavioral norms) approaches and more recent distributional (based on natural language corpora) semantic models, and the potential overlap between the two approaches. Capturing the type of relation between concepts appears to be particularly important in this modeling endeavor. Regarding processes, the review focuses on random walk models and the degree to which retrieval processes demand attention in pursuit of given task goals, which dovetails with the type of relation retrieved during tasks. Ultimately, this review provides a critical assessment of how the network perspective can be reconciled with distributional and machine-learning-based perspectives to meaning representation, and describes how cognitive network science provides a useful conceptual toolkit to probe both the structure and retrieval processes within semantic memory.

Perceptions of AI engaging in human expression.

Though humans should defer to the superior judgement of AI in an increasing number of domains, certain biases prevent us from doing so. Understanding when and why these biases occur is a central challenge for human-computer interaction. One proposed source of such bias is task subjectivity. We test this hypothesis by having both real and purported AI engage in one of the most subjective expressions possible: Humor. Across two experiments, we address the following: Will people rate jokes as less funny if they believe an AI created them? When asked to rate jokes and guess their likeliest source, participants evaluate jokes that they attribute to humans as the funniest and those to AI as the least funny. However, when these same jokes are explicitly framed as either human or AI-created, there is no such difference in ratings. Our findings demonstrate that user attitudes toward AI are more malleable than once thought-even when they (seemingly) attempt the most fundamental of human expressions.

Semantic Memory Search and Retrieval in a Novel Cooperative Word Game: A Comparison of Associative and Distributional Semantic Models.

Considerable work during the past two decades has focused on modeling the structure of semantic memory, although the performance of these models in complex and unconstrained semantic tasks remains relatively understudied. We introduce a two-player cooperative word game, Connector (based on the boardgame Codenames), and investigate whether similarity metrics derived from two large databases of human free association norms, the University of South Florida norms and the Small World of Words norms, and two distributional semantic models based on large language corpora (word2vec and GloVe) predict performance in this game. Participant dyads were presented with 20-item word boards with word pairs of varying relatedness. The speaker received a word pair from the board (e.g., exam-algebra) and generated a one-word semantic clue (e.g., math), which was used by the guesser to identify the word pair on the board across three attempts. Response times to generate the clue, as well as accuracy and latencies for the guessed word pair, were strongly predicted by the cosine similarity between word pairs and clues in random walk-based associative models, and to a lesser degree by the distributional models, suggesting that conceptual representations activated during free association were better able to capture search and retrieval processes in the game. Further, the speaker adjusted subsequent clues based on the first attempt by the guesser, who in turn benefited from the adjustment in clues, suggesting a cooperative influence in the game that was effectively captured by both associative and distributional models. These results indicate that both associative and distributional models can capture relatively unconstrained search processes in a cooperative game setting, and Connector is particularly suited to examine communication and semantic search processes.

Going, going, gone: competitive decision-making in Dutch auctions.

In a Dutch auction, an item is offered for sale at a set maximum price. The price is then gradually lowered over a fixed interval of time until a bid is made, securing the item for the bidder at the current price. Bidders must trade-off between certainty and price: bid early to secure the item and you pay a premium; bid later at a lower price but risk losing to another bidder. These properties of Dutch auctions provide new opportunities to study competitive decision-making in a group setting. We developed a novel computerised Dutch auction platform and conducted a set of experiments manipulating volatility (fixed vs varied number of items for sale) and price reduction interval rate (step-rate). Triplets of participants ([Formula: see text]) competed with hypothetical funds against each other. We report null effects of step-rate and volatility on bidding behaviour. We developed a novel adaptation of prospect theory to account for group bidding behaviour by balancing certainty and subjective expected utility. We show the model is sensitive to variation in auction starting price and can predict the associated changes in group bid prices that were observed in our data.

Inferring latent learning factors in large-scale cognitive training data.

The flexibility to learn diverse tasks is a hallmark of human cognition. To improve our understanding of individual differences and dynamics of learning across tasks, we analyse the latent structure of learning trajectories from 36,297 individuals as they learned 51 different tasks on the Lumosity online cognitive training platform. Through a data-driven modelling approach using probabilistic dimensionality reduction, we investigate covariation across learning trajectories with few assumptions about learning curve form or relationships between tasks. Modelling results show substantial covariation across tasks, such that an entirely unobserved learning trajectory can be predicted by observing trajectories on other tasks. The latent learning factors from the model include a general ability factor that is expressed mostly at later stages of practice and additional task-specific factors that carry information capable of accounting for manually defined task features and task domains such as attention, spatial processing, language and math.

Distant connectivity and multiple-step priming in large-scale semantic networks.

We examined 3 different network models of representing semantic knowledge (5,018-word directed and undirected step distance networks, and an association-correlation network) to predict lexical priming effects. In Experiment 1, participants made semantic relatedness judgments for word pairs with varying path lengths. Response latencies for judgments followed a quadratic relationship with network path lengths, replicating and extending a recent pattern reported by Kenett, Levi, Anaki, and Faust (2017) for an 800-word association-correlation network in Hebrew. In Experiment 2, participants identified target words in a progressive demasking task, immediately following a briefly presented prime (120 ms). Response latencies to identify the target showed a linear trend for all network path lengths. Importantly, there were statistically significant differences between relatively distant words in the step distance networks, for example, path lengths 4 and beyond, suggesting that association networks can indeed capture distant functional semantic relationships. Additional comparisons with 2 distributional models (LSA and word2vec) suggested that distributional models also successfully predicted response latencies, although there appear to be fundamental differences in the types of semantic relationships captured by the different models. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A large-scale analysis of task switching practice effects across the lifespan.

An important feature of human cognition is the ability to flexibly and efficiently adapt behavior in response to continuously changing contextual demands. We leverage a large-scale dataset from Lumosity, an online cognitive-training platform, to investigate how cognitive processes involved in cued switching between tasks are affected by level of task practice across the adult lifespan. We develop a computational account of task switching that specifies the temporal dynamics of activating task-relevant representations and inhibiting task-irrelevant representations and how they vary with extended task practice across a number of age groups. Practice modulates the level of activation of the task-relevant representation and improves the rate at which this information becomes available, but has little effect on the task-irrelevant representation. While long-term practice improves performance across all age groups, it has a greater effect on older adults. Indeed, extensive task practice can make older individuals functionally similar to less-practiced younger individuals, especially for cognitive measures that focus on the rate at which task-relevant information becomes available.

Experimental design modulates variance in BOLD activation: The variance design general linear model.

Typical fMRI studies have focused on either the mean trend in the blood-oxygen-level-dependent (BOLD) time course or functional connectivity (FC). However, other statistics of the neuroimaging data may contain important information. Despite studies showing links between the variance in the BOLD time series (BV) and age and cognitive performance, a formal framework for testing these effects has not yet been developed. We introduce the variance design general linear model (VDGLM), a novel framework that facilitates the detection of variance effects. We designed the framework for general use in any fMRI study by modeling both mean and variance in BOLD activation as a function of experimental design. The flexibility of this approach allows the VDGLM to (a) simultaneously make inferences about a mean or variance effect while controlling for the other and (b) test for variance effects that could be associated with multiple conditions and/or noise regressors. We demonstrate the use of the VDGLM in a working memory application and show that engagement in a working memory task is associated with whole-brain decreases in BOLD variance.

Predicting Task and Subject Differences with Functional Connectivity and Blood-Oxygen-Level-Dependent Variability.

Previous research has found that functional connectivity (FC) can accurately predict the identity of a subject performing a task and the type of task being performed. These results are replicated using a large data set collected at the Ohio State University Center for Cognitive and Behavioral Brain Imaging. This work introduces a novel perspective on task and subject identity prediction: blood-oxygen-level-dependent variability (BV). Conceptually, BV is a region-specific measure based on the variance within each brain region. BV is simple to compute, interpret, and visualize. This work shows that both FC and BV are predictive of task and subject, even across scanning sessions separated by multiple years. Subject differences rather than task differences account for the majority of changes in BV and FC. Similar to results in FC, BV is reduced during cognitive tasks relative to rest.

Monitoring the ebb and flow of attention: Does controlling the onset of stimuli during encoding enhance memory?

Central to the operation of the Atkinson and Shiffrin's (Psychology of learning and motivation, 2, 89-195, 1968) model of human memory are a variety of control processes that manage information flow. Research on metacognition reveals that provision of control in laboratory learning tasks is generally beneficial to memory. In this paper, we investigate the novel domain of attentional fluctuations during study. If learners are able to monitor attention, then control over the onset of stimuli should also improve performance. Across four experiments, we found no evidence that control over the onset of stimuli enhances learning. This result stands in notable contrast to the fact that control over stimulus offset does enhance memory (Experiment 1; Tullis & Benjamin, Journal of memory and language, 64 (2), 109-118, 2011). This null finding was replicated across laboratory and online samples of subjects, and with both words and faces as study material. Taken together, the evidence suggests that people either cannot monitor fluctuations in attention effectively or cannot precisely time their study to those fluctuations.

The joint contribution of participation and performance to learning functions: Exploring the effects of age in large-scale data sets.

Large-scale data sets from online training and game platforms offer the opportunity for more extensive and more precise investigations of human learning than is typically achievable in the laboratory. However, because people make their own choices about participation, any investigation into learning using these data sets must simultaneously model performance-that is, the learning function-and participation. Using a data set of 54 million gameplays from the online brain training site Lumosity, we show that learning functions of participants are systematically biased by participation policies that vary with age. Older adults who are poorer performers are more likely to drop out than older adults who perform well. Younger adults show no such effect. Using this knowledge, we can extrapolate group learning functions that correct for these age-related differences in dropout.