Evaluating the robustness of parameter estimates in cognitive models: A meta-analytic review of multinomial processing tree models across the multiverse of estimation methods.

Researchers have become increasingly aware that data-analysis decisions affect results. Here, we examine this issue systematically for multinomial processing tree (MPT) models, a popular class of cognitive models for categorical data. Specifically, we examine the robustness of MPT model parameter estimates that arise from two important decisions: the level of data aggregation (complete-pooling, no-pooling, or partial-pooling) and the statistical framework (frequentist or Bayesian). These decisions span a multiverse of estimation methods. We synthesized the data from 13,956 participants (164 published data sets) with a meta-analytic strategy and analyzed the magnitude of divergence between estimation methods for the parameters of nine popular MPT models in psychology (e.g., process-dissociation, source monitoring). We further examined moderators as potential sources of divergence. We found that the absolute divergence between estimation methods was small on average (<.04; with MPT parameters ranging between 0 and 1); in some cases, however, divergence amounted to nearly the maximum possible range (.97). Divergence was partly explained by few moderators (e.g., the specific MPT model parameter, uncertainty in parameter estimation), but not by other plausible candidate moderators (e.g., parameter trade-offs, parameter correlations) or their interactions. Partial-pooling methods showed the smallest divergence within and across levels of pooling and thus seem to be an appropriate default method. Using MPT models as an example, we show how transparency and robustness can be increased in the field of cognitive modeling. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Parallel Acquisition of Uncorrelated Sequences does Not Provide Firm Evidence for a Modular Sequence-Learning System.

Dual-systems theories of sequence learning assume that sequence learning may proceed within a unidimensional learning system that is immune to cross-dimensional interference because information is processed and represented in dimension-specific, encapsulated modules. Important evidence for such modularity comes from studies investigating the absence or presence of interference between multiple uncorrelated sequences (e.g., a sequence of color stimuli and a sequence of motor keypresses). Here we question the premise that the parallel acquisition of uncorrelated sequences provides convincing evidence for a modularized learning system. In contrast, we demonstrate that parallel acquisition of multiple uncorrelated sequences is well predicted by a computational model that assumes a single learning system with joint representations of stimulus and response features.

Assumptions of the process-dissociation procedure are violated in implicit sequence learning.

In implicit sequence learning, a process-dissociation (PD) approach has been proposed to dissociate implicit and explicit learning processes. Applied to the popular generation task, participants perform two different task versions: inclusion instructions require generating the transitions that form the learned sequence; exclusion instructions require generating transitions other than those of the learned sequence. Whereas accurate performance under inclusion may be based on either implicit or explicit knowledge, avoiding to generate learned transitions requires controllable explicit sequence knowledge. The PD approach yields separate estimates of explicit and implicit knowledge that are derived from the same task; it therefore avoids many problems of previous measurement approaches. However, the PD approach rests on the critical assumption that the implicit and explicit processes are invariant across inclusion and exclusion conditions. We tested whether the invariance assumptions hold for the PD generation task. Across three studies using first-order as well as second-order regularities, invariance of the controlled process was found to be violated. In particular, despite extensive amounts of practice, explicit knowledge was not exhaustively expressed in the exclusion condition. We discuss the implications of these findings for the use of process-dissociation in assessing implicit knowledge. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Distorted estimates of implicit and explicit learning in applications of the process-dissociation procedure to the SRT task.

We investigated potential biases affecting the validity of the process-dissociation (PD) procedure when applied to sequence learning. Participants were or were not exposed to a serial reaction time task (SRTT) with two types of pseudo-random materials. Afterwards, participants worked on a free or cued generation task under inclusion and exclusion instructions. Results showed that pre-experimental response tendencies, non-associative learning of location frequencies, and the usage of cue locations introduced bias to PD estimates. These biases may lead to erroneous conclusions regarding the presence of implicit and explicit knowledge. Potential remedies for these problems are discussed.