Evaluating the robustness of parameter estimates in cognitive models: A meta-analytic review of multinomial processing tree models across the multiverse of estimation methods.

Researchers have become increasingly aware that data-analysis decisions affect results. Here, we examine this issue systematically for multinomial processing tree (MPT) models, a popular class of cognitive models for categorical data. Specifically, we examine the robustness of MPT model parameter estimates that arise from two important decisions: the level of data aggregation (complete-pooling, no-pooling, or partial-pooling) and the statistical framework (frequentist or Bayesian). These decisions span a multiverse of estimation methods. We synthesized the data from 13,956 participants (164 published data sets) with a meta-analytic strategy and analyzed the magnitude of divergence between estimation methods for the parameters of nine popular MPT models in psychology (e.g., process-dissociation, source monitoring). We further examined moderators as potential sources of divergence. We found that the absolute divergence between estimation methods was small on average (<.04; with MPT parameters ranging between 0 and 1); in some cases, however, divergence amounted to nearly the maximum possible range (.97). Divergence was partly explained by few moderators (e.g., the specific MPT model parameter, uncertainty in parameter estimation), but not by other plausible candidate moderators (e.g., parameter trade-offs, parameter correlations) or their interactions. Partial-pooling methods showed the smallest divergence within and across levels of pooling and thus seem to be an appropriate default method. Using MPT models as an example, we show how transparency and robustness can be increased in the field of cognitive modeling. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

A multinomial modelling approach to face identity recognition during instructed threat.

To organise future behaviour, it is important to remember both the central and contextual aspects of a situation. We examined the impact of contextual threat or safety, learned through verbal instructions, on face identity recognition. In two studies (N = 140), 72 face-context compounds were presented each once within an encoding session, and an unexpected item/source recognition task was performed afterwards (including 24 new faces). Hierarchical multinomial processing tree modelling served to estimate individual parameters of item (face identity) and source memory (threat or safety context) as well as guessing behaviour. Results show that language was highly effective in establishing threatening and safe context conditions. In Study 1, a fleeting picture stream (1 s per picture) led to poor item and source recognition. Prolonged presentation times (Study 2 with 6 s per picture) improved face memory but no contextual modulation was observed. Thus, incidental face learning was surprisingly poor and rapidly changing contextual settings might have interfered with the accurate encoding of face identity information and item-source binding.

Contextual source information modulates neural face processing in the absence of conscious recognition: A threat-of-shock study.

Often the source of information is as important as the information itself. The present study examined the impact of contextual threat and safety signals (source information) on memory encoding and recognition of faces (item information). In two experimental sessions, 30 participants viewed neutral face pictures. In the encoding session, 60 faces were presented with contextual background colors (blue or green, 30 pictures each) which were verbally instructed to signal either threat-of-shock or safety. In the recognition session, the 60 old faces intermixed with 30 new faces were shown while a combined old/new recognition and source memory task was performed. During the encoding session, face processing varied as a function of contextual source information. Confirming successful threat-of-shock manipulation, threatening compared to safe face-context compounds revealed differential neural processing (early parieto-occipital and late fronto-central negativity) as well as pronounced threat ratings. During the recognition session, participants had serious problems identifying old from new faces with poor source memory. Intriguingly, however, brain activity differentiated previously seen faces from newly presented pictures (old/new ERP effect). Moreover, old faces presented within a threat context were associated with distributed late negativities compared to old safe faces. Thus, threat effects not only emerged during face encoding (incidental learning) but also during face recognition, although no valid judgements could be made regarding the threatening or safe sources. These findings support the notion that contextual source information critically modulates person perception and recognition as a form of an expectation based remembering in the absence of conscious recognition.

Testing Hypotheses About Binding in Context Memory With a Hierarchical Multinomial Modeling Approach.

In experiments on multidimensional source memory, a stochastic dependency of source memory for different facets of an episode has been repeatedly demonstrated. This may suggest an integrated representation leading to mutual cuing in context retrieval. However, experiments involving a manipulated reinstatement of one source feature have often failed to affect retrieval of the other feature, suggesting unbound features or rather item-feature binding. The stochastic dependency found in former studies might be a spurious correlation due to aggregation across participants varying in memory strength. We test this artifact explanation by applying a hierarchical multinomial model. Observing stochastic dependency when accounting for interindividual differences would rule out the artifact explanation. A second goal is to elucidate the nature of feature binding: Contrasting encoding conditions with integrated feature judgments versus separate feature judgments are expected to induce different levels of stochastic dependency despite comparable overall source memory if integrated representations include feature-feature binding. The experiment replicated the finding of stochastic dependency and, thus, ruled out an artifact interpretation. However, we did not find different levels of stochastic dependency between conditions. Therefore, the current findings do not reveal decisive evidence to distinguish between the feature-feature binding and the item-context binding account.

Item-method directed forgetting and working memory capacity: A hierarchical multinomial modeling approach.

Intentional forgetting of information that has recently been encoded is regarded an active and adaptive process and is widely studied using the item-method or the list-method directed forgetting (DF) paradigm. In the present research, we tested whether inter-individual differences in working-memory capacity (WMC), that have been identified as a relevant predictor of DF within the list method, are also related to stronger DF effects within the item method. Furthermore, we investigated relationships between WMC and item-method DF at different processing stages by applying the multinomial storage-retrieval model hierarchically to our data. Results showed that individuals with high WMC are better able to store to-be-remembered information than individuals with low WMC, whereas WMC was not related to retrieval of to-be-remembered information or to either storage or retrieval of to-be-forgotten information. Implications for theoretical accounts of item-method DF are discussed.

TreeBUGS: An R package for hierarchical multinomial-processing-tree modeling.

Multinomial processing tree (MPT) models are a class of measurement models that account for categorical data by assuming a finite number of underlying cognitive processes. Traditionally, data are aggregated across participants and analyzed under the assumption of independently and identically distributed observations. Hierarchical Bayesian extensions of MPT models explicitly account for participant heterogeneity by assuming that the individual parameters follow a continuous hierarchical distribution. We provide an accessible introduction to hierarchical MPT modeling and present the user-friendly and comprehensive R package TreeBUGS, which implements the two most important hierarchical MPT approaches for participant heterogeneity-the beta-MPT approach (Smith & Batchelder, Journal of Mathematical Psychology 54:167-183, 2010) and the latent-trait MPT approach (Klauer, Psychometrika 75:70-98, 2010). TreeBUGS reads standard MPT model files and obtains Markov-chain Monte Carlo samples that approximate the posterior distribution. The functionality and output are tailored to the specific needs of MPT modelers and provide tests for the homogeneity of items and participants, individual and group parameter estimates, fit statistics, and within- and between-subjects comparisons, as well as goodness-of-fit and summary plots. We also propose and implement novel statistical extensions to include continuous and discrete predictors (as either fixed or random effects) in the latent-trait MPT model.

Hierarchical Multinomial Modeling Approaches: An Application to Prospective Memory and Working Memory.

Hierarchical extensions of multinomial processing tree (MPT) models have been developed to deal with heterogeneity in participants or items. In this study, the beta-MPT model ( J. B. Smith & Batchelder, 2010 ) and the latent-trait approach ( Klauer, 2010 ) were used to estimate individual model parameters for prospective and retrospective components of prospective memory (PM), which requires remembering to perform an action in the future. The data from two experiments investigating the relationship between PM and working memory ( R. E. Smith & Bayen, 2005 , Experiment 1; R. E. Smith, Persyn, & Butler, 2011 ) were reanalyzed using the two hierarchical modeling approaches, both of which provide parameter estimates for individual participants. The results showed a positive correlation of the prospective component of PM with working-memory span and provide the first direct comparisons of the two hierarchical extensions of an MPT model.

Empirical validation of the diffusion model for recognition memory and a comparison of parameter-estimation methods.

The diffusion model introduced by Ratcliff (Psychol Rev 85:59-108, 1978) has been applied to many binary decision tasks including recognition memory. It describes dynamic evidence accumulation unfolding over time and models choice accuracy as well as response-time distributions. Various parameters describe aspects of decision quality and response bias. In three recognition-memory experiments, the validity of the model was tested experimentally and analyzed with three different programs: fast-dm, EZ, and DMAT. Each of three central model parameters was targeted via specific experimental manipulations. All manipulations affected mainly the corresponding parameters, thus supporting the convergent validity of the measures. There were, however, smaller effects on other parameters, showing some limitations in discriminant validity.

Is prospective memory related to depression and anxiety? A hierarchical MPT modelling approach.

Prospective memory (PM) refers to remembering to perform an action in the future. One hundred and twenty-nine students completed a laboratory event-based PM task as well as depression and anxiety questionnaires. The data were analysed with the beta-MPT version of the multinomial processing tree model of event-based PM. Thereby, the prospective and retrospective components of PM were estimated for each participant and were then correlated with depression and anxiety. State anxiety was negatively correlated with the prospective component of PM. Neither depression nor trait anxiety were related to either component of PM.

Hierarchical modeling of contingency-based source monitoring: a test of the probability-matching account.

According to the probability-matching account of source guessing (Spaniol & Bayen, Journal of Experimental Psychology: Learning, Memory, and Cognition 28:631-651, 2002), when people do not remember the source of an item in a source-monitoring task, they match the source-guessing probabilities to the perceived contingencies between sources and item types. In a source-monitoring experiment, half of the items presented by each of two sources were consistent with schematic expectations about this source, whereas the other half of the items were consistent with schematic expectations about the other source. Participants' source schemas were activated either at the time of encoding or just before the source-monitoring test. After test, the participants judged the contingency of the item type and source. Individual parameter estimates of source guessing were obtained via beta-multinomial processing tree modeling (beta-MPT; Smith & Batchelder, Journal of Mathematical Psychology 54:167-183, 2010). We found a significant correlation between the perceived contingency and source guessing, as well as a correlation between the deviation of the guessing bias from the true contingency and source memory when participants did not receive the schema information until retrieval. These findings support the probability-matching account.