Refining Bayesian hierarchical MPT modeling: Integrating prior knowledge and ordinal expectations.

Multinomial processing tree (MPT) models are a broad class of statistical models used to test sophisticated psychological theories. The research questions derived from these theories often go beyond simple condition effects on parameters and involve ordinal expectations (e.g., the same-direction effect on the memory parameter is stronger in one experimental condition than another) or disordinal expectations (e.g., the effect reverses in one experimental condition). Here, we argue that by refining common modeling practices, Bayesian hierarchical models are well suited to estimate and test these expectations. Concretely, we show that the default priors proposed in the literature lead to nonsensical predictions for individuals and the population distribution, leading to problems not only in model comparison but also in parameter estimation. Rather than relying on these priors, we argue that MPT modelers should determine priors that are consistent with their theoretical knowledge. In addition, we demonstrate how Bayesian model comparison may be used to test ordinal and disordinal interactions by means of Bayes factors. We apply the techniques discussed to empirical data from Bell et al. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 456-472 (2015).

A tutorial on Bayesian model-averaged meta-analysis in JASP.

Researchers conduct meta-analyses in order to synthesize information across different studies. Compared to standard meta-analytic methods, Bayesian model-averaged meta-analysis offers several practical advantages including the ability to quantify evidence in favor of the absence of an effect, the ability to monitor evidence as individual studies accumulate indefinitely, and the ability to draw inferences based on multiple models simultaneously. This tutorial introduces the concepts and logic underlying Bayesian model-averaged meta-analysis and illustrates its application using the open-source software JASP. As a running example, we perform a Bayesian meta-analysis on language development in children. We show how to conduct a Bayesian model-averaged meta-analysis and how to interpret the results.

"This behavior strikes us as ideal": assessment and anticipations of Huisman (2022).

Huisman (Psychonomic Bulletin & Review, 1-10. 2022) argued that a valid measure of evidence should indicate more support in favor of a true alternative hypothesis when sample size is large than when it is small. Bayes factors may violate this pattern and hence Huisman concluded that Bayes factors are invalid as a measure of evidence. In this brief comment we call attention to the following: (1) Huisman's purported anomaly is in fact dictated by probability theory; (2) Huisman's anomaly has been discussed and explained in the statistical literature since 1939; the anomaly was also highlighted in the Psychonomic Bulletin & Review article by Rouder et al. (2009), who interpreted the anomaly as "ideal": an interpretation diametrically opposed to that of Huisman. We conclude that when intuition clashes with probability theory, chances are that it is intuition that needs schooling.

Predictions and rewards affect decision-making but not subjective experience.

To survive, organisms constantly make decisions to avoid danger and maximize rewards in information-rich environments. As a result, decisions about sensory input are not only driven by sensory information but also by other factors, such as the expected rewards of a decision (known as the payoff matrix) or by information about temporal regularities in the environment (known as cognitive priors or predictions). However, it is unknown to what extent these different types of information affect subjective experience or whether they merely result in nonperceptual response criterion shifts. To investigate this question, we used three carefully matched manipulations that typically result in behavioral shifts in decision criteria: a visual illusion (Müller-Lyer condition), a punishment scheme (payoff condition), and a change in the ratio of relevant stimuli (base rate condition). To gauge shifts in subjective experience, we introduce a task in which participants not only make decisions about what they have just seen but are also asked to reproduce their experience of a target stimulus. Using Bayesian ordinal modeling, we show that each of these three manipulations affects the decision criterion as intended but that the visual illusion uniquely affects sensory experience as measured by reproduction. In a series of follow-up experiments, we use computational modeling to show that although the visual illusion results in a distinct drift-diffusion (DDM) parameter profile relative to nonsensory manipulations, reliance on DDM parameter estimates alone is not sufficient to ascertain whether a given manipulation is perceptual or nonperceptual.

Bayesian hierarchical modeling: an introduction and reassessment.

With the recent development of easy-to-use tools for Bayesian analysis, psychologists have started to embrace Bayesian hierarchical modeling. Bayesian hierarchical models provide an intuitive account of inter- and intraindividual variability and are particularly suited for the evaluation of repeated-measures designs. Here, we provide guidance for model specification and interpretation in Bayesian hierarchical modeling and describe common pitfalls that can arise in the process of model fitting and evaluation. Our introduction gives particular emphasis to prior specification and prior sensitivity, as well as to the calculation of Bayes factors for model comparisons. We illustrate the use of state-of-the-art software programs Stan and brms. The result is an overview of best practices in Bayesian hierarchical modeling that we hope will aid psychologists in making the best use of Bayesian hierarchical modeling.

Why many studies of individual differences with inhibition tasks may not localize correlations.

Individual difference exploration of cognitive domains is predicated on being able to ascertain how well performance on tasks covary. Yet, establishing correlations among common inhibition tasks such as Stroop or flanker tasks has proven quite difficult. It remains unclear whether this difficulty occurs because there truly is a lack of correlation or whether analytic techniques to localize correlations perform poorly real-world contexts because of excessive measurement error from trial noise. In this paper, we explore how well correlations may localized in large data sets with many people, tasks, and replicate trials. Using hierarchical models to separate trial noise from true individual variability, we show that trial noise in 24 extant tasks is about 8 times greater than individual variability. This degree of trial noise results in massive attenuation in correlations and instability in Spearman corrections. We then develop hierarchical models that account for variation across trials, variation across individuals, and covariation across individuals and tasks. These hierarchical models also perform poorly in localizing correlations. The advantage of these models is not in estimation efficiency, but in providing a sense of uncertainty so that researchers are less likely to misinterpret variability in their data. We discuss possible improvements to study designs to help localize correlations.

Multibridge: an R package to evaluate informed hypotheses in binomial and multinomial models.

The multibridge R package allows a Bayesian evaluation of informed hypotheses [Formula: see text] applied to frequency data from an independent binomial or multinomial distribution. multibridge uses bridge sampling to efficiently compute Bayes factors for the following hypotheses concerning the latent category proportions 𝜃: (a) hypotheses that postulate equality constraints (e.g., 𝜃1 = 𝜃2 = 𝜃3); (b) hypotheses that postulate inequality constraints (e.g., 𝜃1 < 𝜃2 < 𝜃3 or 𝜃1 > 𝜃2 > 𝜃3); (c) hypotheses that postulate combinations of inequality constraints and equality constraints (e.g., 𝜃1 < 𝜃2 = 𝜃3); and (d) hypotheses that postulate combinations of (a)-(c) (e.g., 𝜃1 < (𝜃2 = 𝜃3),𝜃4). Any informed hypothesis [Formula: see text] may be compared against the encompassing hypothesis [Formula: see text] that all category proportions vary freely, or against the null hypothesis [Formula: see text] that all category proportions are equal. multibridge facilitates the fast and accurate comparison of large models with many constraints and models for which relatively little posterior mass falls in the restricted parameter space. This paper describes the underlying methodology and illustrates the use of multibridge through fully reproducible examples.

Bayes Factors for Mixed Models: Perspective on Responses.

In van Doorn et al. (2021), we outlined a series of open questions concerning Bayes factors for mixed effects model comparison, with an emphasis on the impact of aggregation, the effect of measurement error, the choice of prior distributions, and the detection of interactions. Seven expert commentaries (partially) addressed these initial questions. Surprisingly perhaps, the experts disagreed (often strongly) on what is best practice-a testament to the intricacy of conducting a mixed effect model comparison. Here, we provide our perspective on these comments and highlight topics that warrant further discussion. In general, we agree with many of the commentaries that in order to take full advantage of Bayesian mixed model comparison, it is important to be aware of the specific assumptions that underlie the to-be-compared models.

Does every study? Implementing ordinal constraint in meta-analysis.

The most prominent goal when conducting a meta-analysis is to estimate the true effect size across a set of studies. This approach is problematic whenever the analyzed studies have qualitatively different results; that is, some studies show an effect in the predicted direction while others show no effect and still others show an effect in the opposite direction. In case of such qualitative differences, the average effect may be a product of different mechanisms, and therefore uninterpretable. The first question in any meta-analysis should therefore be whether all studies show an effect in the same, expected direction. To tackle this question a model with ordinal constraints is proposed where the ordinal constraint holds each study in the set. This "every study" model is compared with a set of alternative models, such as an unconstrained model that predicts effects in both directions. If the ordinal constraints hold, one underlying mechanism may suffice to explain the results from all studies, and this result could be supported by reduced between-study heterogeneity. A major implication is then that average effects become interpretable. We illustrate the model comparison approach using Carbajal et al.'s (2021) meta-analysis on the familiar-word-recognition effect, show how predictor analyses can be incorporated in the approach, and provide R-code for interested researchers. As common in meta-analysis, only surface statistics (such as effect size and sample size) are provided from each study, and the modeling approach can be adapted to suit these conditions. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Evaluating multinomial order restrictions with bridge sampling.

Hypotheses concerning the distribution of multinomial proportions typically entail exact equality constraints that can be evaluated using standard tests. Whenever researchers formulate inequality constrained hypotheses, however, they must rely on sampling-based methods that are relatively inefficient and computationally expensive. To address this problem we developed a bridge sampling routine that allows an efficient evaluation of multinomial inequality constraints. An empirical application showcases that bridge sampling outperforms current Bayesian methods, especially when relatively little posterior mass falls in the restricted parameter space. The method is extended to mixtures between equality and inequality constrained hypotheses. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Creative or not? Hierarchical diffusion modeling of the creative evaluation process.

When producing creative ideas (i.e., ideas that are original and useful) two main processes occur: ideation, where people brainstorm ideas, and evaluation, where they decide if the ideas are creative or not. While much is known about the ideation phase, the cognitive processes involved in creativity evaluation are less clear. In this article, we present a novel modeling approach for the evaluation phase of creativity. We apply the drift diffusion model (DDM) to the Creative-or-Not task (CON-task) to study the cognitive basis of evaluation and to examine individual differences in the extent to which people take originality and utility into account when evaluating creative ideas. The CON-task is a timed decision-making task where participants indicate whether they find uses for certain objects creative or not (e.g., using a book as a buoy). The different uses vary on the two creativity dimensions "originality" and "utility." In two studies (n = 293, 17,806 trials; n = 152, 9,291 trials), we found that stimulus originality was strongly related to participants' drift rates but found only weak evidence for an association between stimulus utility and the drift rate. However, participants differed substantially in the effects of originality and utility. Furthermore, the implicit weights assigned to originality and utility on the CON-task were associated with self-reported importance ratings of originality and utility and with divergent thinking performance in the Alternative Uses task (AUT). This research provides a cognitive modeling approach to creativity evaluation and underlines the importance of communicating rating criteria in divergent thinking tasks to ensure a fair assessment of creative ability. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Informed Bayesian survival analysis.

BACKGROUND: We provide an overview of Bayesian estimation, hypothesis testing, and model-averaging and illustrate how they benefit parametric survival analysis. We contrast the Bayesian framework to the currently dominant frequentist approach and highlight advantages, such as seamless incorporation of historical data, continuous monitoring of evidence, and incorporating uncertainty about the true data generating process. METHODS: We illustrate the application of the outlined Bayesian approaches on an example data set, retrospective re-analyzing a colon cancer trial. We assess the performance of Bayesian parametric survival analysis and maximum likelihood survival models with AIC/BIC model selection in fixed-n and sequential designs with a simulation study. RESULTS: In the retrospective re-analysis of the example data set, the Bayesian framework provided evidence for the absence of a positive treatment effect of adding Cetuximab to FOLFOX6 regimen on disease-free survival in patients with resected stage III colon cancer. Furthermore, the Bayesian sequential analysis would have terminated the trial 10.3 months earlier than the standard frequentist analysis. In a simulation study with sequential designs, the Bayesian framework on average reached a decision in almost half the time required by the frequentist counterparts, while maintaining the same power, and an appropriate false-positive rate. Under model misspecification, the Bayesian framework resulted in higher false-negative rate compared to the frequentist counterparts, which resulted in a higher proportion of undecided trials. In fixed-n designs, the Bayesian framework showed slightly higher power, slightly elevated error rates, and lower bias and RMSE when estimating treatment effects in small samples. We found no noticeable differences for survival predictions. We have made the analytic approach readily available to other researchers in the RoBSA R package. CONCLUSIONS: The outlined Bayesian framework provides several benefits when applied to parametric survival analyses. It uses data more efficiently, is capable of considerably shortening the length of clinical trials, and provides a richer set of inferences.

The Einstein effect provides global evidence for scientific source credibility effects and the influence of religiosity.

People tend to evaluate information from reliable sources more favourably, but it is unclear exactly how perceivers' worldviews interact with this source credibility effect. In a large and diverse cross-cultural sample (N = 10,195 from 24 countries), we presented participants with obscure, meaningless statements attributed to either a spiritual guru or a scientist. We found a robust global source credibility effect for scientific authorities, which we dub 'the Einstein effect': across all 24 countries and all levels of religiosity, scientists held greater authority than spiritual gurus. In addition, individual religiosity predicted a weaker relative preference for the statement from the scientist compared with the spiritual guru, and was more strongly associated with credibility judgements for the guru than the scientist. Independent data on explicit trust ratings across 143 countries mirrored our experimental findings. These findings suggest that irrespective of one's religious worldview, across cultures science is a powerful and universal heuristic that signals the reliability of information.

Are There Reliable Qualitative Individual Difference in Cognition?

In this paper we propose a new set of questions that focus on the direction of effects. In almost all studies the direction is important. For example, in a Stroop task we expect responses to incongruent items to be slower than those to congruent ones, and this direction implies one theoretical explanation. Yet, if congruent words are slowed down relative to incongruent words we would have a completely different theoretical explanation. We ask a 'does everybody' question, such as, 'does every individual show a Stroop effect in the same direction?' Or, 'does every individual respond faster to loud tones than soft tones?' If all individuals truly have effects in the same direction that implicate a common theory, we term the differences among them as quantitative individual differences. Conversely, if all individuals truly have effects in different directions that implicate different theories, we term the differences among them as qualitative individual differences. Here, we provide a users guide to the question of whether individual differences are qualitative or quantitative. We discuss theoretical issues, methodological advances, new software for assessment, and, most importantly, how the question impacts theory development in cognitive science. Our hope is that this mode of analysis is a productive tool in researchers' toolkits.

Revisiting the remember-know task: Replications of Gardiner and Java (1990).

One of the most evidential behavioral results for two memory processes comes from Gardiner and Java (Memory & Cognition, 18, 23-30 1990). Participants provided more "remember" than "know" responses for old words but more know than remember responses for old nonwords. Moreover, there was no effect of word/nonword status for new items. The combination of a crossover interaction for old items with an invariance for new items provides strong evidence for two distinct processes while ruling out criteria or bias explanations. Here, we report a modern replication of this study. In three experiments, (Experiments 1, 2, and 4) with larger numbers of items and participants, we were unable to replicate the crossover. Instead, our data are more consistent with a single-process account. In a fourth experiment (Experiment 3), we were able to replicate Gardiner and Java's baseline results with a sure-unsure paradigm supporting a single-process explanation. It seems that Gardiner and Java's remarkable crossover result is not replicable.

The JASP guidelines for conducting and reporting a Bayesian analysis.

Despite the increasing popularity of Bayesian inference in empirical research, few practical guidelines provide detailed recommendations for how to apply Bayesian procedures and interpret the results. Here we offer specific guidelines for four different stages of Bayesian statistical reasoning in a research setting: planning the analysis, executing the analysis, interpreting the results, and reporting the results. The guidelines for each stage are illustrated with a running example. Although the guidelines are geared towards analyses performed with the open-source statistical software JASP, most guidelines extend to Bayesian inference in general.

Beyond overall effects: A Bayesian approach to finding constraints in meta-analysis.

Most meta-analyses focus on the behavior of meta-analytic means. In many cases, however, this mean is difficult to defend as a construct because the underlying distribution of studies reflects many factors, including how we as researchers choose to design studies. We present an alternative goal for meta-analysis. The analyst may ask about relations that are stable across all the studies. In a typical meta-analysis, there is a hypothesized direction (e.g., that violent video games increase, rather than decrease, aggressive behavior). We ask whether all studies in a meta-analysis have true effects in the hypothesized direction. If so, this is an example of a stable relation across all the studies. We propose 4 models: (a) all studies are truly null; (b) all studies share a single true nonzero effect; (c) studies differ, but all true effects are in the same direction; and (d) some study effects are truly positive, whereas others are truly negative. We develop Bayes factor model comparison for these models and apply them to 4 extant meta-analyses to show their usefulness. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Common or distinct attention mechanisms for contrast and assimilation?

The ability to inhibit distractors while focusing on specific targets is crucial. In most tasks, like Stroop or priming, the to-be-ignored distractors affect the response to be more like the distractors. We call this assimilation. Yet, in some tasks, the opposite holds. Constrast occurs when the response is caused to be least like the distractors. Contrast and assimilation are opposing behavioral effects, but they both occur when to-be-ignored information affects judgments. We ask here whether inhibition across contrastive and assimilative tasks is common or distinct. Assimilation and contrast are often thought to have different underlying psychological mechanisms, and we use a correlational analysis with hierarchical Bayesian models as a test of this hypothesis. We designed tasks with large assimilation or contrast effects. The stimuli are morphed letters, and whether there is contrast or assimilation depends on whether the surrounding information is a letter field (contrast) or a word field (assimilation). Critically, a positive correlation was found-individuals who better inhibited contrast-inducing contexts also better inhibited assimilation-inducing contexts. These results indicate that inhibition is common, at least in part, across contrast and assimilation tasks.

A psychometrics of individual differences in experimental tasks.

In modern individual-difference studies, researchers often correlate performance on various tasks to uncover common latent processes. Yet, in some sense, the results have been disappointing as correlations among tasks that seemingly have processes in common are often low. A pressing question then is whether these attenuated correlations reflect statistical considerations, such as a lack of individual variability on tasks, or substantive considerations, such as that inhibition in different tasks is not a unified concept. One problem in addressing this question is that researchers aggregate performance across trials to tally individual-by-task scores. It is tempting to think that aggregation is fine and that everything comes out in the wash. But as shown here, this aggregation may greatly attenuate measures of effect size and correlation. We propose an alternative analysis of task performance that is based on accounting for trial-by-trial variability along with the covariation of individuals' performance across tasks. The implementation is through common hierarchical models, and this treatment rescues classical concepts of effect size, reliability, and correlation for studying individual differences with experimental tasks. Using recent data from Hedge et al. Behavioral Research Methods, 50(3), 1166-1186, 2018 we show that there is Bayes-factor support for a lack of correlation between the Stroop and flanker task. This support for a lack of correlation indicates a psychologically relevant result-Stroop and flanker inhibition are seemingly unrelated, contradicting unified concepts of inhibition.