Raising awareness about measurement error in research on unconscious mental processes.

Experimental psychologists often neglect the poor psychometric properties of the dependent measures collected in their studies. In particular, a low reliability of measures can have dramatic consequences for the interpretation of key findings in some of the most popular experimental paradigms, especially when strong inferences are drawn from the absence of statistically significant correlations. In research on unconscious cognition, for instance, it is commonly argued that the lack of a correlation between task performance and measures of awareness or explicit recollection of the target stimuli provides strong support for the conclusion that the cognitive processes underlying performance must be unconscious. Using contextual cuing of visual search as a case study, we show that given the low reliability of the dependent measures collected in these studies, it is usually impossible to draw any firm conclusion about the unconscious character of this effect from correlational analyses. Furthermore, both a psychometric meta-analysis of the available evidence and a cognitive-modeling approach suggest that, in fact, we should expect to see very low correlations between performance and awareness at the empirical level, even if both constructs are perfectly related at the latent level. Convincing evidence for the unconscious character of contextual cuing and other effects will most likely demand richer and larger data sets, coupled with more powerful analytic approaches.

The Challenge of Inferring Unconscious Mental Processes.

Studies of unconscious mental processes often compare a performance measure (e.g., some assessment of perception or memory) with a measure of awareness (e.g., a verbal report or forced-choice response) of the critical cue or contingency taken either concurrently or separately. The resulting patterns of bivariate data across participants lend themselves to several analytic approaches for inferring the existence of unconscious mental processes, but it is rare for researchers to consider the underlying generative processes that might cause these patterns. We show that bivariate data are generally insufficient to discriminate single-process models, with a unitary latent process determining both performance and awareness, from dual-process models, comprising distinct latent processes for performance and awareness. Future research attempting to isolate and investigate unconscious processes will need to employ richer types of data and analyses.

Correlation analysis to investigate unconscious mental processes: A critical appraisal and mini-tutorial.

As a method to investigate the scope of unconscious mental processes, researchers frequently obtain concurrent measures of task performance and stimulus awareness across participants. Even though both measures might be significantly greater than zero, the correlation between them might not, encouraging the inference that an unconscious process drives task performance. We highlight the pitfalls of this null-correlation approach and provide a mini-tutorial on ways to avoid them. As reference, we use a recent study by Salvador et al. (2018) reporting a non-significant correlation between the extent to which memory was suppressed by a Think/No-Think cue and an index of cue awareness. In the Null Hypothesis Significance Testing (NHST) framework, it is inappropriate to interpret failure to reject the null hypothesis (i.e., correlation = 0) as evidence for the null. Furthermore, psychological measures are often unreliable, which can dramatically attenuate the size of observed correlations. A Bayesian approach can circumvent both problems and compare the extent to which the data provide evidence for the null versus the alternative hypothesis (i.e., correlation > 0), while considering the usually low reliabilities of the variables. Applied to Salvador et al.'s data, this approach indicates no to moderate support for the claimed unconscious nature of participants' memory-suppression performance-depending on the model of the alternative hypothesis. Hence, more reliable data are needed. When analyzing correlational data, we recommend researchers to employ the Bayesian methods developed here (and made freely available as R scripts), rather than standard NHST methods, to take account of unreliability.

On a problematic procedure to manipulate response biases in recognition experiments: the case of "implied" base rates.

The experimental manipulation of response biases in recognition-memory tests is an important means for testing recognition models and for estimating their parameters. The textbook manipulations for binary-response formats either vary the payoff scheme or the base rate of targets in the recognition test, with the latter being the more frequently applied procedure. However, some published studies reverted to implying different base rates by instruction rather than actually changing them. Aside from unnecessarily deceiving participants, this procedure may lead to cognitive conflicts that prompt response strategies unknown to the experimenter. To test our objection, implied base rates were compared to actual base rates in a recognition experiment followed by a post-experimental interview to assess participants' response strategies. The behavioural data show that recognition-memory performance was estimated to be lower in the implied base-rate condition. The interview data demonstrate that participants used various second-order response strategies that jeopardise the interpretability of the recognition data. We thus advice researchers against substituting actual base rates with implied base rates.

No source memory for unrecognized items when implicit feedback is avoided.

In a recent empirical study, Starns, Hicks, Brown, and Martin (Memory & Cognition, 36, 1-8 2008) collected source judgments for old items that participants had claimed to be new and found residual source discriminability depending on the old-new response bias. The authors interpreted their finding as evidence in favor of the bivariate signal-detection model, but against the two-high-threshold model of item/source memory. According to the latter, NEW responses only follow from the state of old-new uncertainty for which no source discrimination is possible, and the probability of entering this state is independent of the old-new response bias. However, when missed old items were presented for source discrimination, the participants could infer that the items had been previously studied. To test whether this implicit feedback led to second retrieval attempts and thus to source memory for presumably unrecognized items, we replicated Starns et al.'s (Memory & Cognition, 36, 1-8 2008) finding and compared their procedure to a procedure without such feedback. Our results challenge the conclusion to abandon discrete processing in source memory; source memory for unrecognized items is probably an artifact of the procedure, by which implicit feedback prompts participants to reconsider their recognition judgment when asked to rate the source of old items in the absence of item memory.