Mean rating difference scores are poor measures of discernment: The role of response criteria.

Many interventions aim to protect people from misinformation. Here, we review common measures used to assess their efficacy. Some measures only assess the target behavior (e.g., ability to spot misinformation) and therefore cannot determine whether interventions have overly general effects (e.g., erroneously identifying accurate information as misinformation). Better measures assess discernment, the ability to discriminate target from non-target content. Discernment can determine whether interventions are overly general but is often measured by comparing differences in mean ratings between target and non-target content. We show how this measure is confounded by the configuration of response criteria, leading researchers to incorrectly conclude that an intervention improves discernment. We recommend using measures from signal detection theory, such as the area under the receiver operating characteristic curve, to assess discernment.

Effects of inductive learning and gamification on news veracity discernment.

This preregistered study tests a novel psychological intervention to improve news veracity discernment. The main intervention involved inductive learning (IL) training (i.e., practice discriminating between multiple true and fake news exemplars with feedback) with or without gamification. Participants (N = 282 Prolific users) were randomly assigned to either a gamified IL intervention, a nongamified version of the same IL intervention, a no-treatment control group, or a Bad News intervention, a notable web-based game designed to tackle online misinformation. Following the intervention (if applicable), all participants rated the veracity of a novel set of news headlines. We hypothesized that the gamified intervention would be the most effective for improving news veracity discernment, followed by its nongamified equivalent, then Bad News, and finally the control group. The results were analyzed with receiver-operating characteristic curve analyses, which have previously never been applied to news veracity discernment. The analyses indicated that there were no significant differences between conditions and the Bayes factor indicated very strong evidence for the null. This finding raises questions about the effectiveness of current psychological interventions and contradicts prior research that has supported the efficacy of Bad News. Age, gender, and political leaning all predicted news veracity discernment. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Gamified inoculation interventions do not improve discrimination between true and fake news: Reanalyzing existing research with receiver operating characteristic analysis.

Gamified inoculation interventions designed to improve the detection of online misinformation are becoming increasingly prevalent. Two of the most notable interventions of this kind are Bad News and Go Viral!. To assess their efficacy, prior research has typically used pre-post designs in which participants rated the reliability or manipulativeness of true and fake news items before and after playing these games, while most of the time also including a control group who played an irrelevant game (Tetris) or did nothing at all. Mean ratings were then compared between pre-tests and post-tests and/or between the control and experimental conditions. Critically, these prior studies have not separated response bias effects (overall tendency to respond "true" or "fake") from discrimination (ability to distinguish between true and fake news, commonly dubbed discernment). We reanalyzed the results from five prior studies using receiver operating characteristic (ROC) curves, a method common to signal detection theory that allows for discrimination to be measured free from response bias. Across the studies, when comparable true and fake news items were used, Bad News and Go Viral! did not improve discrimination, but rather elicited more "false" responses to all news items (more conservative responding). These novel findings suggest that the current gamified inoculation interventions designed to improve fake news detection are not as effective as previously thought and may even be counterproductive. They also demonstrate the usefulness of ROC analysis, a largely unexploited method in this setting, for assessing the effectiveness of any intervention designed to improve fake news detection. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The dark side of corrective feedback: Controlled and automatic influences of retrieval practice.

Corrective feedback is often touted as a critical benefit to learning, boosting testing effects when retrieval is poor and reducing negative testing effects. Here, we explore the dark side of corrective feedback. In three experiments, we found that corrective feedback on multiple-choice (MC) practice questions is later endorsed as the answer to related second-test questions, even though it is no longer correct. We describe this effect as an automatic influence of memory for feedback that participants fail to control. We explored how this influence is affected by the depth of retrieval during practice by successively increasing the retrieval demands of the MC practice test across the three experiments: Experiment 1 used a standard (select a single favorite option) format; Experiment 2 used ranking (rank order the options); and Experiment 3 used elimination testing (provide reasons for rejecting unchosen options). Increasing retrieval depth enhanced controlled influences on a cued-recall second test, evidenced by better accuracy on related versus new questions. However, it did not reduce the automatic influence on accuracy when the second test was MC, partly because repeating the options between practice and test likely led to false recognition of related questions. Together, the results suggest that MC practice tests produce both automatic and controlled memory influences on related second-test questions, with retrieval depth at practice being an important determinant of the controlled influence. However, whether that controlled influence will override the automatic influence of memory for the corrective feedback also depends on the second test format. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

How safe is this trip? Judging personal safety in a pandemic based on information from different sources.

In a preregistered experiment, we presented participants with information about the safety of traveling during a deadly pandemic and during a migration trip using five different sources (a news article, a family member, an official organization, someone with personal experience, and the travel organizer) and four different verbal descriptions of the likelihood of safety (very likely, likely, unlikely, and very unlikely). We found that both for the pandemic and migration contexts, judgments about the likelihood of safely traveling and decisions to travel were most strongly influenced by information from the respective official organizations and that participants also indicated greater willingness to share information from official organizations with others. These results are consistent with the established finding that expert sources are more persuasive. However, we also found that, regardless of source, participants thought that it would be safe to travel even when told that it was unlikely or very unlikely to be safe. Additionally, participants did not discriminate between the grades of likelihood description (such as between likely and very likely or between unlikely and very unlikely), suggesting that in the contexts examined directionality matters much more than attempts to communicate more fine-grained likelihood information with verbal phrases. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Can you trust what you hear? Concurrent misinformation affects recall memory and judgments of guilt.

In most misinformation studies, participants are exposed to a to-be-remembered event and then subsequently given misinformation in textual form. This misinformation impacts people's ability to accurately report the initial event. In this article, we present 2 experiments that explored a different approach to presenting misinformation. In the context of a murder suspect, the to-be-remembered event was audio of a police interview, whereas the misinformation was copresented as subtitles with some words being different to, and more incriminating than, those that were actually said. We refer to this as concurrent misinformation. In Experiment 1, concurrent misinformation was inappropriately reported in a cued-recall test, and inflated participants' ratings of how incriminating the audio was. Experiment 2 attempted to employ warnings to mitigate the influence of concurrent misinformation. Warnings after the to-be-remembered event had no effect, whereas warnings before the event reduced the effect of concurrent misinformation for a subset of participants. Participants that noticed the discrepancy between the audio and the subtitles were also less likely to judge the audio as incriminating. These results were considered in relation to existing theories underlying the misinformation effect, as well as the implication for the use of audio and text in applied contexts. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Repeated exposure to exemplars does not enhance implicit learning: A puzzle for models of learning and memory.

We learn regularities in the world around us, frequently without conscious effort, a phenomenon known as implicit learning. These regularities are often impossible to verbalise. One example of implicit learning is the structural effect, in which participants learn a rule set combining two factors, such as lexical frequency and concreteness. Theories of implicit learning predict that repetition of exemplar words would result in improved learning of the rule set, increasing the magnitude of the structural effect. Over four experiments, we demonstrate that this is, in fact, not the case. In Experiments 1 and 2, three repetitions of exemplar words result in superior item memory, but no change in the magnitude of the structural effect, compared with individually presented words. In Experiments 3 and 4, the structural effect is shown to be invariant to five repetitions of exemplar words and at high and low numbers of exemplars. In all four experiments, participants were unable to describe the rule set underlying the structural effect. However, confidence ratings demonstrated sensitivity to the structure and this sensitivity, unlike endorsements, increased with strength. The results are discussed in reference to differentiation, structural versus judgement knowledge, and flexible learning systems.

New improved gamma: Enhancing the accuracy of Goodman-Kruskal's gamma using ROC curves.

For decades, researchers have debated the relative merits of different measures of people's ability to discriminate the correctness of their own responses (resolution). The probabilistic approach, primarily led by Nelson, has advocated the Goodman-Kruskal gamma coefficient, an ordinal measure of association. The signal detection approach has advocated parametric measures of distance between the evidence distributions or the area under the receiver operating characteristic (ROC) curve. Here we provide mathematical proof that the indices associated with the two approaches are far more similar than has previously been thought: The true value of gamma is equal to twice the true area under the ROC curve minus one. Using this insight, we report 36 simulations involving 3,600,000 virtual participants that pitted gamma estimated with the original concordance/discordance formula against gamma estimated via ROC curves and the trapezoidal rule. In all but five of our simulations-which systematically varied resolution, the number of points on the metacognitive scale, and response bias-the ROC-based gamma estimate deviated less from the true value of gamma than did the traditional estimate. Consequently, we recommend using ROC curves to estimate gamma in the future.

Beyond hypercorrection: remembering corrective feedback for low-confidence errors.

Correcting errors based on corrective feedback is essential to successful learning. Previous studies have found that corrections to high-confidence errors are better remembered than low-confidence errors (the hypercorrection effect). The aim of this study was to investigate whether corrections to low-confidence errors can also be successfully retained in some cases. Participants completed an initial multiple-choice test consisting of control, trick and easy general-knowledge questions, rated their confidence after answering each question, and then received immediate corrective feedback. After a short delay, they were given a cued-recall test consisting of the same questions. In two experiments, we found high-confidence errors to control questions were better corrected on the second test compared to low-confidence errors - the typical hypercorrection effect. However, low-confidence errors to trick questions were just as likely to be corrected as high-confidence errors. Most surprisingly, we found that memory for the feedback and original responses, not confidence or surprise, were significant predictors of error correction. We conclude that for some types of material, there is an effortful process of elaboration and problem solving prior to making low-confidence errors that facilitates memory of corrective feedback.

Effects of postwarning specificity on memory performance and confidence in the eyewitness misinformation paradigm.

The influence of postevent misinformation on memory is typically constrained by postwarnings, but little is known about the effectiveness of particular features of postwarnings, such as their specificity. Experiment 1 compared 2 levels of postwarning specificity: A general postwarning just stated the presence of misinformation, whereas a specific postwarning identified the test items for which misinformation had been presented earlier. The specific postwarning, but not the general postwarning, eliminated both the misinformation effect and its deleterious impact on memory monitoring (using a classic 2-alternative forced-choice recognition procedure). Experiment 2 ruled out an alternative interpretation of these findings and replicated this postwarning specificity pattern using a cued-recall test. We observed, in addition to the moderating influence of task representations on misinformation acceptance, 2 unexpected facilitative effects on event memory caused by misinformation. Misinformation facilitated event memory during narrative encoding if discrepancies between the event and the narrative were detected (Experiment 1) and during retrieval if a specific postwarning was combined with cued recall (Experiment 2). We interpret the facilitative effect of discrepancy detection within a recursive-remindings framework on noticing and recollecting change. (PsycINFO Database Record

Auditory hindsight bias: Fluency misattribution versus memory reconstruction.

We report 4 experiments investigating auditory hindsight bias-the tendency to overestimate the intelligibility of distorted auditory stimuli after learning their identity. An associative priming manipulation was used to vary the amount of processing fluency independently of prior target knowledge. For hypothetical designs, in which hindsight judgments are made for peers in foresight, we predicted that judgments would be based on processing fluency and that hindsight bias would be greater in the unrelated- compared to related-prime context (differential-fluency hypothesis). Conversely, for memory designs, in which foresight judgments are remembered in hindsight, we predicted that judgments would be based on memory reconstruction and that there would be independent effects of prime relatedness and prior target knowledge (recollection hypothesis). These predictions were confirmed. Specifically, we found support for the differential-fluency hypothesis when a hypothetical design was used in Experiments 1 and 2 (hypothetical group). Conversely, when a memory design was used in Experiments 2 (memory group), 3A, and 3B, we found support for the recollection hypothesis. Together, the results suggest that qualitatively different mechanisms create hindsight bias in the 2 designs. The results are discussed in terms of fluency misattributions, memory reconstruction, anchoring-and-adjustment, sense making, and a multicomponent model of hindsight bias. (PsycINFO Database Record

Confidence in forced-choice recognition: What underlies the ratings?

Two-alternative forced-choice recognition tests are commonly used to assess recognition accuracy that is uncontaminated by changes in bias. In such tests, participants are asked to endorse the studied item out of 2 presented alternatives. Participants may be further asked to provide confidence judgments for their recognition decisions. It is often assumed that both recognition decisions and confidence judgments in 2-alternative forced-choice recognition tests depend on participants' assessments of a difference in strength of memory evidence supporting the 2 alternatives-the relative account. In the present study we focus on the basis of confidence judgments and we assess the relative account of confidence against the absolute account of confidence, by which in assigning confidence participants consider only strength of memory evidence supporting the chosen alternative. The results of the study show that confidence in 2-alternative forced-choice recognition decisions is higher when memory evidence is stronger for the chosen alternative and also when memory evidence is stronger for the unchosen alternative. These patterns of results are consistent with the absolute account of confidence in 2-alternative forced-choice recognition but they are inconsistent with the relative account. (PsycINFO Database Record

Memory, metamemory, and social cues: Between conformity and resistance.

When presented with responses of another person, people incorporate these responses into memory reports: a finding termed memory conformity. Research on memory conformity in recognition reveals that people rely on external social cues to guide their memory responses when their own ability to respond is at chance. In this way, conforming to a reliable source boosts recognition performance but conforming to a random source does not impair it. In the present study we assessed whether people would conform indiscriminately to reliable and unreliable (random) sources when they are given the opportunity to exercise metamemory control over their responding by withholding answers in a recognition test. In Experiments 1 and 2, we found the pattern of memory conformity to reliable and unreliable sources in 2 variants of a free-report recognition test, yet at the same time the provision of external cues did not affect the rate of response withholding. In Experiment 3, we provided participants with initial feedback on their recognition decisions, facilitating the discrimination between the reliable and unreliable source. This led to the reduction of memory conformity to the unreliable source, and at the same time modulated metamemory decisions concerning response withholding: participants displayed metamemory conformity to the reliable source, volunteering more responses in their memory report, and metamemory resistance to the random source, withholding more responses from the memory report. Together, the results show how metamemory decisions dissociate various types of memory conformity and that memory and metamemory decisions can be independent of each other.

Judgments of learning index relative confidence, not subjective probability.

The underconfidence-with-practice (UWP) effect is a common finding in calibration studies concerned with judgments of learning (JOLs) elicited on a percentage scale. The UWP pattern is present when, in a procedure consisting of multiple study-test cycles, the mean scale JOLs underestimate the mean recall performance on Cycle 2 and beyond. Although this pattern is present both for items recalled and unrecalled on the preceding cycle, to date research has concentrated mostly on the sources of UWP for the latter type of items. In the present study, we aimed to bridge this gap. In three experiments, we examined calibration on the third of three cycles. The results of Experiment 1 demonstrated the typical pattern of higher recall and scale JOLs for previously recalled items than for unrecalled ones. More importantly, they also revealed that even though the UWP effect was found for items previously recalled both once and twice, its magnitude was greater for the former class of items. Experiments 2 and 3, which employed a binary betting task and a binary 0 %/100 % JOL task, respectively, demonstrated that people can accurately predict future recall for previously recalled items with binary decisions. In both experiments, the UWP effect was absent for both items recalled once and twice. We suggest that the sensitivity of scale JOLs, but not binary judgments, to the number of previous recall successes strengthens the claim of Hanczakowski, Zawadzka, Pasek, and Higham (Journal of Memory and Language 69:429-444, 2013) that scale JOLs reflect confidence in, rather than the subjective probability of, future recall.

Memory and metacognition in dangerous situations: investigating cognitive impairment from gas narcosis in undersea divers.

OBJECTIVE: The current study tested whether undersea divers are able to accurately judge their level of memory impairment from inert gas narcosis. BACKGROUND: Inert gas narcosis causes a number of cognitive impairments, including a decrement in memory ability. Undersea divers may be unable to accurately judge their level of impairment, affecting safety and work performance. METHOD: In two underwater field experiments, performance decrements on tests of memory at 33 to 42 m were compared with self-ratings of impairment and resolution. The effect of depth (shallow [I-II m] vs. deep [33-42 m]) was measured on free-recall (Experiment I; n = 41) and cued-recall (Experiment 2; n = 39) performance, a visual-analogue self-assessment rating of narcotic impairment, and the accuracy of judgements-of-learning JOLs). RESULTS: Both free- and cued-recall were significantly reduced in deep, compared to shallow, conditions. This decrement was accompanied by an increase in self-assessed impairment. In contrast, resolution (based on JOLs) remained unaffected by depth. The dissociation of memory accuracy and resolution, coupled with a shift in a self-assessment of impairment, indicated that divers were able to accurately judge their decrease in memory performance at depth. CONCLUSION: These findings suggest that impaired self-assessment and resolution may not actually be a symptom of narcosis in the depth range of 33 to 42 m underwater and that the divers in this study were better equipped to manage narcosis than prior literature suggested. The results are discussed in relation to implications for diver safety and work performance.

Management of menopausal symptoms for breast cancer patients.

BACKGROUND: Menopausal symptoms are commonly experienced in women treated for breast cancer. This project aimed to identify the types and prevalence of menopausal symptoms women experience and assess how well such symptoms are managed by means of a clinical audit. The authors also wanted to identify whether patients and health professionals require further education in this area to enhance patients' quality of life. METHOD: A pilot audit was initially undertaken. Twenty women were recruited from medical and clinical oncology clinics spanning a 2-week period. The main audit was conducted over a 3-week period (19 March 2012 to 6 April 2012). A total of 215 patients were surveyed from 11 consultant-led and 1 nurse-led clinic per week. A menopause rating scale (MRS) developed by Heinemann et al (2003) was used to assess the types and severity of symptoms. RESULTS: Findings from the main audit provided preliminary evidence that certain breast cancer treatments can cause either the early onset of menopausal symptoms in pre-menopausal women or the return or aggravation of menopausal symptoms in peri-menopausal or post-menopausal women. This indicated that, for many women, symptoms are inadequately managed and supported. DISCUSSION: A more detailed exploratory study of the management of menopausal symptoms is needed. Health professionals should consider discussing such symptoms when patients start treatment and assess these symptoms at follow-up appointments to identify potential interventions.

The dud-alternative effect in memory for associations: putting confidence into local context.

When participants are asked to provide confidence judgments for each provided alternative in a multiple-choice memory task, such judgments are inflated if the assessed alternatives are accompanied by an implausible (dud) alternative. This finding, termed the dud-alternative effect, has been recently documented in a memory setting with a lineup procedure (Charman, Wells, & Joy, Law & Human Behavior 35:479-500 2011). In the present study, we developed a novel paradigm to investigate the dud-alternative effect in memory. The paradigm utilizes a multiple-choice associative recognition task in which dud alternatives can be rejected on the basis of their unfamiliarity. In two experiments, we demonstrated a reliable dud-alternative effect with our novel procedure. The results demonstrated that the dud-alternative effect in episodic memory is not limited to tasks based on perceptual factors, but is a general phenomenon concerning confidence judgments.

A little bias goes a long way: the effects of feedback on the strategic regulation of accuracy on formula-scored tests.

Under formula-scoring rules for multiple-choice exams, a penalty is applied to incorrect responses to reduce noise in the observed score. To avoid the penalty individuals are allowed to "pass," and therefore they must be able to strategically regulate the accuracy of their reporting by deciding which and how many questions to answer. To investigate the effect of bias within this framework, Higham (2007) introduced bias profiles, which show the score obtained under formula scoring (corrected score) as a function of the omission rate. Bias profiles estimate the optimal number of questions that should be answered to maximize the corrected score (i.e., optimal bias). Our initial research showed that individuals tend to be too conservative when setting reporting criteria, "omitting" too many answers. The present three experiments introduced a feedback manipulation whereby participants were informed of the optimal omission rate after completing a test and asked to alter their reporting decisions accordingly. This feedback and concomitant alteration of reporting decisions led to improved corrected scores on true/false (Experiment 1), 2-alternative tests (Experiments 2), and 4-alternative tests (Experiment 3). Importantly, corrected scores at optimal bias also were higher than at forced-report for both true/false and 2-alternative tests. Furthermore, in Experiment 3, feedback based on one test improved scores on a second test, and participants were more likely to perform optimally on a third test without feedback. These effects suggest that optimal-bias feedback may have long-term effects and generalize to new tests.

Dissociating early- and late-selection processes in recall: the mixed blessing of categorized study lists.

Two experiments are reported in which we used type-2 signal detection theory to separate the effects of semantic categorization on early- and late-selection processes in free and cued recall. In Experiment 1, participants studied cue-target pairs for which the targets belonged to two, six, or 24 semantic categories, and later the participants were required to recall the targets either with (cued recall) or without (free recall) the studied cues. A confidence rating and a report decision were also required, so that we could compute both forced-report quantity and metacognitive resolution (type-2 discrimination), which served as our estimates of early- and late-selection processes, respectively. Consistent with prior research, having fewer categories enhanced the early-selection process (in performance, two > six > 24 categories). However, in contrast, the late-selection process was impaired (24 > six = two categories). In Experiment 2, encoding of paired associates, for which the targets belonged to either two or 20 semantic categories, was manipulated by having participants either form interactive images or engage in rote repetition. Having fewer categories again was associated with enhanced early selection (two > 20 categories); this effect was greater for rote repetition than for interactive imagery, and greater for free recall than for cued recall. However, late selection again showed the opposite pattern (20 > two categories), even with interactive-imagery encoding, which formed distinctive, individuated memory traces. The results are discussed in terms of early- and late-selection processes in retrieval, as well as overt versus covert recognition.

Implicit learning of conjunctive rule sets: an alternative to artificial grammars.

A single experiment is reported that investigated implicit learning using a conjunctive rule set applied to natural words. Participants memorized a training list consisting of words that were either rare-concrete and common-abstract or common-concrete and rare-abstract. At test, they were told of the rule set, but not told what it was. Instead, they were shown all four word types and asked to classify words as rule-consistent words or not. Participants classified the items above chance, but were unable to verbalize the rules, even when shown a list that included the categories that made up the conjunctive rule and asked to select them. Most participants identified familiarity as the reason for classifying the items as they did. An analysis of the materials demonstrated that conscious micro-rules (i.e., chunk knowledge) could not have driven performance. We propose that such materials offer an alternative to artificial grammar for studies of implicit learning.