The Effects of Personalized Nudges on Cognitively Disengaged Student Behavior in Low-Stakes Assessments.

In educational settings, students rely on metacognitive processes to determine whether or not to exert effort. We investigated ways to minimize cognitively disengaged responses (i.e., not-fully-effortful responses) during a low-stakes mathematics assessment. Initially, we established theory-driven time thresholds for each item to detect such responses. We then administered the test to 800 eighth-graders across three conditions: (a) control (n = 271); (b) instruction (n = 267); and (c) nudge (n = 262). In the instruction condition, students were told to exert their best effort before starting the assessment. In the nudge condition, students were prompted to give their best effort following each first-attempt response that was both incorrect and not-fully-effortful. Therefore, students had multiple opportunities to adjust their level of effort. Nudges, but not effort instruction, significantly reduced students' not-fully-effortful responses. Neither the nudges nor the effort instruction significantly impacted performance. In a post-test survey, most students reported that they received nudges whenever they did not know the answer (55%). Overall, these findings suggest that while nudges reduce cognitively disengaged responses, most students appear to strategically modulate their level of effort based on self-monitoring their knowledge and response effort.

Memory updating after retrieval: when new information is false or correct.

We conducted three experiments testing the malleability of memory in incorporating new information following retrieval. All experiments used associative lists typical of the DRM paradigm [Deese, J. (1959). On the prediction of occurrence of particular verbal intrusions in immediate recall. Journal of Experimental Psychology, 58(1), 17-22; Roediger, H. L., & McDermott, K. B. (1995). Creating false memories: Remembering words not presented in lists. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(4), 803-814]. This paradigm enabled the evaluation of the integration of false information and correct information with the original information. In Experiment 1, participants studied DRM lists, and in a later phase either retrieved or restudied the lists and were presented with never-presented critical lures. The results of Experiment 1 showed that compared to restudy, retrieval enhanced the integration of subsequent false information, as measured by later recall in a follow-up test. In Experiments 2 and 3, after initial study, participants retrieved or studied incorrect information and received corrective feedback. The results showed that retrieval led to more error correction than restudy, when feedback was presented immediately. In general, this research suggests retrieval facilitates incorporation of new, related information, regardless of whether it is false or correct.

How to bring the news peak-end effects in children's affective responses to peer assessments of their social behavior.

The retrospective evaluation of an event tends to be based on how the experience felt during the most intense moment and the last moment. Two experiments tested whether this so-called peak-end effect influences how primary school students are affected by peer assessments. In both experiments, children (ages 7-12) assessed two classmates on their behaviour in school and then received two manipulated assessments. In Experiment 1 (N = 30), one assessment consisted of four negative ratings and the other of four negative ratings with an extra moderately negative rating added to the end. In Experiment 2 (N = 44), one assessment consisted of four positive ratings, and the other added an extra moderately positive rating to the end. Consistent with the peak-end effect, the extended assessment in Experiment 1 and the short assessment in Experiment 2 were remembered as more pleasant and less difficult to deal with, which shaped children's peer assessment preferences and prospective choices of which assessment to repeat. These findings indicate that the process of peer assessment can be improved by ending the feedback with the most positive part of the assessment.

The effect of unsuccessful retrieval on children's subsequent learning.

It is well known that successful retrieval enhances subsequent adults' learning by promoting long-term retention. Recent research has also found benefits from unsuccessful retrieval, but the evidence is not as clear-cut when the participants are children. In this study, we employed a methodology based on guessing-the weak associate paradigm-to test whether children can learn from generated errors or whether errors are harmful for learning. We tested second- and third-grade children in Experiment 1 and tested preschool and kindergarten children in Experiment 2. With slight differences in the method, in both experiments children heard the experimenter saying one word (cue) and were asked to guess an associate word (guess condition) or to listen to the correspondent target-associated word (study condition), followed by corrective feedback in both conditions. At the end of the guessing phase, the children undertook a cued-recall task in which they were presented with each cue and were asked to say the corrected target. Together, the results showed that older children-those in kindergarten and early elementary school-benefited from unsuccessful retrieval. Older children showed more correct responses and fewer errors in the guess condition. In contrast, preschoolers produced similar levels of correct and error responses in the two conditions. In conclusion, generating errors seems to be beneficial for future learning of children older than 5years.

Is a Computerized Adaptive Test More Motivating Than a Fixed-Item Test?

Computer adaptive tests provide important measurement advantages over traditional fixed-item tests, but research on the psychological reactions of test takers to adaptive tests is lacking. In particular, it has been suggested that test-taker engagement, and possibly test performance as a consequence, could benefit from the control that adaptive tests have on the number of test items examinees answer correctly. However, previous research on this issue found little support for this possibility. This study expands on previous research by examining this issue in the context of a mathematical ability assessment and by considering the possible effect of immediate feedback of response correctness on test engagement, test anxiety, time on task, and test performance. Middle school students completed a mathematics assessment under one of three test type conditions (fixed, adaptive, or easier adaptive) and either with or without immediate feedback about the correctness of responses. Results showed little evidence for test type effects. The easier adaptive test resulted in higher engagement and lower anxiety than either the adaptive or fixed-item tests; however, no significant differences in performance were found across test types, although performance was significantly higher across all test types when students received immediate feedback. In addition, these effects were not related to ability level, as measured by the state assessment achievement levels. The possibility that test experiences in adaptive tests may not in practice be significantly different than in fixed-item tests is raised and discussed to explain the results of this and previous studies.

Hitting a high note on math tests: Remembered success influences test preferences.

Remembered utility is the retrospective evaluation about the pleasure and pain associated with a past experience. It can influence choices about repeating or avoiding similar situations in the future (Kahneman, 2000). A set of 5 experiments explored the remembered utility of effortful test episodes and how it impacted future test choices. Experiments 1-3 mimicked Kahneman, Fredrickson, Schreiber, and Redelmeier's (1993) cold pressor study, but used a challenging test experience in place of submerging one's hand in painfully cold ice water. Participants took a short and an extended test of difficult math problems. The short test was made of 30 difficult math problems. The extended test used 30 difficult problems plus 10 moderately difficult problems. Participants made retrospective evaluations about the tests and chose which kind of test (short or extended) they wanted for their next test and for a hypothetical test 1 day later. Results showed that a challenging test episode extended by a more moderate interval was preferred to an unextended interval. Future test choices reflected this preference. This pattern was evidenced even when points were awarded only to correctly answered difficult items but not to moderately answered difficult items. Experiment 4 directly compared preferences for a moderate end versus a moderate start test, and Experiment 5 contrasted those test conditions with a moderate middle test condition. The findings demonstrated test-takers' bias in favor of test experiences that start or end on a high note. (PsycINFO Database Record

Interfering effects of retrieval in learning new information.

In 7 experiments, we explored the role of retrieval in associative updating, that is, in incorporating new information into an associative memory. We tested the hypothesis that retrieval would facilitate incorporating a new contextual detail into a learned association. Participants learned 3 pieces of information-a person's face, name, and profession (in Experiments 1-5). In the 1st phase, participants in all conditions learned faces and names. In the 2nd phase, participants either restudied the face-name pair (the restudy condition) or were given the face and asked to retrieve the name (the test condition). In the 3rd phase, professions were presented for study just after restudy or testing. Our prediction was that the new information (the profession) would be more readily learned following retrieval of the face-name association compared to restudy of the face-name association. However, we found that the act of retrieval generally undermined acquisition of new associations rather than facilitating them. This detrimental effect emerged on both immediate and delayed tests. Further, the effect was not due to selective attention to feedback because we found impairment whether or not feedback was provided after the Phase 2 test. The data are novel in showing that the act of retrieving information can inhibit the ability to learn new information shortly thereafter. The results are difficult to accommodate within current theories that mostly emphasize benefits of retrieval for learning.

Reconsolidation from negative emotional pictures: is successful retrieval required?

Finn and Roediger (Psychological science 22:781-786, 2011) found that when a negative emotional picture was presented immediately after a successful retrieval, later test performance was enhanced as compared to when a neutral picture or a blank screen had been shown. This finding implicates the period immediately following retrieval as playing an important role in determining later retention via reconsolidation. In two new experiments, we investigated whether successful retrieval was required to show the enhancing effect of negative emotion on later recall. In both experiments, the participants studied Swahili-English vocabulary pairs, took an intervening cued-recall test, and were given a final cued-recall test on all items. In Experiment 1, we tested a distinctiveness explanation of the effect. The results showed that neither presentation of a negative picture just prior to successful retrieval nor presentation of a positive picture after successful retrieval produced the enhancing effect that was seen when negative pictures were presented after successful retrieval. In Experiment 2, we tested whether the enhancing effect would occur when a negative picture followed an unsuccessful retrieval attempt with feedback, and a larger enhancement effect occurred after errors of commission than after errors of omission. These results indicate that effort in retrieving is critical to the enhancing effect shown with negative pictures; whether the target is produced by the participant or given by an external source following a commission error does not matter. We interpret these results as support for semantic enrichment as a key element in producing the enhancing effect of negative pictures that are presented after a retrieval attempt.

Metacognitive judgments of repetition and variability effects in natural concept learning: evidence for variability neglect.

In four experiments, we examined the effects of repetitions and variability on the learning of bird families and metacognitive awareness of such effects. Of particular interest was the accuracy of, and bases for, predictions regarding classification of novel bird species, referred to as category learning judgments (CLJs). Participants studied birds in high repetitions and high variability conditions. These conditions differed in the number of presentations of each bird (repetitions) and the number of unique species from each family (variability). After study, participants made CLJs for each family and were then tested. Results from a classification test revealed repetition benefits for studied species and variability benefits for novel species. In contrast with performance, CLJs did not reflect the benefits of variability. Results showed that CLJs were susceptible to accessibility-based metacognitive illusions produced by additional repetitions of studied items.

Enhancing retention through reconsolidation: negative emotional arousal following retrieval enhances later recall.

When information is retrieved from memory, it enters a labile state rendering it amenable to change. This process of reconsolidation may explain, in part, the benefits that are observed in later retention following retrieval of information on an initial test. We examined whether the benefits of retrieval could be modulated by an emotional event occurring after retrieval. Participants studied Swahili-English vocabulary pairs. On a subsequent cued-recall test, each retrieval was followed by a blank screen, a neutral picture, or a picture inducing negative affect. Performance on a final cued-recall test was best for items whose initial retrieval was followed by negative pictures. This outcome occurred when a negative picture was presented immediately after (Experiment 1) or 2 s after (Experiment 2) successful retrieval, but not when it was presented after restudy of the vocabulary pair (Experiment 3). Postretrieval reconsolidation via emotional processing may enhance the usual positive effects of retrieval.

People's hypercorrection of high-confidence errors: did they know it all along?

This study investigated the "knew it all along" explanation of the hypercorrection effect. The hypercorrection effect refers to the finding that when people are given corrective feedback, errors that are committed with high confidence are easier to correct than low-confidence errors. Experiment 1 showed that people were more likely to claim that they knew it all along when they were given the answers to high-confidence errors as compared with low-confidence errors. Experiments 2 and 3 investigated whether people really did know the correct answers before being told or whether the claim in Experiment 1 was mere hindsight bias. Experiment 2 showed that (a) participants were more likely to choose the correct answer in a 2nd guess multiple-choice test when they had expressed an error with high rather than low confidence and (b) that they were more likely to generate the correct answers to high-confidence as compared with low-confidence errors after being told they were wrong and to try again. Experiment 3 showed that (c) people were more likely to produce the correct answer when given a 2-letter cue to high- rather than low-confidence errors and that (d) when feedback was scaffolded by presenting the target letters 1 by 1, people needed fewer such letter prompts to reach the correct answers when they had committed high- rather than low-confidence errors. These results converge on the conclusion that when people said that they knew it all along, they were right. This knowledge, no doubt, contributes to why they are able to correct those high-confidence errors so easily.

Does easily learned mean easily remembered?: it depends on your beliefs about intelligence.

Because numerous studies have shown that feelings of encoding fluency are positively correlated with judgments of learning, a single dominant heuristic, easily learned = easily remembered (ELER), has been posited to explain how people interpret encoding fluency when assessing their own memory. However, the inferences people draw from feelings of encoding fluency may vary with their beliefs about why information is easy or effortful to encode. We conducted two experiments in which participants studied word lists and then predicted their future recall of those items. Results revealed that subjects who viewed intelligence as fixed, and who tended to interpret effortful encoding as indicating that they had reached the limits of their ability, used the ELER heuristic to make judgments of learning. However, subjects who viewed intelligence as malleable, and who tended to interpret effortful encoding as indicating greater engagement in learning, did not use the ELER heuristic and at times predicted greater memory for items that they found more effortful to learn.

Scaffolding feedback to maximize long-term error correction.

Scaffolded feedback was tested against three other feedback presentation methods (standard corrective feedback, minimal feedback, and answer-until-correct multiple-choice feedback) over both short- and long-term retention intervals in order to assess which method would produce the most robust gains in error correction. Scaffolded feedback was a method designed to take advantage of the benefits of retrieval practice by providing incremental hints until the correct answer could be self-generated. In Experiments 1 and 3, on an immediate test, final memory for the correct answer was lowest for questions given minimal feedback, moderate for the answer-until-correct condition, and equally high in the scaffolded feedback condition and the standard feedback condition. However, tests of the maintenance of the corrections over a 30-min delay (Experiment 2) and over a 1-day delay (Experiment 3) demonstrated that scaffolded feedback gave rise to the best memory for the correct answers at a delay.

Ending on a high note: adding a better end to effortful study.

Remembered utility is the retrospective evaluation about the pleasure and pain associated with a past experience. It has been shown to influence prospective choices about whether to repeat or to avoid similar situations in the future (D. Kahneman 2000; D. Kahneman, D. L. Fredrickson, C. A. Schreiber, & D. A. Redelmeier, 1993). Evaluations about our hedonic past often disregard the duration of the experience and are influenced more by the peak and the final levels of discomfort (B. L. Fredrickson & D. Kahneman, 1993). Two experiments explored the remembered discomfort of an effortful learning experience and the influence of this evaluation on prospective study choices. The design of the studies mimicked D. Kahneman et al.'s (1993) cold-pressor study, but used an exceptionally challenging learning experience in place of the painful experience of submerging one's hand in ice water. An extremely effortful study episode extended by a more moderate interval was preferred to a shorter, unextended interval, despite better test performance following the shorter interval. Future study choices reflected this preference. These findings suggest that the act of acquiring knowledge has value in the learning process.

Delayed versus immediate feedback in children's and adults' vocabulary learning.

We investigated whether the superior memory performance sometimes seen with delayed rather than immediate feedback was attributable to the shorter retention interval (or lag to test) from the last presentation of the correct information in the delayed condition. Whether lag to test was controlled or not, delayed feedback produced better final test performance than did immediate feedback, which in turn produced better performance than did no feedback at all, when we tested Grade 6 children learning school-relevant vocabulary. With college students learning GRE-level words, however, delayed feedback produced better performance than did immediate feedback (and both were better than no feedback) when lag to test was uncontrolled, but there was no difference between the delayed and immediate feedback conditions when the lag to test was controlled.

Judgments of Learning are Influenced by Memory for Past Test.

The Underconfidence with Practice (UPW) effect (Koriat, Sheffer & Ma'ayan, 2002), found in multi-trial learning, is marked by a pattern of underconfidence accompanied by an increase in resolution between the judgments and test on and after the second trial. We tested whether the memory for past test (MPT) heuristic (Finn & Metcalfe, 2007) could explain the resolution and calibration effects. To selectively alter Trial 1 test performance, and hence MPT, we manipulated the number of repetitions (Experiment 1) or the study time (Experiment 2) on Trial 1, but then the manipulation was reversed on Trial 2, thereby equating final performance. Despite equivalent Trial 2 recall performance, Trial 2 JOLs reflected the manipulated Trial 1 test performance, providing support for the MPT hypothesis. Follow up experiments tested alternative explanations. We found that people could remember past test and that use of this information would produce both underconfidence and improved resolution. In contrast, neither memory for Trial 1 encoding fluency nor memory for Trial 1 JOLs was able to explain both aspects of the UWP effect. These experiments support the proposal that people use the Memory for Past Test heuristic to make second trial immediate JOLs, and that its use can account for the UWP effect.

Familiarity and retrieval processes in delayed judgments of learning.

Two processes are postulated to underlie delayed judgments of learning (JOLs)--cue familiarity and target retrievability. The two processes are distinguishable because the familiarity-based judgments are thought to be faster than the retrieval-based processes, because only retrieval-based JOLs should enhance the relative accuracy of the correlations between the JOLs and criterion test performance, and because only retrieval-based judgments should enhance memory. To test these predictions, in three experiments, the authors either speeded people's JOLs or allowed them to be unspeeded. The relative accuracy of the JOLs in predicting performance on the criterion test was higher for the unspeeded JOLs than for the speeded JOLs, as predicted. The unspeeded JOL conditions showed enhanced memory as compared with the speeded JOL conditions, as predicted. Finally, the unspeeded JOLs were sensitive to manipulations that modified recallability of the target, whereas the speeded JOLs were selectively sensitive to experimental variations in the familiarity of the cues. Thus, all three of the predictions about the consequences of the two processes potentially underlying delayed JOLs were borne out. A model of the processes underlying delayed JOLs based on these and earlier results is presented.

Framing effects on metacognitive monitoring and control.

Three experiments explored the contribution of framing effects on metamemory judgments. In Experiment 1, participants studied word pairs. After each presentation, they made an immediate judgment of learning (JOL), framed in terms of either remembering or forgetting. In the remember frame, participants made judgments about how likely it was that they would remember each pair on the upcoming test. In the forget frame, participants made judgments about how likely it was that they would forget each pair. Confidence differed as a result of the frame. Forget frame JOLs, equated to the remember frame JOL scale by a 1-judgment conversion, were lower and demonstrated a smaller overconfidence bias than did remember frame JOLs. When judgments were made at a delay, framing effects did not occur. In Experiment 2, people chose to restudy more items when choices were made within a forget frame. In Experiment 3, people studied Spanish-English vocabulary pairs ranging in difficulty. The framing effect was replicated with judgments and choices. Moreover, forget frame participants included more easy and medium items to restudy. These results demonstrated the important consequences of framing effects on assessment and control of study.

Evidence that judgments of learning are causally related to study choice.

Three experiments investigated whether study choice was directly related to judgments of learning (JOLs) by examining people's choices in cases in which JOLs were dissociated from recall. In Experiment 1, items were given either three repetitions or one repetition on Trial 1. Items given three repetitions received one on Trial 2, and those given one repetition received three on Trial 2-equating performance at the end of Trial 2, but yielding different immediate Trial 2 JOLs. Study choice followed the "illusory" JOLs. A delayed JOL condition in Experiment 2 did not show this JOL bias and neither did study choice. Finally, using a paradigm (Koriat & Bjork, 2005) in which similar JOLs are given to forward and backward associative pairs, despite much worse performance on the backward pairs, study choice again followed the mistaken JOLs. We concluded that JOLs-what people believe they know-directly influence people's study choices.

The role of memory for past test in the underconfidence with practice effect.

According to the Memory for Past Test (MPT) heuristic, judgments of learning (JOLs) may be based, in part, on memory for the correctness of answers on a previous test. The authors explored MPT as the source of the underconfidence with practice effect (UWP; A. Koriat, L. Sheffer, & H. Ma'ayan, 2002), whereby Trial 1 overconfidence switches to underconfidence by Trial 2. Immediate and delayed JOLs were contrasted because only immediate JOLs demonstrate UWP. Consistent with MPT for immediate JOLs, Trial 1 test performance better predicted Trial 2 JOLs than did Trial 2 test performance. Delayed JOLs showed the reverse. Furthermore, items forgotten on Trial 1 but remembered on Trial 2 contributed disproportionately to UWP, but only with immediate JOLs.