Does Using None-of-the-Above (NOTA) Hurt Students' Confidence?

Students claim that multiple-choice questions can be tricky, particularly those with competitive incorrect choices or choices like none-of-the-above (NOTA). Additionally, assessment researchers suggest that using NOTA is problematic for assessment. In experiments conducted online (with trivia questions) and in the classroom (with course-related questions), I investigated the effects of including NOTA as a multiple-choice choice alternative on students' confidence and performance. In four experiments, participants answered two types of questions: basic multiple-choice questions (basic condition) and equivalent questions in which one incorrect choice was replaced with NOTA (NOTA condition). Immediately after answering each question, participants rated their confidence in their answer to that question (item-by-item confidence). At the end of the experiments, participants made aggregate confidence judgments for the two types of questions and provided additional comments about the use of NOTA as an alternative. Surprisingly, I found no significant differences in item-by-item confidence or performance between the two conditions in any of the experiments. However, across all four experiments, when making aggregate judgments, participants provided lower confidence estimates in the NOTA condition than in the basic condition. Although people often report that NOTA questions hurt their confidence, the present results suggest that they might not-at least not on a question-by-question basis.

How "none of the above" (NOTA) affects the accessibility of tested and related information in multiple-choice questions.

Answering multiple-choice questions increases accessibility of the tested information and can improve accessibility of related information. However, multiple-choice questions with "none of the above" (NOTA) as a choice hurts accessibility of the previously tested information when NOTA serves as the correct answer (NOTAcorrect). Would prompting participants to recall an answer when choosing NOTA, and then providing feedback, reduce the costs of NOTAcorrect items? In the present experiments, participants answered general knowledge questions in multiple-choice (with a NOTA alternative) or cued-recall formats; half of the participants, when choosing NOTA, were prompted to provide an answer. Half of the participants received feedback. On a final cued-recall test assessing performance for previously tested and nontested related information, we found that NOTAcorrect questions hurt performance for previously tested items as compared to the cued-recall condition; but they facilitated recall of related information. Feedback improved performance for all previously tested information, but performance was still worse for NOTAcorrect questions than for cued-recall questions. The addition of a cued-recall component on NOTA questions did not influence accessibility. The results have implications for using NOTA in practice tests and help explain why using NOTAcorrect items hurt learning, even with feedback, supporting a proactive interference account.

Online and Clicker Quizzing on Jargon Terms Enhances Definition-Focused but Not Conceptually Focused Biology Exam Performance.

Mastery of jargon terms is an important part of student learning in biology and other science, technology, engineering, and mathematics domains. In two experiments, we investigated whether prelecture quizzes enhance memory for jargon terms, and whether that enhanced familiarity can facilitate learning of related concepts that are encountered during subsequent lectures and readings. Undergraduate students enrolled in neuroanatomy and physiology courses completed 10-minute low-stakes quizzes with feedback on jargon terms either online (experiment 1) or using in-class clickers (experiment 2). Quizzes occurred before conventional course instruction in which the terms were used. On exams occurring up to 12 weeks later, we observed improved student performance on questions that targeted memory of previously quizzed jargon terms and their definitions relative to questions on terms that were not quizzed. This pattern occurred whether those questions were identical (experiment 1) or different (experiment 2) from those used during quizzing. Benefits of jargon quizzing did not consistently generalize, however, to exam questions that assessed conceptual knowledge but not necessarily jargon knowledge. Overall, this research demonstrates that a brief and easily implemented jargon-quizzing intervention, deliverable via Internet or in-class platforms, can yield substantial improvements in students' course-relevant scientific lexica, but does not necessarily impact conceptual learning.

The role of retrieval in answering multiple-choice questions.

Taking a test improves memory for that tested information, a finding referred to as the testing effect. Multiple-choice tests tend to produce smaller testing effects than do cued-recall tests, and this result is largely attributed to the different processing that the two formats are assumed to induce. Specifically, it is generally assumed that the multiple-choice format bypasses the need to retrieve information. Research suggests, however, that multiple-choice questions can be constructed to induce retrieval of information pertaining to the incorrect alternatives. In the present research, we investigated the processes that individuals use to answer multiple-choice questions and how those processes relate to later memory, particularly for information pertaining to the incorrect alternatives. Most critically, we found that participants sometimes spontaneously recall information pertaining to incorrect alternatives, and these spontaneous retrievals are associated with retention of those alternatives as correct answers to related questions later. Although multiple-choice questions can be constructed so as to bypass retrieval, they can also be constructed to induce retrieval, and when they are, learning benefits are likely to occur. The present work has practical implications for how instructors can create multiple-choice questions to induce processes that facilitate learning. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Multiple-choice pretesting potentiates learning of related information.

Although the testing effect has received a substantial amount of empirical attention, such research has largely focused on the effects of tests given after study. The present research examines the effect of using tests prior to study (i.e., as pretests), focusing particularly on how pretesting influences the subsequent learning of information that is not itself pretested but that is related to the pretested information. In Experiment 1, we found that multiple-choice pretesting was better for the learning of such related information than was cued-recall pretesting or a pre-fact-study control condition. In Experiment 2, we found that the increased learning of non-pretested related information following multiple-choice testing could not be attributed to increased time allocated to that information during subsequent study. Last, in Experiment 3, we showed that the benefits of multiple-choice pretesting over cued-recall pretesting for the learning of related information persist over 48 hours, thus demonstrating the promise of multiple-choice pretesting to potentiate learning in educational contexts. A possible explanation for the observed benefits of multiple-choice pretesting for enhancing the effectiveness with which related nontested information is learned during subsequent study is discussed.

Category learning strategies in younger and older adults: Rule abstraction and memorization.

Despite the fundamental role of category learning in cognition, few studies have examined how this ability differs between younger and older adults. The present experiment examined possible age differences in category learning strategies and their effects on learning. Participants were trained on a category determined by a disjunctive rule applied to relational features. The utilization of rule- and exemplar-based strategies was indexed by self-reports and transfer performance. Based on self-reported strategies, the frequencies of rule- and exemplar-based learners were not significantly different between age groups, but there was a significantly higher frequency of intermediate learners (i.e., learners not identifying with a reliance on either rule- or exemplar-based strategies) in the older than younger adult group. Training performance was higher for younger than older adults regardless of the strategy utilized, showing that older adults were impaired in their ability to learn the correct rule or to remember exemplar-label associations. Transfer performance converged with strategy reports in showing higher fidelity category representations for younger adults. Younger adults with high working memory capacity were more likely to use an exemplar-based strategy, and older adults with high working memory capacity showed better training performance. Age groups did not differ in their self-reported memory beliefs, and these beliefs did not predict training strategies or performance. Overall, the present results contradict earlier findings that older adults prefer rule- to exemplar-based learning strategies, presumably to compensate for memory deficits. (PsycINFO Database Record

Can Multiple-Choice Testing Induce Desirable Difficulties? Evidence from the Laboratory and the Classroom.

The term desirable difficulties (Bjork, 1994) refers to conditions of learning that, though often appearing to cause difficulties for the learner and to slow down the process of acquisition, actually improve long-term retention and transfer. One known desirable difficulty is testing (as compared with restudy), although typically it is tests that clearly involve retrieval--such as free and cued recall tests--that are thought to induce these learning benefits and not multiple-choice tests. Nonetheless, multiple-choice testing is ubiquitous in educational settings and many other high-stakes situations. In this article, we discuss research, in both the laboratory and the classroom, exploring whether multiple-choice testing can also be fashioned to promote the type of retrieval processes known to improve learning, and we speculate about the necessary properties that multiple-choice questions must possess, as well as the metacognitive strategy students need to use in answering such questions, to achieve this goal.

Metamemory monitoring and control following retrieval practice for text.

Test-taking is assumed to help learners diagnose what they do and do not know, and by so doing improve the effectiveness of their subsequent study. Previous work has examined metamemory monitoring (e.g., predictions of future performance) and control (e.g., restudy decisions) following testing or retrieval practice with relatively simple materials (e.g., word pairs). There is reason to believe, however, that such monitoring and control decisions might be more difficult with text materials, even after retrieval practice, owing perhaps to difficulty in accurately assessing one's performance on the retrieval-practice test. In two experiments, participants read texts about world regions, then engaged in retrieval practice or rereading of the information in those texts, made estimates about future performance, and then received an opportunity to restudy the texts before taking a final recall test, with self-paced restudy enabling an examination of control processes. Memory predictions were more accurate in the retrieval-practice than in the rereading condition, and learners in both conditions allocated restudy time on the basis of their predictions. Additionally, restudy provided a greater benefit following retrieval practice than following rereading. The present study has implications for how students can use retrieval practice with text to foster subsequent learning.

Optimizing multiple-choice tests as tools for learning.

Answering multiple-choice questions with competitive alternatives can enhance performance on a later test, not only on questions about the information previously tested, but also on questions about related information not previously tested-in particular, on questions about information pertaining to the previously incorrect alternatives. In the present research, we assessed a possible explanation for this pattern: When multiple-choice questions contain competitive incorrect alternatives, test-takers are led to retrieve previously studied information pertaining to all of the alternatives in order to discriminate among them and select an answer, with such processing strengthening later access to information associated with both the correct and incorrect alternatives. Supporting this hypothesis, we found enhanced performance on a later cued-recall test for previously nontested questions when their answers had previously appeared as competitive incorrect alternatives in the initial multiple-choice test, but not when they had previously appeared as noncompetitive alternatives. Importantly, however, competitive alternatives were not more likely than noncompetitive alternatives to be intruded as incorrect responses, indicating that a general increased accessibility for previously presented incorrect alternatives could not be the explanation for these results. The present findings, replicated across two experiments (one in which corrective feedback was provided during the initial multiple-choice testing, and one in which it was not), thus strongly suggest that competitive multiple-choice questions can trigger beneficial retrieval processes for both tested and related information, and the results have implications for the effective use of multiple-choice tests as tools for learning.

Individual differences in category learning: memorization versus rule abstraction.

Although individual differences in category-learning tasks have been explored, the observed differences have tended to represent different instantiations of general processes (e.g., learners rely upon different cues to develop a rule) and their consequent representations. Additionally, studies have focused largely on participants' categorizations of transfer items to determine the representations that they formed. In the present studies, we used a convergent-measures approach to examine participants' categorizations of transfer items in addition to their self-reported learning orientations and response times on transfer items, and in doing so, we garnered evidence that qualitatively distinct approaches in explicit strategies for category learning (i.e., memorization vs. abstracting an articulable rule) and consequent representations might emerge in a single task. Participants categorized instances that followed a categorization rule (in Study 1, we used a relational rule; in Study 2, an additional task with a single-feature rule). Critically, for both tasks, some transfer items differed from trained instances on only one attribute (but otherwise were perceptually similar), rendering the item a member of the opposing category on the basis of the rule (i.e., termed ambiguous items). Some learners categorized ambiguous items on the basis of perceptual similarity, whereas others categorized them on the basis of an abstracted rule. Self-reported learning orientation (i.e., memorization vs. rule abstraction) predicted categorizations and response times on transfer items. Differences in learning orientations were not associated with performance on other cognitive measures (i.e., working memory capacity and Raven's Advanced Progressive Matrices). This work suggests that individuals may have different predispositions toward memorization versus rule abstraction in a single categorization task.

Multiple-choice tests exonerated, at least of some charges: fostering test-induced learning and avoiding test-induced forgetting.

Among the criticisms of multiple-choice tests is that-by exposing the correct answer as one of the alternatives-such tests engage recognition processes rather than the productive retrieval processes known to enhance later recall. We tested whether multiple-choice tests could trigger productive retrieval processes-provided the alternatives were made plausible enough to enable test takers to retrieve both why the correct alternatives were correct and why the incorrect alternatives were incorrect. In two experiments, we found not only that properly constructed multiple-choice tests can indeed trigger productive retrieval processes, but also that they had one potentially important advantage over cued-recall tests. Both testing formats fostered retention of previously tested information, but multiple-choice tests also facilitated recall of information pertaining to incorrect alternatives, whereas cued-recall tests did not. Thus, multiple-choice tests can be constructed so that they exercise the very retrieval processes they have been accused of bypassing.

The costs and benefits of testing text materials.

Tests have been shown to improve the later recall of tested information, a result known as the testing effect. Tests, however, can also impair the later recall of related information, an effect known as retrieval-induced forgetting. Although retrieval-induced forgetting has been demonstrated using a wide variety of materials, recent work suggests that learning information in the context of a coherent text passage may afford protection from retrieval-induced forgetting. In four experiments we explored the conditions under which retrieval-induced forgetting does and does not occur with such materials. We found that two factors-the coherence of the to-be-learned material and the competitiveness of retrieval practice-are important in determining whether retrieval-induced forgetting does or does not occur. Furthermore, even if retrieval-induced forgetting does occur, having the opportunity to restudy the forgotten information can prevent that forgetting from persisting. Taken together, these findings provide greater understanding of the costs and benefits of testing text materials, with possible implications for the optimisation of testing as a tool for learning in educational contexts.