RESUMO
OBJECTIVE: To determine the impact of item-writing flaws and cognitive level on student performance metrics in one course series across two semesters at a single institution. METHODS: Four investigators reviewed 928 multiple-choice items from an integrated therapeutics course series. Differences in performance metrics were examined between flawed and standard items, flawed stems and flawed answer choices, and cognitive levels. RESULTS: Reviewers found that 80% of the items were flawed, with the most common types being implausible distractors and unfocused stems. Flawed items were easier than standard ones, with items becoming easier as they accumulate more flaws. These flaws benefited lower-performing students more than higher-performing ones, particularly up to two flaws. Items with flawed stems had the same difficulty as standard items, but those with flawed answer choices were significantly easier. Most items tested lower-level skills and have more flaws than higher-level items. There was no significant difference in difficulty between lower- and higher-level cognitive items, and higher-level items were more likely to have answer flaws than stem flaws. CONCLUSION: Item-writing flaws differently impact student performance. Implausible distractors artificially lower the difficulty of questions, even those designed to assess higher-level skills. This effect contributes to a lack of significant difference in difficulty between higher- and lower-level items. Unfocused stems, on the other hand, likely increase confusion and hinder performance regardless of the question's cognitive complexity.
RESUMO
OBJECTIVE: The purpose of this analysis was to identify, analyze, and report patterns (or themes) of planning and preparation considerations of students that scored less than the historic average score on the Pre-NAPLEX® exam. METHODS: This qualitative study was a retrospective, inductive thematic analysis of de-identified semi-structured interview field notes collected from student interviews for those students that scored less than the historic average score on the Pre-NAPLEX® exam. RESULTS: Ninety-one students were initially contacted based on their score on the Pre-NAPLEX® exam to participate in one-on-one virtual discussions (i.e., interviews) with faculty members. Fifty-two responded and participated with their responses analyzed and included in thematic categorization. Four major themes were identified during the analysis. These include 1) Organization and Messaging of NAPLEX® Preparation Efforts, 2) Time Management during Competing Obligations, 3) Test Taking Experience, and 4) Curricular Disconnect. CONCLUSION: Student performance on the NAPLEX licensing exam is of great concern to many colleges of pharmacy. As a result, many institutions are looking at root-causes for poor performance and working to implement structural changes at their institution to address these concerns. This investigation identified four major themes surrounding the preparation and planning for the Pre-NAPLEX® for students that scored less than the historic average score on the Pre-NAPLEX®. These include 1) Organization and Messaging of NAPLEX® Preparation Efforts, 2) Time Management during Competing Obligations, 3) Test Taking Experience, and 4) Curricular Disconnect. Each of these themes provides potentially actionable items to improve how students prepare and plan for the Pre-NAPLEX®, which may be translatable to informing actions to improve results on the actual NAPLEX exam itself.
