Evaluating construct equivalence and criterion-related validity for repeat examinees on a standardized patient examination.

PURPOSE: Prior studies report large score gains for examinees who fail and later repeat standardized patient (SP) assessments. Although research indicates that score gains on SP exams cannot be attributed to memorizing previous cases, no studies have investigated the empirical validity of scores for repeat examinees. This report compares single-take and repeat examinees in terms of both internal (construct) validity and external (criterion-related) validity. METHOD: Data consisted of test scores for examinees who took the United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam between July 16, 2007, and September 12, 2009. The sample included 12,090 examinees who completed Step 2 CS on one occasion and another 4,030 examinees who completed the exam on two occasions. The internal measures included four separately scored performance domains of the Step 2 CS examination, whereas the external measures consisted of scores on three written assessments of medical knowledge (Step 1, Step 2 clinical knowledge, and Step 3). The authors subjected the four Step 2 CS domains to confirmatory factor analysis and evaluated correlations between Step 2 CS scores and the three written assessments for single-take and repeat examinees. RESULTS: The factor structure for repeat examinees on their first attempt was markedly different from the factor structure for single-take examinees, but it became more similar to that for single-take examinees by their second attempt. Scores on the second attempt correlated more highly with all three external measures. CONCLUSIONS: The findings support the validity of scores for repeat examinees on their second attempt.

The impact of repeat information on examinee performance for a large-scale standardized-patient examination.

PURPOSE: The United States Medical Licensing Examination series Step 2 Clinical Skills (CS) examination is a high-stakes performance assessment that uses standardized patients (SPs) to assess the clinical skills of physicians. Each Step 2 CS examination form involves 12 SPs, each of whom portrays a different clinical scenario or case. Examinees who fail and repeat the examination may encounter repeat information--the same SP, the same case, or the same SP portraying the same case. The goal of this study was twofold: to investigate score gains for all repeat examinees, regardless of whether they experienced repeat information, and to perform additional analyses for only those examinees who did encounter repeat information. METHOD: The dataset consisted of 3,045 Step 2 CS repeat examinees who initially tested between April 2005 and December 2007. The authors used paired t tests and analysis of variance models to assess mean score gains (first attempt versus second attempt) and to determine standardized mean differences between encounters with repeat information and those without. The authors ran each set of analyses by test score component and by examinee subgroup. RESULTS: The authors observed significant mean score increases on second attempt examinations for the entire group of repeat examinees. However, they observed no significant score increases for the subgroup of examinees who encountered repeat information. CONCLUSIONS: Examinees taking Step 2 CS for the second time improve on average, and those with prior exposure to exam information do not appear to benefit unfairly from this exposure.