Does 'summative' count? The influence of the awarding of study credits on feedback use and test-taking motivation in medical progress testing.

Despite the increasing implementation of formative assessment in medical education, its' effect on learning behaviour remains questionable. This effect may depend on how students value formative, and summative assessments differently. Informed by Expectancy Value Theory, we compared test preparation, feedback use, and test-taking motivation of medical students who either took a purely formative progress test (formative PT-group) or a progress test that yielded study credits (summative PT-group). In a mixed-methods study design, we triangulated quantitative questionnaire data (n = 264), logging data of an online PT feedback system (n = 618), and qualitative interview data (n = 21) to compare feedback use, and test-taking motivation between the formative PT-group (n = 316), and the summative PT-group (n = 302). Self-reported, and actual feedback consultation was higher in the summative PT-group. Test preparation, and active feedback use were relatively low and similar in both groups. Both quantitative, and qualitative results showed that the motivation to prepare and consult feedback relates to how students value the assessment. In the interview data, a link could be made with goal orientation theory, as performance-oriented students perceived the formative PT as not important due to the lack of study credits. This led to low test-taking effort, and feedback consultation after the formative PT. In contrast, learning-oriented students valued the formative PT, and used it for self-study or self-assessment to gain feedback. Our results indicate that most students are less motivated to put effort in the test, and use feedback when there are no direct consequences. A supportive assessment environment that emphasizes recognition of the value of formative testing is required to motivate students to use feedback for learning.

Progress Testing in UK Medical Education: Evaluating Its Impact and Potential.

This comprehensive review critically examines the UK medical curriculum, with a particular focus on progress testing as an innovative assessment strategy. The curriculum, evolving from foundational sciences to practical applications, is encapsulated in the integrated curriculum model (ICM). This model adeptly combines theoretical knowledge with clinical practice, fostering cognitive, affective, and psychomotor skills among medical students. Central to this review is an exploration of progress testing. This method, grounded in constructivist learning theories, emphasises continuous assessment and professional development. Progress testing's regular, comprehensive examinations are instrumental in guiding students through the progressive stages of competence, as outlined in Miller's pyramid, from foundational knowledge to clinical proficiency. The review also addresses the broader impacts of progress testing on teaching approaches, student feedback, academic and pastoral support, and quality assurance. By aligning with the dynamic requirements of 21st-century medical training, progress testing not only nurtures well-rounded professionals but also ensures compliance with regulatory bodies like the General Medical Council. Its emphasis on continuous evaluation aligns with the practical realities of a medical career, driving curricular innovation and aligning with regulatory standards. The implementation of progress testing marks a significant advancement in medical education. Its continuous, holistic nature benefits both students and educators, nurturing a more engaged learning attitude and meeting evolving medical needs. The adoption of this assessment strategy is seen as pivotal in shaping competent medical professionals, ready to face the challenges of modern medical practice.

Progress testing of an objective structured clinical examination during undergraduate clinical clerkship: a mixed-methods pilot study.

BACKGROUND: Progress testing is an assessment method in which an examination reflecting competencies at graduation is regularly administered to students over multiple years, thereby facilitating self-directed learning. However, the significance of the objective structured clinical examination as a progress test in undergraduate education, needs to be determined. This study provides evidence of the role of the objective structured clinical examination for progress testing and optimal scoring methods for assessing students in different academic years. METHODS: We conducted a sequential explanatory mixed-methods pilot study. Participants were assessed using the Item Rating Scale, the year-adjusted Global Rating Scale, and the Training Level Rating Scale. The characteristics of each scale were compared quantitatively. In addition, the influence of the objective structured clinical examination as a progress test on learning attitudes was examined. Qualitative data from a post-examination questionnaire were analyzed, using content analysis to explore influences on self-directed learning. RESULTS: Sixth and fifth year clinical students (n = 235) took the objective structured clinical examination progress test. The total Item Rating Scales were recorded (%) as 59.03 ± 5.27 and 52.64 ± 5.08 (p < 0.01); Training Level Rating Scale was 3.94 ± 0.39 vs 3.22 ± 0.42 (p < 0.01); and the year-adjusted Global Rating Scale was 4.25 ± 0.44 vs 4.32 ± 0.52 (no significant difference), for the sixth and fifth year students, respectively. The correlations across stations and the reliability of each station were satisfactory. Four categories were identified in the qualitative analysis: "motivation to learn during the clinical clerkship was promoted," "dissatisfied with being asked about things they had not experienced," "confusion about being unable to use conventional test-taking strategies," and "insufficient understanding of competencies at graduation." The scores indicated significant differences in performance according to training year. CONCLUSIONS: This study provides evidence that the objective structured clinical examination can be used as a progress testing tool for undergraduate clinical clerkships. Further enhancement of training opportunities and dissemination of performance competency goals in clerkship curricula are required if we intend to promote self-directed learning through progress testing.

A narrative review of adaptive testing and its application to medical education.

Adaptive testing has a long but largely unrecognized history. The advent of computer-based testing has created new opportunities to incorporate adaptive testing into conventional programmes of study. Relatively recently software has been developed that can automate the delivery of summative assessments that adapt by difficulty or content. Both types of adaptive testing require a large item bank that has been suitably quality assured. Adaptive testing by difficulty enables more reliable evaluation of individual candidate performance, although at the expense of transparency in decision making, and requiring unidirectional navigation. Adaptive testing by content enables reduction in compensation and targeted individual support to enable assurance of performance in all the required outcomes, although at the expense of discovery learning. With both types of adaptive testing, candidates are presented a different set of items to each other, and there is the potential for that to be perceived as unfair. However, when candidates of different abilities receive the same items, they may receive too many they can answer with ease, or too many that are too difficult to answer. Both situations may be considered unfair as neither provides the opportunity to demonstrate what they know. Adapting by difficulty addresses this. Similarly, when everyone is presented with the same items, but answer different items incorrectly, not providing individualized support and opportunity to demonstrate performance in all the required outcomes by revisiting content previously answered incorrectly could also be considered unfair; a point addressed when adapting by content. We review the educational rationale behind the evolution of adaptive testing and consider its inherent strengths and limitations. We explore the continuous pursuit of improvement of examination methodology and how software can facilitate personalized assessment. We highlight how this can serve as a catalyst for learning and refinement of curricula; fostering engagement of learner and educator alike.

Development of a shared item repository for progress testing in veterinary education.

Introduction: Progress testing in education is an assessment principle for the measurement of students' progress over time, e.g., from start to graduation. Progress testing offers valid longitudinal formative measurement of the growth in the cognitive skills of the individual students within the subjects of the test as well as a tool for educators to monitor potential educational gaps and mismatches within the curriculum in relation to the basic veterinary learning outcomes. Methods: Six veterinary educational establishments in Denmark, Finland, Germany (Hannover), the Netherlands, Norway, and Sweden established in cooperation with the European Association of Establishments for Veterinary Education (EAEVE) a common veterinary item repository that can be used for progress testing in European Veterinary Education Establishments (VEEs), linear as well as computer adaptive, covering the EAEVE veterinary subjects and theoretical "Day One Competencies." First, a blueprint was created, suitable item formats were identified, and a quality assurance process for reviewing and approving items was established. The items were trialed to create a database of validated and calibrated items, and the responses were subsequently psychometrically analyzed according to Modern Test Theory. Results: In total, 1,836 items were submitted of which 1,342 were approved by the reviewers for trial testing. 1,119 students from all study years and all partners VEEs participated in one or more of six item trials, and 1,948 responses were collected. Responses were analyzed using Rasch Modeling (analysis of item-fit, differential item function, item-response characteristics). A total of 821 calibrated items of various difficulty levels matching the veterinary students' abilities and covering the veterinary knowledge domains have been banked. Discussion: The item bank is now ready to be used for formative progress testing in European veterinary education. This paper presents and discusses possible pitfalls, problems, and solutions when establishing an international veterinary progress test.

Professional Identity Formation, Professionalism, Leadership and Resilience (PILLAR) in Medical Students: Methodology and Early Results.

Objectives: The fundamental role of medical education is the transformation of students to doctors, through a process of education and professional identity formation (PIF), which can be informed by several educational, behavioural and emotional factors. PIF has been deemed to be of equal importance to the acquisition of clinical knowledge and skills and includes constructs such as professionalism, leadership and resilience. We aimed to assess professional identity formation, professionalism, leadership and resilience (PILLAR) in the junior years of medical school in the 2020/2021 academic year and illustrate the potential role of quantitative assessment to demonstrate progression in these areas. In this research, we provide the methods and baseline results for the PILLAR study. Methods: We implemented a compulsory assessment in pre-clinical years of graduate entry and direct entry medicine at the Royal College of Surgeons in Ireland University of Medicine and Health Sciences, Dublin, Ireland. Validated scales were used to assess students' PILLAR. Descriptive and univariable statistical techniques were used to compare student scores between respective years. Results: A total of 1311 students (92% response rate) provided their consent for research. For the psychometric scales, there were no evident trends among the years on these assessment measures. Results indicated significant differences in all measures, however, these did not correspond to ascending years of seniority. Conclusion: The PILLAR methodology provides important information on the challenges of quantitatively assessing medical students in the four key areas of PIF, professionalism, leadership, and resilience. Our cross-sectional results point to cohort effects, without the expected progression per year in the cross-sectional data, or suggest that the chosen quantitative measures may be problematic for these constructs in pre-clinical students. Therefore, while we believe that PILLAR has potential as a progress test for these constructs, this will only truly be elucidated by repeated measures of each cohort over time.

Relationship between medical programme progress test performance and surgical clinical attachment timing and performance.

PURPOSE: Progress tests (PTs) assess applied knowledge, promote knowledge integration, and facilitate retention. Clinical attachments catalyse learning through an appropriate learning context. The relationship between PT results and clinical attachment sequence and performance are under-explored. Aims: (1) Determine the effect of Year 4 general surgical attachment (GSA) completion and sequence on overall PT performance, and for surgically coded items; (2) Determine the association between PT results in the first 2 years and GSA assessment outcomes. MATERIALS AND METHODS: All students enrolled in the medical programme, who started Year 2 between January 2013 and January 2016, were included; with follow up until December 2018. A linear mixed model was applied to study the effect of undertaking a GSA on subsequent PT results. Logistic regressions were used to explore the effect of past PT performance on the likelihood of a student receiving a distinction grade in the GSA. RESULTS: 965 students were included, representing 2191 PT items (363 surgical items). Sequenced exposure to the GSA in Year 4 was associated with increased performance on surgically coded PT items, but not overall performance on the PT, with the difference decreasing over the year. PT performance in Years 2-3 was associated with an increased likelihood of being awarded a GSA distinction grade (OR 1.62, p < 0.001), with overall PT performance a better predictor than performance on surgically coded items. CONCLUSIONS: Exposure to a surgical attachment improves PT results in surgically coded PT items, although with a diminishing effect over time, implying clinical exposure may accelerate subject specific learning. Timing of the GSA did not influence end of year performance in the PT. There is some evidence that students who perform well on PTs in preclinical years are more likely to receive a distinction grade in a surgical attachment than those with lower PT scores.

Progress testing: An educational perspective exploring the rationale for progress testing and its introduction into a Diagnostic Radiography curriculum.

INTRODUCTION: In March 2020, the first diagnostic radiography degree apprenticeship programme in England was launched at the authors' institution. As part of the programme development and design, the programme development team explored and then implemented progress testing into a strand of the programme. The objective of this educational perspective is to scrutinise the literature around the use of progress testing in higher education programmes, namely medicine, to explain how and why this decision was reached. METHODS: The initial search strategy was developed using the electronic databases CINHAL Complete and SCOPUS. Key words included 'progress test' and 'medicine' or 'health' or 'education' or 'higher education'. Eliminating articles that were not relevant, and also identifying and adding additional articles by key authors and experts resulted in thirty-three key articles being considered for review. RESULTS: The thirty-three articles were a mixture of review articles, empirical research, case studies and conference presentations. Five key themes were identified which are discussed in this article; the evolution of progress testing; advantages of progress testing, disadvantages of progress testing, developing a test framework and academic progression and student feedback. DISCUSSION: Progress testing is now well established in pre-registration medical programmes globally. The advantages of progress testing and the use of frequent look rapid remediation appear to be undisputed. Key disadvantages with progress testing were identified as it being an administrative heavy assessment process as well as a perceived bias towards male students undertaking this type of assessment. CONCLUSION: Now this assessment practice is established within medicine, it seems reasonable to explore its use in other areas of healthcare, such as radiography.

Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools.

BACKGROUND: Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive. METHODS: We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests. RESULTS: The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, ηp2 = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, ηp2 = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, ηp2 = 0.002. CONCLUSIONS: Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.

Retrieval practice and spaced learning: preventing loss of knowledge in Dutch medical sciences students in an ecologically valid setting.

INTRODUCTION: Knowledge, once acquired, degrades over time. Exams that contain questions related to previously acquired knowledge ('retrieval practice questions') may promote retrieval practice and spaced learning, and subsequently prevent knowledge loss. To investigate this hypothesis, we compare the score of retrieval practice questions to regular questions in exams of a two-year (bio)medical study program. METHODS: The two-year "Mechanisms of Health and Disease"-program for biomedical sciences and medical students in Nijmegen (the Netherlands) contains 14 spaced exams of 80 questions each. The percentages of correct-, false-, and non-answers were compared between regular questions and retrieval practice questions. Using Pearson correlations between question scores and exam scores (RiT-values), the impact of retrieval practice questions on the internal consistency of exams was determined. Mixed model analyses determined changes in outcomes across time. RESULTS: Analysis of 2006 regular questions and 1728 retrieval practice questions revealed a significantly higher percentage of correct and false answers, and a significantly lower percentage of non-answers, in retrieval practice questions versus regular questions (all P < 0.05). Scores did not change across time. RiT-values were slightly lower in retrieval practice questions, with a small inverse trend across time. CONCLUSION: Our data indicate preservation of knowledge, possibly related to retrieval practice and/or spaced learning. Although the RiT-values of retrieval practice questions were slightly lower than those of regular questions, the discriminative capacity was well within acceptable range. These data highlight the potency of retrieval practice questions to prevent knowledge decrement, without altering exam quality.

Anatomy (knowledge) in postgraduate obstetrics and gynaecology training: Is it sufficient enough?

BACKGROUND: Due to the importance, anatomy training is worldwide recognizable in virtually all undergraduate curricula and many postgraduate surgical curricula (Estai and Bunt, 2016; Older, 2004). The postgraduate curriculum of Obstetrics and Gynaecology (O&G) is such a surgical curriculum. It is a diverse branch of medicine and the role of anatomy in O&G is versatile. In the Netherlands nor in Europe the expectations of knowledge on anatomy are specified in the current training program, making trainees insecure about their performance in anatomy knowledge ("Better Education for Obsetrics and Gynaecology,"). Therefore, we recently performed a Delphi study to determine which anatomical structures should be taught to ensure safe and competent practice among general gynaecologists (Koppes et al., 2020). The aim of this study is the determination of the anatomical knowledge level in postgraduate training for O&G. Our hypothesis is that the trainees possess a good knowledge of anatomy and on average at least 80% of correct answers on core knowledge is shown. METHODS: A longitudinal knowledge analysis was performed under Dutch Trainees Obstetrics and Gynaecology. The anatomy questions of the annual progress tests from 2010 to 2019 were analysed. Anatomy questions were selected and assessed on relevance based on the previous performed Delphi study which identified 86 structures which are essential to perform safe and competent practice as a general gynaecologist. Scores on relevant anatomy questions were calculated. RESULTS: In 10-year 3136 trainees performed the annual progress test. 54 Anatomy related questions were asked on a total of 1637 questions (3.3%). Of these 54 questions, 38 (70%) were concerned as relevant questions. Overall 10-year score was 64.5%. CONCLUSIONS: The anatomy knowledge of trainees' Obstetrics and Gynaecology is insufficient. Our results are a step in the awareness of testing and improving anatomy knowledge of postgraduate O&G training.

Using longitudinal progress test data to determine the effect size of learning in undergraduate medical education - a retrospective, single-center, mixed model analysis of progress testing results.

Medical education research focuses on the development of efficient learning methods promoting the acquisition of student's knowledge and competencies. Evaluation of any modification of educational approaches needs to be evaluated accordingly and a reliable effect size needs to be reached. Our aim is to provide a methodological basis to calculate effect sizes from longitudinal progress test data that can be used as reference values in further research. We used longitudinally collected progress test data and evaluated the increasing knowledge of medical students from the first to the fifth academic year. Students were asked to participate in the progress test, which consists of 200 multiple-choice questions in single best answer format with an additional 'don't know' option. All available individual test scores of all progress tests (n = 10) administered between April 2012 and October 2017 were analyzed. Due to the large amount of missing test results, e.g., from students at the beginning of their studies, a linear mixed model was fitted to include all collected data. In total, we analyzed 6324 test scores provided by 2587 medical students. Mean score for medical knowledge (% correct answers) increases from 16.6% (SD: 10.8%) to 51.0% (SD: 15.7%, overall effects size using linear mixed models d = 1.55). Medical students showed a learning effect of d = 0.54 (total gain: 6.9%) between the 1st and 2nd, d = 0.88 (total gain: 12.0%) between the 2nd and 3rd, d = 0.60 (total gain: 7.9%) between the 3rd and 4th and d = 0.58 (total gain: 7.9%) between the 4th and 5th study year. We demonstrated that incomplete data from longitudinally collected progress tests can be used to acquire reliable effect size estimates. The demonstrated effects size between d = 0.53-0.9 by study year may help researchers to design studies in medical education.

Twelve tips for embedding assessment for and as learning practices in a programmatic assessment system.

Programmatic assessment supports the evolution from assessment of learning to fostering assessment for learning and as learning practices. A well-designed programmatic assessment system aligns educational objectives, learning opportunities, and assessments with the goals of supporting student learning, making decisions about student competence and promotion decisions, and supporting curriculum evaluation. We present evidence-based guidance for implementing assessment for and as learning practices in the pre-clinical knowledge assessment system to help students learn, synthesize, master and retain content for the long-term so that they can apply knowledge to patient care. Practical tips are in the domains of culture and motivation of assessment, including how an honour code and competency-based grading system can support an assessment system to develop student self-regulated learning and professional identity, curricular assessment structure, such as how and when to utilize low-stakes and cumulative assessment to drive learning, exam and question structure, including what authentic question and exam types can best facilitate learning, and assessment follow-up and review considerations, such exam retake processes to support learning, and academic success structures. A culture change is likely necessary for administrators, faculty members, and students to embrace assessment as most importantly a learning tool for students and programs.

University Admission Test Associates with Academic Performance at the End of Medical Course in a PBL Medical Hybrid Curriculum.

PURPOSE: Most studies assessing the value of the university admissions test (UAT) to predict academic performance at the end of a medical course were carried out on lecture-based medical courses. However, the association between performance in the UAT with academic achievement at the end of medical course in a problem-based learning (PBL) medical hybrid curriculum remains controversial. The aim of this study was to correlate marks in the UAT with those obtained in the Organized Structured Clinical Examination (OSCE), in the progress testing (PT), and in the final marks of the clerkship (FMC). METHODS: We used data from 48 medical students. A single and a multiple dependency studies were performed to assess bivariate and multiple correlation between the UAT or the essay scores (dependent variables) and the OSCE, PT, and FMC (independent variables). Pearson test, multiple linear regression, and ANOVA tests were used and a p-value < 0.05 was considered significant. RESULTS: In the bivariate analysis, only the UAT and FMC marks were correlated (r=0.34; p=0.02). However, the multiple dependency study showed a moderate correlation among UAT, OSCE, PT, and FMC marks (r=0.46; p=0.01). No correlation was found between the essay scores and PT, FMC, and OSCE scores. CONCLUSION: Our study shows that UAT marks, but not essay scores, can predict academic achievement, particularly in terms of clinical competence (FMC) at the end of a medical course in a PBL hybrid curriculum.

Entrustment within an objective structured clinical examination (OSCE) progress test: Bridging the gap towards competency-based medical education.

PURPOSE: Progress testing aligns well with competency-based medical education (CBME) frameworks, which stress the importance of continuous improvement. Entrustment is a useful assessment concept in CBME models. The purpose of this study was to explore the use of an entrustability rating scale within the context of an objective structured clinical examination (OSCE) Progress Test. METHODS: A 9-case OSCE Progress Test was administered to Internal Medicine residents (PGYs 1-4). Residents were assessed using a checklist (CL), global rating scale (GRS), training level rating scale (TLRS), and entrustability scale (ENT). Reliability was calculated using Cronbach's alpha. Differences in performance by training year were explored using ANOVA and effect sizes were calculated using partial eta-squared. Examiners completed a post-examination survey. RESULTS: Ninety one residents and forty two examiners participated in the OSCE. Inter-station reliability was high for all instruments. There was an overall effect of training level for all instruments (p < 0.001). Effect sizes were large. 88% of examiners completed the survey. Most (62%) indicated feeling comfortable in making entrustment decisions during the OSCE. CONCLUSIONS: An entrustability scale can be used in an OSCE Progress Test to generate highly reliable ratings that discriminate between learners at different levels of training.

Progress testing anytime and anywhere - Does a mobile-learning approach enhance the utility of a large-scale formative assessment tool?

BACKGROUND: The widespread use of mobile devices among students favors the use of mobile learning scenarios at universities. In this study, we explore whether a time- and location-independent variant of a formative progress test has an impact on the students' acceptance, its validity and reliability and if there is a difference in response processes between the two exam conditions. METHODS: Students were randomly assigned to two groups of which one took the test free of local or temporal fixations, while the other group took the test at the local testing center under usual examination conditions. Beside the generated test data, such as test score, time-on-test, and semester status, students also evaluated the settings. RESULTS: While there was no significant effect on the test score between the two groups, students in the mobile group spent more time on the test and were more likely to use the help of books or online resources. The results of the evaluation show that the acceptability among students is increased by a mobile version of the formative progress test. CONCLUSIONS: The results suggest that the acceptance and motivation to participate in formative tests is enhanced by lifting local and temporal restrictions. The mobile version nonetheless does not have an impact on the students' performance.

START - evaluating a novel assessment of consultant readiness in paediatrics: The entry not the exit.

The Royal College of Paediatrics and Child Health (RCPCH) incepted a new end-of-training assessment in 2012, known as START, the Speciality Trainee Assessment of Readiness for Tenure [as a Consultant]. It is a novel, formative, multi-scenario, OSCE-style, out-of-workplace assessment using unseen scenarios with generic, external assessors undertaken in the trainees' penultimate training year. This study considers whether this assessment assists in preparing senior paediatric trainees for consultant working. A mixed qualitative and quantitative study in the post-positivist paradigm was designed. Subjects were paediatricians who have taken START and completed their paediatric training. Methods were an on-line questionnaire survey and a key informant interview. The assessment is viewed positively, but some trainees report negative experiences. They find value in the formative feedback which generally helps direct trainees towards focussing their training in their final year before ending their training and consultant appointment. For many respondents, the assessment highlighted areas for further development, was relevant for consultant working and useful for consultant interview preparation. Of least value was travelling, cost, assessor performance, feedback quality, feeling like a summative exam and sub-speciality involvement. Many respondents felt the assessment highlighted areas to develop in their subsequent training. Overall START supports transition to consultant working.

Assessing the growth in clinical skills using a progress clinical skills examination.

BACKGROUND: This study evaluates the generalizability of an eight-station progress clinical skills examination and assesses the growth in performance for six clinical skills domains among first- and second-year medical students over four time points during the academic year. METHODS: We conducted a generalizability study for longitudinal and cross-sectional comparisons and assessed growth in six clinical skill domains via repeated measures ANOVA over the first and second year of medical school. RESULTS: The generalizability of the examination domain scores was low but consistent with previous studies of data gathering and communication skills. Variations in case difficulty across administrations of the examination made it difficult to assess longitudinal growth. It was possible to compare students at different training levels and the interaction of training level and growth. Second-year students outperformed first-year students, but first-year students' clinical skills performance grew faster than second-year students narrowing the gap in clinical skills over the students' first year of medical school. CONCLUSIONS: Case specificity limits the ability to assess longitudinal growth in clinical skills through progress testing. Providing students with early clinical skills training and authentic clinical experiences appears to result in the rapid growth of clinical skills during the first year of medical school.

Relationships between Bloom's taxonomy, judges' estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study

ABSTRACT BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Introducing Summative Progress Testing in Radiology Residency: Little Change in Residents' Test Results After Transitioning from Formative Progress Testing.

INTRODUCTION: Educational effects of transitioning from formative to summative progress testing are unclear. Our purpose was to investigate whether such transitioning in radiology residency is associated with a change in progress test results. METHODS: We investigated a national cohort of radiology residents (N > 300) who were semi-annually assessed through a mandatory progress test. Until 2014, this test was purely formative for all residents, but in 2014/2015, it was transitioned (as part of a national radiology residency program revision) to include a summative pass requirement for new residents. In 7 posttransitioning tests in 2015-2019, including summatively and formatively tested residents who followed the revised and pre-transitioning residency program, respectively, we assessed residents' relative test scores and percentage of residents that reached pass standards. RESULTS: Due to our educational setting, most posttransitioning tests had no residents in the summative condition in postgraduate year 4-5, nor residents in the formative condition in year 0.5-2. Across the 7 tests, relative test scores in postgraduate year 1-3 of the summative resident group and year 3.5-4.5 of the formative group differed significantly (p < 0.01 and p < 0.05, respectively, Kruskal-Wallis test). However, scores fluctuated without consistent time trends and without consistent differences between both resident groups. Percentage of residents reaching the pass standard did not differ significantly across tests or between groups. DISCUSSION: Transitioning from formative to summative progress testing was associated with overall steady test results of the whole resident group in 4 post-transitioning years. We do not exclude that transitioning may have positive educational effects for resident subgroups.