Discovering unknown response patterns in progress test data to improve the estimation of student performance.

BACKGROUND: The Progress Test Medizin (PTM) is a 200-question formative test that is administered to approximately 11,000 students at medical universities (Germany, Austria, Switzerland) each term. Students receive feedback on their knowledge (development) mostly in comparison to their own cohort. In this study, we use the data of the PTM to find groups with similar response patterns. METHODS: We performed k-means clustering with a dataset of 5,444 students, selected cluster number k = 5, and answers as features. Subsequently, the data was passed to XGBoost with the cluster assignment as target enabling the identification of cluster-relevant questions for each cluster with SHAP. Clusters were examined by total scores, response patterns, and confidence level. Relevant questions were evaluated for difficulty index, discriminatory index, and competence levels. RESULTS: Three of the five clusters can be seen as "performance" clusters: cluster 0 (n = 761) consisted predominantly of students close to graduation. Relevant questions tend to be difficult, but students answered confidently and correctly. Students in cluster 1 (n = 1,357) were advanced, cluster 3 (n = 1,453) consisted mainly of beginners. Relevant questions for these clusters were rather easy. The number of guessed answers increased. There were two "drop-out" clusters: students in cluster 2 (n = 384) dropped out of the test about halfway through after initially performing well; cluster 4 (n = 1,489) included students from the first semesters as well as "non-serious" students both with mostly incorrect guesses or no answers. CONCLUSION: Clusters placed performance in the context of participating universities. Relevant questions served as good cluster separators and further supported our "performance" cluster groupings.

Communication skills training in undergraduate medical education at Charité - Universitätsmedizin Berlin.

Objective: The objective of this article is a description of the longitudinal communication curriculum in the Model Medicine Curriculum (MSM) at Charité - Universitätsmedizin Berlin. The authors describe the planning and integration of the curriculum into the study program, outline how communicative competence is taught and evaluated in the MSM, and identify which challenges need to be mastered in the process. Project description: Starting with the introduction of the MSM in 2010, students have been spending 102 class hours, spread out over seven semesters, practicing social and communicative competences in the interactive small group format "Communication, Interaction & Teamwork (KIT)". The course contents are closely linked to the topics covered each semester and increase in complexity over the course of their studies. The contents are selected by the KIT planning group whose members continually check the curriculum's timeliness and determine any changes. Students as well as instructors have opportunities for evaluating KIT throughout, and their evaluations are taken into consideration as KIT continues to be updated. Instructors from different disciplines teach KIT courses. They participate in mandatory didactic trainings that prepare them to teach KIT. During their 4th and 9th semesters, respectively, students take summative exams that test their communicative competence. Results: According to the semester evaluations by students and instructors, students participating in KIT improved their conversation management skills (students: M=2.2, SD=1.1, instructors: M=1.9, SD=0.7, on a scale of 1-5). In addition, students and graduates rate KIT to be (very) relevant, consider the degree to which it is taught in the MSM to be (very) high, and consider KIT to be a meaningful part of the curriculum. Students taking the summative exams in their 4th and 9th semesters achieve a mean score of 75.9%, respectively 76.9%, in the purely communicative stations and 82.6%, respectively 83.3%, in the global evaluation of communicative competence in clinical-practical stations. Discussion: Survey and exam results alike indicate that the communication training is well accepted by students and instructors and that the training led to an improvement in general and specific communicative skills. Due to a lack of control groups or a pre-post design, it has thus far not been possible to unequivocally demonstrate a causal relationship between communicative competence trainings and good test results. Quality control measures, such as trainings for instructors and regular course evaluations, have been designed to address any challenges in the implementation of the communication curriculum at the faculty level. Conclusion: Building on the experience with the Charité's Reformed Medical Curriculum, a longitudinal, competence-based communication curriculum was integrated into the MSM's overall curriculum. This measure remedied a gap in the medical training that many graduates of regular study programs had previously bemoaned (Jansen 2010 [1]).

Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination.

BACKGROUND: The concept of EPAs is increasingly applied to assess trainees' workplace performance by means of entrustment ratings. OSCEs assess performance in a simulated setting, and it is unclear whether entrustment ratings can be integrated into these exams. This study explores the introduction of an entrustment rating scale into an existing OSCE. METHODS: A 6-point entrustment scale was added to the standard ratings in an OSCE administered prior to students' final clerkship year in an undergraduate medical programme. Standard OSCE ratings assess clinical and communication skills. Assessors (n = 54) rated students' performance (n = 227) on a diverse set of clinical tasks and evaluated the addition of entrustment scales to OSCEs. Descriptive and inferential statistics were calculated for analyses. RESULTS: Student performance varied across the stations, as reflected in both the standard OSCE ratings and the added entrustment ratings. Students received generally high standard OSCE ratings, whereas entrustment ratings were more widely distributed. All students passed the OSCE, and only a small proportion of students did not reach the expected pass threshold of 60% on the standard ratings in the single stations. The proportion of students who did not reach the expected entrustment level in the respective stations was noticeably higher. Both the clinical and communication skill ratings were related to the entrustment rating in most OSCE stations. A majority of the assessors positively evaluated the addition of entrustment ratings into the OSCE. DISCUSSION: The findings provide an empirical basis to broaden our understanding of the potential use of entrustment ratings in existing OSCEs. They provide directions for future, more specific studies. The ratings might be used for formative feedback on students' readiness for workplace practice.