ChatGPT in medical school: how successful is AI in progress testing?

BACKGROUND: As generative artificial intelligence (AI), ChatGPT provides easy access to a wide range of information, including factual knowledge in the field of medicine. Given that knowledge acquisition is a basic determinant of physicians' performance, teaching and testing different levels of medical knowledge is a central task of medical schools. To measure the factual knowledge level of the ChatGPT responses, we compared the performance of ChatGPT with that of medical students in a progress test. METHODS: A total of 400 multiple-choice questions (MCQs) from the progress test in German-speaking countries were entered into ChatGPT's user interface to obtain the percentage of correctly answered questions. We calculated the correlations of the correctness of ChatGPT responses with behavior in terms of response time, word count, and difficulty of a progress test question. RESULTS: Of the 395 responses evaluated, 65.5% of the progress test questions answered by ChatGPT were correct. On average, ChatGPT required 22.8 s (SD 17.5) for a complete response, containing 36.2 (SD 28.1) words. There was no correlation between the time used and word count with the accuracy of the ChatGPT response (correlation coefficient for time rho = -0.08, 95% CI [-0.18, 0.02], t(393) = -1.55, p = 0.121; for word count rho = -0.03, 95% CI [-0.13, 0.07], t(393) = -0.54, p = 0.592). There was a significant correlation between the difficulty index of the MCQs and the accuracy of the ChatGPT response (correlation coefficient for difficulty: rho = 0.16, 95% CI [0.06, 0.25], t(393) = 3.19, p = 0.002). CONCLUSION: ChatGPT was able to correctly answer two-thirds of all MCQs at the German state licensing exam level in Progress Test Medicine and outperformed almost all medical students in years 1-3. The ChatGPT answers can be compared with the performance of medical students in the second half of their studies.

Discovering unknown response patterns in progress test data to improve the estimation of student performance.

BACKGROUND: The Progress Test Medizin (PTM) is a 200-question formative test that is administered to approximately 11,000 students at medical universities (Germany, Austria, Switzerland) each term. Students receive feedback on their knowledge (development) mostly in comparison to their own cohort. In this study, we use the data of the PTM to find groups with similar response patterns. METHODS: We performed k-means clustering with a dataset of 5,444 students, selected cluster number k = 5, and answers as features. Subsequently, the data was passed to XGBoost with the cluster assignment as target enabling the identification of cluster-relevant questions for each cluster with SHAP. Clusters were examined by total scores, response patterns, and confidence level. Relevant questions were evaluated for difficulty index, discriminatory index, and competence levels. RESULTS: Three of the five clusters can be seen as "performance" clusters: cluster 0 (n = 761) consisted predominantly of students close to graduation. Relevant questions tend to be difficult, but students answered confidently and correctly. Students in cluster 1 (n = 1,357) were advanced, cluster 3 (n = 1,453) consisted mainly of beginners. Relevant questions for these clusters were rather easy. The number of guessed answers increased. There were two "drop-out" clusters: students in cluster 2 (n = 384) dropped out of the test about halfway through after initially performing well; cluster 4 (n = 1,489) included students from the first semesters as well as "non-serious" students both with mostly incorrect guesses or no answers. CONCLUSION: Clusters placed performance in the context of participating universities. Relevant questions served as good cluster separators and further supported our "performance" cluster groupings.

Evaluation of a 3D-MC examination format in anatomy.

INTRODUCTION: The quality of education in medical anatomy is a fundamental pillar of good clinical practice. Current reforms of the medical curriculum have resulted in major methodological changes in the teaching and testing of anatomy. A number of recent studies have however described a decrease in positive metrics of anatomical knowledge among students so taught. It has been suggested that the reduced anatomical knowledge measured in these studies may endanger patient safety. As proxy measures of exam quality, evaluation of the levels of students 'achievement in the examinations, assessment of the subjectively perceived level of question difficulty and analysis of exam satisfaction are each suitable parameters of investigation of medical education. MATERIAL AND METHODS: To address these issues with regard to medical education at the Charité-Universitätsmedizin Berlin, we have analyzed students' levels of achievement in the anatomical Three Dimensional Multiple Choice (hereafter, 3D-MC)-examination of 2,015 students matriculated in medical studies from Summer Semester of 2014 through Summer Semester of 2017. We either compared students' achievement levels of identical 3D-MC questions using models or prepared anatomical specimen. Furthermore, we have analyzed the type and frequency of cognitive levels used in the anatomical questions in relation to the students' level of achievement. Finally, we conducted an anonymous survey to measure students' (n = 207) and instructors' (n = 16) satisfaction with the 3D-MC-examination in comparison to other employed anatomical testing strategies. RESULTS: Students' achievement is significantly enhanced with anatomical questions using models relative to those utilizing anatomical specimen. Over 80% of the anatomical questions in the 3D-MC-examinations assessed the lowest cognitive levels and higher cognitive question levels were accompanied by a significant decrease of the levels of students' performance. Our survey further revealed that both, students and instructors preferred the practical examinations in anatomy and that the difficulty levels of the 3D-MC-examination was perceived as being the lowest in comparison to the other anatomical testing strategies. DISCUSSION: Testing levels of anatomical understanding using anatomical models is not comparable to human specimen, and thus using specimen before models should be preferred to learn and test close to an authentic medical situation. The application of anatomical models and low cognitive question levels in the examination reduces the subjectively perceived level of difficulty, encourages superficial learning, and therefore decreases the retention of anatomical knowledge. CONCLUSION: Although students and instructors prefer practical examinations in anatomy, the current development does not reflect these results. Therefore, it would be recommendable to rethink the development of anatomical testing strategies based on the existing evidence.

Self-monitoring accuracy does not increase throughout undergraduate medical education.

CONTEXT: Accurate self-assessment of one's performace on a moment-by-moment basis (ie, accurate self-monitoring) is vital for the self-regulation of practising physicians and indeed for the effective regulation of self-directed learning during medical education. However, little is currently known about the functioning of self-monitoring and its co-development with medical knowledge across medical education. This study is the first to simultaneously investigate a number of relevant aspects and measures that have so far been studied separately: different measures of self-monitoring for a broad area of medical knowledge across 10 different performance levels. METHODS: This study assessed the self-monitoring accuracy of medical students (n = 3145) across 10 semesters. Data collected during the administration of the formative Berlin Progress Test Medicine (PTM) were analysed. The PTM comprises 200 multiple-choice questions covering all major medical disciplines and organ systems. A self-report indicator (ie, confidence) and two behavioural indicators of self-monitoring accuracy (ie, response time and the likelihood of changing an initial answer to a correct rather than an incorrect item) were examined for their development over semesters. RESULTS: Analyses of more than 390 000 observations (of approximately 250 students per semester) showed that confidence was higher for correctly than for incorrectly answered items and that 86% of items answered with high confidence were indeed correct. Response time and the likelihood of the initial answer being changed were higher when the initial answer was incorrect than when it was correct. Contrary to expectations, no differences in self-monitoring accuracy were observed across semesters. CONCLUSIONS: Convergent evidence from different measures of self-monitoring suggests that medical students self-monitor their knowledge on a question-by-question basis well, although not perfectly, and to the same degree as has been found in studies outside medicine. Despite large differences in performance, no variations in self-monitoring across semesters (with the exception of the first semester) were observed.

Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination.

BACKGROUND: The concept of EPAs is increasingly applied to assess trainees' workplace performance by means of entrustment ratings. OSCEs assess performance in a simulated setting, and it is unclear whether entrustment ratings can be integrated into these exams. This study explores the introduction of an entrustment rating scale into an existing OSCE. METHODS: A 6-point entrustment scale was added to the standard ratings in an OSCE administered prior to students' final clerkship year in an undergraduate medical programme. Standard OSCE ratings assess clinical and communication skills. Assessors (n = 54) rated students' performance (n = 227) on a diverse set of clinical tasks and evaluated the addition of entrustment scales to OSCEs. Descriptive and inferential statistics were calculated for analyses. RESULTS: Student performance varied across the stations, as reflected in both the standard OSCE ratings and the added entrustment ratings. Students received generally high standard OSCE ratings, whereas entrustment ratings were more widely distributed. All students passed the OSCE, and only a small proportion of students did not reach the expected pass threshold of 60% on the standard ratings in the single stations. The proportion of students who did not reach the expected entrustment level in the respective stations was noticeably higher. Both the clinical and communication skill ratings were related to the entrustment rating in most OSCE stations. A majority of the assessors positively evaluated the addition of entrustment ratings into the OSCE. DISCUSSION: The findings provide an empirical basis to broaden our understanding of the potential use of entrustment ratings in existing OSCEs. They provide directions for future, more specific studies. The ratings might be used for formative feedback on students' readiness for workplace practice.

Shame in Medical Education: A Randomized Study of the Acquisition of Intimate Examination Skills and Its Effect on Subsequent Performance.

THEORY: Although medical students are exposed to a variety of emotions, the impact of emotions on learning has received little attention so far. Shame-provoking intimate examinations are among the most memorable events for students. Their emotions, however, are rarely addressed during training, potentially leading to withdrawal and avoidance and, consequently, performance deficits. However, emotions of negative valance such as shame may be particularly valuable for learning, as they might prompt mental rehearsal. We investigated the effect of shame on learning from the perspective of cognitive load theory. HYPOTHESES: We hypothesized that (a) training modality determines state shame, (b) state shame directly affects the quality of a clinical breast examination as one example of a shame-provoking exam, and (c) students who experience shame during training outperform those who just discuss the emotion during subsequent performance assessments. METHOD: Forty-nine advanced medical students participated in a randomized controlled, single-blinded study. After a basic, low-fidelity breast examination training, students were randomized to further practice either on a high-fidelity mannequin including a discussion of their emotions or by examining a standardized patient's real breasts. Last, all students conducted a breast examination in a simulated doctor's office. Dependent variables were measures of outcome and process quality and of situational shame. RESULTS: Students training with a standardized patient experienced more shame during training (p < .001, d = 2.19), spent more time with the patient (p = .005, d = 0.89), and documented more breast lumps (p = .026, d = 0.65) than those training on a mannequin. Shame interacted with training modality, F(1, 45) = 21.484, p < .001, η2 = 0.323, and differences in performance positively correlated to decline in state shame (r = .335, p = .022). CONCLUSIONS: Students experiencing state shame during training do reenact their training and process germane load-in other words, learn. Furthermore, altering simulation modality offers a possibility for educators to adjust the affective component of training to their objectives.

Development and Evaluation of Two Canine Low-Fidelity Simulation Models.

Two self-made low-fidelity models for simulation of canine intubation and canine female urinary catheterization were developed and evaluated. We used a study design that compares acquired skills of two intervention groups and one control group in a practical examination. Fifty-eight second-year veterinary medicine students received a theoretical introduction to intubation and were randomly divided into three groups. Group I (high-fidelity) was then trained on a commercially available Intubation Training Manikin (item #2006, Veteffects), group II (low-fidelity) was trained on our low-fidelity model, and group III (text) read a text describing intubation of the dog. Forty-seven fifth-year veterinary medicine students followed the same procedure for training urinary catheterization using the commercially available Female Urinary Catheter Training Manikin (Paws 2 Claws), our self-made model, and text. Outcomes were assessed in a practical examination on a cadaver using an Objective Structured Clinical Examination (OSCE) checklist. Considering a value of p≤.05 significant, intervention groups performed significantly better than the text groups. Group I (high-fidelity) and group II (low-fidelity) showed no significant differences (p≤.684, intubation; p≤.901, urinary catheterization). We thereby conclude that low-fidelity models can be as effective as high-fidelity models for clinical skills training.