Objective Assessment of the Entrustable Professional Activity Handover in Undergraduate and Postgraduate Surgical Learners.

OBJECTIVE: This study used a virtual patient simulation (VPS) to quantifiably and objectively assess undergraduate (UG) to postgraduate (PG) medical learners' acquisition of the entrustable professional activity (EPA) "handover," focusing particularly on the transition to residency. This EPA is critical because it is part of a core competency for UG and PG training in both the United States and Canada, and is essential for patient safety and comprehensive professional communication. DESIGN: Data were collected from 3 separate groups of participants: 2 UG cohorts from an earlier study, as well as a PG cohort at the beginning of residency. All participants completed the same trauma VPS, which required a free text summary statement that was used as a surrogate for an oral handover. These were collected and scored independently, using previously developed validated rubrics, one procedural and the second semantic. SETTING: All study participants were from one site. The VPS case was completed online. PARTICIPANTS: Two different UG groups, one designated junior (N = 52), was studied at the beginning of their clerkship year, a second group, designated senior (N = 30), was studied at the end of their clerkship year. These groups were compared to a third group of PG learners (N = 31) during the initial 2 weeks of their residency. Informed consent was obtained from all participants. RESULTS: A procedural rubric assessed learners' cognitive knowledge of trauma care-management. A semantic rubric assessed their use of the professional language necessary for a safe and succinct clinical handover communication. An Analysis of Variance comparing scores on the procedural rubric was highly significant with Tukey LSD tests indicating that all 3 groups were significantly different. Students increased their scores on the procedural rubric at each stage of their training. A parallel Analysis of Variance comparing students' scores on the semantic rubric revealed no significant increase in scores, indicating that students did not improve in their capacity to communicate professionally as they progressed through their training. CONCLUSIONS: Taken together, these results demonstrate that training was successful in teaching cognitive-based procedures, but not effective in teaching professional communication, which is critical to the EPA handover. Greater emphasis needs to be placed on ensuring the acquisition of professional communication skills throughout the continuum of UG and PG clinical activities. Faculty development should serve as a support to assist medical educators to address this requirement. These results also demonstrate that VPS with associated objective and validated rubrics can be used as an assessment methodology to quantifiably measure learner performance with respect to the EPA handover. A similar strategy should be considered across the UG and PG continuum for other EPAs and could form the nexus for further research.

Development and Validation of a Tool to Evaluate the Evolution of Clinical Reasoning in Trauma Using Virtual Patients.

CONTEXT: Undergraduate medical students at a large academic trauma center are required to manage a series of online virtual trauma patients as a mandatory exercise during their surgical rotation. PURPOSE: Clinical reasoning during undergraduate medical education can be difficult to assess. The purpose of the study was to determine whether we could use components of the students' virtual patient management to measure changes in their clinical reasoning over the course of the clerkship year. In order to accomplish this, we decided to determine if the use of scoring rubrics could change the traditional subjective assessment to a more objective evaluation. BASIC PROCEDURES: Two groups of students, one at the beginning of clerkship (Juniors) and one at the end of clerkship (Seniors), were chosen. Each group was given the same virtual patient case, a clinical scenario based on the Advanced Trauma Life Support (ATLS) Primary Trauma Survey, which had to be completed during their trauma rotation. The learner was required to make several key patient management choices based on their clinical reasoning, which would take them along different routes through the case. At the end of the case they had to create a summary report akin to sign-off. These summaries were graded independently by two domain "Experts" using a traditional subjective surgical approach to assessment and by two "Non-Experts" using two internally validated scoring rubrics. One rubric assessed procedural or domain knowledge (Procedural Rubric), while the other rubric highlighted semantic qualifiers (Semantic Rubric). Each of the rubrics was designed to reflect established components of clinical reasoning. Student's t-tests were used to compare the rubric scores for the two groups and Cohen's d was used to determine effect size. Kendall's τ was used to compare the difference between the two groups based on the "Expert's" subjective assessment. Inter-rater reliability (IRR) was determined using Cronbach's alpha. MAIN FINDINGS: The Seniors did better than the Juniors with respect to "Procedural" issues but not for "Semantic" issues using the rubrics as assessed by the "Non-Experts". The average Procedural rubric score for the Senior group was 59% ± 13% while for the junior group, it was 51% ± 12% (t(80)= 2.715; p = 0.008; Cohen's d = 1.53). The average Semantic rubric score for the Senior group was 31% ± 15% while for the Junior group, it was 28% ± 14% (t(80) = 1.010; p = .316, ns). There was no statistical difference in the marks given to the Senior versus Junior groups by the "Experts" (Kendall's τ = 0.182, p = 0.07). The IRR between the "Non-Experts" using the rubrics was higher than the IRR of the "Experts" using the traditional surgical approach to assessment. The Cronbach's alpha for the Procedural and Semantic rubrics was 0.94 and 0.97, respectively, indicating very high IRR. The correlation between the Procedural rubric scores and "Experts" assessment was approximately r = 0.78, and that between the Semantic rubric and the "Experts" assessment was roughly r = 0.66, indicating high concurrent validity for the Procedural rubric and moderately high validity for the Semantic rubric. PRINCIPLE CONCLUSION: Clinical reasoning, as measured by some of its "procedural" features, improves over the course of the clerkship year. Rubrics can be created to objectively assess the summary statement of an online interactive trauma VP for "procedural" issues but not for "semantic" issues. Using IRR as a measure, the quality of assessment is improved using the rubrics. The "Procedural" rubric appears to measure changes in clinical reasoning over the course of 3rd-year undergraduate clinical studies.

Sim one, do one, teach one: considerations in designing training curricula for surgical simulation.

Although there is considerable interest in the use of simulation for the acquisition of fundamental surgical skills through goal-directed practice in a safe environment, there is little evidence guiding educators on how best to implement simulation within surgical skills curricula. This article reviews the application of the expert performance model in surgery and the role of simulation in surgical skills acquisition. The focus is on implementation of deliberate practice, highlighting the principles of part-task training, proficiency-based training and overtraining. With resident and educator time at a premium, the identification of an optimally effective and efficient training strategy has significant implications for how surgical skills training is incorporated into residency programs, which is critical in today's environment.