Response Process Validity Evidence for Video Commentary Assessment in Surgery: A Qualitative Study.

OBJECTIVES: Well-developed mental representations of a task are fundamental to proficient performance. 'Video Commentary' (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees' actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees' response processes in VC. DESIGN: Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions. SETTING: Academic surgical residency program. PARTICIPANTS: Forty-eight surgical residents (PGY-1 to PGY-5). RESULTS: Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions. CONCLUSIONS: Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments.

Simulation as Soothsayer: Simulated Surgical Skills MMIs During Residency Interviews are Associated With First Year Residency Performance.

OBJECTIVE: The main consideration during residency recruitment is identifying applicants who will succeed during residency. However, few studies have identified applicant characteristics that are associated with competency development during residency, such as the Accreditation Council for Graduate Medical Education milestones. As mini multiple interviews (MMIs) can be used to assess various competencies, we aimed to determine if simulated surgical skills MMI scores during a general surgery residency interview were associated with Accreditation Council for Graduate Medical Education milestone ratings at the conclusion of intern year. DESIGN: Retrospective cohort study. Interns' Step 1 and 2 clinical knowledge (CK) scores, interview day simulated surgical skills MMI overall score, traditional faculty interview scores, average overall milestone ratings in the spring of residency, and intern American Board of Surgery In-Training Examination (ABSITE) percentile scores were gathered. Two multiple linear regression were performed analyzing the association between Step 1, Step 2 CK, MMI, and traditional faculty interview scores with (1) average overall milestone rating and (2) ABSITE percentile scores, controlling for categorical/preliminary intern classification. SETTING: One academic medical center PARTICIPANTS: General surgery interns matriculating in 2020-2021 RESULTS: Nineteen interns were included. Multiple linear regression revealed that higher overall simulated surgical skills MMI score was associated with higher average milestone ratings (ß = .45, p = 0.03) and higher ABSITE score (ß = .43, p = 0.02) while neither Step 1, Step 2 CK, nor faculty interview scores were significantly associated with average milestone ratings. CONCLUSIONS: Surgical residency programs invest a tremendous amount of effort into training residents, thus metrics for predicting applicants that will succeed are needed. Higher scores on a simulated surgical skills MMIs are associated with higher milestone ratings 1 year into residency and higher intern ABSITE percentiles. These results indicate a noteworthy method, simulated surgical skills MMIs, as an additional metric that may select residents that will have early success in residency.

Video Commentary & Machine Learning: Tell Me What You See, I Tell You Who You Are.

BACKGROUND & OBJECTIVE: Teaching and assessment of complex problem solving are a challenge for medical education. Integrating Machine Learning (ML) into medical education has the potential to revolutionize teaching and assessment of these problem-solving processes. In order to demonstrate possible applications of ML to education, we sought to apply ML in the context of a structured Video Commentary (VC) assessment, using ML to predict residents' training level. SETTING: A secondary analysis of multi-institutional, IRB approved study. Participants had completed the VC assessment consisting of 13 short (20-40 seconds) operative video clips. They were scored in real-time using an extensive checklist by an experienced proctor in the assessment. A ML model was developed using TensorFlow and Keras. The individual scores of the 13 video clips from the VC assessment were used as the inputs for the ML model as well as for regression analysis. PARTICIPANTS: A total of 81 surgical residents of all postgraduate years (PGY) 1-5 from 7 institutions constituted the study sample. RESULTS: Scores from individual VC clips were strongly positively correlated with PGY level (p = 0.001). Some video clips were identified to be strongly correlated with a higher total score on the assessment; others had significant influence when used to predict trainees' PGY levels. Using a supervised machine learning model to predict trainees' PGY resulted in a 40% improvement over traditional statistical analysis. CONCLUSIONS: Performing better in a few select video clips was key to obtaining a higher total score but not necessarily foretelling of a higher PGY level. The use of the total score as a sole measure may fail to detect deeper relationships. Our ML model is a promising tool in gauging learners' levels on an assessment as extensive as VC. The model managed to approximate residents' PGY levels with a lower MAE than using traditional statistics. Further investigations with larger datasets are needed.

LEGO®-based communication assessment in virtual general surgery residency interviews.

Purpose: Effective communication skills are a critical quality and skill that is highly sought after for surgeons which largely impacts patient outcomes. Residency programs design their interview processes to select the best candidates. LEGO®-based activities have been frequently used to enhance communication skills and team building. This study investigates the effectiveness and reliability of a novel LEGO®-based communication assessment in interviews for surgical residencies and the feasibility of implementing it in a virtual setting. Methods: This study conducted a retrospective analysis of a LEGO®-based communication assessment at the program's 2020/2021 residency interviews. Each applicant was assessed on a different model. The total scores were analyzed for consistency among raters and correlated to faculty interviews. Furthermore, the impact of the assessment structure, scoring criteria, and range of models' difficulties on the total scores were explored. Results: A total of 54 categorical and 55 preliminary applicants interviewed on 2 days. The assessment on different models and had no impact on applicants' total scores for either categorical and preliminary groups (p = 0.791 and 0.709, respectively). The communication components of the assessment showed high consistency between the raters. The two applicant groups displayed a statistically significant difference (p = 0.004) in the communication evaluation and model accuracy components. Total scores did not correlate with the faculty interviews of standardized questions in either group. Conclusion: This novel LEGO®-based communication assessment showed high reliability and promising results as a tool to assess communication and problem solving for residency interviews that can be readily implemented in a virtual setting. Supplementary Information: The online version contains supplementary material available at 10.1007/s44186-022-00021-4.

Conducting Virtual Simulated Skills Multiple Mini-Interviews for General Surgery Residency Interviews.

OBJECTIVE: We describe the feasibility of transitioning simulated skills assessments during general surgery interviews from an in-person to virtual format. DESIGN/SETTING: Technical and nontechnical skill multiple mini-interviews (MMIs) were performed virtually and assessed for 109 applicants during virtual general surgery interviews over 2 days at a tertiary academic medical center. RESULTS: We demonstrate the feasibility of virtually assessing general surgery residency applicants' technical and non-technical skills. Using a virtual MMI format during general surgery interviews, we assessed communication, emotional intelligence, anatomical knowledge, interpretation of medical tests, knot tying, and suturing. Four tasks (communication, emotional intelligence , anatomical knowledge, and interpretation of tests) were assessed synchronously by trained general surgery interns. Applicants submitted a recording of themselves performing knot tying and suturing tasks, which were asynchronously assessed after the interview day. Applicants rated the MMI experience highly (4.3/5) via postinterview day survey and the majority of applicants felt that station objectives were met in the virtual format. CONCLUSIONS: We report a successful experience implementing technical and nontechnical virtual MMIs with capacity for 120 applicants during general surgery residency interviews. In the midst of a COVID-19 pandemic, the ability to assess surgical leaners virtually is essential. Virtual skills assessments may provide a more comprehensive picture of applicants and enable residency programs to better assess residents when gathering in person is not feasible.

Evaluating Non-operative Robotic Skills in Colorectal Surgical Training.

BACKGROUND: Robotic-assisted surgery has become a common platform for performing colorectal procedures. Educators must determine how best to teach and train residents to use the technology safely. There is a paucity of literature on how non-operative skills are being taught and integrated into colorectal training. Herein we describe the implementation and assessment of a pilot simulation-based Robotic Colorectal Surgery Non-Technical Skills curriculum. MATERIALS AND METHODS: Since 2017 six colon and rectal surgery residents participated in two scenarios: pelvic bleeding and CO2 embolism. The scenarios were administered in a simulated operating room twice during the academic year (fall and spring), and audio-video recorded. In addition to self-assessment, videos were evaluated by faculty utilizing the validated Interpersonal and Cognitive Assessment for Robotic Surgery system. To understand the role of scenario difficulty with respect to perceived cognitive workload and performance residents completed a NASA-Task Load Index assessment form. RESULTS: Between the fall and spring sessions residents significantly improved in intraoperative leadership skills for both the CO2 embolism and bleeding scenarios, and decision-making and situational awareness for the embolism case. Assessment between resident (self) and expert (faculty) did not correlate (P < 0.05) for either scenario during the fall session. A correlation for both scenarios was appreciated following the spring session revealing resident non-technical skills improved over time. Other than for physical demand, NASA-Task Load Index scores were similar for both scenarios. CONCLUSIONS: We were able to successfully develop and implement a pilot Robotic Colorectal Surgery Non-Technical Skills curriculum in a risk-free simulated environment. Non-technical skill curriculums should be considered for both training and assessment in robotic surgery.

Video Commentary: Rapid-Fire Assessment of Surgical Cognitive Skills.

OBJECTIVE: We describe the use of "Video Commentary", an assessment using a set of operative videos, to assess trainees' surgical cognitive skills (operative knowledge, spatial awareness, and surgical insight). DESIGN & SETTING: The Video Commentary assessment has been routinely administered to Postgraduate Year (PGY) 1-5 general surgery residents since 2014 as part of a biannual multistation, OSCE-type exam at Mayo Clinic, Rochester, MN. Video Commentary is a rapid-fire, 6-minute assessment, where trainees watch a series of 20 to 30 second operative video clips and comment on them as they play. Each clip varies in procedure, approach, difficulty, and complexity. The combination of video clips differs according to trainees' PGY level except for a few videos that overlap among PGY groups. The name of the procedure is provided at the beginning of each clip with a countdown timer showing in the corner of the screen. A comprehensive checklist is used to score trainees' performance in real-time. DISCUSSION: Assessment of trainees of different levels and staff surgeons show a positive correlation with the experience level (p = 0.0001). The assessment provides a safe alternative to assess trainees in the operating room and encourages them to become more effective communicators. With the use of technology, large video databases can be created to provide just-in-time tailored feedback to the trainees. CONCLUSIONS: Video Commentary can serve as a time and resource-efficient assessment of trainees' surgical cognitive skills and insight. The use and demand of real-time commentary on operative videos may provide a viable approach to help surgeon educators determine trainees' baseline, progression, and readiness to advance.

Inspiring Young Learners Toward a Career in Healthcare Using Simulation.

STATEMENT: To inspire young learners toward a career in healthcare and provide them opportunities to learn lifesaving skills, a hospital-based simulation center collaborated with a local middle school to develop a 88-minute simulation-based educational activity. The activity consisted of eight 10-minute stations on lifesaving and basic medical skills. One hundred fifty students participated. Evaluation surveys showed students favored stations with extensive hands-on activities and valued the opportunity to interact with health professionals. Students also reported more interest in science careers after the visit. This collaborative effort is time-efficient and low-budget simulation-based learning experience that had an immediate impact on middle school students.

What Are We REALLY Looking for in a Letter of Recommendation?

OBJECTIVES: The letter of recommendation (LOR) is an important component of a residency application. There is great subjectivity in the interpretation of a letter writer's narrative and many residencies have moved toward standardized LORs (sLOR). We aimed to scrutinize the importance afforded to specific content and applicant attributes in an LOR in hopes of decoding this time-honored process. DESIGN: A 35-question survey comprised of nonidentifying general questions, and participant evaluation of applicant characteristics and LOR phrases were administered (cross-sectional design). Evaluations were scored both objectively on a 10-point Likert scale and subjectively using a relative ranking. SETTING: Academic, tertiary care center with a large general surgery residency program (Mayo Clinic, Rochester, MN). PARTICIPANTS: Surgery attendings and general surgery residents (n = 122). RESULTS: Seventy-two attendings and 50 general surgery residents completed the survey. Faculty ranked strong work ethic/hard working (median rank = 1) and inquisitive/hungry learner (median rank = 3) as the top 2 attributes. "We will plan to recruit this candidate" (median rank = 1.5) and "I give my highest recommendation" (median rank = 2) were the top 2 phrases. Residents valued strong work ethic and collaborative/team player as the top 2 applicant attributes. Seventy-three percent of attendings and 82% of residents agreed that LOR allows the applicant pool to be divided into upper and lower halves. Only 17% of faculty and 18% of residents agreed that an LOR allowed candidate stratification into quartiles. CONCLUSIONS: Elaborating the most favorable applicant characteristics and highly regarded content in an LOR will help truly exceptional candidates obtain letters that make them stand out in the eyes of their evaluators. Since LORs are mostly considered to be able to stratify only upper and lower halves of the applicant pool, it is imperative to move toward LORs which portray superior applicant qualities, and can provide more objective evaluation of a candidate.

Low-cost materials yield high resolution assessment of anatomic knowledge in surgical residents.

INTRODUCTION: Knowledge of anatomy is essential for surgeons. We sought to determine whether it is possible to effectively assess and differentiate the anatomic knowledge of general surgery residents, using pieces of fabric and yarn. METHODS: Postgraduate years 2, 3, and 4 general surgery residents were assessed during a simulation-based assessment known as the Surgical X-Games. Residents were allowed 3 minutes to assemble the anatomic structures of the right upper quadrant region and 2 minutes to create the right lower quadrant of the abdomen, using colored felt and yarn. One point (each) was given for naming and placing the structures in the correct position. A checklist was used to assess trainees with a maximum combined score of 150 points. RESULTS: A total of 34 residents (postgraduate year 2 = 16, postgraduate year 3 = 8, postgraduate year 4 = 10) participated in the 2017 fall Surgical X-Games and 31 residents (postgraduate year 2 = 13, postgraduate year 3 = 9, postgraduate year 4 = 9) participated in the spring Surgical X-Games. Total scores increased respective to the level of clinical training in both the fall Surgical X-Games (postgraduate year 2 = 77, postgraduate year 3 = 84, postgraduate year 4 = 93, P = .04) and the spring Surgical X-Games (postgraduate year 2 = 94, postgraduate year 3 = 101, postgraduate year 4 = 109). We observed significant improvement in the right upper quadrant, right lower quadrant, and total scores from the fall to the spring postgraduate year (P < .001). CONCLUSION: Surgical residents showed an increase in mean anatomical scores from postgraduate years 2 to postgraduate years 3 to postgraduate years 4, using low-fidelity models. This inexpensive, 5-minute test based on a simple checklist may offer surgical educators insight in to residents' anatomic knowledge and potential readiness for clinical rotations.

Surgical interns in 2018: Objective assessment suggests they are better but still lack critical knowledge and skill.

BACKGROUND: The transition from fourth-year medical student to surgical intern is difficult. A lack of repetitions, experience, and knowledge is problematic. We report our experience using simulation-based technical and nontechnical skills to assess the competency of surgical interns in July and January of their intern year. DESIGN: As part of a larger assessment effort, our general surgery interns (2010-2016) have been tested on performing an emergent cricothyrotomy, interpreting 2 arterial blood gases, and reading 3 chest x-rays in fewer than 7 minutes. We retrospectively analyzed general surgery interns' performance on these 3 tests (total score = 20). RESULTS: A total of 210 interns completed both July and January (identical) assessments. Overall mean scores improved from July (12.62 ± 3.44) to January (16.5 ± 2.46; P < .05). During the study period general surgery interns' mean baseline scores improved in both July (P < .05) and in January (P < .05). Although most individual general surgery interns did improve their total scores (92% improved, 3% same, 5% worse) between July and January (P < .05), in January 40% could not perform an emergent cricothyrotomy swiftly, and 6% missed a tension pneumothorax on chest x-ray. CONCLUSION: Our data suggest that surgical interns start residency training with low levels of skill and comprehension with emergent cricothyrotomy, arterial blood gas, and chest x-ray. They improve with 6 months of clinical and simulation training. Encouragingly, overall scores for both July and January assessments have improved during the study period. Given that some interns still struggle in January to perform these three tasks, we believe that 2018 interns are better, but still potentially lack critical knowledge and skill.

Outcomes of laparoscopic and open CAPD catheter placement: A single-center experience.

BACKGROUND: Continuous Ambulatory Peritoneal Dialysis (CAPD) catheter placement is typically a straightforward surgical procedure performed on chronically ill patients with end-stage renal disease (ESRD). Post-operative outcomes and reoperative rates vary greatly in the medical literature. We report our experience using both minimally invasive and open techniques in placing CAPD catheters and offer our surgical outcomes. METHODS: This study is an IRB-approved, retrospective review (2005-2018) of all patients undergoing CAPD catheter placement at Mayo Clinic-Rochester. Analysis focused on specific patient outcomes, including early (< 30 days) versus late (≥ 30 days) complication and reoperation rates. RESULTS: A total of 173 patients with ESRD (mean ASA score = 3.1) underwent laparoscopic (n = 22) and open (n = 151) CAPD catheter placement (mean follow-up = 309 days; range: 1-3497 days). The total index operation complication rate was 41%. The total index reoperation rate was 37% and was similar in open and laparoscopic approaches. CAPD catheters malfunctioned in 19 patients (11% of total) and each underwent reoperation. CAPD catheter infections occurred in 30 patients (17% of total), and 24 required reoperation; 6 patients were treated successfully with antibiotics. CAPD catheter migrations occurred in 21 patients (12% of total) and all underwent reoperation. CONCLUSION: Although CAPD catheter placements in patients with ESRD are technically easy to accomplish, the long term outcomes suggest as many as one in three patients will struggle with catheter function or infection. This study has led to changes in our technical CAPD catheter placement procedures, as well as the post-operative patient care algorithm.

Surgery Website as a 24/7 Adjunct to a Surgical Curriculum.

OBJECTIVE: Successfully teaching duty hour restricted trainees demands engaging learning opportunities. Our surgical educational website and its associated assets were assessed to understand how such a resource was being used. DESIGN: Our website was accessible to all Mayo Clinic employees via the internal web network. Website access data from April 2015 through October 2016 were retrospectively collected using Piwik. SETTING: Academic, tertiary care referral center with a large general surgery training program. Mayo Clinic, Rochester, MN. PARTICIPANTS: A total of 257 Mayo Clinic employees used the website. RESULTS: The website had 48,794 views from 6313 visits by 257 users who spent an average of 14 ± 11 minutes on the website. Our website houses 295 videos, 51 interactive modules, 14 educational documents, and 7 flashcard tutorials. The most popular content type was videos, with a total of 30,864 views. The most popular visiting time of the day was between 8 pm and 9 pm with 6358 views (13%), and Thursday was the most popular day with 17,907 views (37%).  A total of 78% of users accessed content beyond the homepage. Average visits peaked in relation to 2 components of our curriculum: a 240% increase one day before our biannual intern simulation assessments, and a 61% increase one day before our weekly conducted Friday simulation sessions. Interns who rotated on the service of the staff surgeon who actively endorses the website had 93% more actions per visit as compared to other users. The highest clicks were on the home banner for our weekly simulation session pre-emptive videos, followed by "groin anatomy," and "TEP hernia repair" videos. CONCLUSIONS: Our website acted as a "just-in-time" accessible portal to reliable surgical information. It supplemented the time sensitive educational needs of our learners by serving as a heavily used adjunct to 3 components of our surgical education curriculum: weekly simulation sessions, biannual assessments, and clinical rotations.

Accurate surgical skills evaluation: Does it mandate raters have a medical background?

BACKGROUND: Surgeons rarely have time to assess/rate trainee performance. From a 10 year-experience of implementing OSCE style assessments, we hypothesize that the accurate scoring of interns in selected tests is not affected by the rater's medical background. METHODS: A prospective collection of quantitative scoring data by both medical school graduates and college students was compared. Each rater underwent training and then watched three videos of actors performing in each of two OSCE stations and individually scored them. RESULTS: Twelve college students and 16 medical graduates participated. There was no difference in the mean scores between rating groups for chest tube insertion (Video 1: 1.7 vs. 2.0; Video 2: 2.9 vs 3.1; Video 3: 6.1 vs 6.1; p = 0.8) and cricothyrotomy (Video 1: 4.0 vs 4.5; Video 2: 4.8 vs 5.1; Video 3: 9.2 vs 9.1; p = 0.7). CONCLUSION: Accurate scoring of surgical performance does not mandate a medical background. Given the limited availability of attending surgeons for assessments, use of validated, simple checklists can help raters with minimal medical experience perform assessments proficiently.