Response Process Validity Evidence for Video Commentary Assessment in Surgery: A Qualitative Study.

OBJECTIVES: Well-developed mental representations of a task are fundamental to proficient performance. 'Video Commentary' (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees' actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees' response processes in VC. DESIGN: Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions. SETTING: Academic surgical residency program. PARTICIPANTS: Forty-eight surgical residents (PGY-1 to PGY-5). RESULTS: Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions. CONCLUSIONS: Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments.

Video Commentary & Machine Learning: Tell Me What You See, I Tell You Who You Are.

BACKGROUND & OBJECTIVE: Teaching and assessment of complex problem solving are a challenge for medical education. Integrating Machine Learning (ML) into medical education has the potential to revolutionize teaching and assessment of these problem-solving processes. In order to demonstrate possible applications of ML to education, we sought to apply ML in the context of a structured Video Commentary (VC) assessment, using ML to predict residents' training level. SETTING: A secondary analysis of multi-institutional, IRB approved study. Participants had completed the VC assessment consisting of 13 short (20-40 seconds) operative video clips. They were scored in real-time using an extensive checklist by an experienced proctor in the assessment. A ML model was developed using TensorFlow and Keras. The individual scores of the 13 video clips from the VC assessment were used as the inputs for the ML model as well as for regression analysis. PARTICIPANTS: A total of 81 surgical residents of all postgraduate years (PGY) 1-5 from 7 institutions constituted the study sample. RESULTS: Scores from individual VC clips were strongly positively correlated with PGY level (p = 0.001). Some video clips were identified to be strongly correlated with a higher total score on the assessment; others had significant influence when used to predict trainees' PGY levels. Using a supervised machine learning model to predict trainees' PGY resulted in a 40% improvement over traditional statistical analysis. CONCLUSIONS: Performing better in a few select video clips was key to obtaining a higher total score but not necessarily foretelling of a higher PGY level. The use of the total score as a sole measure may fail to detect deeper relationships. Our ML model is a promising tool in gauging learners' levels on an assessment as extensive as VC. The model managed to approximate residents' PGY levels with a lower MAE than using traditional statistics. Further investigations with larger datasets are needed.

LEGO®-based communication assessment in virtual general surgery residency interviews.

Purpose: Effective communication skills are a critical quality and skill that is highly sought after for surgeons which largely impacts patient outcomes. Residency programs design their interview processes to select the best candidates. LEGO®-based activities have been frequently used to enhance communication skills and team building. This study investigates the effectiveness and reliability of a novel LEGO®-based communication assessment in interviews for surgical residencies and the feasibility of implementing it in a virtual setting. Methods: This study conducted a retrospective analysis of a LEGO®-based communication assessment at the program's 2020/2021 residency interviews. Each applicant was assessed on a different model. The total scores were analyzed for consistency among raters and correlated to faculty interviews. Furthermore, the impact of the assessment structure, scoring criteria, and range of models' difficulties on the total scores were explored. Results: A total of 54 categorical and 55 preliminary applicants interviewed on 2 days. The assessment on different models and had no impact on applicants' total scores for either categorical and preliminary groups (p = 0.791 and 0.709, respectively). The communication components of the assessment showed high consistency between the raters. The two applicant groups displayed a statistically significant difference (p = 0.004) in the communication evaluation and model accuracy components. Total scores did not correlate with the faculty interviews of standardized questions in either group. Conclusion: This novel LEGO®-based communication assessment showed high reliability and promising results as a tool to assess communication and problem solving for residency interviews that can be readily implemented in a virtual setting. Supplementary Information: The online version contains supplementary material available at 10.1007/s44186-022-00021-4.

Evaluating Non-operative Robotic Skills in Colorectal Surgical Training.

BACKGROUND: Robotic-assisted surgery has become a common platform for performing colorectal procedures. Educators must determine how best to teach and train residents to use the technology safely. There is a paucity of literature on how non-operative skills are being taught and integrated into colorectal training. Herein we describe the implementation and assessment of a pilot simulation-based Robotic Colorectal Surgery Non-Technical Skills curriculum. MATERIALS AND METHODS: Since 2017 six colon and rectal surgery residents participated in two scenarios: pelvic bleeding and CO2 embolism. The scenarios were administered in a simulated operating room twice during the academic year (fall and spring), and audio-video recorded. In addition to self-assessment, videos were evaluated by faculty utilizing the validated Interpersonal and Cognitive Assessment for Robotic Surgery system. To understand the role of scenario difficulty with respect to perceived cognitive workload and performance residents completed a NASA-Task Load Index assessment form. RESULTS: Between the fall and spring sessions residents significantly improved in intraoperative leadership skills for both the CO2 embolism and bleeding scenarios, and decision-making and situational awareness for the embolism case. Assessment between resident (self) and expert (faculty) did not correlate (P < 0.05) for either scenario during the fall session. A correlation for both scenarios was appreciated following the spring session revealing resident non-technical skills improved over time. Other than for physical demand, NASA-Task Load Index scores were similar for both scenarios. CONCLUSIONS: We were able to successfully develop and implement a pilot Robotic Colorectal Surgery Non-Technical Skills curriculum in a risk-free simulated environment. Non-technical skill curriculums should be considered for both training and assessment in robotic surgery.

Video Commentary: Rapid-Fire Assessment of Surgical Cognitive Skills.

OBJECTIVE: We describe the use of "Video Commentary", an assessment using a set of operative videos, to assess trainees' surgical cognitive skills (operative knowledge, spatial awareness, and surgical insight). DESIGN & SETTING: The Video Commentary assessment has been routinely administered to Postgraduate Year (PGY) 1-5 general surgery residents since 2014 as part of a biannual multistation, OSCE-type exam at Mayo Clinic, Rochester, MN. Video Commentary is a rapid-fire, 6-minute assessment, where trainees watch a series of 20 to 30 second operative video clips and comment on them as they play. Each clip varies in procedure, approach, difficulty, and complexity. The combination of video clips differs according to trainees' PGY level except for a few videos that overlap among PGY groups. The name of the procedure is provided at the beginning of each clip with a countdown timer showing in the corner of the screen. A comprehensive checklist is used to score trainees' performance in real-time. DISCUSSION: Assessment of trainees of different levels and staff surgeons show a positive correlation with the experience level (p = 0.0001). The assessment provides a safe alternative to assess trainees in the operating room and encourages them to become more effective communicators. With the use of technology, large video databases can be created to provide just-in-time tailored feedback to the trainees. CONCLUSIONS: Video Commentary can serve as a time and resource-efficient assessment of trainees' surgical cognitive skills and insight. The use and demand of real-time commentary on operative videos may provide a viable approach to help surgeon educators determine trainees' baseline, progression, and readiness to advance.

Inspiring Young Learners Toward a Career in Healthcare Using Simulation.

STATEMENT: To inspire young learners toward a career in healthcare and provide them opportunities to learn lifesaving skills, a hospital-based simulation center collaborated with a local middle school to develop a 88-minute simulation-based educational activity. The activity consisted of eight 10-minute stations on lifesaving and basic medical skills. One hundred fifty students participated. Evaluation surveys showed students favored stations with extensive hands-on activities and valued the opportunity to interact with health professionals. Students also reported more interest in science careers after the visit. This collaborative effort is time-efficient and low-budget simulation-based learning experience that had an immediate impact on middle school students.

What Are We REALLY Looking for in a Letter of Recommendation?

OBJECTIVES: The letter of recommendation (LOR) is an important component of a residency application. There is great subjectivity in the interpretation of a letter writer's narrative and many residencies have moved toward standardized LORs (sLOR). We aimed to scrutinize the importance afforded to specific content and applicant attributes in an LOR in hopes of decoding this time-honored process. DESIGN: A 35-question survey comprised of nonidentifying general questions, and participant evaluation of applicant characteristics and LOR phrases were administered (cross-sectional design). Evaluations were scored both objectively on a 10-point Likert scale and subjectively using a relative ranking. SETTING: Academic, tertiary care center with a large general surgery residency program (Mayo Clinic, Rochester, MN). PARTICIPANTS: Surgery attendings and general surgery residents (n = 122). RESULTS: Seventy-two attendings and 50 general surgery residents completed the survey. Faculty ranked strong work ethic/hard working (median rank = 1) and inquisitive/hungry learner (median rank = 3) as the top 2 attributes. "We will plan to recruit this candidate" (median rank = 1.5) and "I give my highest recommendation" (median rank = 2) were the top 2 phrases. Residents valued strong work ethic and collaborative/team player as the top 2 applicant attributes. Seventy-three percent of attendings and 82% of residents agreed that LOR allows the applicant pool to be divided into upper and lower halves. Only 17% of faculty and 18% of residents agreed that an LOR allowed candidate stratification into quartiles. CONCLUSIONS: Elaborating the most favorable applicant characteristics and highly regarded content in an LOR will help truly exceptional candidates obtain letters that make them stand out in the eyes of their evaluators. Since LORs are mostly considered to be able to stratify only upper and lower halves of the applicant pool, it is imperative to move toward LORs which portray superior applicant qualities, and can provide more objective evaluation of a candidate.

Outcomes of laparoscopic and open CAPD catheter placement: A single-center experience.

BACKGROUND: Continuous Ambulatory Peritoneal Dialysis (CAPD) catheter placement is typically a straightforward surgical procedure performed on chronically ill patients with end-stage renal disease (ESRD). Post-operative outcomes and reoperative rates vary greatly in the medical literature. We report our experience using both minimally invasive and open techniques in placing CAPD catheters and offer our surgical outcomes. METHODS: This study is an IRB-approved, retrospective review (2005-2018) of all patients undergoing CAPD catheter placement at Mayo Clinic-Rochester. Analysis focused on specific patient outcomes, including early (< 30 days) versus late (≥ 30 days) complication and reoperation rates. RESULTS: A total of 173 patients with ESRD (mean ASA score = 3.1) underwent laparoscopic (n = 22) and open (n = 151) CAPD catheter placement (mean follow-up = 309 days; range: 1-3497 days). The total index operation complication rate was 41%. The total index reoperation rate was 37% and was similar in open and laparoscopic approaches. CAPD catheters malfunctioned in 19 patients (11% of total) and each underwent reoperation. CAPD catheter infections occurred in 30 patients (17% of total), and 24 required reoperation; 6 patients were treated successfully with antibiotics. CAPD catheter migrations occurred in 21 patients (12% of total) and all underwent reoperation. CONCLUSION: Although CAPD catheter placements in patients with ESRD are technically easy to accomplish, the long term outcomes suggest as many as one in three patients will struggle with catheter function or infection. This study has led to changes in our technical CAPD catheter placement procedures, as well as the post-operative patient care algorithm.

Surgery Website as a 24/7 Adjunct to a Surgical Curriculum.

OBJECTIVE: Successfully teaching duty hour restricted trainees demands engaging learning opportunities. Our surgical educational website and its associated assets were assessed to understand how such a resource was being used. DESIGN: Our website was accessible to all Mayo Clinic employees via the internal web network. Website access data from April 2015 through October 2016 were retrospectively collected using Piwik. SETTING: Academic, tertiary care referral center with a large general surgery training program. Mayo Clinic, Rochester, MN. PARTICIPANTS: A total of 257 Mayo Clinic employees used the website. RESULTS: The website had 48,794 views from 6313 visits by 257 users who spent an average of 14 ± 11 minutes on the website. Our website houses 295 videos, 51 interactive modules, 14 educational documents, and 7 flashcard tutorials. The most popular content type was videos, with a total of 30,864 views. The most popular visiting time of the day was between 8 pm and 9 pm with 6358 views (13%), and Thursday was the most popular day with 17,907 views (37%).  A total of 78% of users accessed content beyond the homepage. Average visits peaked in relation to 2 components of our curriculum: a 240% increase one day before our biannual intern simulation assessments, and a 61% increase one day before our weekly conducted Friday simulation sessions. Interns who rotated on the service of the staff surgeon who actively endorses the website had 93% more actions per visit as compared to other users. The highest clicks were on the home banner for our weekly simulation session pre-emptive videos, followed by "groin anatomy," and "TEP hernia repair" videos. CONCLUSIONS: Our website acted as a "just-in-time" accessible portal to reliable surgical information. It supplemented the time sensitive educational needs of our learners by serving as a heavily used adjunct to 3 components of our surgical education curriculum: weekly simulation sessions, biannual assessments, and clinical rotations.