Reducing trainee mistakes. Better performance with changing to a high-fidelity simulation system?

BACKGROUND: Postpartum hemorrhage is a significant cause of both maternal morbidity and mortality worldwide and is increasing in incidence. This study aimed to assess improvement and identify shortcomings in trainee performance in different simulation systems in the management of postpartum hemorrhage. OBJECTIVE: To perform a pilot study evaluating and comparing high- and low-fidelity simulation models, assessing improvement in repeated performance with high-fidelity mode and identifying mistakes made assessed using Objective Structured Assessment of Technical Skills and thereby exploring what aspects of emergency management of postpartum hemorrhage should be prioritized in teaching settings and assessing what simulation setup is most effective in achieving competence. STUDY DESIGN: This was a prospective randomized, single-blinded, single-institution trial in a population of 17 junior obstetrical trainees at the Charité University Hospital Obstetric Simulation Center in Berlin. Trainees were randomized into 2 groups, with either initial low-fidelity simulation or high-fidelity simulation, followed by repeated assessment of performance, using the high-fidelity model simulation system. Individual simulation sessions were video-recorded and transcribed, and the timing of interventions was documented. Strandardized Objective Structured Assessment of Technical Skills forms were used as a checklist for performance. RESULTS: There was a statistically significant general improvement in performance (P=.02; 24.7-27.2 of 31.0 points; average of 8.7%) in the second cycle of simulation assessment and a statistically significant training effect (P=.043; 24.4-28.4 of 31.0 points; average of 12.9%) in the group that underwent repeat simulation assessment from the initial low-fidelity system to the high-fidelity system compared with the group using the same high-fidelity setup (P=.276; 25.0-25.8 of 31.0; average of 2.4%). CONCLUSION: There was an improvement in the performance when trainees underwent a repeated cycle of simulation assessment changing from a low-fidelity system to a high-fidelity system. Simulation assessment can identify mistakes and learning gaps that are important for obstetrical trainees. This study found that trainees make the same mistakes, regardless of which simulation model was initially used.

Origami Box Folding Exercise: A laparoscopic box simulation training and assessment method in facilitating laparoscopic psychomotor skills with minimal preparation.

OBJECTIVE: Laparoscopic box simulation training is widely recognized as an assessment tool to facilitate psychomotor skills especially for novice surgeons. However, current commercialized training modules including pegs, gauze, clips, pins etc. are generally costly and relatively inaccessible. We introduce a simple and pioneer surgical training drill, the Origami Box Folding Exercise (OBFE), based on the validated evaluating system of objective structured assessment of technical skills (OSATS) constructed with the scoring system of procedure-specific checklist (PSC) and global rating scale (GRS). MATERIALS AND METHODS: Face and content validation of the OBFE and OSATS are evaluated by five endoscopic experts from two medical centers in Taiwan. This is a prospective observational study analyzing the pre-test/post-test result of OBFE from 37 participants in two individual workshops as training and evaluating method for laparoscopic psychomotor skills. Both the pre and post tests are video recorded with a time limit of 5 min graded by two independent evaluators based on the OSATS scoring system. RESULTS: The reliability of PSC, GRS, and intergroup value between PSC and GRS were 0.923, 0.926 and 0.933, respectively. Inter-rater reliability of PSC, GRS, and both were 0.985, 0.932 and 0.977, respectively. Construct validity of PSC and GRS were statistically significant, with p-value 0.006 and 0.001, respectively. CONCLUSION: OBFE enhances laparoscopic psychomotor skills with requirement of a single piece of paper. The associated OSATS tool for a 5-min OBFE test was validated. OBFE training is an efficient training and assessment system to promote psychomotor skills in laparoscopic box simulation drill which requires simple and economical preparation.

Effects of a structural simulation curriculum for the repair of obstetrical anal sphincter injuries.

OBJECTIVE: To assess the short-term and skill-retention effects of a structural simulation curriculum for the repair of obstetrical anal sphincter injuries (OASIS). METHODS: This prepost experimental study recruited 31 obstetrics and gynecology residents from Peking Union Medical College Hospital. OASIS repair skill evaluations (at all timepoints) comprised a task-specific checklist (TSC), global rating scale (GRS), and pass/fail grade. A beef-tongue model was adopted in the simulation curriculum. After baseline testing, residents completed a structured 1-h workshop. Two weeks later, a second test was performed, followed by a one-to-one teaching workshop, with the proper completion of a checklist as the end-point. Three months later, residents completed a third test. RESULTS: The overall pass rate at baseline was 16.1%, with higher GRS scores (p = 0.035) in senior residents. Additionally, previewing the procedure before class and a history of OASIS repair observation were associated with higher TSC and GRS scores. Significant skill improvement was observed in the second and third tests, with pass rates of 96.8% and 93.5% respectively. Both TSC and GRS scores were improved compared with baseline (p < 0.001). CONCLUSION: A structural simulation curriculum integrating deliberate practice under supervision improves OASIS repair competence and achieves satisfactory skill retention.

Comparison of teaching effect of hands-on instruction versus video demonstration in forceps delivery for training residents / 中华全科医师杂志

Objective:To evaluate the effectiveness of hands-on training and video demonstration in training of forceps delivery for residents.Methods:Forty nine residents who were rotating in the obstetrics department of Peking Union Medical College Hospital from 2019 to 2021 were enrolled. The residents were randomly divided into two groups: the instructor group ( n=24) was taught by hands-on training of forceps delivery and the video group ( n=25) was instructed by watching video demonstration. All the trainees completed the self-confidence questionnaire survey, and were evaluated by written tests and objective structured assessment of technical skills scoring system. Results:The scores of self-confidence in each item after the simulation training were higher than those before training in both groups; and there were no significant differences between two groups in the increment of scores(mastering knowledge: 1.54±0.98 vs. 1.40±0.71, U=266.68, P=0.480;mastering operation skills: 1.42±0.93 vs.1.80±0.87, U=233.47, P=0.161; mastering forceps structure: 1.63±1.10 vs. 1.88±0.93, U=261.63, P=0.416; confidence in independent operation: 1.13±0.90 vs. 1.00±1.08, U=287.74, P=0.799; evaluation of simulation training: 0.21±0.51 vs. 0.16±0.55, U=288.27, P=0.776). In the written tests, the scores of the instructor group were significantly higher than those of the video group (83.00±7.18 vs.70.56±10.37; t=4.86, P<0.001). In the practical operation, the instructor group significantly outperformed the video group in items of “right blade placement” (0.71±0.46 vs. 0.20±0.41, U=147.54, P<0.001), “objective total score” (6.17±1.46 vs. 4.72±1.65, U=155.49, P=0.003) and “correct traction” (0.85±0.31 vs. 0.56±0.51, U=213.86, P=0.036). Conclusion:Training delivered via hands-on instruction and demonstration was generally more effective than that delivered via video, although both groups show a increased self-confidence in learning and performing forceps delivery.

Impact of Rater Training on Residents Technical Skill Assessments: A Randomized Trial.

OBJECTIVE: The ACS/APDS Resident Skills Curriculum's Objective Structured Assessment of Technical Skills (OSATS) consists of task-specific checklists and a global rating scale (GRS) completed by raters. Prior work demonstrated a need for rater training. This study evaluates the impact of a rater-training curriculum on scoring discrimination, consistency, and validity for handsewn bowel anastomosis (HBA) and vascular anastomosis (VA). DESIGN/ METHODS: A rater training video model was developed, which included a GRS orientation and anchoring performances representing the range of potential scores. Faculty raters were randomized to rater training or no rater training and were asked to score videos of resident HBA/VA. Consensus scores were assigned to each video using a modified Delphi process (Gold Score). Trained and untrained scores were analyzed for discrimination and score spread and compared to the Gold Score for relative agreement. RESULTS: Eight general and eight vascular surgery faculty were randomized to score 24 HBA/VA videos. Rater training increased rater discrimination and decreased rating scale shrinkage for both VA (mean trained score: 2.83, variance 1.88; mean untrained score: 3.1, variance 1.14, p = 0.007) and HBA (mean trained score: 3.52, variance 1.44; mean untrained score: 3.42, variance 0.96, p = 0.033). On validity analyses, a comparison between each rater group vs Gold Score revealed a moderate training impact for VA, trained κ=0.65 vs untrained κ=0.57 and no impact for HBA, R1 κ = 0.71 vs R2 κ = 0.73. CONCLUSION: A rater-training curriculum improved raters' ability to differentiate performance levels and use a wider range of the scoring scale. However, despite rater training, there was persistent disagreement between faculty GRS scores with no groups reaching the agreement threshold for formative assessment. If technical skill exams are incorporated into high stakes assessments, consensus ratings via a standard setting process are likely a more valid option than individual faculty ratings.

Surgical skill and oncological outcome of laparoscopic radical hysterectomy: JGOG1081s-A1, an ancillary analysis of the Japanese Gynecologic Oncology Group Study JGOG1081.

OBJECTIVES: We investigated whether surgical skill and procedure were related to oncological outcomes in cervical cancer patients who underwent Laparoscopic Radical Hysterectomy (LRH). METHODS: We previously assessed data of LRH from 251 patients with FIGO stage (2009) IA2, IB1and IIA1 cervical cancer collected for JGOG 1081s study. 1) The JGOG 1081s cohort study was re-examined to refine the surgical details and extend the follow-up period as chart review. 2) Unedited videos for recurrent cases and matched non-recurrent control cases were newly compared by experts for various surgical skills and surgical procedures using the modified Objective Structured Assessment of Technical Skills (OSATS) tool, without awareness of the recurrence status as video review. RESULTS: After a median follow-up of 46 months, tumors had recurred in 31 of the 251 patients. The five-year Recurrence-Free Survival rate was 86.9% (81.8-90.6) and five-year Overall Survival rate was 93.7% (87.5-96.8). Multivariate analysis from chart reviews found that an experience with LRH of less than 20 cases per institution was an independent prognostic factor for recurrence (Hazard Ratio (HR) 2.49, 95%CI 1.12-5.53, p = 0.025). For the surgical video review, we compared 23 videos of recurrent cases with 23 background-matched non-recurrent controls. Lower modified OSATS scores from the video review were consistently trended to have a higher risk of recurrence. CONCLUSIONS: Our new study has found that LRH surgical experience and skill trended to have better oncological outcomes.

Gynecologic surgical skill acquisition through simulation with outcomes at the time of surgery: a systematic review and meta-analysis.

OBJECTIVE: To evaluate the effect of simulation training vs traditional hands-on surgical instruction on learner operative skills and patient outcomes in gynecologic surgeries. DATA SOURCES: PubMed, Embase, ClinicalTrials.gov, and the Cochrane Central Register of Controlled Trials from inception to January 12, 2021. STUDY ELIGIBILITY CRITERIA: Randomized controlled trials, prospective comparative studies, and prospective single-group studies with pre- and posttraining assessments that reported surgical simulation-based training before gynecologic surgery were included. METHODS: Reviewers independently identified the studies, obtained data, and assessed the study quality. The results were analyzed according to the type of gynecologic surgery, simulation, comparator, and outcome data, including clinical and patient-related outcomes. The maximum likelihood random effects model meta-analyses of the odds ratios and standardized mean differences were calculated with estimated 95% confidence intervals. RESULTS: Twenty studies, including 13 randomized controlled trials, 1 randomized crossover trial, 5 nonrandomized comparative studies, and 1 prepost study were identified. Most of the included studies (14/21, 67%) were on laparoscopic simulators and had a moderate quality of evidence. Meta-analysis showed that compared with traditional surgical teaching, high- and low-fidelity simulators improved surgical technical skills in the operating room as measured by global rating scales, and high-fidelity simulators decreased the operative time. Moderate quality evidence was found favoring warm-up exercises before laparoscopic surgery. There was insufficient evidence to conduct a meta-analysis for other gynecologic procedures. CONCLUSION: Current evidence supports incorporating simulation-based training for a variety of gynecologic surgeries to increase technical skills in the operating room, but data on patient-related outcomes are lacking.

Comparison of a practice-based versus theory-based training program for conducting vacuum-assisted deliveries: a randomized-controlled trial.

PURPOSE: Vacuum-assisted deliveries (VAD) are complex procedures that require training and experience to be performed proficiently. We aimed to evaluate if a more resource intensive practice-based training program for conducting VAD is more efficient compared to a purely theory-based training program, with respect to immediate training effects and persistence of skills 4-8 weeks after the initial training. METHODS: In this randomized-controlled study conducted in maternity staff, participants performed a simulated low-cavity non-rotational vacuum delivery before (baseline test) and immediately after the training (first post-training test) as well as 4-8 weeks thereafter (second post-training test). The study's primary endpoint was to compare training effectiveness between the two study groups using a validated objective structured assessment of technical skills (OSATS) rating scale. RESULTS: Sixty-two participants were randomized to either the theory-based group (n = 31) or the practice-based group (n = 31). Total global and specific OSATS scores, as well as distance of cup application to the flexion point improved significantly from baseline test to the first post-training test in both groups (pall < 0.007). Skill deterioration after 4-8 weeks was only found in the theory-based group, whereas skills remained stable in the practice-based group. CONCLUSION: A practice-based training program for conducting VAD results in comparable immediate improvement of skills compared to a theory-based training program, but the retention of skills 4-8 weeks after training is superior in a practice-based program. Future studies need to evaluate, whether VAD simulation training improves maternal and neonatal outcome after VAD.

Remote simulator training of coronary artery bypass grafting during the coronavirus disease 2019 pandemic.

OBJECTIVES: The coronavirus disease 2019 pandemic presents in-person exposure risk during surgical education. We aimed to validate the feasibility of fully remote faculty-supervised surgical training sessions focused on coronary artery bypass grafting using a synthetic simulator and online videochat software. METHODS: This observational study organized 24 sessions of 2-hour remote training. Each session involved 3 trainees, 1 faculty member, and 1 host. A total of 70 trainees and 24 faculty members were enrolled. The participants joined the remote sessions via online videochat and performed focused training in coronary artery anastomosis using a commercially available simulator. A survey was conducted to validate the feasibility of the remote sessions. Performance improvement of the trainees who repeatedly participated (n = 13) were analyzed comparing initial and final scores of various performance indicators. RESULTS: All trainees and faculty members were satisfied with the efficacy of the remote session. Additionally, most trainees (79%) and faculty members (95%) agreed that the remote training sessions were equivalent to conventional onsite training seminars. A significant improvement between initial and last sessions was observed in the scoring components of near side (3.4 ± 1.0 vs 4.1 ± 0.9; P = .02), far side (3.3 ± 0.8 vs 3.9 ± 0.8; P = .03), external appearance (3.5 ± 0.8 vs 4.2 ± 0.7; P = .01), and internal appearance (2.8 ± 0.9 vs 4.0 ± 0.9; P = .004). CONCLUSIONS: Faculty-supervised remote surgical training sessions were executed with satisfactory results. This methodology may have important implications for surgical education during the coronavirus disease 2019 pandemic.

Working Towards Competence: A Novel Application of Borderline Regression to a Task-Specific Checklist for Technical Skills in Novices.

OBJECTIVE: Demonstrated competence through frequent assessment is an expected goal for progressive development in competency-based medical education curricula. The Objective Structured Assessment of Technical Skill (OSATS) is considered a valid method of formative assessment, but in few instances have standards been set for determining competence. The present study used borderline regression methods to examine standard setting of performance on a complex technical task with novices assessed using an OSATS checklist. METHODS: This was a single institution prospective single arm experimental design study. Participants were 58 non-medical undergraduate students with no previous surgical experience, who observed a computer-based training module on end-to-side vascular anastomosis. Subsequently, participants were provided two 20-minute training sessions, two weeks apart where they received expert feedback whilst performing the task on a low-fidelity model. After each training session, participants completed the task unaided. Sessions were recorded and assessed using an OSATS checklist retrospectively by experts. RESULTS: Paired t-test analyses indicate that for both the checklist total score (t(52) = 8.05, p < 0.001) and the global rating score (t(53) = 8.15, p < 0.001), individuals performed significantly better in Phase 2. Borderline regression analyses indicated that in Phase 1 (R2 = .60) and Phase 2 (R2 = .75), the OSATS checklist could adequately capture variation in performance in novices. Further, the checklist could reliably classify novices at three of the five global rating performance levels. Pass rates determined by regression equations improved from Phase 1 to Phase 2 on all global rating levels. CONCLUSIONS: With the increasing focus on competency-based medical education, it is imperative that training programs have the capacity to accurately assess outcomes and set minimum performance standards. Borderline regression methods can accurately differentiate novice learners of varying performance levels before and after training on a complex technical skill task using an OSATS checklist.

Competency-Based Assessment Tool for Pediatric Esophagoscopy: International Modified Delphi Consensus.

OBJECTIVES/HYPOTHESIS: Create a competency-based assessment tool for pediatric esophagoscopy with foreign body removal. STUDY DESIGN: Blinded modified Delphi consensus process. SETTING: Tertiary care center. METHODS: A list of 25 potential items was sent via the Research Electronic Data Capture database to 66 expert surgeons who perform pediatric esophagoscopy. In the first round, items were rated as "keep" or "remove" and comments were incorporated. In the second round, experts rated the importance of each item on a seven-point Likert scale. Consensus was determined with a goal of 7 to 25 final items. RESULTS: The response rate was 38/64 (59.4%) in the first round and returned questionnaires were 100% complete. Experts wanted to "keep" all items and 172 comments were incorporated. Twenty-four task-specific and 7 previously-validated global rating items were distributed in the second round, and the response rate was 53/64 (82.8%) with questionnaires returned 97.5% complete. Of the task-specific items, 9 reached consensus, 7 were near consensus, and 8 did not achieve consensus. For global rating items that were previously validated, 6 reached consensus and 1 was near consensus. CONCLUSIONS: It is possible to reach consensus about the important steps involved in rigid esophagoscopy with foreign body removal using a modified Delphi consensus technique. These items can now be considered when evaluating trainees during this procedure. This tool may allow trainees to focus on important steps of the procedure and help training programs standardize how trainees are evaluated. LEVEL OF EVIDENCE: 5. Laryngoscope, 131:1168-1174, 2021.

The Oral and Maxillofacial Objective Structured Assessment of Technical Skills (OMOSATS) examination: a pilot study.

The aim of this study was to develop a simulated technical skills examination specific to oral and maxillofacial surgery, assessing the validity of the test and its psychometric properties and obtaining feedback on the examination from both resident trainees and expert evaluators. The Oral and Maxillofacial Objective Structured Assessment of Technical Skills (OMOSATS) examination is a simulated procedure-based eight-station technical skills examination. Fourteen oral and maxillofacial surgery (OMFS) residents of various levels participated in 12-minute surgical tasks and were evaluated by blinded expert OMFS surgeons. The tasks included suturing, tracheostomy, arthroscopy, dental implant placement, plate bending and lag screw application, microvascular anastomosis, and skin grafting. These tasks reflect a broad range of technical milestones necessary for the practice of OMFS. Tasks were evaluated using a validated global rating scale and task-specific checklists. The results of the pilot study indicated strong face and construct validity for seven of the eight OMOSATS stations in discerning the level of training based on technical skill demonstrated at each of the eight surgical stations. Additionally, through one-way ANOVA, P-values were observed to confirm the level of discernment. The OMOSATS examination is a novel way to assess the technical skills of OMFS residents. This type of performance-based assessment has been used in other surgical domains for many years. There is strong evidence to support the use of technical skills examinations to help evaluate, teach, and provide surgeon educators with training outcomes for our OMFS trainees.

Use of high-fidelity 3-dimensional-printed models for training novice residents in basic nasal endoscopic skills.

BACKGROUND: The use of 3-dimensional (3D)-printed models is promising in nasal endoscopic technique training. Here, we aimed to develop postsurgical simulants for use in conjunction with 3D-printed nasal models and to assess their usefulness in helping residents transfer basic endoscopic skills acquired during simulation training to clinical situations. METHODS: The secretion simulant was prepared via a crosslinked reaction between sodium alginate and acrylamide, whereas the packing simulant was prepared using a superabsorbent polymer. After the simulants' fidelity and utility were evaluated by 5 rhinologists using a 5-point Likert scale, 46 novice residents were trained using the 3D-printed nasal models and postsurgical simulants for 2 weeks. A checklist and Global Rating Scale (GRS) were used to assess their performances before and after training, and the time to finish each task was also recorded. Following training, the qualified trainees operated on real patients and were reevaluated. RESULTS: The simulants' similarity and usefulness scored ≥4.0, and the training cost was 28 CNY ($4 USD) per session. Following training, the checklist and GRS scores increased, and the operation time decreased (all p < 0.05). There were no statistical differences between the trainees' performances on the models with the simulants and on patients (all p > 0.05). CONCLUSION: The low-cost simulated secretion and dressing are safe to use. The application of the simulants in conjunction with that of 3D-printed nasal models in a simulated task setting can help residents in transferring endoscopic skills acquired during simulation teaching to real patients.

Effect of a Surgical Teaching Video on Resident Performance of a Laparoscopic Salpingo-oophorectomy: A Randomized Controlled Trial.

STUDY OBJECTIVE: To assess the effect of a surgical teaching video on junior resident knowledge and performance of a laparoscopic salpingo-oophorectomy (LSO). DESIGN: Randomized controlled trial. SETTING: Urban tertiary care academic obstetrics and gynecology department. PATIENTS: First- and second-year gynecology residents. INTERVENTIONS: Access to an education video on LSO for 1 week before performing this surgery in the operating room. MEASUREMENTS AND MAIN RESULTS: Twenty-four junior residents were recruited and randomized to either the educational video group or traditional residency training group. All participants completed a demographic survey and knowledge questionnaire before performing an LSO, which was video-recorded. Video recordings of surgical performance were analyzed using the Objective Structured Assessment of Technical Skills (OSATS; 20 points) and an LSO-specific tool (30 points). Participants completed a self-assessment questionnaire before completing the procedure. The primary outcome measure was the difference in OSATS scores. The secondary outcomes were the knowledge questionnaire scores and self-assessed confidence scores. There were no significant differences between demographic variables of the 2 groups. The primary outcome revealed no significant differences in mean (standard deviation) OSATS scores (10.64 [2.05] vs 11.55 [1.85], p = .3) or LSO-specific tool scores (16.45 [2.68] vs 17.85 [2.63], p = .24). However, there was a significant difference in mean knowledge scores between the video and the traditional training (8.42 [0.79] vs 7.11 [1.36], p = .01) groups. In addition, residents in the video group had more confidence in their knowledge of pelvic anatomy (3.83 [0.39] vs 3.00 [1.00] out of 5.00, p = .04). CONCLUSION: For junior learners, the use of an LSO video improved knowledge and confidence in anatomy but did not translate to improved surgical performance in the operating room. Surgical videos are a useful adjunct and complement hands-on technical teaching.

Competency-Based Assessment Tool for Pediatric Tracheotomy: International Modified Delphi Consensus.

OBJECTIVES/HYPOTHESIS: Create a competency-based assessment tool for pediatric tracheotomy. STUDY DESIGN: Blinded, modified, Delphi consensus process. METHODS: Using the REDCap database, a list of 31 potential items was circulated to 65 expert surgeons who perform pediatric tracheotomy. In the first round, items were rated as "keep" or "remove," and comments were incorporated. In the second round, experts were asked to rate the importance of each item on a seven-point Likert scale. Consensus criteria were determined a priori with a goal of 7 to 25 final items. RESULTS: The first round achieved a response rate of 39/65 (60.0%), and returned questionnaires were 99.5% complete. All items were rated as "keep," and 137 comments were incorporated. In the second round, 30 task-specific and seven previously validated global rating items were distributed, and the response rate was 44/65 (67.7%), with returned questionnaires being 99.3% complete. Of the Task-Specific Items, 13 reached consensus, 10 were near consensus, and 7 did not achieve consensus. For the 7 previously validated global rating items, 5 reached consensus and two were near consensus. CONCLUSIONS: It is feasible to reach consensus on the important steps involved in pediatric tracheotomy using a modified Delphi consensus process. These items can now be considered to create a competency-based assessment tool for pediatric tracheotomy. Such a tool will hopefully allow trainees to focus on the important aspects of this procedure and help teaching programs standardize how they evaluate trainees during this procedure. LEVEL OF EVIDENCE: 5 Laryngoscope, 130:2700-2707, 2020.

Crowdsourced Assessment of Inanimate Biotissue Drills: A Valid and Cost-Effective Way to Evaluate Surgical Trainees.

OBJECTIVE: Providing feedback to surgical trainees is a critical component for assessment of technical skills, yet remains costly and time consuming. We hypothesize that statistical selection can identify a homogenous group of nonexpert crowdworkers capable of accurately grading inanimate surgical video. DESIGN: Applicants auditioned by grading 9 training videos using the Objective Structured Assessment of Technical Skills (OSATS) tool and an error-based checklist. The summed OSATS, summed errors, and OSATS summary score were tested for outliers using Cronbach's Alpha and single measure intraclass correlation. Accepted crowdworkers then submitted grades for videos in 3 different compositions: full video 1× speed, full video 2× speed, and critical section segmented video. Graders were blinded to this study and a similar statistical analysis was performed. SETTING: The study was conducted at the University of Pittsburgh Medical Center (Pittsburgh, PA), a tertiary care academic teaching hospital. PARTICIPANTS: Thirty-six premedical students participated as crowdworker applicants and 2 surgery experts were compared as the gold-standard. RESULTS: The selected hire intraclass correlation was 0.717 for Total Errors and 0.794 for Total OSATS for the first hire group and 0.800 for Total OSATS and 0.654 for Total Errors for the second hire group. There was very good correlation between full videos at 1× and 2× speed with an interitem statistic of 0.817 for errors and 0.86 for OSATS. Only moderate correlation was found with critical section segments. In 1 year 275hours of inanimate video was graded costing $22.27/video or $1.03/minute. CONCLUSIONS: Statistical selection can be used to identify a homogenous cohort of crowdworkers used for grading trainees' inanimate drills. Crowdworkers can distinguish OSATS metrics and errors in full videos at 2× speed but were less consistent with segmented videos. The program is a comparatively cost-effective way to provide feedback to surgical trainees.

Research progress and prospect of objective structured assessment of technical skills (OSATS) in the quality control of thoracic surgical procedures / 中国胸心血管外科临床杂志

@#It has been absent from an accepted criteria for normalization and quality control of the thoracic surgery until now. The ideal assessing instrument which will be used to evaluate the technical skills and surgical procedures should present a few vital characterizations below: objectivity, speciality in the content, detailed structure, and quantifiability. Objective structured assessment of technical skills (OSATS) has developed as a reliable method of surgical skills measurement. This article focuses on the history of OSATS and its prospect in the thoracic surgery area by reviewing relevant literatures.

Validation of the Omni: A Novel, Multimodality, and Longitudinal Surgical Skills Assessment.

OBJECTIVE: The breadth of technical skills included in general surgery training continues to expand. The current competency-based training model requires assessment tools to measure acquisition, learning, and mastery of technical skill longitudinally in a reliable and valid manner. This study describes a novel skills assessment tool, the Omni, which evaluates performance in a broad range of skills over time. DESIGN: The 5 Omni tasks, consisting of open bowel anastomosis, knot tying, laparoscopic clover pattern cut, robotic needle drive, and endoscopic bubble pop, were developed by general surgery faculty. Component performance metrics assessed speed, accuracy, and quality, which were scaled into an overall score ranging from 0 to 10 for each task. For each task, ANOVAs with Scheffé's post hoc comparisons and Pearson's chi-squared tests compared performance between 6 resident cohorts (clinical years (CY1-5) and research fellows (RF)). Paired samples t-tests evaluated changes in performance across academic years. Cronbach's alpha coefficient determined the internal consistency of the Omni as an overall assessment. SETTING: The Omni was developed by the Department of Surgery at Duke University. Annual assessment and this research study took place in the Surgical Education and Activities Lab. PARTICIPANTS: All active general surgery residents in 2 consecutive academic years spanning 2015 to 2017. RESULTS: A total of 62 general surgery residents completed the Omni and 39 (67.2%) of those residents completed the assessment in 2 consecutive years. Based on data from all residents' first assessment, statistically significant differences (p < 0.05) were observed among CY cohorts for bowel anastomosis, robotic, and laparoscopic task metrics. By pair-wise comparisons, mean bowel anastomosis scores distinguished CY1 from CY3-5 and CY2 from CY5. Mean robotic scores distinguished CY1 from RF, and mean laparoscopic scores distinguished CY1 from RF, CY3, and CY5 in addition to CY2 from CY3. Mean scores in performance on the knot tying and endoscopic tasks were not significantly different. Statistically significant improvement in mean scores was observed for all tasks from year 1 to year 2 (all p < 0.02). The internal consistency analysis revealed an alpha coefficient of 0.656. CONCLUSIONS: The Omni is a novel composite assessment tool for surgical technical skill that utilizes objective measures and scoring algorithms to evaluate performance. In this pilot study, 3 tasks demonstrated discriminative ability of performance by CY, and all 5 tasks demonstrated construct validity by showing longitudinal improvement in performance. Additionally, the Omni has adequate internal consistency for a formative assessment. These results suggest the Omni holds promise for the evaluation of resident technical skill and early identification of outliers requiring intervention.

Creation of a novel simulator for minimally invasive neurosurgery: fusion of 3D printing and special effects.

OBJECTIVE Recent advances in optics and miniaturization have enabled the development of a growing number of minimally invasive procedures, yet innovative training methods for the use of these techniques remain lacking. Conventional teaching models, including cadavers and physical trainers as well as virtual reality platforms, are often expensive and ineffective. Newly developed 3D printing technologies can recreate patient-specific anatomy, but the stiffness of the materials limits fidelity to real-life surgical situations. Hollywood special effects techniques can create ultrarealistic features, including lifelike tactile properties, to enhance accuracy and effectiveness of the surgical models. The authors created a highly realistic model of a pediatric patient with hydrocephalus via a unique combination of 3D printing and special effects techniques and validated the use of this model in training neurosurgery fellows and residents to perform endoscopic third ventriculostomy (ETV), an effective minimally invasive method increasingly used in treating hydrocephalus. METHODS A full-scale reproduction of the head of a 14-year-old adolescent patient with hydrocephalus, including external physical details and internal neuroanatomy, was developed via a unique collaboration of neurosurgeons, simulation engineers, and a group of special effects experts. The model contains "plug-and-play" replaceable components for repetitive practice. The appearance of the training model (face validity) and the reproducibility of the ETV training procedure (content validity) were assessed by neurosurgery fellows and residents of different experience levels based on a 14-item Likert-like questionnaire. The usefulness of the training model for evaluating the performance of the trainees at different levels of experience (construct validity) was measured by blinded observers using the Objective Structured Assessment of Technical Skills (OSATS) scale for the performance of ETV. RESULTS A combination of 3D printing technology and casting processes led to the creation of realistic surgical models that include high-fidelity reproductions of the anatomical features of hydrocephalus and allow for the performance of ETV for training purposes. The models reproduced the pulsations of the basilar artery, ventricles, and cerebrospinal fluid (CSF), thus simulating the experience of performing ETV on an actual patient. The results of the 14-item questionnaire showed limited variability among participants' scores, and the neurosurgery fellows and residents gave the models consistently high ratings for face and content validity. The mean score for the content validity questions (4.88) was higher than the mean score for face validity (4.69) (p = 0.03). On construct validity scores, the blinded observers rated performance of fellows significantly higher than that of residents, indicating that the model provided a means to distinguish between novice and expert surgical skills. CONCLUSIONS A plug-and-play lifelike ETV training model was developed through a combination of 3D printing and special effects techniques, providing both anatomical and haptic accuracy. Such simulators offer opportunities to accelerate the development of expertise with respect to new and novel procedures as well as iterate new surgical approaches and innovations, thus allowing novice neurosurgeons to gain valuable experience in surgical techniques without exposing patients to risk of harm.