Verification of Proficiency in Basic Skills for PGY-1 Surgical Residents: 10-Year Update.

OBJECTIVE: The American College of Surgeons and the Association of Program Directors in Surgery developed a curriculum in 2001 that involved instructional modules for 11 basic surgical skills and a standardized Verification of Proficiency (VOP) evaluation instrument. Our institution continues to employ a modified version of this curriculum and the purpose of this study was to provide a 10-year update on our VOP evaluation instrument used to assess postgraduate year 1 (PGY-1) residents on surgical skills. DESIGN: All PGY-1 surgical residents over the past 10 years at our institution have completed the American College of Surgeons/the Association of Program Directors in Surgery-adapted basic surgical skills curriculum and VOP assessment. Retrospective analysis of VOP data for all residents was subjected to statistical analysis for internal validity and level of correlation. SETTING: Department of Surgery at Southern Illinois University School of Medicine located in Springfield, Illinois. PARTICIPANTS: All PGY-1 surgical residents (per year: 4 general surgery, 3 orthopedic surgery, 2 plastic surgery, 2 urology, 2 ENT, 1 vascular surgery, and 1 neurosurgery) over the past 10 years. RESULTS: One hundred and thirty five residents underwent VOP evaluation over 10 years; 92 (68%) failed at least 1 module and 40 (30%) failed at least 2 modules. Residents who failed to demonstrate proficiency were mandated to complete remediation and retested until their scores were considered proficient. Performance on checklist items showed moderate internal consistency (⍺ ≥ 0.50) on 9 of 11 modules. Poor internal consistency (⍺ < 0.30) was noted for overall proficiency across all modules. Combined performance on checklist items and economy of time and motion demonstrated significant positive correlation (p < 0.05) with overall proficiency in every module. CONCLUSIONS: The VOP instrument offers an internally valid means of assessing distinct basic skills of PGY-1 residents at basic surgical skills. The instrument provides critical formative and summative feedback on surgical skill performance to trainees.

Verification of proficiency in basic skills for postgraduate year 1 residents.

BACKGROUND: The American College of Surgeons and Association of Program Directors in Surgery Phase 1 curriculum involves basic surgical skills instructional modules and Verification of Proficiency. This article is a study and revision of beta versions of the Verification of Proficiency instruments. METHODS: Postgraduate year 1 residents were tested on 11 skills after undergoing lab instruction and practice. Deidentified videotaped performances were scored and data were analyzed to identify correlations between individual checklist items and failure. RESULTS: In all, 23 residents underwent Verification of Proficiency over 2 years; 8 (35%) passed all Verification of Proficiency examinations at the first attempt, 15 (65%) failed at least 1 module, and 11 (48%) failed at least 2 modules. Residents who failed to demonstrate proficiency underwent mandatory remediation and retested until their scores were considered proficient. Scrutiny of the results revealed checklist items that were predictive independently of overall failure. The pass rate was significantly greater in 2009 compared with 2008 after the introduction of rater training and consequences for failure. CONCLUSION: Verification of Proficiency provides a framework to evaluate learner progress toward skills proficiency. That we achieved 100% faculty compliance with more than 250 performances speaks to the feasibility of Verification of Proficiency. This approach should facilitate a more widespread Verification of Proficiency acceptance as a step closer to developing a final proficiency examination for basic surgical skills in postgraduate year 1 residents.

Skills coaches as part of the educational team: a randomized controlled trial of teaching of a basic surgical skill in the laboratory setting.

BACKGROUND: The aim of this study was to compare the laboratory teaching of a basic technical skill by a nonphysician skills coach and a faculty surgeon. METHODS: Medical students were randomized to instruction of skin suturing in the skills laboratory by a faculty surgeon or by a nonphysician skills coach. Testing of performance occurred at 3 time points. Other faculty surgeons, blinded to identities and training groups, rated performance. RESULTS: Forty-nine students participated. Baseline fourth-year student mean scores showed no significant difference between training groups. Third-year and fourth-year student performance showed no difference between training groups on postintervention testing. Delayed testing also showed no difference in third-year student scores. CONCLUSIONS: Training by either a nonsurgeon skills coach or a faculty surgeon resulted in no difference in performance on a basic surgical skill. This was true for students with and without prior experience and was also true after subsequent clinical experiences. Nonphysician coaches may ease the teaching burden of surgical faculty members while providing similar quality of instruction for trainees.

Refining the evaluation of operating room performance.

PURPOSE: An accurate and consistent evaluation of resident operative performance is necessary but difficult to achieve. This study continues the examination of the Southern Illinois University (SIU) operative performance rating system (OPRS) by studying additional factors that may influence reliability, accuracy, and interpretability of results. METHODS: OPRS evaluations of surgical residents by faculty at SIU, from 2001 to 2008, were analyzed for the most frequently rated procedures to determine (1) the elapsed time from the procedure until completion of rating, (2) the patterns in responses of procedure-specific and global surgical skills items, and (3) whether particular evaluating surgeons differed in their stringency of ratings of resident operative performance. RESULTS: In all, 566 evaluations were analyzed, which consisted of open colectomy (n = 125), open inguinal hernia (n = 103), laparoscopic cholecystectomy (n = 199), and excisional biopsy (n = 139). The number of residents evaluated per training level (PGY) ranged from 88 to 161. The median time to completion of evaluations was 11 days, 9 hours. The quickest evaluation was 18 hours after assignment. Most were completed within 4.5 to 22 days. Procedure-specific and global scale scores resulted in similar rank-ordering of performances (single-measure intraclass correlation using the consistency model = 0.88; 95% confidence interval [CI] = 0.87-0.90) and similar absolute OPRS scores (single-measure intraclass correlation using the consistency model = 0.89; 95% CI, 0.87-0.90). Evaluating surgeons differed in stringency of ratings across procedures (average difference = 1.4 points of 5 possible points). Resident performance improved with increasing PGY level for all 4 procedures. CONCLUSIONS: Substantial time elapses between performance in the operating room and the completion of the evaluation. This raises the question of whether surgeons remember the nuances of the procedure well enough to rate performance accurately. The item type used for rating does not affect the absolute rating assigned or the rank ordering of the performance. Differences in stringency of evaluators indicate the need for multiple resident performance observations by multiple surgeons. These findings are the foundation for an upcoming multi-institutional trial.

A theory-based curriculum for enhancing surgical skillfulness.

BACKGROUND: Curricula for surgical technical skills laboratories have traditionally been designed to accommodate the clinical activities of residents, so they typically consist of individual, episodic training sessions. We believe that the skills laboratory offers an opportunity to design a surgical skills curriculum based on the fundamental elements known to be important for motor skill instruction. We hypothesized that training novices with such a curriculum for a 1-month period would yield skills performance levels equivalent to those of second year surgery residents who had trained in a traditional program. STUDY DESIGN: Fourth-year medical students served as study subjects (novice group) during a 4-week senior elective. They were taught each skill during a 1-week period. Subjects received instruction by a content expert followed by a 1-week period of deliberate practice with feedback. The novice performances were videotaped both before and after the intervention, and each videotape was evaluated in a blinded fashion by experts using a validated evaluation instrument. These results were compared with skill performance ratings of first- and second-year surgery residents that had been accumulated over the previous 3 years. RESULTS: Average performance ratings for the novices substantially improved for all four skills after training. There was no marked difference between average performance ratings of postintervention novice scores when compared with the average scores in the resident group. Inter-rater agreement in scoring for the videotaped novice performances exceeded 0.87 (intraclass correlation) for all ratings of pre- and posttraining. CONCLUSIONS: These results demonstrate the effectiveness of a laboratory-based training program that includes fundamentals of motor skills acquisition.

Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents.

BACKGROUND: Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format. METHODS: Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations. RESULTS: Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable. CONCLUSIONS: Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.

Accreditation council on graduate medical education technical skills competency compliance: urologic surgical skills.

BACKGROUND: In accordance with new mandates implemented by the Accreditation Council on Graduate Medical Education, reliance on operative case logs as demonstration of residents' surgical competence will no longer be adequate. We describe the implementation of a comprehensive, year-round, mandatory skills laboratory curriculum as an integral component of our urology residency training program. STUDY DESIGN: We developed eight laboratory practicums using primarily nonhuman models: basic endoscopy, advanced endoscopy, ureteroscopy, percutaneous renal surgery, basic laparoscopy, advanced laparoscopy, urologic use of the gastrointestinal tract, and cadaveric pelvic dissection. RESULTS: Anonymous evaluations submitted by all training session participants indicate that acquisition of surgical skills is facilitated through participation in laboratory practicums. An incremental progression in proficiency was observed by all of the instructors and students who participated. There was a high degree of satisfaction with model fidelity and the value of technical experience gained. CONCLUSIONS: Our urologic surgery skills laboratory curriculum is an effective means of skills acquisition and maintenance for a wide variety of urologic techniques, including complex endourologic procedures. Patient care can safely be of secondary importance with respect to trainee experience in a low-stress environment that provides an opportunity for supervised repetitive performance of essential technical skills. We describe effective models, with high fidelity-to-cost ratio, that incorporate laboratory-based surgical skills training and evaluation into urology residency programs, with the aim of Accreditation Council on Graduate Medical Education competency guideline compliance.

A new approach to urology training: a laboratory model for percutaneous nephrolithotomy.

PURPOSE: The efficacy of traditional operating room based training of urology residents is being reevaluated. The development of hands-on laboratory practicums to facilitate the acquisition of skills by surgical residents lessens learning curves and hastens familiarity with tissue and instrument handling. We describe an innovative model for simulated percutaneous renal access and nephrolithotomy. MATERIALS AND METHODS: Porcine kidneys pre-implanted with artificial stone material were placed within intact chicken carcasses as a model for percutaneous nephrolithotomy. Urology residents were taught needle access, tract dilation and renal access sheath insertion using fluoroscopy. Training in percutaneous nephrolithotomy with the nephroscope, graspers and stone fragmentation methods followed. RESULTS: This simple, cost-effective model closely simulates percutaneous nephrolithotomy. Anonymous evaluations submitted by training session participants revealed a high degree of satisfaction with model effectiveness in the application of percutaneous renal access and nephrolithotomy techniques. CONCLUSIONS: Our percutaneous nephrolithotomy laboratory model is an effective means of skills acquisition for a complex endourological procedure. Patient care can safely be of secondary importance with respect to trainee experience in a low stress environment that provides an opportunity for supervised, repetitive performance of essential technical skills. We describe an effective percutaneous renal access and nephrolithotomy surgical training model of original design.

A senior elective designed to prepare medical students for surgical residency.

BACKGROUND: The transition from medical student to surgery internship can be stressful. The goal of this project was to design, implement, and evaluate a 1-month long elective course that would meet the majority of the American College of Surgeons Graduate Medical Education Committee prerequisites for graduate surgical education METHODS: The major elements of the curriculum included faculty- and resident-facilitated case-based sessions and cadaver dissections. In addition, the students participated in skills laboratory experiences, Intensive Care Unit rounds, and mock interviews and clinical pages. The students took a knowledge pretest and post-test that was compared with the performance of 8 surgical interns on the same examination. RESULTS: The highest rated elements of the course were those that provided hands-on experience or practical knowledge. The post-test knowledge examination scores were significantly higher than pretest scores and surgical intern scores. CONCLUSIONS: It was possible to develop a 1-month senior medical student elective course that provided students with the essential prerequisites believed to be essential for all surgical interns.

Surface landmarks to locate the thenar branch of the median nerve: an anatomical study.

The thenar branch of the median nerve can be injured during carpal tunnel release. The purpose of this study was to identify surface landmarks to consistently predict the location of the thenar branch of the median nerve. Surface landmarks were marked and incised in 28 cadaveric hands. The incisions were made along the longitudinal line of the third web space and the horizontal cardinal line from the hamate hook to the ulnar border of the thumb. The origin of the thenar branch was determined in relation to these longitudinal and horizontal vectors. The origin of the thenar nerve branch was consistently observed in the radial proximal quadrant formed by the aforementioned longitudinal and horizontal vectors. The thenar branch origin was observed to be an average of 8.6 +/- 1.9 mm radial to the longitudinal axis along the third web space. The origin of the thenar branch was observed to be an average of 6.3 +/- 2.0 mm proximal to the horizontal axis between the hamate hook and the ulnar border of the thumb. The thenar branch was observed precisely at the intersection of the longitudinal vector from the second web space to the scaphoid tubercle and the horizontal vector from the hamate hook to the radial edge of the proximal metacarpophalangeal crease in all 28 cadaveric hands. On the basis of these 28 cadaveric dissections, the location of the thenar branch of the median nerve can be predicted by the intersection of the longitudinal vector from the second web space to the scaphoid tubercle and the horizontal vector from the hamate hook to the radial aspect of the metacarpophalangeal crease.