Establishing Validity for a Vaginal Hysterectomy Simulation Model for Surgical Skills Assessment.

OBJECTIVE: To use the Messick validity framework for a simulation-based assessment of vaginal hysterectomy skills. METHODS: Video recordings of physicians at different levels of training and experience performing vaginal hysterectomy on a high-fidelity vaginal surgery model were objectively assessed using a modified 10-item Vaginal Surgical Skills Index, a one-item global scale of overall performance, and a pass-fail criterion. Participants included obstetrics and gynecology trainees and faculty from five institutions. Video recordings were independently assessed by expert surgeons blinded to the identities of the study participants. RESULTS: Fifty surgeons (11 faculty, 39 trainees) were assessed. Experience level correlated strongly with both the modified Vaginal Surgical Skills Index and global scale score, with more experienced participants receiving higher scores (Pearson r=0.81, P<.001; Pearson r=0.74, P<.001). Likewise, surgical experience was also moderately correlated with the modified Vaginal Surgical Skills Index and global scale score (Pearson r=0.55, P<.001; Pearson r=0.58, P<.001). The internal consistency of the modified Vaginal Surgical Skills Index was excellent (Cronbach's alpha=0.97). Interrater reliability of the modified Vaginal Surgical Skills Index and global scale score, as measured by the intraclass correlation coefficient, was moderate to good (0.49-0.95; 0.50-0.87). Using the receiver operating characteristic curve and the pass-fail criterion, a modified Vaginal Surgical Skills Index cutoff score of 27 was found to most accurately (area under the curve 0.951, 95% CI 0.917-0.983) differentiate competent from noncompetent surgeons. CONCLUSION: We demonstrated validity evidence for using a high-fidelity vaginal surgery model with the modified Vaginal Surgical Skills Index or global scale score to assess vaginal hysterectomy skills.

Validation of an educational simulation model for vaginal hysterectomy training: a pilot study.

INTRODUCTION AND HYPOTHESIS: The Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC, USA) was designed as a realistic vaginal surgery simulation model. Our aim was to describe this model and present pilot data on validity and reliability of the model as an assessment tool of vaginal hysterectomy skills. METHODS: We video recorded ten obstetrics and gynecology residents (novice group) and ten practicing gynecologists (expert group) performing vaginal hysterectomy using the Miya model. Blood loss and time taken to complete the procedure were documented. Participants evaluated the model using a postsimulation survey. In addition, two experienced gynecologic surgeons independently evaluated video recordings of each participant's performance using two previously validated global rating scales: Reznick's Objective Structured Assessment of Technical Skill (OSATS) and Vaginal Surgical Skills Index (VSSI). RESULTS: Most participants (80% of novice and 100% of expert group) rated the model as effective or highly effective for vaginal hysterectomy training and assessment. Median time to procedure completion was significantly higher in the novice group, whereas median estimated blood loss was no different between groups. No significant differences were observed in the composite median OSATS or VSSI scores between groups. The interrater reliability indices for subscales and composite scores of the OSATS and VSSI were high and ranged from 0.79 to 0.90 and 0.77 to 0.93, respectively. CONCLUSIONS: With further study, the Miya Model may be a useful tool for teaching and assessing vaginal surgical skills.