Competency-Based Training and Simulation: Making a "Valid" Argument.

The use of simulation as an assessment tool is much more controversial than is its utility as an educational tool. However, without valid simulation-based assessment tools, the ability to objectively assess technical skill competencies in a competency-based medical education framework will remain challenging. The current literature in urologic simulation-based training and assessment uses a definition and framework of validity that is now outdated. This is probably due to the absence of awareness rather than an absence of comprehension. The following review article provides the urologic community an updated taxonomy on validity theory as it relates to simulation-based training and assessments and translates our simulation literature to date into this framework. While the old taxonomy considered validity as distinct subcategories and focused on the simulator itself, the modern taxonomy, for which we translate the literature evidence, considers validity as a unitary construct with a focus on interpretation of simulator data/scores.

Optimizing the Timing of Expert Feedback During Simulation-Based Spaced Practice of Endourologic Skills.

INTRODUCTION: Provision of expert feedback is widely acknowledged to be an essential component of simulation-based training. However, little is known about the most effective and efficient ways to provide feedback to novices. Optimizing the timing of expert feedback may improve outcomes while reducing resource requirements. The main objective of this study was to determine the impact of providing early versus late expert feedback to novice learners engaged in a flexible ureteroscopy (fURS) training curriculum. METHODS: Senior medical students were recruited to participate in this study. Each student participated in a comprehensive fURS training curriculum that included 3 deliberate, independent practice sessions. Baseline and postcourse fURS skill was assessed for each student using a standardized fURS test task. Each student was randomized to either an early feedback group (EFG) or late feedback group (LFG). The EFG participants were provided expert feedback immediately after the baseline skill test, whereas LFG participants were given feedback before their final deliberate, independent practice session. RESULTS: Eighteen senior medical students completed the study (9 EFG and 9 LFG participants). There were no discernible demographic differences between the groups at baseline. When controlling for pretest performance, early rather than late feedback was associated with both shorter postcourse time to completion of the task (19.2 vs. 21.5 minutes, P < 0.01) and higher performance scores (13.1 vs. 10.5, P < 0.01). CONCLUSIONS: This study offers preliminary evidence that novice learners may benefit more from early feedback when learning a novel skill. Further study is required.

Anthropometric Renal Anatomic Alterations Between Supine and Prone Positions in Percutaneous Renal Ablation for Renal Cortical Neoplasms.

OBJECTIVES: To establish patterns of anatomic changes relevant to the kidney and colon during positional change between the supine and prone positions as noted on CT scans performed during percutaneous cryoablation for renal cortical neoplasms (RCN). METHODS: Nineteen patients undergoing percutaneous cryoablation for RCN with abdominal CT scan in both the supine and prone positions were included in the study. We documented the anterior/posterior, medial/lateral, and cranial/caudal anatomic changes of the kidney, kidney rotation, and the proportion of the kidney whose access was limited by the liver, spleen, and lung. We also calculated the length of the percutaneous access tract and the distance between the colon and kidney in hilar position as well as the anterior/posterior location of the colon relative to the kidney. RESULTS: In the prone position, the kidney lies significantly more anteriorly on both sides: 4.7 cm vs 4.3 cm (L) and 4.4 cm vs 4.1 cm (R) (p = 0.02 and p = 0.03, respectively). On prone CT images, both kidneys are more cranial when compared with the supine position: 80.4 mm vs 60.8 mm (L) and 87.2 mm vs 57.4 mm (R) (p = 0.002 and p < 0.001, respectively). The skin to tumor distance is significantly shorter in the prone position (p < 0.0001 [L], p = 0.005 [R]). The colon lies closer to the hilum of the kidney and is more posteriorly located in the prone position: 1.21 cm vs 1.04 cm (L) and 0.80 cm vs 0.70 cm (R) (p = 0.005 and p = 0.005, respectively). In the prone position, the lung covers a significantly larger proportion of the right kidney (27.3 mm vs 6.05 mm, p = 0.0001). CONCLUSIONS: We documented clinically significant anatomic alterations between supine and prone CT imaging. The changes associated with the prone position modify percutaneous access, particularly for right upper pole tumors. Prone imaging before surgery may be helpful in selected cases.

Development of a novel iPad-based laparoscopic trainer and comparison with a standard laparoscopic trainer for basic laparoscopic skills testing.

INTRODUCTION: We developed the iTrainer (iT) as a portable laparoscopic trainer, which incorporates the iPad tablet. We then compared the iT with a standard pelvic trainer (SPT) to assess surgical skills as well as its image quality, resolution, brightness, comfort, and overall performance. MATERIALS AND METHODS: We designed and constructed the iT to be compatible with the Apple iPad 3 and standard laparoscopic instruments. Participants were assigned to perform the thread-the-loops task on both trainers and were prospectively randomized to start on either the iT or the SPT. Each participant was allowed a 2-minute warm-up before the 2-minute testing period. We scored participants using the product of skill quality (0-4 scale) and quantity of loops threaded (0-10 scale). Participants then rated each trainer on image quality, resolution, brightness, comfort, and overall performance on a 5-point Likert scale. RESULTS: A total of 45 subjects including 10 undergraduates, 10 medical students, 10 general surgery and urology residents, and 15 experts (fellows and attending surgeons) participated in this study. There was no significant difference between thread-the-loops task scores completed on the iT when compared with the SPT for all groups tested (p > 0.05) with the exception of the medical student group, who performed better on the SPT (p < 0.05). On evaluation of each trainer, participants rated the iT as having superior image quality and resolution when compared with the SPT (p < 0.05) but rated the SPT higher in overall performance (p < 0.05). Brightness and comfort were rated similarly for both trainers. CONCLUSIONS: We have demonstrated face validity and criterion validity for the thread-the-loops task on the iT. The iT rated superior in image quality and resolution but inferior in overall performance compared with the SPT. The iT provides trainees a unique advantage over SPT as an additional resource to laparoscopic training as it is inexpensive, portable, and can be readily available for training.

Evaluation of the impact of three-dimensional vision on laparoscopic performance.

INTRODUCTION: Recent technological advancements have led to the introduction of new three-dimensional (3D) cameras in laparoscopic surgery. The 3D view has been touted as useful during robotic surgery, however, there has been limited investigation into the utility of 3D in laparoscopy. MATERIALS AND METHODS: We performed a prospective, randomized crossover trial comparing a 0° 3D camera with a conventional 0° two-dimensional (2D) camera using a high definition monitor (Karl Storz, Tuttlingen, Germany). All participants completed six standardized basic skills tasks. Quality testing scores were measured by the number of drops, grasping attempts, and precision of needle entry and exiting. Additionally, resolution, color distribution, depth of field and distortion were measured using optical test targets. RESULTS: In this pilot study, we evaluated 10 medical students, 7 residents, and 7 expert surgeons. There was a significant difference in the performance in all the six skill tasks, for the three levels of surgical expertise and training levels in 2D vs 3D except for the cut the line quality score and the peg transfer quality score. Adjusting for the training level, 3D camera image results were superior for the number of rings left (p=0.041), ring transfer quality score (p=0.046), thread the rings (no. of rings) (p=0.0004), and thread the rings quality score (p=0.0002). The 3D camera image was also superior for knot tying (quality score) (p=0.004), peg transfer (time in seconds) (p=0.047), peg transfer pegs left (p=0.012), and for peg transfer quality score (p=0.001). The 3D camera system showed significantly less distortion (p=0.0008), a higher depth of field (p=0.0004) compared with the 2D camera system. CONCLUSION: 3D laparoscopic camera equipment results in a significant improvement in depth perception, spatial location, and precision of surgical performance compared with the conventional 2D camera equipment. With this improved quality of vision, even expert laparoscopic surgeons may benefit from 3D imaging.

High-fidelity simulation-based team training in urology: evaluation of technical and nontechnical skills of urology residents during laparoscopic partial nephrectomy.

INTRODUCTION: The use of low-risk simulation training for resident education is rapidly expanding as teaching centers integrate simulation-based team training (SBTT) sessions into their education curriculum. SBTT is a valuable tool in technical and communication skills training and assessment for residents. We created a unique SBTT scenario for urology residents involving a laparoscopic partial nephrectomy procedure. MATERIALS AND METHODS: Urology residents were randomly paired with a certified registered nurse anesthetists or an anesthesia resident. The scenario incorporated a laparoscopic right partial nephrectomy utilizing a unique polyvinyl alcohol kidney model with an embedded 3cm lower pole exophytic tumor and the high-fidelity SimMan3G mannequin. The Urology residents were instructed to pay particular attention to the patient's identifying information provided at the beginning of the case. Two scripted events occurred, the patient had an anaphylactic reaction to a drug and, after tumor specimen was sent for a frozen section, the confederate pathologist called into the operating room (OR) twice, first with the wrong patient name and subsequently with the wrong specimen. After the scenario was complete, technical performance and nontechnical performance were evaluated and assessed. A debriefing session followed the scenario to discuss and assess technical performance and interdisciplinary nontechnical communication between the team. RESULTS: All Urology residents (n = 9) rated the SBTT scenario as a useful tool in developing communication skills among the OR team and 88% rated the model as useful for technical skills training. Despite cuing to note patient identification, only 3 of 9 (33%) participants identified that the wrong patient information was presented when the confederate "pathologist" called in to report pathology results. CONCLUSION: All urology residents rated SBTT sessions as useful for the development of communication skills between different team members and making residents aware of unlikely but potential critical errors in the OR. We will continue to use SBTT as a useful method to develop resident technical and nontechnical skills outside of the high-risk operating environment.

Evaluation of potential distractors in the urology operating room.

BACKGROUND AND PURPOSE: Surgical outcomes depend on patient and disease-related factors, as well as the technical skill of the surgeon. Various distractions in the operating room (OR) environment have been shown to negatively impact a surgeon's performance. A survey was conducted with the objective to evaluate and characterize distractions during urologic surgery. METHODS: An Internet-based survey was distributed to 2057 international urologists via email between April and October 2011; questions focused on a variety of disruptive factors postulated to have a negative impact on surgical performance. RESULTS: Of the 523 (25%) respondents, 58% practiced in North America, 42% were from an academic institution, and 68% had completed a clinical fellowship. In an average year, 83% reported having operated at least once while sleep deprived, 84% when significantly ill, 55% with a musculoskeletal injury, and 65% under significant social stress. Up to 38% reported that on at least one occasion, such "internal distractions" had significantly affected surgical performance and 14% perceived that at least one surgical complication was caused mainly by an internal distraction. Less than 50% had ever cancelled surgery because of an internal distraction. Music was routinely played in the OR by 57% of respondents, >67% reported answering pages and discussing consults while operating, and 25% reported "commonly" working with scrub nurses/techs that were unfamiliar with the procedure and/or instruments. Only 44% had consistent individual(s) assisting, and 27% reported that the scrub nurse/tech would "commonly" scrub out during a critical portion of the procedure. Overall, 14.5% reported that at least one complication had occurred mainly because of such "external" or "interactive" distractions. CONCLUSIONS: Urologists face various distractions in the OR that can negatively impact surgical performance, potentially compromising patient outcomes and safety. Further studies are needed to elucidate the true impact of such distractions and to develop strategies to mitigate their effects.

Robotic surgical simulation.

Robotic surgery has undergone exponential growth and has ever developing utilization. The explosion of new technologies and regulation have led to challenges in training surgeons who desire this skill set. We review the current state of robotic simulation and incorporation of simulation into surgical training curricula. In addition to the literature review, results of a questionnaire survey study of 21 expert and novice surgeons attending a Urologic Robotic Oncology conference using 3 different robotic skill simulation devices are discussed. An increasing number of robotic surgery simulators have had some degree of validation study of their use in surgical education curricula and proficiency testing. Although simulators are advantageous, confirmation of construct and predictive validity of robotic simulators and their reliability as a training tool will be necessary before they are integrated into the surgical credentialing process.

Surgical skills acquisition among left-handed trainees-true inferiority or unfair assessment: a preliminary study.

INTRODUCTION: Studies involving the formal assessment of surgical skills have often reported inferior abilities among left-handed surgical trainees (LHT). Most surgical training curricula and assessment methods, however, are inherently geared toward right-handed trainees (RHT); potentially placing LHT at both a training and assessment disadvantage. We evaluated the effect of a hand dominance-based curriculum for acquisition of basic suturing and knot tying skills among medical students. METHODS: After Institutional Review Board approval, first- and second-year medical students from the University of California, Irvine School of Medicine were recruited to participate in a basic suturing and knot tying skills course. Consenting students were randomized to either a left-handed curriculum or a right-handed curriculum consisting of (1) a 30-minute introductory video and (2) a 2-hour instructor-led, hands-on training session on basic suturing and knot tying. All instructional methods, instruments, and instructors were exclusively right-handed or left-handed for the right-handed curriculum or left-handed curriculum, respectively. Students were assessed on the performance of 2 suturing tasks, continuous running suturing and instrument knot tying, and performance assessments were conducted both immediately and 2 weeks posttraining. RESULTS: A total of 19 students completed the training course and both assessments (8 LHT, 11 RHT). Students randomized to a curriculum "concordant" with their hand dominance performed significantly better than those randomized to a "discordant" curriculum on both tasks (p < 0.01). This difference was found at both immediate and 2 weeks posttraining assessments. Within concordant and discordant groups, there were no significant differences between LHT and RHT. CONCLUSIONS: This preliminary study demonstrates that medical students, both LHT and RHT, immersed in a training environment that is discordant with their hand dominance might have inferior acquisition of basic suturing and knot tying skills.

In vitro evaluation of ScopeSafe fibers and the scope guardian sheath in prevention of ureteroscope endolumenal working damage.

BACKGROUND AND PURPOSE: Ureteroscope breakage is commonly related to laser fiber damage. Often, the damage is mechanical and not energy related. As such, we evaluated a novel laser fiber and sheath system in preventing mechanical ureteroscope damage during fiber insertion. MATERIALS AND METHODS: We assessed 200-µm/272/3-µm laser fibers with the laser sheath in a flexible ureteroscope. Diminishment of active deflection and irrigation flow rates were compared with a standard laser fiber alone. Eight nonassembled working channel components were tested in a 0-degree/90-degree/210-degree deflection model. After insertion cycles, external and endolumenal damage to the working channel were classified. We also tested the sheath system in a 0-degree/90-degree/210-degree deflection model for fiber failure and laser damage. RESULTS: In all test trials with the sheath and for standard laser fibers in the 0-degree model, there were no channel perforations or damage. With standard laser fibers, in the 210-degree model, superficial scratches and demarcated abrasions were visible after 10 and 60 to 70 insertions for the 273-µm laser fiber and after 30 insertions (superficial scratches) for the 200-µm laser fiber. In the 90-degree model, superficial scratches occurred after 20 insertions for the 273-µm fibers and after 40 insertions for the 200-µm laser fibers. No demarcated abrasions were seen after 100 insertions. In the 210-degree model, there was one perforation with the 272-µm fiber, but none with 200-µm fiber. There were no fiber failures with sheath use; however, the sheath did not prevent laser energy damage. The laser sheath resulted in a 4.7-degree/3.8-degree (1.2%/1.5%) diminishment in deflection (up/down) for the 200 µm and a 3.5-degree/4.3-degree (1.8%/1.5%) diminishment for 272-µm laser fiber compared with standard 200/272-µm laser fiber. Irrigation flow was diminished with the sheath on both the 200-µm and 272-µm laser fiber by 28.7% and 32.6%, respectively. CONCLUSION: The Scope Guardian Sheath prevented mechanical working channel damage with minimal diminishment of deflection and irrigation flow.

Effect of the American Urological Association Annual Review course on the results of the urology qualifying examination.

PURPOSE: In this study we assess the impact of a urology dedicated review course on the scores of the corresponding board qualifying examination for attendees of the urology review course. MATERIALS AND METHODS: The ABU (American Board of Urology) Qualifying Examination scores from 2009, 2010 and 2011 were categorized into group 1 candidates who attended the AUA (American Urological Association) Annual Review Course the same year, and group 2 candidates who did not attend the AUA Annual Review Course that same year, and were compared. The scores of the preceding year's In-Service Examination were also compared for the same groups of candidates and compared to their subsequent first time taken Qualifying Examination scores. RESULTS: There was no difference in Qualifying Examination scores of resident candidates attending vs not attending the AUA Annual Review Course in all 3 years. The overall failure rate was low, and essentially the same for all candidates in all years regardless of attendance at the AUA Annual Review Course at 2% in 2009, 2% in 2010 and 4% in 2011. Of group 1 candidates the majority (80% to 98%) considered the Annual Review Course helpful or very helpful in preparation for the Qualifying Examination. CONCLUSIONS: The majority of candidates are adequately prepared to pass their Qualifying Examination at the conclusion of their residency training program regardless of their attendance of the AUA Annual Review Course. This course may help bolster the confidence of the candidate preparing for their Qualifying Examination.

Validation, correlation, and comparison of the da Vinci trainer(™) and the daVinci surgical skills simulator(™) using the Mimic(™) software for urologic robotic surgical education.

PURPOSE: Virtual reality simulators with self-assessment software may assist novice robotic surgeons to augment direct proctoring in robotic surgical skill acquisition. We compare and correlate the da Vinci Trainer™ (dVT) and da Vinci Surgical Skills Simulators (dVSSS) in subjects with varying robotic experience. MATERIALS AND METHODS: Students, urology residents, fellows, and practicing urologists with varying robotic experience were enrolled after local institutional review board approval. Three virtual reality tasks were preformed in sequential order (pegboard 1, pegboard 2, and tubes)-initially on the dVSSS and then on the dVT. The Mimic™ software used on both systems provides raw values and percent scores that were used in statistical evaluation. Statistical analysis was performed with the two-tailed independent t-test, analysis of variance, Tukey, and the Pearson rank correlation coefficient where appropriate. RESULTS: Thirty-two participants were recruited for this study and separated into five groups based on robotic surgery experience. In regards to construct validity, both simulators were able to differentiate differences among the five robotic surgery experience groups in the tubes suturing task (p≤0.00). Sixty-seven percent (4/6) robotic experts thought that surgical simulation should be implemented in residency training. The overall cohort considered both platforms easy to learn and use. CONCLUSIONS: Although performance scores were less in the dVT compared with the dVSSS, both simulators demonstrate good content and construct validity. The simulators appear to be equivalent for assessing surgeon proficiency and either can be used for robotic skills training with self-assessment feedback.

Genitourinary exam skills training curriculum for medical students: a follow-up study of comfort and skill utilization.

PURPOSE: We developed a genitourinary skills training (GUST) curriculum for incoming third year medical students (MS3) and performed a follow-up study of comfort with and utilization of these skills. MATERIALS AND METHODS: GUST consisted of a didactic lecture followed by skills sessions including standardized patient testicular examination (TE) and digital rectal examination (DRE), male and female Foley catheter (MFC and FFC) placement training, suture-knot tying, and a faculty-directed small group learning session. Precourse and postcourse, and 6 and 18 months after the course, MS3 rated comfort with each skill (Likert scale 0-5), and quantified skill usage. Results were compared with 4th year students (MS4) who had not undergone GUST. RESULTS: Participants were 281 MS3 GUST students and 44 MS4. Post-GUST, mean comfort on a Likert scale (0=uncomfortable) increased for all four skills (88.2%-96.9% vs 8.3%-18.5%, P<0.0001). This was maintained at the 6-month and 18-month follow up time points (P<0.0001). At 18 months, MS3 trended toward higher comfort with TE compared with MS4 (74 vs 54%, P=0.068), while with the other skills, both groups showed equal comfort. MS4 learned exam skills from faculty and MFC and FFC from nurses on the wards. Eleven percent of MS4 were never formally taught TE or DRE. MS3 and MS4 performed TE and/or DRE on <8% of newly admitted patients. CONCLUSIONS: MS3 described improved comfort with the GU skills at all time points during follow-up. This was particularly important because both MS3 and MS4 reported using their skills infrequently during their clinical training years.

Positive correlation between motion analysis data on the LapMentor virtual reality laparoscopic surgical simulator and the results from videotape assessment of real laparoscopic surgeries.

PURPOSE: We studied the construct validity of the LapMentor, a virtual reality laparoscopic surgical simulator, and the correlation between the data collected on the LapMentor and the results of video assessment of real laparoscopic surgeries. MATERIALS AND METHODS: Ninety-two urologists were tested on basic skill tasks No. 3 (SK3) to No. 8 (SK8) on the LapMentor. They were divided into three groups: Group A (n=25) had no experience with laparoscopic surgeries as a chief surgeon; group B (n=33) had <35 experiences; and group C (n=34) had ≥35 experiences. Group scores on the accuracy, efficacy, and time of the tasks were compared. Forty physicians with ≥20 experiences supplied unedited videotapes showing a laparoscopic nephrectomy or an adrenalectomy in its entirety, and the videos were assessed in a blinded fashion by expert referees. Correlations between the videotape score (VS) and the performances on the LapMentor were analyzed. RESULTS: Group C showed significantly better outcomes than group A in the accuracy (SK5) (P=0.013), efficacy (SK8) (P=0.014), or speed (SKs 3 and 8) (P=0.009 and P=0.002, respectively) of the performances of LapMentor. Group B showed significantly better outcomes than group A in the speed and efficacy of the performances in SK8 (P=0.011 and P=0.029, respectively). Analyses of motion analysis data of LapMentor demonstrated that smooth and ideal movement of instruments is more important than speed of the movement of instruments to achieve accurate performances in each task. Multiple linear regression analysis indicated that the average score of the accuracy in SK4, 5, and 8 had significant positive correlation with VS (P=0.01). CONCLUSIONS: This study demonstrated the construct and predictive validity of the LapMentor basic skill tasks, supporting their possible usefulness for the preclinical evaluation of laparoscopic skills.