Modeling Framework Used to Analyze and Describe Junctional Tourniquet Skills.

Medical educators have acknowledged the importance of simulation training in developing procedural skills. While simulation training in other disciplines has benefitted from evaluations of users' skill acquisition, the majority of medical training simulators continue to be developed from overly simplified descriptions of procedures, such as techniques prescribed by existing instructional material. Our objective was to use a modeling framework to characterize the skill of various users in applying junctional tourniquets in order to design an effective training simulator. We recorded 46 medical first responders performing training exercises applying a junctional tourniquet and used coded video and sensor data to identify the hierarchy of actions they performed in the process. The model provides several insights into trainee performance, such as the way in which advanced users perform more tasks in parallel, or areas where advanced users employ situational awareness to identify ways they can deviate from recommended protocol to improve outcomes. The model successfully identifies variations in tourniquet application technique that correlates with improvement on clinically relevant metrics including application speed, pressure applied, and tourniquet placement stability. This methodology can improve medical training simulations by indicating changes during the course of learning a new task, such as helpful deviations from instructional protocol.

Dynamic Visual Feedback During Junctional Tourniquet Training.

BACKGROUND: This project involved the development and evaluation of a new visual bleeding feedback (VBF) system for tourniquet training. We hypothesized that dynamic VBF during junctional tourniquet training would be helpful and well received by trainees. MATERIALS AND METHODS: We designed the VBF to simulate femoral bleeding. Medical students (n = 15) and emergency medical service (EMS) members (n = 4) were randomized in a single-blind, crossover study to the VBF or without feedback groups. Poststudy surveys assessing VBF usefulness and recommendations were conducted along with participants' reported confidence using a 7-point Likert scale. Data from the different groups were compared using Wilcoxon signed-rank and rank-sum tests. RESULTS: Participants rated the helpfulness of the VBF highly (6.53/7.00) and indicated they were very likely to recommend the VBF simulator to others (6.80/7.00). Pre- and post-VBF confidence were not statistically different (P = 0.59). Likewise, tourniquet application times for VBF and without feedback before crossover were not statistically different (P = 0.63). Although participant confidence did not change significantly from beginning to end of the study (P = 0.46), application time was significantly reduced (P = 0.001). CONCLUSIONS: New tourniquet learners liked our VBF prototype and found it useful. Although confidence did not change over the course of the study for any group, application times improved. Future studies using outcomes of this study will allow us to continue VBF development as well as incorporate other quantitative measures of task performance to elucidate VBF's true benefit and help trainees achieve mastery in junctional tourniquet skills.

Improving diagnosis in healthcare: Local versus national adoption of recommended guidelines for the clinical breast examination.

BACKGROUND: This study explores the long-term effectiveness of a newly developed clinical skills curriculum. METHODS: Students (N = 40) were exposed to a newly developed, simulation-based, clinical breast exam (CBE) curriculum. The same students returned one year later to perform the CBE and were compared to a convenience sample of medical students (N = 15) attending a national conferences. All students were given a clinical vignette and performed the CBE. CBE techniques were video recorded. Chi-squared tests were used to assess differences in CBE technique. RESULTS: Students exposed to a structured curriculum performed physical examination techniques more consistent with national guidelines than the random, national student sample. Structured curriculum students were more organized, likely to use two hands, a linear search pattern, and include the nipple-areolar complex during the CBE compared to national sample (p < 0.01). CONCLUSIONS: Students exposed to a structured skills curriculum more consistently performed the CBE according to national guidelines. The variability in technique compared with the national sample of students calls for major improvements in adoption and implementation of structured skills curricula.

Rescuing the Clinical Breast Examination: Advances in Classifying Technique and Assessing Physician Competency.

OBJECTIVE: Develop new performance evaluation standards for the clinical breast examination (CBE). SUMMARY BACKGROUND DATA: There are several, technical aspects of a proper CBE. Our recent work discovered a significant, linear relationship between palpation force and CBE accuracy. This article investigates the relationship between other technical aspects of the CBE and accuracy. METHODS: This performance assessment study involved data collection from physicians (n = 553) attending 3 different clinical meetings between 2013 and 2014: American Society of Breast Surgeons, American Academy of Family Physicians, and American College of Obstetricians and Gynecologists. Four, previously validated, sensor-enabled breast models were used for clinical skills assessment. Models A and B had solitary, superficial, 2 cm and 1 cm soft masses, respectively. Models C and D had solitary, deep, 2 cm hard and moderately firm masses, respectively. Finger movements (search technique) from 1137 CBE video recordings were independently classified by 2 observers. Final classifications were compared with CBE accuracy. RESULTS: Accuracy rates were model A = 99.6%, model B = 89.7%, model C = 75%, and model D = 60%. Final classification categories for search technique included rubbing movement, vertical movement, piano fingers, and other. Interrater reliability was (k = 0.79). Rubbing movement was 4 times more likely to yield an accurate assessment (odds ratio 3.81, P < 0.001) compared with vertical movement and piano fingers. Piano fingers had the highest failure rate (36.5%). Regression analysis of search pattern, search technique, palpation force, examination time, and 6 demographic variables, revealed that search technique independently and significantly affected CBE accuracy (P < 0.001). CONCLUSIONS: Our results support measurement and classification of CBE techniques and provide the foundation for a new paradigm in teaching and assessing hands-on clinical skills. The newly described piano fingers palpation technique was noted to have unusually high failure rates. Medical educators should be aware of the potential differences in effectiveness for various CBE techniques.

Junctional and Inguinal Hemorrhage Simulation: Tourniquet Master Training.

Hemorrhages are the leading cause of potentially survivable combat mortalities when patients are unable to reach a treatment facility in time. New tourniquet devices have been developed to combat hemorrhages in the field. However, there is a lack in training systems to properly teach and assess users on tourniquet device application. We have developed an objective feedback system applicable to various full body manikins. We tested the system with expert users and received improvement feedback and verified the system's usefulness in instructing and assessing correct tourniquet device use.

Evaluation of Simulated Clinical Breast Exam Motion Patterns Using Marker-Less Video Tracking.

OBJECTIVE: This study investigates using marker-less video tracking to evaluate hands-on clinical skills during simulated clinical breast examinations (CBEs). BACKGROUND: There are currently no standardized and widely accepted CBE screening techniques. METHODS: Experienced physicians attending a national conference conducted simulated CBEs presenting different pathologies with distinct tumorous lesions. Single hand exam motion was recorded and analyzed using marker-less video tracking. Four kinematic measures were developed to describe temporal (time pressing and time searching) and spatial (area covered and distance explored) patterns. RESULTS: Mean differences between time pressing, area covered, and distance explored varied across the simulated lesions. Exams were objectively categorized as either sporadic, localized, thorough, or efficient for both temporal and spatial categories based on spatiotemporal characteristics. The majority of trials were temporally or spatially thorough (78% and 91%), exhibiting proportionally greater time pressing and time searching (temporally thorough) and greater area probed with greater distance explored (spatially thorough). More efficient exams exhibited proportionally more time pressing with less time searching (temporally efficient) and greater area probed with less distance explored (spatially efficient). Just two (5.9 %) of the trials exhibited both high temporal and spatial efficiency. CONCLUSIONS: Marker-less video tracking was used to discriminate different examination techniques and measure when an exam changes from general searching to specific probing. The majority of participants exhibited more thorough than efficient patterns. APPLICATION: Marker-less video kinematic tracking may be useful for quantifying clinical skills for training and assessment.

Development and evaluation of a simulation-based continuing medical education course: beyond lectures and credit hours.

BACKGROUND: The aim of our study was to modify our previously developed laparoscopic ventral hernia (LVH) simulator to increase difficulty and then reassess validity and feasibility for using the simulator in a newly developed simulation-based continuing medical education course. METHODS: Participants (N = 30) were practicing surgeons who signed up for a hands-on postgraduate laparoscopic hernia course. An LVH simulator, with prior validity evidence, was modified for the course to increase difficulty. Participants completed 1 of the 3 variations in hernia anatomy: incarcerated omentum, incarcerated bowel, and diffuse adhesions. During the procedure, course faculty and peer observers rated surgeon performance using Global Operative Assessment of Laparoscopic Skills-Incisional Hernia and Global Operative Assessment of Laparoscopic Skills rating scales with prior validity evidence. Rating scale reliability was reassessed for internal consistency. Peer and faculty raters' scores were compared. In addition, quality and completeness of the hernia repairs were rated. RESULTS: Internal consistency on the general skills performance (peer α = .96, faculty α = .94) and procedure-specific performance (peer α = .91, faculty α = .88) scores were high. Peers were more lenient than faculty raters on all LVH items in both the procedure-specific skills and general skills ratings. Overall, participants scored poorly on the quality and completeness of their hernia repairs (mean = 3.90/16, standard deviation = 2.72), suggesting a mismatch between course attendees and hernia difficulty and identifying a learning need. CONCLUSIONS: Simulation-based continuing medical education courses provide hands-on experiences that can positively affect clinical practice. Although our data appear to show a significant mismatch between clinical skill and simulator difficulty, these findings also underscore significant learning needs in the surgical community.

The use of error analysis to assess resident performance.

BACKGROUND: The aim of this study was to assess validity of a human factors error assessment method for evaluating resident performance during a simulated operative procedure. METHODS: Seven postgraduate year 4-5 residents had 30 minutes to complete a simulated laparoscopic ventral hernia (LVH) repair on day 1 of a national, advanced laparoscopic course. Faculty provided immediate feedback on operative errors and residents participated in a final product analysis of their repairs. Residents then received didactic and hands-on training regarding several advanced laparoscopic procedures during a lecture session and animate lab. On day 2, residents performed a nonequivalent LVH repair using a simulator. Three investigators reviewed and coded videos of the repairs using previously developed human error classification systems. RESULTS: Residents committed 121 total errors on day 1 compared with 146 on day 2. One of 7 residents successfully completed the LVH repair on day 1 compared with all 7 residents on day 2 (P = .001). The majority of errors (85%) committed on day 2 were technical and occurred during the last 2 steps of the procedure. There were significant differences in error type (P ≤ .001) and level (P = .019) from day 1 to day 2. The proportion of omission errors decreased from day 1 (33%) to day 2 (14%). In addition, there were more technical and commission errors on day 2. CONCLUSION: The error assessment tool was successful in categorizing performance errors, supporting known-groups validity evidence. Evaluating resident performance through error classification has great potential in facilitating our understanding of operative readiness.

Idle time: an underdeveloped performance metric for assessing surgical skill.

BACKGROUND: The aim of this study was to evaluate validity evidence using idle time as a performance measure in open surgical skills assessment. METHODS: This pilot study tested psychomotor planning skills of surgical attendings (n = 6), residents (n = 4) and medical students (n = 5) during suturing tasks of varying difficulty. Performance data were collected with a motion tracking system. Participants' hand movements were analyzed for idle time, total operative time, and path length. We hypothesized that there will be shorter idle times for more experienced individuals and on the easier tasks. RESULTS: A total of 365 idle periods were identified across all participants. Attendings had fewer idle periods during 3 specific procedure steps (P < .001). All participants had longer idle time on friable tissue (P < .005). CONCLUSIONS: Using an experimental model, idle time was found to correlate with experience and motor planning when operating on increasingly difficult tissue types. Further work exploring idle time as a valid psychomotor measure is warranted.

Use of decision-based simulations to assess resident readiness for operative independence.

BACKGROUND: Recent literature has called into question resident readiness for operative independence at the end of general surgery training. METHODS: We used a simulation-based exit examination to assess resident readiness. Six chief residents performed 3 simulated procedures: bowel anastomosis, laparoscopic ventral hernia (LVH) repair, and pancreaticojejunostomy. Faculty assessed resident performance using task-specific checklists, Objective Structured Assessment of Technical Skills (OSATS), and final product analysis. RESULTS: Residents' individual task-specific checklist scores ranged from 25% to 100% across all 3 procedures. Mean OSATS scores ranged from 4.06 to 4.23/5.0. Residents scored significantly higher on "instrument knowledge" (mean = 4.78, standard deviation [SD] = 23) than "time and motion" (mean = 3.94, SD = .48, P = .025) and "ability to adapt to individual pathologic circumstances" (mean = 4.06, SD =.12, P = .002). Final product analysis revealed a range of errors, including incorrect technique and poor intraoperative planning. CONCLUSIONS: Despite relatively high OSATS ratings, residents had a wide range of errors and procedure outcomes. Exit assessments using multiple evaluation metrics may improve awareness of residents' learning needs.

A comparative assessment and gap analysis of commonly used team rating scales.

BACKGROUND: The purpose of this article was to conduct a gap analysis of important team constructs that may be absent in widely used team assessments. METHODS AND MATERIALS: Two assessment tools with known validity evidence (1) Non-Technical Skills for Surgeons (NOTSS) and (2) the Cannon-Bowers Scale were used to evaluate 11 teams of surgical residents (n = 33) performing simulated laparoscopic hernia repairs. Faculty raters' scores were used to compare the surveys and assess validity and reliability. Raters' detailed observation notes were used to indicate important behavioral constructs that were missing from the team rating scales. RESULTS: When assessing inter-item correlations (reliability) four of five NOTSS' scale items had significant correlations (r = 0.9-1.0, P < 0.05) with the Cannon-Bowers items. While the correlations were only noted for three of six Cannon-Bowers items, in each instance the same four of five NOTSS items correlated with the three Cannon-Bowers items, thus providing further validity evidence for both scales. When evaluating the gap, key emerging themes included the need to focus on critical team errors, individual team member contributions, task performance, and overall team performance. These gaps, plus items from the NOTSS and Cannon-Bowers scales, were incorporated into a new rating scale. CONCLUSIONS: Despite continued evidence of validity and reliability, there were several behavioral constructs that were not represented when using the NOTSS and Cannon-Bowers scales. Critical team errors, individual team member contributions, task performance, and overall team performance appear important in our ability to understand teams and teamwork.

Modification of the pelvic examination simulator for the developing world.

The clinical pelvic exam is a critical examination for external and internal inspection of female reproductive organs. A sensor enabled pelvic examination simulator was developed to provide immediate visual performance feedback. The simulator was modified for rural area usage, where electricity supply and PC display may not be available. We succeeded at replacing key components while maintaining functionality.

Validity of a Newly Developed Tri-axial Sensor for Clinical Breast Examination Skills Assessment.

The clinical breast examination is a critical exam for early detection of breast cancer. Assessment methods are needed to determine competency and skill mastery for experts and novices. The 3D sensor was developed to capture hand shear and normal forces conducted during an exam. Trials were conducted to record exploratory maneuvers used during the exam. The sensor system was found to be a reliable method for capturing exploratory maneuvers.

Application of a new adaptable thyroid model for ultrasound and hands-on skill assessment.

The thyroid exam is an important clinical skill that is widely practiced by both primary care and specialty physicians. Despite this, there are few teaching models that allow for practice and assessment of proper examination technique. We developed a sensorized thyroid exam model and accompanying survey to capture clinical decision making and hands-on performance. Two endocrinologists performed the thyroid exam on the models. There was verbal consensus on clinical findings; however estimates on thyroid properties and size differed. Additional experiments with an ultrasound provided feedback on use of the model for hands-on and ultrasound-based examinations.

Multimodality approach to classifying hand utilization for the clinical breast examination.

The clinical breast examination (CBE) is performed to detect breast pathology. However, little is known regarding clinical technique and how it relates to diagnostic accuracy. We sought to quantify breast examination search patterns and hand utilization with a new data collection and analysis system. Participants performed the CBE while the sensor mapping and video camera system collected performance data. From this data, algorithms were developed that measured the number of hands used during the exam and active examination time. This system is a feasible and reliable method to collect new information on CBE techniques.

Sensor-based assessment of cast placement and removal.

Appropriate pressure during the application of a cast is critical to provide adequate stabilization of fractures. Force-sensing resistors (FSR) were used to measure pressure during cast placement and removal. The data demonstrated a signature pattern of skin pressure during the different steps of cast placement and removal. This reproducible signal provides validity evidence for our model.

Tourniquet Master Training for Junctional and Inguinal Hemorrhage Control (TMT).

Junctional and inguinal bleeding is a significant and challenging problem on the battlefield. Inventors have developed new types of tourniquets, including the Abdominal Aortic Tourniquet™ (AAT) and the Combat Ready Clamp™ (CRoC) to address these abdominal and pelvic injuries. While these hemorrhage control technologies have been developed, validated, and approved for use, training systems that teach and refresh skills related to these technologies have not been developed. These training systems are vital because these skills can be difficult to train and are infrequently used. To address these needs, a sensor-enabled manikin was designed. Using the sensor data, the different phases of applying the tourniquet were captured and key performance was measured.

Evaluation of Hands-On Clinical Exam Performance Using Marker-less Video Tracking.

This study investigates the potential of using marker-less video tracking of the hands for evaluating hands-on clinical skills. Experienced family practitioners attending a national conference were recruited and asked to conduct a breast examination on a simulator that simulates different clinical presentations. Videos were made of the clinician's hands during the exam and video processing software for tracking hand motion to quantify hand motion kinematics was used. Practitioner motion patterns indicated consistent behavior of participants across multiple pathologies. Different pathologies exhibited characteristic motion patterns in the aggregate at specific parts of an exam, indicating consistent inter-participant behavior. Marker-less video kinematic tracking therefore shows promise in discriminating between different examination procedures, clinicians, and pathologies.

Use of simulation to understand the effects of task complexity and time away on clinical confidence.

In prior studies, mannequin-based simulation training has been used to help decrease student anxiety toward intimate clinical examinations. Using time away as an independent variable, the aim of this study was to assess decay of clinical confidence for four procedural tasks that vary in procedural complexity. Clinical confidence with intimate examinations, after a standardized mannequin-based simulation curriculum, decays over time. This decay is noted after two months of time away. Longer periods of time away did not show increased differences.