Assessment of Open Surgery Suturing Skill: Image-based Metrics Using Computer Vision.

OBJECTIVE: This paper presents a computer vision algorithm for extraction of image-based metrics for suturing skill assessment and the corresponding results from an experimental study of resident and attending surgeons. DESIGN: A suturing simulator that adapts the radial suturing task from the Fundamentals of Vascular Surgery (FVS) skills assessment is used to collect data. The simulator includes a camera positioned under the suturing membrane, which records needle and thread movement during the suturing task. A computer vision algorithm processes the video data and extracts objective metrics inspired by expert surgeons' recommended best practice, to "follow the curvature of the needle." PARTICIPANTS AND RESULTS: Experimental data from a study involving subjects with various levels of suturing expertise (attending surgeons and surgery residents) are presented. Analysis shows that attendings and residents had statistically different performance on 6 of 9 image-based metrics, including the four new metrics introduced in this paper: Needle Tip Path Length, Needle Swept Area, Needle Tip Area and Needle Sway Length. CONCLUSION AND SIGNIFICANCE: These image-based process metrics may be represented graphically in a manner conducive to training. The results demonstrate the potential of image-based metrics for assessment and training of suturing skill in open surgery.

Measuring hand movement for suturing skill assessment: A simulation-based study.

BACKGROUND: To maximize patient safety, surgical skills education is increasingly adopting simulation-based curricula for formative skills assessment and training. However, many standardized assessment tools rely on human raters for performance assessment, which is resource-intensive and subjective. Simulators that provide automated and objective metrics from sensor data can address this limitation. We present an instrumented bench suturing simulator, patterned after the clock face radial suturing model from the Fundamentals of Vascular Surgery, for automated and objective assessment of open suturing skills. METHODS: For this study, 97 participants (35 attending surgeons, 32 residents, and 30 novices) were recruited at national vascular conferences. Automated hand motion metrics, especially focusing on rotational motion analysis, were developed from the inertial measurement unit attached to participants' hands, and the proposed suite of metrics was used to differentiate between the skill levels of the 3 groups. RESULTS: Attendings' and residents' performances were found to be significantly different from novices for all metrics. Moreover, most of our novel metrics could successfully distinguish between finer skill differences between attending and resident groups. In contrast, traditional operative skill metrics, such as time and path length, were unable to distinguish attendings from residents. CONCLUSION: This study provides evidence for the effectiveness of rotational motion analysis in assessing suturing skills. The suite of inertial measurement unit-based hand motion metrics introduced in this study allows for the incorporation of hand movement data for suturing skill assessment.

Cannulation Skill Assessment Using Functional Data Analysis.

OBJECTIVE: A clinician's operative skill-the ability to safely and effectively perform a procedure-directly impacts patient outcomes and well-being. Therefore, it is necessary to accurately assess skill progression during medical training as well as develop methods to most efficiently train healthcare professionals. METHODS: In this study, we explore whether time-series needle angle data recorded during cannulation on a simulator can be analyzed using functional data analysis methods to (1) identify skilled versus unskilled performance and (2) relate angle profiles to degree of success of the procedure. RESULTS: Our methods successfully differentiated between types of needle angle profiles. In addition, the identified profile types were associated with degrees of skilled and unskilled behavior of subjects. Furthermore, the types of variability in the dataset were analyzed, providing particular insight into the overall range of needle angles used as well as the rate of change of angle as cannulation progressed in time. Finally, cannulation angle profiles also demonstrated an observable correlation with degree of cannulation success, a metric that is closely related to clinical outcome. CONCLUSION: In summary, the methods presented here enable rich assessment of clinical skill since the functional (i.e., dynamic) nature of the data is duly considered.

Relating process and outcome metrics for meaningful and interpretable cannulation skill assessment: A machine learning paradigm.

BACKGROUND AND OBJECTIVES: The quality of healthcare delivery depends directly on the skills of clinicians. For patients on hemodialysis, medical errors or injuries caused during cannulation can lead to adverse outcomes, including potential death. To promote objective skill assessment and effective training, we present a machine learning approach, which utilizes a highly-sensorized cannulation simulator and a set of objective process and outcome metrics. METHODS: In this study, 52 clinicians were recruited to perform a set of pre-defined cannulation tasks on the simulator. Based on data collected by sensors during their task performance, the feature space was then constructed based on force, motion, and infrared sensor data. Following this, three machine learning models- support vector machine (SVM), support vector regression (SVR), and elastic net (EN)- were constructed to relate the feature space to objective outcome metrics. Our models utilize classification based on the conventional skill classification labels as well as a new method that represents skill on a continuum. RESULTS: With less than 5% of trials misplaced by two classes, the SVM model was effective in predicting skill based on the feature space. In addition, the SVR model effectively places both skill and outcome on a fine-grained continuum (versus discrete divisions) that is representative of reality. As importantly, the elastic net model enabled the identification of a set of process metrics that highly impact outcomes of the cannulation task, including smoothness of motion, needle angles, and pinch forces. CONCLUSIONS: The proposed cannulation simulator, paired with machine learning assessment, demonstrates definite advantages over current cannulation training practices. The methods presented here can be adopted to drastically increase the effectiveness of skill assessment and training, thereby potentially improving clinical outcomes of hemodialysis treatment.

Simulator-Based Metrics for Quantifying Vascular Palpation Skill for Cannulation.

Palpation is essential for accurate diagnosis and treatment in many clinical examinations and procedures. Specifically, vascular palpation is used to diagnose cardiovascular health issues and identify anatomical landmarks in the peripheral vascular system. However, little attention has been given to quantifying what comprises skilled vascular palpation; therefore, this study aims to objectively quantify the differences between high performer (HP), mid performer (MP), and low performer (LP) behavior towards understanding vascular palpation skills. Eleven HPs, twenty-five MPs, and ten LPs completed sixteen trials on our simulator under various conditions. There were four fistulas, two skin thicknesses, and two motor vibration intensities. Finger force and location data were recorded for each trial on the simulator. We examined three types of palpation metrics: time, force, and location. All three types of metrics demonstrated statistically significant differences between HP and LP palpation behavior. Therefore, these metrics could be used for structured and standardized palpation skills training in the future, potentially improving patient outcomes.

Assessment of open surgery suturing skill: Simulator platform, force-based, and motion-based metrics.

Objective: This paper focuses on simulator-based assessment of open surgery suturing skill. We introduce a new surgical simulator designed to collect synchronized force, motion, video and touch data during a radial suturing task adapted from the Fundamentals of Vascular Surgery (FVS) skill assessment. The synchronized data is analyzed to extract objective metrics for suturing skill assessment. Methods: The simulator has a camera positioned underneath the suturing membrane, enabling visual tracking of the needle during suturing. Needle tracking data enables extraction of meaningful metrics related to both the process and the product of the suturing task. To better simulate surgical conditions, the height of the system and the depth of the membrane are both adjustable. Metrics for assessment of suturing skill based on force/torque, motion, and physical contact are presented. Experimental data are presented from a study comparing attending surgeons and surgery residents. Results: Analysis shows force metrics (absolute maximum force/torque in z-direction), motion metrics (yaw, pitch, roll), physical contact metric, and image-enabled force metrics (orthogonal and tangential forces) are found to be statistically significant in differentiating suturing skill between attendings and residents. Conclusion and significance: The results suggest that this simulator and accompanying metrics could serve as a useful tool for assessing and teaching open surgery suturing skill.

Measuring Cannulation Skills for Hemodialysis: Objective Versus Subjective Assessment.

Lack of cannulation skill during hemodialysis treatments results in poor clinical outcomes due to infiltration and other cannulation-related trauma. Unfortunately, training of patient care technicians and nurses, specifically on the "technical" aspects of cannulation, has traditionally not received much attention. Simulators have been successfully deployed in many medical specialties for assessment and training of clinical skills. However, simulators have not been as widely used in nursing, especially in the context of training clinical personnel in the dialysis unit. We designed a state-of-the-art simulator for quantifying skill for hemodialysis cannulation. In this study, 52 nurses and patient care technicians with varying levels of clinical experience performed 16 cannulations on the simulator with different fistula properties. We formulated a composite metric for objectively measuring overall success of cannulation and compared this metric with subjective assessment by experts. In addition, we examined if years of clinical experience correlated with objective and subjective scores for cannulation skill. Results indicated that, while subjective and objective metrics generally correlated with each other, the objective metric was more precise and better suited for quantifying cannulation skill. Further, the simulator-based objective metric provides several advantages over subjective ratings, including providing fine-grained assessment of skill, consistency in measurement unaffected by subjective biases, and basing assessment on a more complete evaluation of performance. Years of clinical experience, however, demonstrated little correlation with either method of skill assessment. The methods presented for cannulation skill assessment in this study, if widely applied, could result in improved cannulation skill among our PCTs and nurses, which could positively impact patient outcomes in a tangible way.

Motion Smoothness Metrics for Cannulation Skill Assessment: What Factors Matter?

Medical training simulators have the potential to provide remote and automated assessment of skill vital for medical training. Consequently, there is a need to develop "smart" training devices with robust metrics that can quantify clinical skills for effective training and self-assessment. Recently, metrics that quantify motion smoothness such as log dimensionless jerk (LDLJ) and spectral arc length (SPARC) are increasingly being applied in medical simulators. However, two key questions remain about the efficacy of such metrics: how do these metrics relate to clinical skill, and how to best compute these metrics from sensor data and relate them with similar metrics? This study addresses these questions in the context of hemodialysis cannulation by enrolling 52 clinicians who performed cannulation in a simulated arteriovenous (AV) fistula. For clinical skill, results demonstrate that the objective outcome metric flash ratio (FR), developed to measure the quality of task completion, outperformed traditional skill indicator metrics (years of experience and global rating sheet scores). For computing motion smoothness metrics for skill assessment, we observed that the lowest amount of smoothing could result in unreliable metrics. Furthermore, the relative efficacy of motion smoothness metrics when compared with other process metrics in correlating with skill was similar for FR, the most accurate measure of skill. These results provide guidance for the computation and use of motion-based metrics for clinical skill assessment, including utilizing objective outcome metrics as ideal measures for quantifying skill.

Simulator-based hemodialysis cannulation skills training: a new horizon?

In accordance with the recently released Kidney Disease Outcomes Quality Initiative  (KDOQI) guidelines, there is a significant need for focused efforts on improving hemodialysis cannulation outcomes. Toward this, structured and meaningful training of our clinical personnel who cannulate in dialysis clinics is a priority. With the availability of advanced sensors and computing methods, simulators could be indispensable tools for standardized skills assessment and training. In this article we present ways in which sensor data could be used to quantify cannulation skill. As with many other medical specialties, implementation of simulator-based training holds the promise of much-needed improvement in end-stage kidney disease patient outcomes.

Is Experience in Hemodialysis Cannulation Related to Expertise? A Metrics-based Investigation for Skills Assessment.

Cannulation is not only one of the most common medical procedures but also fraught with complications. The skill of the clinician performing cannulation directly impacts cannulation outcomes. However, current methods of teaching this skill are deficient, relying on subjective demonstrations and unrealistic manikins that have limited utility for skills training. Furthermore, of the factors that hinders effective continuing medical education is the assumption that clinical experience results in expertise. In this work, we examine if objective metrics acquired from a novel cannulation simulator are able to distinguish between experienced clinicians and established experts, enabling the measurement of true expertise. Twenty-two healthcare professionals, who practiced cannulation with varying experience, performed a simulated arteriovenous fistula cannulation task on the simulator. Four clinicians were peer-identified as experts while the others were designated to the experienced group. The simulator tracked the motion of the needle (via an electromagnetic sensor), rendered blood flashback function (via an infrared light sensor), and recorded pinch forces exerted on the needle (via force sensing elements). Metrics were computed based on motion, force, and other sensor data. Results indicated that, with near 80% of accuracy using both logistic regression and linear discriminant analysis, the objective metrics differentiated between experts and the experienced, including identifying needle motion and finger force as two prominent features that distinguished between the groups. Furthermore, results indicated that expertise was not correlated with years of experience, validating the central hypothesis of the study. These insights contribute to structured and standardized medical skills training by enabling a meaningful definition of expertise and could potentially lead to more effective skills training methods.

A Method for Segmenting the Process of Needle Insertion during Simulated Cannulation using Sensor Data.

Cannulation is a routine yet challenging medical procedure resulting in a direct impact on patient outcomes. While current training programs provide guidelines to learn this complex procedure, the lack of objective and quantitative feedback impedes learning this skill more effectively. In this paper, we present a simulator for performing hemodialysis cannulation that captures the process using multiple sensing modalities that provide a multi-faceted assessment of cannulation. Further, we describe an algorithm towards segmenting the cannulation process using specific events in the sensor data for detailed analysis. Results from three participants with varying levels of clinical cannulation expertise are presented along with a metric that successfully differentiates the three participants. This work could lead to sensor-based cannulation skill assessment and training in the future potentially resulting in improved patient outcomes.

Examining the Effect of Haptic Factors for Vascular Palpation Skill Assessment Using an Affordable Simulator.

GOAL: Simulators that incorporate haptic feedback for clinical skills training are increasingly used in medical education. This study addresses the neglected aspect of rendering simulated feedback for vascular palpation skills training by systematically examining the effect of haptic factors on performance. METHODS: A simulator-based approach to examine palpation skill is presented. Novice participants with and without minimal previous palpation training performed a palpation task on a simulator that rendered controlled vibratory feedback under various conditions. RESULTS: Five objective metrics were employed to analyze participants' performance that yielded key findings in quantifying palpation performance. Participants' palpation accuracy was influenced by all three haptic factors, ranging from moderate to statistically significant. Duration, Total Path Length and Ration of Correct Movement also demonstrated utility for quantifying performance. CONCLUSIONS: We demonstrate that our affordable simulator is capable of rendering controlled haptic feedback suitable for skills training. Further, metrics presented in this study can be used for structured palpation skills assessment and training, potentially improving healthcare delivery.

Extracting Subtask-specific Metrics Toward Objective Assessment of Needle Insertion Skill for Hemodialysis Cannulation.

About 80% of all in-hospital patients require vascular access cannulation for treatments. However, there is a high rate of failure for vascular access cannulation, with several studies estimating up to a 50% failure rate for these procedures. Hemodialysis cannulation (HDC) is arguably one of the most difficult of these procedures with a steep learning curve and an extremely high failure rate. In light of this, there is a critical need that clinicians performing HDC have requisite skills. In this work, we present a method that combines the strengths of simulator-based objective skill quantification and task segmentation for needle insertion skill assessment at the subtask level. The results from our experimental study with seven novice nursing students on the cannulation simulator demonstrate that the simulator was able to segment needle insertion into subtask phases. In addition, most metrics were significantly different between the two phases, indicating that there may be value in evaluating participants' behavior at the subtask level. Further, the outcome metric (risk of infiltrating the simulated blood vessel) was successfully predicted by the process metrics in both phases. The implications of these results for skill assessment and training are discussed, which could potentially lead to improved patient outcomes if more extensive validation is pursued.

Surgical Suturing with Depth Constraints: Image-based Metrics to Assess Skill.

Suturing is one of the most fundamental surgical skills, requiring careful and systematic instruction for skilled performance. In this paper, we evaluate the performance of attending surgeons and surgical residents on an open surgery suturing task to examine if the introduction of different depth levels affects their performance. A vision algorithm is used to extract metrics meaningful in the assessment of suturing skill. As subjects perform a suturing task on the platform, our vision algorithm computes metrics identified to be potentially useful in assessing suturing skill: distances from optimal entry and optimal exit points, stitch length, stitch time, idle time, needle swept area, needle tip trace distance, needle tip area, and needle sway length. Preliminary experimental data from a study with 5 attending surgeons and 7 surgical residents are presented. Results demonstrate that the metrics of distance from optimal exit points, idle time, needle swept area, needle tip trace distance, needle tip area, and needle sway length are useful in quantifying the effect of depth constraints on suturing performance.

A Novel Platform for Assessment of Surgical Suturing Skill: Preliminary Results.

In this work, we describe a novel platform for quantifying surgical suturing skill. Forces and user movements are recorded using sensors during suturing maneuvers performed on a suture patch. Preliminary results from a pilot experiment suggest that force data could be used for objective assessment of suturing skill.

A Capacitance-Based Sensor for Hemodialysis Cannulation Training: A Proof-of-Concept Study.

In this work, we present a novel capacitance-based sensor for teaching hemodialysis cannulation skills. Initial results from a pilot experiment suggest that data from the sensor could be used to indicate puncture events in tissue-like materials.

Simulator-based assessment of haptic surgical skill: a comparative study.

The aim of this study was to examine if the forces applied by users of a haptic simulator could be used to distinguish expert surgeons from novices. Seven surgeons with significant operating room expertise and 9 novices with no surgical experience participated in this study. The experimental task comprised exploring 4 virtual materials with the haptic device and learning the precise forces required to compress the materials to various depths. The virtual materials differed in their stiffness and force-displacement profiles. The results revealed that for nonlinear virtual materials, surgeons applied significantly greater magnitudes of force than novices. Furthermore, for the softer nonlinear and linear materials, surgeons were significantly more accurate in reproducing forces than novices. The results of this study suggest that the magnitudes of force measured using haptic simulators may be used to objectively differentiate experts' haptic skill from that of novices. This knowledge can inform the design of virtual reality surgical simulators and lead to the future incorporation of haptic skills training in medical school curricula.

Endovascular seldinger needle placement: a simulator for examining haptic skills.

In this work, we develop an affordable haptic simulator for examining haptic skills required for endovascular Seldinger needle placement.

A perspective on the role and utility of haptic feedback in laparoscopic skills training.

Laparoscopic surgery is a minimally invasive surgical technique with significant potential benefits to the patient, including shorter recovery time, less scarring, and decreased costs. There is a growing need to teach surgical trainees this emerging surgical technique. Simulators, ranging from simple "box" trainers to complex virtual reality (VR) trainers, have emerged as the most promising method for teaching basic laparoscopic surgical skills. Current box trainers require oversight from an expert surgeon for both training and assessing skills. VR trainers decrease the dependence on expert teachers during training by providing objective, real-time feedback and automatic skills evaluation. However, current VR trainers generally have limited credibility as a means to prepare new surgeons and have often fallen short of educators' expectations. Several researchers have speculated that the missing component in modern VR trainers is haptic feedback, which refers to the range of touch sensations encountered during surgery. These force types and ranges need to be adequately rendered by simulators for a more complete training experience. This article presents a perspective of the role and utility of haptic feedback during laparoscopic surgery and laparoscopic skills training by detailing the ranges and types of haptic sensations felt by the operating surgeon, along with quantitative studies of how this feedback is used. Further, a number of research studies that have documented human performance effects as a result of the presence of haptic feedback are critically reviewed. Finally, key research directions in using haptic feedback for laparoscopy training simulators are identified.