Enhancing User Performance by Adaptively Changing Haptic Feedback Cues in a Fitts's Law Task.

Enhancing human user performance in some complex task is an important research question in many domains from skilled manufacturing to rehabilitation and surgical training. Many examples in the literature explore the effects of both haptic assistance or guidance to complete a task, as well as haptic hindrance to temporarily increase task difficulty for the ultimate goal of faster learning. Studies also suggest adaptively changing guidance based on expertise may be most effective. However, to our knowledge, there has not yet been a conclusive study evaluating these enhancement modes in a systematic experiment. In this article, we evaluate learning outcomes for 24 human subjects in a randomized control trial performing a Fitt's law reaching task under various haptic feedback conditions including: no haptics, assistive haptics, resistive haptics, and adaptively changing haptics tied to current performance measures. Subjects each performed 400 trials total and this paper reports results for 40 pre-test and 40 post-test trials. While most conditions did show improvements in performance, we found statistically significant results indicating that our adaptive haptic feedback condition leads to faster and more effective learning as evidenced by metrics of movement time, overshoot, performance index, and speed when compared to the other groups.

Frame-wise detection of surgeon stress levels during laparoscopic training using kinematic data.

PURPOSE: Excessive stress experienced by the surgeon can have a negative effect on the surgeon's technical skills. The goal of this study is to evaluate and validate a deep learning framework for real-time detection of stressed surgical movements using kinematic data. METHODS: 30 medical students were recruited as the subjects to perform a modified peg transfer task and were randomized into two groups, a control group (n=15) and a stressed group (n=15) that completed the task under deteriorating, simulated stressful conditions. To classify stressed movements, we first developed an attention-based Long-Short-Term-Memory recurrent neural network (LSTM) trained to classify normal/stressed trials and obtain the contribution of each data frame to the stress level classification. Next, we extracted the important frames from each trial and used another LSTM network to implement the frame-wise classification of normal and stressed movements. RESULTS: The classification between normal and stressed trials using attention-based LSTM model reached an overall accuracy of 75.86% under Leave-One-User-Out (LOUO) cross-validation. The second LSTM classifier was able to distinguish between the typical normal and stressed movement with an accuracy of 74.96% with an 8-second observation under LOUO. Finally, the normal and stressed movements in stressed trials could be classified with the accuracy of 68.18% with a 16-second observation under LOUO. CONCLUSION: In this study, we extracted the movements which are more likely to be affected by stress and validated the feasibility of using LSTM and kinematic data for frame-wise detection of stress level during laparoscopic training. The proposed classifier could be potentially be integrated with robot-assisted surgery platforms for stress management purposes.

Determining the Significant Kinematic Features for Characterizing Stress during Surgical Tasks Using Spatial Attention.

It has been shown that intraoperative stress can have a negative effect on surgeon surgical skills during laparoscopic procedures. For novice surgeons, stressful conditions can lead to significantly higher velocity, acceleration, and jerk of the surgical instrument tips, resulting in faster but less smooth movements. However, it is still not clear which of these kinematic features (velocity, acceleration, or jerk) is the best marker for identifying the normal and stressed conditions. Therefore, in order to find the most significant kinematic feature that is affected by intraoperative stress, we implemented a spatial attention-based Long-Short-Term-Memory (LSTM) classifier. In a prior IRB approved experiment, we collected data from medical students performing an extended peg transfer task who were randomized into a control group and a group performing the task under external psychological stresses. In our prior work, we obtained "representative" normal or stressed movements from this dataset using kinematic data as the input. In this study, a spatial attention mechanism is used to describe the contribution of each kinematic feature to the classification of normal/stressed movements. We tested our classifier under Leave-One-User-Out (LOUO) cross-validation, and the classifier reached an overall accuracy of 77.11% for classifying "representative" normal and stressed movements using kinematic features as the input. More importantly, we also studied the spatial attention extracted from the proposed classifier. Velocity and acceleration on both sides had significantly higher attention for classifying a normal movement (p <= 0.0001); Velocity (p <= 0.015) and jerk (p <= 0.001) on non-dominant hand had significant higher attention for classifying a stressed movement, and it is worthy noting that the attention of jerk on non-dominant hand side had the largest increment when moving from describing normal movements to stressed movements (p = 0.0000). In general, we found that the jerk on non-dominant hand side can be used for characterizing the stressed movements for novice surgeons more effectively.

Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery.

Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during RAS. We modify the original Transformer architecture to be able to generate the current gesture sequence, future gesture sequence, and future trajectory sequence estimations using only the current kinematic data of the surgical robot end-effectors. We evaluate our proposed models on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) and use Leave-One-User-Out (LOUO) cross validation to ensure generalizability of our results. Our models achieve up to 89.3% gesture recognition accuracy, 84.6% gesture prediction accuracy (1 second ahead) and 2.71mm trajectory prediction error (1 second ahead). Our models are comparable to and able to outperform state-of-the-art methods while using only the kinematic data channel. This approach can enabling near-real time surgical activity recognition and prediction.

Toward Correcting Anxious Movements Using Haptic Cues on the Da Vinci Surgical Robot.

Surgical movements have an important stylistic quality that individuals without formal surgical training can use to identify expertise. In our prior work, we sought to characterize quantitative metrics associated with surgical style and developed a near-real-time detection framework for stylistic deficiencies using a commercial haptic device. In this paper, we implement bimanual stylistic detection on the da Vinci Research Kit (dVRK) and focus on one stylistic deficiency, "Anxious", which may describe movements under stressful conditions. Our goal is to potentially correct these "Anxious" movements by exploring the effects of three different types of haptic cues (time-variant spring, damper, and spring-damper feedback) on performance during a basic surgical training task using the da Vinci Research Kit (dVRK). Eight subjects were recruited to complete peg transfer tasks using a randomized order of haptic cues and with baseline trials between each task. Overall, all cues lead to a significant improvement over baseline economy of volume and time-variant spring haptic cues lead to significant improvements in reducing the classified "Anxious" movements and also corresponded with significantly lower path length and economy of volume for the non-dominant hand. This work is the first step in evaluating our stylistic detection model on a surgical robot and could lay the groundwork for future methods to actively and adaptively reduce the negative effect of stress in the operating room.

Developing a novel force forecasting technique for early prediction of critical events in robotics.

Safety critical events in robotic applications can often be characterized by forces between the robot end-effector and the environment. One application in which safe interaction between the robot and environment is critical is in the area of medical robots. In this paper, we propose a novel Compact Form Dynamic Linearization Model-Free Prediction (CFDL-MFP) technique to predict future values of any time-series sensor data, such as interaction forces. Existing time series forecasting methods have high computational times which motivates the development of a novel technique. Using Autoregressive Integrated Moving Average (ARIMA) forecasting as benchmark, the performance of the proposed model was evaluated in terms of accuracy, computation efficiency, and stability on various force profiles. The proposed algorithm was 11% more accurate than ARIMA and maximum computation time of CFDL-MFP was 4ms, compared to ARIMA (7390ms). Furthermore, we evaluate the model in the special case of predicting needle buckling events, before they occur, by using only axial force and needle-tip position data. The model was evaluated experimentally for robustness with steerable needle insertions into different tissues including gelatin and biological tissue. For a needle insertion velocity of 2.5mm/s, the proposed algorithm was able to predict needle buckling 2.03s sooner than human detections. In biological tissue, no false positive or false negative buckling detections occurred and the rates were low in artificial tissue. The proposed forecasting model can be used to ensure safe robot interactions with delicate environments by predicting adverse force-based events before they occur.

Design and Evaluation of Haptic Constraints for Laparoscopic Instrument Handling.

Learning and mastering laparoscopic skills is an involved and complicated process, especially in the case of pediatric surgery due to extremely fragile tissues and small spaces. Given these constraints, precise, controlled, and gentle laparoscopic tool motions are essential. Proper handling and ergonomics of laparoscopic tool handling are often overlooked when training novice surgeons. Novice surgeons tend to overgrip the tool, which may lead to applying excessive amounts of force and potential surgical complications. We have developed two constraint mechanisms to enable proper tool handling: one mechanism is a passive kinematic constraint which physically prevents the user from over-gripping the tool. The second mechanism operates under dynamic resistive control. An elastic silicone membrane, secured by a hard plastic clip to the finger loop of the laparoscopic tool, actively resists the user's overgrip. These constraint devices were tested in a series of human subject studies with novice learners. The resulting data shows a both a significant reduction in over-grip and overall task completion time when using the passive constraint. The left index, right middle, and right ring fingers are shown to have the least amount of over-grip, as well as the lowest time of non-contact with the tool, indicating the importance of these fingers for laparoscopic tool control.

SATR-DL: Improving Surgical Skill Assessment And Task Recognition In Robot-Assisted Surgery With Deep Neural Networks.

PURPOSE: This paper focuses on an automated analysis of surgical motion profiles for objective skill assessment and task recognition in robot-assisted surgery. Existing techniques heavily rely on conventional statistic measures or shallow modelings based on hand-engineered features and gesture segmentation. Such developments require significant expert knowledge, are prone to errors, and are less efficient in online adaptive training systems. METHODS: In this work, we present an efficient analytic framework with a parallel deep learning architecture, SATR-DL, to assess trainee expertise and recognize surgical training activity. Through an end-to-end learning technique, abstract information of spatial representations and temporal dynamics is jointly obtained directly from raw motion sequences. RESULTS: By leveraging a shared highlevel representation learning, the resulting model is successful in the recognition of trainee skills and surgical tasks, suturing, needle-passing, and knot-tying. Meanwhile, we explore the use of ensemble in classification at the trial level, where the SATR-DL outperforms state-of-the-art performance by achieving accuracies of 0.960 and 1.000 in skill assessment and task recognition, respectively. CONCLUSION: This study highlights the potential of SATR-DL to provide improvements for an efficient data-driven assessment in intelligent robotic surgery.

Automatic Surgical Skill Rating Using Stylistic Behavior Components.

A gold standard in surgical skill rating and evaluation is direct observation, which a group of experts rate trainees based on a likert scale, by observing their performance during a surgical task. This method is time and resource intensive. To alleviate this burden, many studies have focused on automatic surgical skill assessment; however, the metrics suggested by the literature for automatic evaluation do not capture the stylistic behavior of the user. In addition very few studies focus on automatic rating of surgical skills based on available likert scales. In a previous study we presented a stylistic behavior lexicon for surgical skill. In this study we evaluate the lexicon's ability to automatically rate robotic surgical skill, based on the 6 domains in the Global Evaluative Assessment of Robotic Skills (GEARS). 14 subjects of different skill levels performed two surgical tasks on da Vinci surgical simulator. Different measurements were acquired as subjects performed the tasks, including limb (hand and arm) kinematics and joint (shoulder, elbow, wrist) positions. Posture videos of the subjects performing the task, as well as videos of the task being performed were viewed and rated by faculty experts based on the 6 domains in GEARS. The paired videos were also rated via crowd-sourcing based on our stylistic behavior lexicon. Two separate regression learner models, one using the sensor measurements and the other using crowd ratings for our proposed lexicon, were trained for each domain in GEARS. The results indicate that the scores predicted from both prediction models are in agreement with the gold standard faculty ratings.