Camera-Robot Calibration for the da Vinci® Robotic Surgery System.

The development of autonomous or semi-autonomous surgical robots stands to improve the performance of existing teleoperated equipment, but requires fine hand-eye calibration between the free-moving endoscopic camera and patient-side manipulator arms (PSMs). A novel method of solving this problem for the da Vinci® robotic surgical system and kinematically similar systems is presented. First, a series of image-processing and optical-tracking operations are performed to compute the coordinate transformation between the endoscopic camera view frame and an optical-tracking marker permanently affixed to the camera body. Then, the kinematic properties of the PSM are exploited to compute the coordinate transformation between the kinematic base frame of the PSM and an optical marker permanently affixed thereto. Using these transformations, it is then possible to compute the spatial relationship between the PSM and the endoscopic camera using only one tracker snapshot of the two markers. The effectiveness of this calibration is demonstrated by successfully guiding the PSM end effector to points of interest identified through the camera. Additional tests on a surgical task, namely grasping a surgical needle, are also performed to validate the proposed method. The resulting visually-guided robot positioning accuracy is better than the earlier hand-eye calibration results reported in the literature for the da Vinci® system, while supporting intraoperative update of the calibration and requiring only devices that are already commonly used in the surgical environment.

Real-Time Visual Tracking of Dynamic Surgical Suture Threads.

In order to realize many of the potential benefits associated with robotically assisted minimally invasive surgery, the robot must be more than a remote controlled device. Currently, using a surgical robot can be challenging, fatiguing, and time consuming. Teaching the robot to actively assist surgical tasks, such as suturing, has the potential to vastly improve both patient outlook and the surgeon's efficiency. One obstacle to completing surgical sutures autonomously is the difficulty in tracking surgical suture threads. This paper presents novel stereo image processing algorithms for the detection, initialization, and tracking of a surgical suture thread. A Non Uniform Rational B-Spline (NURBS) curve is used to model a thin, deformable, and dynamic length thread. The NURBS model is initialized and grown from a single selected point located on the thread. The NURBS curve is optimized by minimizing the image matching energy between the projected stereo NURBS image and the segmented thread image. The algorithms are evaluated using suture threads, a calibrated test pattern, and a simulated thread image. Additionally, the accuracy of the algorithms presented are validated as they track a suture thread undergoing translation, deformation, and apparent length changes. All of the tracking is in real-time. Note to Practioners: Abstract-The problem of tracking a surgical suture thread was addressed in this work. Since the suture thread is highly deformable, any tracking algorithm must be robust to intersections, occlusions, knot tying, and length changes. The detection algorithm introduced in this paper is capable of distinguishing different threads when they intersect. The tracking algorithm presented here demonstrate that it is possible, using polynomial curves, to track a suture thread as it deforms, becomes occluded, changes length, and even ties a knot in real time. The detection algorithm can enhance directional thin features while the polynomial curve modeling can track any string like structure. Further integration of the polynomial curve with a feed-forward thread model could improve the stability and robustness of the thread tracking.

Three-Dimensional Surgical Needle Localization and Tracking Using Stereo Endoscopic Image Streams.

This paper presents algorithms for three-dimensional tracking of surgical needles using the stereo endoscopic camera images obtained from the da Vinci ® Surgical Robotic System. The proposed method employs Bayesian state estimation, computer vision techniques, and robot kinematics. A virtual needle rendering procedure is implemented to create simulated images of the surgical needle under the da Vinci ® robot endoscope, which makes it possible to measure the similarity between the rendered needle image and the real needle. A particle filter algorithm using the mentioned techniques is then used for tracking the surgical needle. The performance of the tracking is experimentally evaluated using an actual da Vinci ® surgical robotic system and quantitatively validated in a ROS/Gazebo simulation thereof.

Analysis of Dynamic Response of an MRI-Guided Magnetically-Actuated Steerable Catheter System.

This paper presents a free-space open-loop dynamic response analysis for an MRI-guided magnetically-actuated steerable intra-vascular catheter system. The catheter tip is embedded with a set of current carrying micro-coils. The catheter is directly actuated via the magnetic torques generated on these coils by the magnetic field of the magnetic resonance imaging (MRI) scanner. The relationship between the input current commands and catheter tip deflection angle presents an inherent nonlinearity in the proposed catheter system. The system nonlinearity is analyzed by utilizing a pendulum model. The pendulum model is used to describe the system nonlinearity and to perform an approximate input-output linearization. Then, a black-box system identification approach is performed for frequency response analysis of the linearized dynamics. The optimal estimated model is reduced by observing the modes and considering the Nyquist frequency of the camera system that is used to track the catheter motion. The reduced model is experimentally validated with 3D open-loop Cartesian free-space trajectories. This study paves the way for effective and accurate free-space closed-loop control of the robotic catheter with real-time feedback from MRI guidance in subsequent research.

Catadioptric Stereo Tracking for Three Dimensional Shape Measurement of MRI Guided Catheters.

The recent introduction of Magnetic Resonance Imaging (MRI)-actuated steerable catheters lays the ground work for increasing the efficacy of cardiac catheter procedures. The MRI, while capable of imaging the catheter for tracking and control, does not fulfill all of the needs required to identify and develop a complete catheter model. Specifially, the frequency response of the catheter must be identified to ensure stable control of the catheter system. This requires a higher frequency imaging than the MRI can achieve. This work uses a catadioptric stereo camera system consisting of a mirror and a single camera in order to track a MRI actuated catheter inside a MRI machine. The catadioptric system works in parallel to the MRI and is capable of recording the catheter at 60 fps for post processing. The accuracy of the catadioptric system is verified in imaging conditions that would be found inside the MRI. The stereo camera is then used to track a catheter as it is actuated inside the MRI.

Needle-Tissue Interaction Force State Estimation for Robotic Surgical Suturing.

Robotically Assisted Minimally Invasive Surgery (RAMIS) offers many advantages over manual surgical techniques. Most of the limitations of RAMIS stem from its non-intuitive user interface and costs. One way to mitigate some of the limitations is to automate surgical subtasks (e.g. suturing) such that they are performed faster while allowing the surgeon to plan the next step of the procedure. One component of successful suture automation is minimizing the internal tissue deformation forces generated by driving a needle through tissue. Minimizing the internal tissue forces requires segmenting the tissue deformation forces from other components of the needle tissue interaction (e.g. friction force). This paper proposes an Unscented Kalman Filter which can successfully model the force components, in particular the internal deformation force, generated by a needle as it is driven through a sample of tissue.

Automatic Initialization and Dynamic Tracking of Surgical Suture Threads.

In order to realize many of the potential benefits associated with robotically assisted minimally invasive surgery, the robot must be more than a remote controlled device. Currently using a surgical robot can be challenging, fatiguing, and time consuming. Teaching the robot to actively assist surgical tasks, such as suturing, has the potential to vastly improve both patient outlook and the surgeon's efficiency. One obstacle to completing surgical sutures autonomously is the difficulty in tracking surgical suture threads. This paper proposes an algorithm which uses a Non-Uniform Rational B-Spline (NURBS) curve to model a suture thread. The NURBS model is initialized from a single selected point located on the thread. The NURBS curve is optimized by minimizing the image match energy between the projected stereo NURBS image and the segmented thread image. The algorithm is able to accurately track a suture thread as it translates, deforms, and changes length in real-time.

Needle Path Planning for Autonomous Robotic Surgical Suturing.

This paper develops a path plan for suture needles used with solid tissue volumes in endoscopic surgery. The path trajectory is based on the best practices that are used by surgeons. The path attempts to minimize the interaction forces between the tissue and the needle. Using surgical guides as a basis, two different techniques for driving a suture needle are developed. The two techniques are compared in hardware experiments by robotically driving the suture needle using both of the motion plans.

Modeling of Needle-Tissue Interaction Forces During Surgical Suturing.

This paper presents a model of needle tissue interaction forces that a rigid suture needle experiences during surgical suturing. The needle-tissue interaction forces are modeled as the sum of lumped parameters. The model has three main components; friction, tissue compression, and cutting forces. The tissue compression force uses the area that the needle sweeps out during a suture to estimate both the force magnitude and force direction. The area that the needle sweeps out is a direct result of driving the needle in a way that does not follow the natural curve of the needle. The friction force is approximated as a static friction force along the shaft of the needle. The cutting force acts only on the needle tip. The resulting force and torque model is experimentally validated using a tissue phantom. These results indicate that the proposed lumped parameter model is capable of accurately modeling the forces experienced during a suture.