Real-time vessel segmentation and reconstruction for virtual fixtures for an active handheld microneurosurgical instrument.

PURPOSE: Complications related to vascular damage such as intra-operative bleeding may be avoided during neurosurgical procedures such as petroclival meningioma surgery. To address this and improve the patient's safety, we designed a real-time blood vessel avoidance strategy that enables operation on deformable tissue during petroclival meningioma surgery using Micron, a handheld surgical robotic tool. METHODS: We integrated real-time intra-operative blood vessel segmentation of brain vasculature using deep learning, with a 3D reconstruction algorithm to obtain the vessel point cloud in real time. We then implemented a virtual-fixture-based strategy that prevented Micron's tooltip from entering a forbidden region around the vessel, thus avoiding damage to it. RESULTS: We achieved a median Dice similarity coefficient of 0.97, 0.86, 0.87 and 0.77 on datasets of phantom blood vessels, petrosal vein, internal carotid artery and superficial vessels, respectively. We conducted trials with deformable clay vessel phantoms, keeping the forbidden region 400 [Formula: see text]m outside and 400 [Formula: see text]m inside the vessel. Micron's tip entered the forbidden region with a median penetration of just 8.84 [Formula: see text]m and 9.63 [Formula: see text]m, compared to 148.74 [Formula: see text]m and 117.17 [Formula: see text]m without our strategy, for the former and latter trials, respectively. CONCLUSION: Real-time control of Micron was achieved at 33.3 fps. We achieved improvements in real-time segmentation of brain vasculature from intra-operative images and showed that our approach works even on non-stationary vessel phantoms. The results suggest that by enabling precise, real-time control, we are one step closer to using Micron in real neurosurgical procedures.

Monocular Vision-Based Retinal Membrane Peeling With a Handheld Robot.

Retinal membrane peeling requires delicate manipulation. The presence of the surgeon's physiological tremor, the high variability and often low quality of the ophthalmic image, and excessive forces make the tasks more challenging. Preventing unintended movement caused by tremor and unintentional forces can reduce membrane injury. With the use of an actively stabilized handheld robot, we employ a monocular camera-based surface reconstruction method to estimate the retinal plane and we propose the use of a virtual fixture with the application of a hard stop and motion scaling to improve control of the tool tip during delaminating in a laboratory simulation of retinal membrane peeling. A hard stop helps to limit downward force exerted on the surface. Motion scaling also improves the user's control of contact force when delaminating. We demonstrate a reduction of maximum force and maximum surface-penetration distance from the estimated retinal plane using the proposed technique.

Toward Monocular Camera-Guided Retinal Vein Cannulation with an Actively Stabilized Handheld Robot.

In this paper we describe work towards retinal vessel cannulation using an actively stabilized handheld robot, guided by monocular vision. We employ a previously developed monocular camera based surface reconstruction method using automated laser beam scanning over the retina. We use the reconstructed plane to find a coordinate transform between the 2D image plane coordinate system and the global 3D frame. Within a hemispherical region around the target, we use motion scaling for higher precision. The contribution of this work is the homography matrix estimation using monocular vision and application of the previously developed laser surface reconstruction to Micron guided vein cannulation. Experiments are conducted in a wet eye phantom to show the higher accuracy of the surface reconstruction as compared to standard stereo reconstruction. Further, experiments to show the increased surgical accuracy due to motion scaling are also carried out.

Multirate Kalman Filter Rejects Impulse Noise in Frequency-Domain-Multiplexed Tracker Measurements.

Frequency domain multiplexing (FDM) is a useful for making multiple measurements simultaneously, such as in optical and electromagnetic position trackers. Much interference is periodic (e.g., AC power harmonics), and is rejected well by FDM, but impulse disturbances are also common. Impulses corrupt the entire spectrum for a short period, and are better rejected in the time domain. Nonlinear blanking is a simple way to suppress impulses, but cannot be easily realized when the required dynamic range is large, and problematic noise may be far smaller than the signal. The described multi-rate Kalman filter upsamples the prediction to the input rate so that impulse departures from the predicted signal are easily detected and blanked out. Also, noise levels in unused adjacent channels are used to estimate measurement noise so that the Kalman filter adapts more slowly when noise is high, keeping output noise roughly constant even in the presence of longer noise bursts.

Comparative Evaluation of Handheld Robot-Aided Intraocular Laser Surgery.

This paper presents robot-aided intraocular laser surgery using a handheld robot known as Micron. The micromanipulator incorporated in Micron enables visual servoing of a laser probe, while maintaining a constant distance of the tool tip from the retinal surface. The comparative study was conducted with various control methods for evaluation of robot-aided intraocular laser surgery.

Hybrid position/force control of an active handheld micromanipulator for membrane peeling.

BACKGROUND: Peeling procedures in retinal surgery require micron-scale manipulation and control of sub-tactile forces. METHODS: Hybrid position/force control of an actuated handheld microsurgical instrument is presented as a means for simultaneously improving positioning accuracy and reducing forces to prevent avoidable trauma to tissue. The system response was evaluated, and membrane-peeling trials were performed by four test subjects in both artificial and animal models. RESULTS: Maximum force was reduced by 56% in both models compared with position control. No statistically significant effect on procedure duration was observed. CONCLUSIONS: A hybrid position/force control system has been implemented that successfully attenuates forces and minimizes unwanted excursions during microsurgical procedures such as membrane peeling. Results also suggest that improvements in safety using this technique may be attained without increasing the duration of the procedure.

Electromagnetic Tracker for Active Handheld Robotic Systems.

We describe the development of the In-Loop Electromagnetic Tracker (ILEMT), designed to meet the demanding latency and resolution requirements for active stabilization of hand motion during precision manipulations such as microsurgery. The prototype surpasses the fastest commercial EM trackers by > 4× in root bandwidth/resolution and 2× in latency. The use of two widely spaced carrier frequencies (e.g., 300 Hz and 10 kHz) enables a particularly simple way of reducing the eddy-current interference caused by nonferrous metals present in the workspace. Previously, metal compatibility has only been achieved at a large cost to measurement speed.

Manipulator Design and Operation for a Six-Degree-of-Freedom Handheld Tremor-Canceling Microsurgical Instrument.

This paper presents the design and actuation of a six-degree-of-freedom (6-DOF) manipulator for a handheld instrument, known as "Micron," which performs active tremor compensation during microsurgery. The design incorporates a Gough-Stewart platform based on piezoelectric linear motor, with a specified minimum workspace of a cylinder 4 mm long and 4 mm in diameter at the end-effector. Given the stall force of the motors and the loading typically encountered in vitreoretinal microsurgery, the dimensions of the manipulator are optimized to tolerate a transverse load of 0.2 N on a remote center of motion near the midpoint of the tool shaft. The optimization yields a base diameter of 23 mm and a height of 37 mm. The fully handheld instrument includes a custom-built optical tracking system for control feedback, and an ergonomic housing to serve as a handle. The manipulation performance was investigated in both clamped and handheld conditions. In positioning experiments with varying side loads, the manipulator tolerates side load up to 0.25 N while tracking a sinusoidal target trajectory with less than 20 µm error. Physiological hand tremor is reduced by about 90% in a pointing task, and error less than 25 µm is achieved in handheld circle-tracing.

Toward Hybrid Position/Force Control for an Active Handheld Micromanipulator.

Vitreoretinal microsurgery requires precise hand-eye coordination to manipulate delicate structures within the eye on the order of tens of microns. To achieve these tasks, surgeons use tools of diameter 0.9 mm or less to access the eye's interior structures. The level of force required during these manipulations is often below the human tactile threshold, requiring the surgeon to rely on subtle visual cues or to apply larger forces above the tactile threshold for feedback. However, both of these methods can lead to tissue damage. Excursions can be made into tissues which are not felt by the surgeon, while larger forces have a higher chance of damaging tissue within the eye. To prevent damage to the retina and other anatomy, we present the implementation of hybrid position/force control operating in the sub-tactile force range for a handheld robotic system. This approach resulted in a 42% reduction in the mean force and 52% reduction in maximum force during peeling tasks.

Toward Automated Intraocular Laser Surgery Using a Handheld Micromanipulator.

This paper presents a technique for automated intraocular laser surgery using a handheld micromanipulator known as Micron. The novel handheld manipulator enables the automated scanning of a laser probe within a cylinder of 4 mm long and 4 mm in diameter. For the automation, the surface of the retina is reconstructed using a stereomicroscope, and then preplanned targets are placed on the surface. The laser probe is precisely located on the target via visual servoing of the aiming beam, while maintaining a specific distance above the surface. In addition, the system is capable of tracking the surface of the eye in order to compensate for any eye movement introduced during the operation. We compared the performance of the automated scanning using various control thresholds, in order to find the most effective threshold in terms of accuracy and speed. Given the selected threshold, we conducted the handheld operation above a fixed target surface. The average error and execution time are reduced by 63.6% and 28.5%, respectively, compared to the unaided trials. Finally, the automated laser photocoagulation was demonstrated also in an eye phantom, including compensation for the eye movement.

Performance of a 6-degree-of-freedom active microsurgical manipulator in handheld tasks.

This paper presents the first experimental results from human users of a new 6-degree-of-freedom handheld micromanipulator. This is the latest prototype of a fully-handheld system, known as "Micron," which performs active compensation of hand tremor for microsurgery. The manipulator is a miniature Gough-Stewart platform incorporating linear ultrasonic motors that provide a cylindrical workspace 4 mm long and 4 mm wide. In addition, the platform allows the possibility of imposing a remote center of motion for controlling motion not only at the tip but also at the entry point in the sclera of the eye. We demonstrate hand tremor reduction in both static and dynamic micromanipulation tasks on a rubber pad. The handheld performance is also evaluated in an artificial eye model while imposing a remote center of motion. In all cases, hand tremor is significantly reduced.

Improvement of optical coherence tomography using active handheld micromanipulator in vitreoretinal surgery.

An active handheld micromanipulator has been developed to cancel hand tremor during microsurgery. The micromanipulator is also applicable in optical coherence tomography to improve the quality of scanning and minimize surgical risks during the scans. The manipulator can maneuver the tool tip with six degrees of freedom within a cylindrical workspace 4 mm in diameter and 4 mm high. The imaging system is equipped with a 25-gauge Fourier-domain common-path OCT probe. This paper introduces the handheld OCT imaging system and techniques involved and presents stabilized OCT images of A-mode and M-mode scans in air and live rabbit eyes. We show the first demonstration of OCT imaging using the active handheld micromanipulator in vivo.

Comparison of Baseline Tremor Under Various Microsurgical Conditions.

This paper presents the characterization and comparison of physiological tremor for pointing tasks in multiple environments, as a baseline for performance evaluation of microsurgical robotics. Previous studies have examined the characteristics of physiological tremor under laboratory settings as well as different operating conditions. However, different test methods make the comparison of results across trials and conditions difficult. Two vitroretinal microsurgeons were evaluated while performing a pointing task with no entry-point constraint, constrained by an artificial eye model, and constrained by a rabbit eye in vivo. For the three respective conditions the 3D RMS positioning error was 144 µm, 258 µm, and 285 µm, and maximum 3D error was 349 µm, 647 µm, and 696 µm. A spectral analysis was also performed, confirming a distinct peak near in the 6-12 Hz frequency range, characteristic of hand tremor during tasks in all three environments.

Vision-Based Control of a Handheld Surgical Micromanipulator with Virtual Fixtures.

Performing micromanipulation and delicate operations in submillimeter workspaces is difficult because of destabilizing tremor and imprecise targeting. Accurate micromanipulation is especially important for microsurgical procedures, such as vitreoretinal surgery, to maximize successful outcomes and minimize collateral damage. Robotic aid combined with filtering techniques that suppress tremor frequency bands increases performance; however, if knowledge of the operator's goals is available, virtual fixtures have been shown to further improve performance. In this paper, we derive a virtual fixture framework for active handheld micromanipulators that is based on high-bandwidth position measurements rather than forces applied to a robot handle. For applicability in surgical environments, the fixtures are generated in real-time from microscope video during the procedure. Additionally, we develop motion scaling behavior around virtual fixtures as a simple and direct extension to the proposed framework. We demonstrate that virtual fixtures significantly outperform tremor cancellation algorithms on a set of synthetic tracing tasks (p < 0.05). In more medically relevant experiments of vein tracing and membrane peeling in eye phantoms, virtual fixtures can significantly reduce both positioning error and forces applied to tissue (p < 0.05).

Micron: an Actively Stabilized Handheld Tool for Microsurgery.

We describe the design and performance of a hand-held actively stabilized tool to increase accuracy in micro-surgery or other precision manipulation. It removes involuntary motion such as tremor by actuating the tip to counteract the effect of the undesired handle motion. The key components are a three-degree-of-freedom piezoelectric manipulator that has 400 µm range of motion, 1 N force capability, and bandwidth over 100 Hz, and an optical position measurement subsystem that acquires the tool pose with 4 µm resolution at 2000 samples/s. A control system using these components attenuates hand motion by at least 15 dB (a fivefold reduction). By considering the effect of the frequency response of Micron on the human visual feedback loop, we have developed a filter that reduces unintentional motion, yet preserves intuitive eye-hand coordination. We evaluated the effectiveness of Micron by measuring the accuracy of the human/machine system in three simple manipulation tasks. Handheld testing by three eye surgeons and three non-surgeons showed a reduction in position error of between 32% and 52%, depending on the error metric.

Optical coherence tomography scanning with a handheld vitreoretinal micromanipulator.

An active handheld micromanipulator has been developed that is capable of automated intraocular acquisition of B-mode and C-mode optical coherence tomography scans that are up to 4 mm wide. The manipulator is a handheld Gough-Stewart platform actuated by ultrasonic linear motors. The manipulator has been equipped with a Fourier-domain common-path intraocular OCT probe that fits within a 25-gauge needle. The paper describes the systems and techniques involved, and presents preliminary results of B-mode and C-mode scans.

Design and Analysis of 6 DOF Handheld Micromanipulator.

This paper presents the design and analysis of a handheld manipulator for vitreoretinal microsurgery and other biomedical applications. The design involves a parallel micromanipulator utilizing six piezoelectric linear actuators, combining compactness with a large range of motion and relatively high stiffness. Given the available force of the actuators, the overall dimension of the micromanipulator was optimized considering realistic external loads on a remote center of motion representing the point of expected contact with the sclera of the eye during microsurgery. Based on optimization and workspace analysis, a benchtop version of the micromanipulator was built with a base diameter of 25 mm and a height of 50 mm. It provides a hemispherical workspace of 4.0 mm diameter at the tool tip. The manipulation performance of the constructed manipulator was measured under a lateral load applied at the remote center of motion. The micromanipulator tolerated side loads up to 200 mN.

Towards Vision-Based Control of a Handheld Micromanipulator for Retinal Cannulation in an Eyeball Phantom.

Injecting clot-busting drugs such as t-PA into tiny vessels thinner than a human hair in the eye is a challenging procedure, especially since the vessels lie directly on top of the delicate and easily damaged retina. Various robotic aids have been proposed with the goal of increasing safety by removing tremor and increasing precision with motion scaling. We have developed a fully handheld micromanipulator, Micron, that has demonstrated reduced tremor when cannulating porcine retinal veins in an "open sky" scenario. In this paper, we present work towards handheld robotic cannulation with the goal of vision-based virtual fixtures guiding the tip of the cannula to the vessel. Using a realistic eyeball phantom, we address sclerotomy constraints, eye movement, and non-planar retina. Preliminary results indicate a handheld micromanipulator aided by visual control is a promising solution to retinal vessel occlusion.

Position-Based Virtual Fixtures for Membrane Peeling with a Handheld Micromanipulator.

Peeling delicate retinal membranes, which are often less than 5 µm thick, is one of the most challenging retinal surgeries. Preventing rips and tears caused by tremor and excessive force can decrease injury and reduce the need for follow up surgeries. We propose the use of a fully handheld microsurgical robot to suppress tremor while enforcing helpful constraints on the motion of the tool. Using stereo vision and tracking algorithms, the robot activates motion-scaled behavior as the tip reaches the surface, providing finer control during the critical step of engaging the membrane edge. A hard virtual fixture just below the surface limits the total downward force that can be applied. Furthermore, velocity limiting during the peeling helps the surgeon maintain a smooth, constant force while lifting and delaminating the membrane. On a phantom consisting of plastic wrap stretched across a rubber slide, we demonstrate our approach reduces maximum force by 40-70%.

Handheld Micromanipulation with Vision-Based Virtual Fixtures.

Precise movement during micromanipulation becomes difficult in submillimeter workspaces, largely due to the destabilizing influence of tremor. Robotic aid combined with filtering techniques that suppress tremor frequency bands increases performance; however, if knowledge of the operator's goals is available, virtual fixtures have been shown to greatly improve micromanipulator precision. In this paper, we derive a control law for position-based virtual fixtures within the framework of an active handheld micromanipulator, where the fixtures are generated in real-time from microscope video. Additionally, we develop motion scaling behavior centered on virtual fixtures as a simple and direct extension to our formulation. We demonstrate that hard and soft (motion-scaled) virtual fixtures outperform state-of-the-art tremor cancellation performance on a set of artificial but medically relevant tasks: holding, move-and-hold, curve tracing, and volume restriction.