Stochastic Force-based Insertion Depth and Tip Position Estimations of Flexible FBG-Equipped Instruments in Robotic Retinal Surgery.

Vitreoretinal surgery is among the most delicate surgical tasks during which surgeon hand tremor may severely attenuate surgeon performance. Robotic assistance has been demonstrated to be beneficial in diminishing hand tremor. Among the requirements for reliable assistance from the robot is to provide precise measurements of system states e.g. sclera forces, tool tip position and tool insertion depth. Providing this and other sensing information using existing technology would contribute towards development and implementation of autonomous robot-assisted tasks in retinal surgery such as laser ablation, guided suture placement/assisted needle vessel cannulation, among other applications. In the present work, we use a state-estimating Kalman filtering (KF) to improve the tool tip position and insertion depth estimates, which used to be purely obtained by robot forward kinematics (FWK) and direct sensor measurements, respectively. To improve tool tip localization, in addition to robot FWK, we also use sclera force measurements along with beam theory to account for tool deflection. For insertion depth, the robot FWK is combined with sensor measurements for the cases where sensor measurements are not reliable enough. The improved tool tip position and insertion depth measurements are validated using a stereo camera system through preliminary experiments and a case study. The results indicate that the tool tip position and insertion depth measurements are significantly improved by 77% and 94% after applying KF, respectively.

ExteriorTag: Automatic Semantic Annotation of BIM Building Exterior Via Voxel Index Analysis.

The growing demand for building information modeling (BIM) data and ubiquitous applications make it increasingly necessary to establish a reliable way to share the models on lightweight devices. Building scenes have strong occlusion features and the building exterior plays an important role in digital devices with limited computational resources. This allows the possibility to reduce the resource consumption while roaming in outdoor scenes by culling away the interior building data. This article addresses the task of automatic annotation of BIM building exterior via voxel index analysis. We showcase the research of using industry foundation classes (IFC) and other mainstream formats as our input data and proposed an automatic algorithm for annotating the building exterior. Afterward, a practical and accurate voxel index analysis procedure is designed for frequently flawed models. The annotation can be added directly into the original data file under the same IFC standard, avoiding the complex procedure and information loss in semantics mapping between different standards. The final examinations show the robustness of our algorithm and the capability of handling large BIM building models.

A study for lens capsule tearing during capsulotomy by finite element simulation.

BACKGROUND AND OBJECTIVE: During capsulotomy, the force applied to the anterior capsule is a crucial parameter controlling capsule tears, that affects the clinical performance. This study aims to investigate the tear force in capsulotomy and analyze the effects of different tearing conditions on the tear force. METHODS: A three-dimensional model of the human lens was constructed based on published clinical data using the finite element (FE) method. The lens model consisted of four layers: the anterior and posterior lens capsule, the cortex, and the nucleus. Distortion energy failure criterion combined with the bilinear interface law was used to express the crack propagation process at the edge of the anterior lens capsule. At the clamping position, a local coordinate system was established to parameterize the capsule tearing. The simulation results were then validated by conducting a capsulorhexis experiment using isolated porcine eyes with force-sensing forceps. RESULTS: The simulation results showed a good agreement with the experimental data of two porcine specimens (No. 6 and 9) during a stable tearing process (p-values = 0.76 and 0.10). The mean force differences between the experimental data and the simulation were 3.10 ± 2.24 mN and 2.14 ± 1.73 mN, respectively. The tear direction with a minimum mean tear force was at θ1 = 0° and θ2 = 30°. The tear velocity was not significantly different to the variation in the tear force. However, an appropriate capsulorhexis diameter was found to contribute to the reduction of tear force. CONCLUSIONS: The outcome of this paper demonstrates that our FE model could be used in modeling lens capsule tearing and the theoretical study of tear mechanism.

[Design and Implementation of User-oriented Auxiliary Treatment Instrument for Meibomian Gland Dysfunction].

Dry eye is a common ophthalmic disease caused by eye maladjustment due to meibomian gland dysfunction (MGD), which is often accompanied by symptoms such as increased tear film osmotic pressure and ocular surface inflammation. In the treatment of dry eye patients, dredging gland obstruction caused by meibomian gland secretion is an effective treatment method. Based on electrothermal effect and hyperelasticity of the silicone, an auxiliary treatment instrument for MGD is designed, which can improve the blood circulation of the glands through heat compress and massage to achieve the purpose of dredging the meibomian glands. The therapy device can display the temperature and pressure during the treatment in real time, so that the surgeon can grasp the progress of the treatment in real time. The therapy device constructs a user-oriented interactive interface based on parametric modeling method, which can be customized by 3D printing according to the user's eyeball geometric parameters. The designed therapeutic device was finally tested on New Zealand white rabbits. The experimental results show that the therapeutic device has significant effectiveness and safety, as well as clinical application prospects.

Automatic Light Pipe Actuating System for Bimanual Robot-Assisted Retinal Surgery.

Retinal surgery is a bimanual operation in which surgeons operate with an instrument in their dominant hand (more capable hand) and simultaneously hold a light pipe (illuminating pipe) with their non-dominant hand (less capable hand) to provide illumination inside the eye. Manually holding and adjusting the light pipe places an additional burden on the surgeon and increases the overall complexity of the procedure. To overcome these challenges, a robot-assisted automatic light pipe actuating system is proposed. A customized light pipe with force-sensing capability is mounted at the end effector of a follower robot and is actuated through a hybrid force-velocity controller to automatically illuminate the target area on the retinal surface by pivoting about the scleral port (incision on the sclera). Static following-accuracy evaluation and dynamic light tracking experiments are carried out. The results show that the proposed system can successfully illuminate the desired area with negligible offset (the average offset is 2.45 mm with standard deviation of 1.33 mm). The average scleral forces are also below a specified threshold (50 mN). The proposed system not only can allow for increased focus on dominant hand instrument control, but also could be extended to three-arm procedures (two surgical instruments held by surgeon plus a robot-holding light pipe) in retinal surgery, potentially improving surgical efficiency and outcome.

A Comparison of Manual and Robot Assisted Retinal Vein Cannulation in Chicken Chorioallantoic Membrane.

Retinal vein occlusion (RVO) is a vision threatening condition occurring in the central or the branch retinal veins. Risk factors include but are not limited to hypercoagulability, thrombus or other cause of low blood flow. Current clinically proven treatment options limit complications of vein occlusion without treating the causative occlusion. In recent years, a more direct approach called Retinal Vein Cannulation (RVC) has been explored both in animal and human eye models. Though RVC has demonstrated potential efficacy, it remains a challenging and risky procedure that demands precise needle manipulation to achieve safely. During RVC, a thin cannula (diameter 70-110 µm) is delicately inserted into a retinal vein. Its intraluminal position is maintained for up to 2 minutes while infusion of a therapeutic drug occurs. Because the tool-tissue interaction forces at the needle tip are well below human tactile perception, a robotic assistant combined with a force sensing microneedle could alleviate the challenges of RVC. In this paper we present a comparative study of manual and robot assisted retinal vein cannulation in chicken chorioallantoic membrane (CAM) using a force sensing microneedle tool. The results indicate that the average puncture force and average force during the infusion period are larger in manual mode than in robot assisted mode. Moreover, retinal vein cannulation was more stable during infusion, in robot assisted mode.

Real Time Prediction of Sclera Force with LSTM Neural Networks in Robot-Assisted Retinal Surgery.

Retinal microsurgery is one of the most technically demanding surgeries, during which the surgical tool needs to be inserted into the eyeball and is constantly constrained by the sclerotomy port. During the surgery, any unexpected manipulation could cause extreme tool-sclera contact force leading to sclera damage. Although, a robot assistant could reduce hand tremor and improve the tool positioning accuracy, it cannot prevent or alarm the surgeon about the upcoming danger caused by surgeon's misoperations, i.e., applying excessive force on the sclera. In this paper, we present a new method based on a Long Short Term Memory recurrent neural network for predicting the user behavior, i.e., the contact force between the tool and sclera (sclera force) and the insertion depth of the tool from sclera contact point (insertion depth) in real time (40Hz). The predicted force information is provided to the user through auditory feedback to alarm any unexpected sclera force. The user behavior data is collected in a mock retinal surgical operation on a dry eye phantom with Steady Hand Eye Robot and a novel multi-function sensing tool. The Long Short Term Memory recurrent neural network is trained on the collected time series of sclera force and insertion depth. The network can predict the sclera force and insertion depth 100 milliseconds in the future with 95.29% and 96.57% accuracy, respectively, and can help reduce the fraction of unsafe sclera forces from 40.19% to 15.43%.

Spotlight-based 3D Instrument Guidance for Retinal Surgery.

Retinal surgery is a complex activity that can be challenging for a surgeon to perform effectively and safely. Image guided robot-assisted surgery is one of the promising solutions that bring significant surgical enhancement in treatment outcome and reduce the physical limitations of human surgeons. In this paper, we demonstrate a novel method for 3D guidance of the instrument based on the projection of spotlight in the single microscope images. The spotlight projection mechanism is firstly analyzed and modeled with a projection on both a plane and a sphere surface. To test the feasibility of the proposed method, a light fiber is integrated into the instrument which is driven by the Steady-Hand Eye Robot (SHER). The spot of light is segmented and tracked on a phantom retina using the proposed algorithm. The static calibration and dynamic test results both show that the proposed method can easily archive 0.5 mm of tip-to-surface distance which is within the clinically acceptable accuracy for intraocular visual guidance.

Force-based Safe Vein Cannulation in Robot-assisted Retinal Surgery: A Preliminary Study.

Retinal vein cannulation (RVC) is a potential treatment for retinal vein occlusion (RVO). Manual surgery has limitations in RVC due to extremely small vessels and instruments involved, as well as the presence of physiological hand tremor. Robot-assisted retinal surgery may be a better approach to smooth and accurate instrument manipulation during this procedure. Motion of the retina and cornea related to heartbeat may be associated with unexpected forces between the tool and eyeball. In this paper, we propose a force-based control strategy to automatically compensate for the movement of the retina maintaining the tip force and sclera force in a predetermined small range. A dual force-sensing tool is used to monitor the tip force, sclera force and tool insertion depth, which will be used to derive a desired joint velocity for the robot via a modified admittance controller. Then the tool is manipulated to compensate for the movement of the retina as well as reduce the tip force and sclera force. Quantitative experiments are conducted to verify the efficacy of the control strategy and a user study is also conducted by a retinal surgeon to demonstrate the advantages of our automatic compensation approach.

Autonomously Navigating a Surgical Tool Inside the Eye by Learning from Demonstration.

A fundamental challenge in retinal surgery is safely navigating a surgical tool to a desired goal position on the retinal surface while avoiding damage to surrounding tissues, a procedure that typically requires tens-of-microns accuracy. In practice, the surgeon relies on depth-estimation skills to localize the tool-tip with respect to the retina in order to perform the tool-navigation task, which can be prone to human error. To alleviate such uncertainty, prior work has introduced ways to assist the surgeon by estimating the tool-tip distance to the retina and providing haptic or auditory feedback. However, automating the tool-navigation task itself remains unsolved and largely unexplored. Such a capability, if reliably automated, could serve as a building block to streamline complex procedures and reduce the chance for tissue damage. Towards this end, we propose to automate the tool-navigation task by learning to mimic expert demonstrations of the task. Specifically, a deep network is trained to imitate expert trajectories toward various locations on the retina based on recorded visual servoing to a given goal specified by the user. The proposed autonomous navigation system is evaluated in simulation and in physical experiments using a silicone eye phantom. We show that the network can reliably navigate a needle surgical tool to various desired locations within 137 µm accuracy in physical experiments and 94 µm in simulation on average, and generalizes well to unseen situations such as in the presence of auxiliary surgical tools, variable eye backgrounds, and brightness conditions.

Towards Bimanual Vein Cannulation: Preliminary Study of a Bimanual Robotic System With a Dual Force Constraint Controller.

Retinal vein cannulation is a promising approach for treating retinal vein occlusion that involves injecting medicine into the occluded vessel to dissolve the clot. The approach remains largely unexploited clinically due to surgeon limitations in detecting interaction forces between surgical tools and retinal tissue. In this paper, a dual force constraint controller for robot-assisted retinal surgery was presented to keep the tool-to-vessel forces and tool-to-sclera forces below prescribed thresholds. A cannulation tool and forceps with dual force-sensing capability were developed and used to measure force information fed into the robot controller, which was implemented on existing Steady Hand Eye Robot platforms. The robotic system facilitates retinal vein cannulation by allowing a user to grasp the target vessel with the forceps and then enter the vessel with the cannula. The system was evaluated on an eye phantom. The results showed that, while the eyeball was subjected to rotational disturbances, the proposed controller actuates the robotic manipulators to maintain the average tool-to-vessel force at 10.9 mN and 13.1 mN and the average tool-to-sclera force at 38.1 mN and 41.2 mN for the cannula and the forcpes, respectively. Such small tool-to-tissue forces are acceptable to avoid retinal tissue injury. Additionally, two clinicians participated in a preliminary user study of the bimanual cannulation demonstrating that the operation time and tool-to-tissue forces are significantly decreased when using the bimanual robotic system as compared to freehand performance.

Toward Safe Retinal Microsurgery: Development and Evaluation of an RNN-Based Active Interventional Control Framework.

OBJECTIVE: Robotics-assisted retinal microsurgery provides several benefits including improvement of manipulation precision. The assistance provided to the surgeons by current robotic frameworks is, however, a "passive" support, e.g., by damping hand tremor. Intelligent assistance and active guidance are, however, lacking in the existing robotic frameworks. In this paper, an active interventional control framework (AICF) has been presented to increase operation safety by actively intervening the operation to avoid exertion of excessive forces to the sclera. METHODS: AICF consists of the following four components: first, the steady-hand eye robot as the robotic module; second, a sensorized tool to measure tool-to-sclera forces; third, a recurrent neural network to predict occurrence of undesired events based on a short history of time series of sensor measurements; and finally, a variable admittance controller to command the robot away from the undesired instances. RESULTS: A set of user studies were conducted involving 14 participants (with four surgeons). The users were asked to perform a vessel-following task on an eyeball phantom with the assistance of AICF as well as other two benchmark approaches, i.e., auditory feedback (AF) and real-time force feedback (RF). Statistical analysis shows that AICF results in a significant reduction of proportion of undesired instances to about 2.5%, compared with 38.4% and 26.2% using AF and RF, respectively. CONCLUSION: AICF can effectively predict excessive-force instances and augment performance of the user to avoid undesired events during robot-assisted microsurgical tasks. SIGNIFICANCE: The proposed system may be extended to other fields of microsurgery and may potentially reduce tissue injury.

Preliminary study of an RNN-based active interventional robotic system (AIRS) in retinal microsurgery.

PURPOSE: Retinal microsurgery requires highly dexterous and precise maneuvering of instruments inserted into the eyeball through the sclerotomy port. During such procedures, the sclera can potentially be injured from extreme tool-to-sclera contact force caused by surgeon's unintentional misoperations. METHODS: We present an active interventional robotic system to prevent such iatrogenic accidents by enabling the robotic system to actively counteract the surgeon's possible unsafe operations in advance of their occurrence. Relying on a novel force sensing tool to measure and collect scleral forces, we construct a recurrent neural network with long short-term memory unit to oversee surgeon's operation and predict possible unsafe scleral forces up to the next 200 ms. We then apply a linear admittance control to actuate the robot to reduce the undesired scleral force. The system is implemented using an existing "steady hand" eye robot platform. The proposed method is evaluated on an artificial eye phantom by performing a "vessel following" mock retinal surgery operation. RESULTS: Empirical validation over multiple trials indicates that the proposed active interventional robotic system could help to reduce the number of unsafe manipulation events. CONCLUSIONS: We develop an active interventional robotic system to actively prevent surgeon's unsafe operations in retinal surgery. The result of the evaluation experiments shows that the proposed system can improve the surgeon's performance.

Dual-Stiffness Force-Sensing Cannulation Tool for Retinal Microsurgery.

Retinal vein cannulation is a promising treatment for retinal vein occlusion that involves the injection of an anticoagulant directly into the occluded vein to dissolve the blockage. However, excessive forces applied by the injection tool during the procedure, at either the scleral incision or injection site, can result in injury to the eye. Furthermore, the force required to puncture retinal veins (around 10 mN) is well below human sensing ability and an order of magnitude smaller than those that can be safely applied at the sclera (around 100 mN). Detection and management of tool-to-tissue forces on these different scales are some of the most challenging aspects of the cannulation procedure. This work describes the development of a sensorized cannulation tool capable of detecting both tool-to-vein puncture forces and tool-to-sclera contact forces. By combining two materials, nitinol alloy for the tool tip and stainless steel for the tool shaft, to achieve dual stiffness, the tool possesses a flexible tip to capture small vein puncture forces and a stiffer shaft to maintain straightness during use. Three segments of fiber Bragg grating sensors are calibrated to measure the transverse forces at both the tool tip and sclerotomy, as well as to determine the tool insertion depth within the eye. The results of the validation experiments show that the root mean square error of the measurements for the force at the tip, the force at the sclerotomy, and the tool position are 0.70 mN, 1.59 mN, and 0.69 mm, respectively.

Sclera Force Evaluation During Vitreoretinal Surgeries in Ex Vivo Porcine Eye Model.

Vitreoretinal surgery is among the most challenging microsurgical procedures as it requires precise tool manipulation in a constrained environment, while the tool-tissue interaction forces are at the human perception limits. While tool tip forces are certainly important, the scleral forces at the tool insertion ports are also important. Clinicians often rely on these forces to manipulate the eyeball position during surgery. Measuring sclera forces could enable valuable sensory input to avoid tissue damage, especially for a cooperatively controlled robotic assistant that otherwise removes the sensation of these familiar intraoperative forces. Previously, our group has measured sclera forces in phantom experiments. However, to the best of our knowledge, there are no published data measuring scleral forces in biological (ex-vivo/in-vivo) eye models. In this paper, we measured sclera forces in ex-vivo porcine eye model. A Fiber Bragg Grating (FBG) based force sensing instrument with a diameter of ~900 µm and a resolution of ~1 mN was used to measure the forces while the clinician-subject followed retinal vessels in manual and robot-assisted modes. Analysis of measured forces show that the average sclera force in manual mode was 133.74 mN while in robot-assisted mode was 146.03 mN.

Towards securing the sclera against patient involuntary head movement in robotic retinal surgery.

Retinal surgery involves manipulating very delicate tissues within the confined area of eyeball. In such demanding practices, patient involuntary head movement might abruptly raise tool-to-eyeball interaction forces which would be detrimental to eye. This study is aimed at implementing different force control strategies and evaluating how they contribute to attaining sclera force safety while patient head drift is present. To simulate patient head movement, a piezoelectric-actuated linear stage is used to produce random motions in a single direction in random time intervals. Having an eye phantom attached to the linear stage then an experienced eye surgeon is asked to manipulate the eye and repeat a mock surgical task both with and without the assist of the Steady-Hand Eye Robot. For the freehand case, warning sounds were provided to the surgeon as auditory feedback to alert him about excessive slclra forces. For the robot-assisted experiments two variants of an adaptive sclera force control and a virtual fixture method were deployed to see how they can maintain eye safety under head drift circumstances. The results indicate that the developed robot control strategies are able to compensate for head drift and keep the sclera forces under safe levels as well as the free hand operation.

Enabling Technology for Safe Robot-Assisted Retinal Surgery: Early Warning for Unsafe Scleral Force.

Retinal microsurgery is technically demanding and requires high surgical skill with very little room for manipulation error. During surgery the tool needs to be inserted into the eyeball while maintaining constant contact with the sclera. Any unexpected manipulation could cause extreme tool-sclera contact force (scleral force) thus damage the sclera. The introduction of robotic assistance could enhance and expand the surgeon's manipulation capabilities during surgery. However, the potential intra-operative danger from surgeon's misoperations remains difficult to detect and prevent by existing robotic systems. Therefore, we propose a method to predict imminent unsafe manipulation in robot-assisted retinal surgery and generate feedback to the surgeon via auditory substitution. The surgeon could then react to the possible unsafe events in advance. This work specifically focuses on minimizing sclera damage using a force-sensing tool calibrated to measure small scleral forces. A recurrent neural network is designed and trained to predict the force safety status up to 500 milliseconds in the future. The system is implemented using an existing "steady hand" eye robot. A vessel following manipulation task is designed and performed on a dry eye phantom to emulate the retinal surgery and to analyze the proposed method. Finally, preliminary validation experiments are performed by five users, the results of which indicate that the proposed early warning system could help to reduce the number of unsafe manipulation events.

Adaptive Control of Sclera Force and Insertion Depth for Safe Robot-Assisted Retinal Surgery.

One of the significant challenges of moving from manual to robot-assisted retinal surgery is the loss of perception of forces applied to the sclera (sclera forces) by the surgical tools. This damping of force feedback is primarily due to the stiffness and inertia of the robot. The diminished perception of tool-to-eye interactions might put the eye tissue at high risk of injury due to excessive sclera forces or extreme insertion of the tool into the eye. In the present study therefore a 1-dimensional adaptive control method is customized for 3-dimensional control of sclera force components and tool insertion depth and then implemented on the velocity-controlled Johns Hopkins Steady-Hand Eye Robot. The control method enables the robot to perform autonomous motions to make the sclera force and/or insertion depth of the tool tip to follow pre-defined desired and safe trajectories when they exceed safe bounds. A robotic light pipe holding application in retinal surgery is also investigated using the adaptive control method. The implementation results indicate that the adaptive control is able to achieve the imposed safety margins and prevent sclera forces and insertion depth from exceeding safe boundaries.

Sclera Force Control in Robot-assisted Eye Surgery: Adaptive Force Control vs. Auditory Feedback.

Surgeon hand tremor limits human capability during microsurgical procedures such as those that treat the eye. In contrast, elimination of hand tremor through the introduction of microsurgical robots diminishes the surgeons tactile perception of useful and familiar tool-to-sclera forces. While the large mass and inertia of eye surgical robot prevents surgeon microtremor, loss of perception of small scleral forces may put the sclera at risk of injury. In this paper, we have applied and compared two different methods to assure the safety of sclera tissue during robot-assisted eye surgery. In the active control method, an adaptive force control strategy is implemented on the Steady-Hand Eye Robot in order to control the magnitude of scleral forces when they exceed safe boundaries. This autonomous force compensation is then compared to a passive force control method in which the surgeon performs manual adjustments in response to the provided audio feedback proportional to the magnitude of sclera force. A pilot study with three users indicate that the active control method is potentially more efficient.

Artificial intelligence, robotics and eye surgery: are we overfitted?

Eye surgery, specifically retinal micro-surgery involves sensory and motor skill that approaches human boundaries and physiological limits for steadiness, accuracy, and the ability to detect the small forces involved. Despite assumptions as to the benefit of robots in surgery and also despite great development effort, numerous challenges to the full development and adoption of robotic assistance in surgical ophthalmology, remain. Historically, the first in-human-robot-assisted retinal surgery occurred nearly 30 years after the first experimental papers on the subject. Similarly, artificial intelligence emerged decades ago and it is only now being more fully realized in ophthalmology. The delay between conception and application has in part been due to the necessary technological advances required to implement new processing strategies. Chief among these has been the better matched processing power of specialty graphics processing units for machine learning. Transcending the classic concept of robots performing repetitive tasks, artificial intelligence and machine learning are related concepts that has proven their abilities to design concepts and solve problems. The implication of such abilities being that future machines may further intrude on the domain of heretofore "human-reserved" tasks. Although the potential of artificial intelligence/machine learning is profound, present marketing promises and hype exceeds its stage of development, analogous to the seventieth century mathematical "boom" with algebra. Nevertheless robotic systems augmented by machine learning may eventually improve robot-assisted retinal surgery and could potentially transform the discipline. This commentary analyzes advances in retinal robotic surgery, its current drawbacks and limitations, and the potential role of artificial intelligence in robotic retinal surgery.