Autonomous control of an ultrasound probe for intra-operative ultrasonography using vision-based shape sensing of pneumatically attachable flexible rails.

PURPOSE: In robotic-assisted minimally invasive surgery, surgeons often use intra-operative ultrasound to visualise endophytic structures and localise resection margins. This must be performed by a highly skilled surgeon. Automating this subtask may reduce the cognitive load for the surgeon and improve patient outcomes. METHODS: We demonstrate vision-based shape sensing of the pneumatically attachable flexible (PAF) rail by using colour-dependent image segmentation. The shape-sensing framework is evaluated on known curves ranging from r = 30 to r = 110 mm, replicating curvatures in a human kidney. The shape sensing is then used to inform path planning of a collaborative robot arm paired with an intra-operative ultrasound probe. We execute 15 autonomous ultrasound scans of a tumour-embedded kidney phantom and retrieve viable ultrasound images, as well as seven freehand ultrasound scans for comparison. RESULTS: The vision-based sensor is shown to have comparable sensing accuracy with FBGS-based systems. We find the RMSE of the vision-based shape sensing of the PAF rail compared with ground truth to be 0.4975 ± 0.4169 mm. The ultrasound images acquired by the robot and by the human were evaluated by two independent clinicians. The median score across all criteria for both readers was '3-good' for human and '4-very good' for robot. CONCLUSION: We have proposed a framework for autonomous intra-operative US scanning using vision-based shape sensing to inform path planning. Ultrasound images were evaluated by clinicians for sharpness of image, clarity of structures visible, and contrast of solid and fluid areas. Clinicians evaluated that robot-acquired images were superior to human-acquired images in all metrics. Future work will translate the framework to a da Vinci surgical robot.

The Application of Machine Perfusion as an Enhanced ex vivo Model for Optical Imaging.

Optical imaging techniques such as spectral imaging show promise for the assessment of tissue health during surgery; however, the validation and translation of such techniques into clinical practise is limited by the lack of representative tissue models. In this paper, we demonstrate the application of an organ perfusion machine as an ex vivo tissue model for optical imaging. Three porcine livers are perfused at stepped blood oxygen saturations. Over the duration of each perfusion, spectral data of the tissue are captured via diffuse optical spectroscopy and multispectral imaging. These data are synchronised with blood oxygen saturation measurements recorded by the perfusion machine. Shifts in the optical properties of the tissue are demonstrated over the duration of the each perfusion, as the tissue becomes reperfused and as the oxygen saturation is varied.

Ex Vivo Perfusion of Porcine Pancreas and Liver Sourced from Commercial Abattoirs after Circulatory Death as a Research Resource: A Methodological Study.

BACKGROUND: Machine perfusion (MP) is increasingly used for human transplant organ preservation. The use of MP for research purposes is another opportunity for this technology. The porcine pancreas and liver are similar in anatomical size and function to their human counterparts, making them an excellent resource for research, but they have some important differences from human organs which can influence their research use. In this paper, we describe a technique developed and tested for the retrieval of porcine organs for use in research on perfused viable organs. METHODS: Whole-organ porcine pancreata and livers were harvested at a commercial abattoir, following standard slaughterhouse processes. The standard slaughterhouse process involved a thoracotomy and mid-line laparotomy, and all the thoracoabdominal organs were removed. The pancreas, fixed in the retroperitoneum, was carefully dissected from its attachments to the surrounding structures, and tissue planes between the pancreas, spleen, duodenum, and colon were meticulously identified and dissected. Vessel exposure and division: The aorta, portal vein (PV), hepatic vein (HV), and hepatic artery (HA) were dissected and isolated, preserving the input and output channels for the liver and pancreas. A distal 3 cm of the aorta was preserved and divided and served as the input for the pancreas perfusions. The liver, PV, HV, and HA were preserved and divided to preserve the physiological channels of the input (PV and HA) and output (HV) for the liver perfusions. The porcine hepatic and pancreas anatomy shares significant resemblance with the gross anatomy found in humans, and this was taken into consideration when designing the perfusion circuitry. The porcine pancreas and spleen shared a common blood supply, with branches arising from the splenic artery. The organs were flushed with cold, heparinised normal saline and transported in a temperature-regulated receptacle maintained at a core temperature between 4 and 8 °C, in line with the standards of static cold storage (SCS), to a dedicated perfusion lab and perfused using our novel perfusion machine with autologous, heparinised porcine blood, also collected at the abattoir.

Learning intraoperative organ manipulation with context-based reinforcement learning.

PURPOSE: Automation of sub-tasks during robotic surgery is challenging due to the high variability of the surgical scenes intra- and inter-patients. For example, the pick and place task can be executed different times during the same operation and for distinct purposes. Hence, designing automation solutions that can generalise a skill over different contexts becomes hard. All the experiments are conducted using the Pneumatic Attachable Flexible (PAF) rail, a novel surgical tool designed for robotic-assisted intraoperative organ manipulation. METHODS: We build upon previous open-source surgical Reinforcement Learning (RL) training environment to develop a new RL framework for manipulation skills, rlman. In rlman, contextual RL agents are trained to solve different aspects of the pick and place task using the PAF rail system. rlman is implemented to support both low- and high-dimensional state information to solve surgical sub-tasks in a simulation environment. RESULTS: We use rlman to train state of the art RL agents to solve four different surgical sub-tasks involving manipulation skills using the PAF rail. We compare the results with state-of-the-art benchmarks found in the literature. We evaluate the ability of the agent to be able to generalise over different aspects of the targeted surgical environment. CONCLUSION: We have shown that the rlman framework can support the training of different RL algorithms for solving surgical sub-task, analysing the importance of context information for generalisation capabilities. We are aiming to deploy the trained policy on the real da Vinci using the dVRK and show that the generalisation of the trained policy can be transferred to the real world.

Autonomous pick-and-place using the dVRK.

PURPOSE: Robotic-assisted partial nephrectomy (RAPN) is a tissue-preserving approach to treating renal cancer, where ultrasound (US) imaging is used for intra-operative identification of tumour margins and localisation of blood vessels. With the da Vinci Surgical System (Sunnyvale, CA), the US probe is inserted through an auxiliary access port, grasped by the robotic tool and moved over the surface of the kidney. Images from US probe are displayed separately to the surgical site video within the surgical console leaving the surgeon to interpret and co-registers information which is challenging and complicates the procedural workflow. METHODS: We introduce a novel software architecture to support a hardware soft robotic rail designed to automate intra-operative US acquisition. As a preliminary step towards complete task automation, we automatically grasp the rail and position it on the tissue surface so that the surgeon is then able to manipulate manually the US probe along it. RESULTS: A preliminary clinical study, involving five surgeons, was carried out to evaluate the potential performance of the system. Results indicate that the proposed semi-autonomous approach reduced the time needed to complete a US scan compared to manual tele-operation. CONCLUSION: Procedural automation can be an important workflow enhancement functionality in future robotic surgery systems. We have shown a preliminary study on semi-autonomous US imaging, and this could support more efficient data acquisition.

Ultrasound 3D reconstruction of malignant masses in robotic-assisted partial nephrectomy using the PAF rail system: a comparison study.

PURPOSE: In robotic-assisted partial nephrectomy (RAPN), the use of intraoperative ultrasound (IOUS) helps to localise and outline the tumours as well as the blood vessels within the kidney. The aim of this work is to evaluate the use of the pneumatically attachable flexible (PAF) rail system for US 3D reconstruction of malignant masses in RAPN. The PAF rail system is a novel device developed and previously presented by the authors to enable track-guided US scanning. METHODS: We present a comparison study between US 3D reconstruction of masses based on: the da Vinci Surgical System kinematics, single- and stereo-camera tracking of visual markers embedded on the probe. An US-realistic kidney phantom embedding a mass is used for testing. A new design for the US probe attachment to enhance the performance of the kinematic approach is presented. A feature extraction algorithm is proposed to detect the margins of the targeted mass in US images. RESULTS: To evaluate the performance of the investigated approaches the resulting 3D reconstructions have been compared to a CT scan of the phantom. The data collected indicates that single camera reconstruction outperformed the other approaches, reconstructing with a sub-millimetre accuracy the targeted mass. CONCLUSIONS: This work demonstrates that the PAF rail system provides a reliable platform to enable accurate US 3D reconstruction of masses in RAPN procedures. The proposed system has also the potential to be employed in other surgical procedures such as hepatectomy or laparoscopic liver resection.

Real-Time Vision-Based Stiffness Mapping †.

This paper presents new findings concerning a hand-held stiffness probe for the medical diagnosis of abnormalities during palpation of soft-tissue. Palpation is recognized by the medical community as an essential and low-cost method to detect and diagnose disease in soft-tissue. However, differences are often subtle and clinicians need to train for many years before they can conduct a reliable diagnosis. The probe presented here fills this gap providing a means to easily obtain stiffness values of soft tissue during a palpation procedure. Our stiffness sensor is equipped with a multi degree of freedom (DoF) Aurora magnetic tracker, allowing us to track and record the 3D position of the probe whilst examining a tissue area, and generate a 3D stiffness map in real-time. The stiffness probe was integrated in a robotic arm and tested in an artificial environment representing a good model of soft tissue organs; the results show that the sensor can accurately measure and map the stiffness of a silicon phantom embedded with areas of varying stiffness.

A Novel Concept for Safe, Stiffness-Controllable Robot Links.

The recent decade has seen an astounding increase of interest and advancement in a new field of robotics, aimed at creating structures specifically for the safe interaction with humans. Softness, flexibility, and variable stiffness in robotics have been recognized as highly desirable characteristics for many applications. A number of solutions were proposed ranging from entirely soft robots (such as those composed mainly from soft materials such as silicone), via flexible continuum and snake-like robots, to rigid-link robots enhanced by joints that exhibit an elastic behavior either implemented in hardware or achieved purely by means of intelligent control. Although these are very good solutions paving the path to safe human-robot interaction, we propose here a new approach that focuses on creating stiffness controllability for the linkages between the robot joints. This article proposes a replacement for the traditionally rigid robot link-the new link is equipped with an additional capability of stiffness controllability. With this added feature, a robot can accurately carry out manipulation tasks (high stiffness), but can virtually instantaneously reduce its stiffness when a human is nearby or in contact with the robot. The key point of the invention described here is a robot link made of an airtight chamber formed by a soft and flexible, but high-strain resistant combination of a plastic mesh and silicone wall. Inflated with air to a high pressure, the mesh silicone chamber behaves like a rigid link; reducing the air pressure, softens the link and rendering the robot structure safe. This article investigates a number of link prototypes and shows the feasibility of the new concept. Stiffness tests have been performed, showing that a significant level of stiffness can be achieved-up to 40 N reaction force along the axial direction, for a 25-mm-diameter sample at 60 kPa, at an axial deformation of 5 mm. The results confirm that this novel concept to linkages for robot manipulators exhibits the beam-like behavior of traditional rigid links when fully pressurized and significantly reduced stiffness at low pressure. The proposed concept has the potential to easily create safe robots, augmenting traditional robot designs.

Fingertip Fiber Optical Tactile Array with Two-Level Spring Structure.

Tactile perception is a feature benefiting reliable grasping and manipulation. This paper presents the design of an integrated fingertip force sensor employing an optical fiber based approach where applied forces modulate light intensity. The proposed sensor system is developed to support grasping of a broad range of objects, including those that are hard as well those that are soft. The sensor system is comprised of four sensing elements forming a tactile array integrated with the tip of a finger. We investigate the design configuration of a separate force sensing element with the aim to improve its measurement range. The force measurement of a single tactile element is based on a two-level displacement that is achieved thanks to a hybrid sensing structure made up of a stiff linear and flexible ortho-planar spring. An important outcome of this paper is a miniature tactile fingertip sensor that is capable of perceiving light contact, typically occurring during the initial stages of a grasp, as well as measuring higher forces, commonly present during tight grasps.