Real-time surgical tool detection with multi-scale positional encoding and contrastive learning.

Real-time detection of surgical tools in laparoscopic data plays a vital role in understanding surgical procedures, evaluating the performance of trainees, facilitating learning, and ultimately supporting the autonomy of robotic systems. Existing detection methods for surgical data need to improve processing speed and high prediction accuracy. Most methods rely on anchors or region proposals, limiting their adaptability to variations in tool appearance and leading to sub-optimal detection results. Moreover, using non-anchor-based detectors to alleviate this problem has been partially explored without remarkable results. An anchor-free architecture based on a transformer that allows real-time tool detection is introduced. The proposal is to utilize multi-scale features within the feature extraction layer and at the transformer-based detection architecture through positional encoding that can refine and capture context-aware and structural information of different-sized tools. Furthermore, a supervised contrastive loss is introduced to optimize representations of object embeddings, resulting in improved feed-forward network performances for classifying localized bounding boxes. The strategy demonstrates superiority to state-of-the-art (SOTA) methods. Compared to the most accurate existing SOTA (DSSS) method, the approach has an improvement of nearly 4% on mAP and a reduction in the inference time by 113%. It also showed a 7% higher mAP than the baseline model.

Robust endoscopic image mosaicking via fusion of multimodal estimation.

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics.

Optimization and fabrication of programmable domains for soft magnetic robots: A review.

Driven by the aim of realizing functional robotic systems at the milli- and submillimetre scale for biomedical applications, the area of magnetically driven soft devices has received significant recent attention. This has resulted in a new generation of magnetically controlled soft robots with patterns of embedded, programmable domains throughout their structures. This type of programmable magnetic profiling equips magnetic soft robots with shape programmable memory and can be achieved through the distribution of discrete domains (voxels) with variable magnetic densities and magnetization directions. This approach has produced highly compliant, and often bio-inspired structures that are well suited to biomedical applications at small scales, including microfluidic transport and shape-forming surgical catheters. However, to unlock the full potential of magnetic soft robots with improved designs and control, significant challenges remain in their compositional optimization and fabrication. This review considers recent advances and challenges in the interlinked optimization and fabrication aspects of programmable domains within magnetic soft robots. Through a combination of improvements in the computational capacity of novel optimization methods with advances in the resolution, material selection and automation of existing and novel fabrication methods, significant further developments in programmable magnetic soft robots may be realized.

Robotic Autonomy for Magnetic Endoscope Biopsy.

Magnetically actuated endoscopes are currently transitioning in to clinical use for procedures such as colonoscopy, presenting numerous benefits over their conventional counterparts. Intelligent and easy-to-use control strategies are an essential part of their clinical effectiveness due to the un-intuitive nature of magnetic field interaction. However, work on developing intelligent control for these devices has mainly been focused on general purpose endoscope navigation. In this work, we investigate the use of autonomous robotic control for magnetic colonoscope intervention via biopsy, another major component of clinical viability. We have developed control strategies with varying levels of robotic autonomy, including semi-autonomous routines for identifying and performing targeted biopsy, as well as random quadrant biopsy. We present and compare the performance of these approaches to magnetic endoscope biopsy against the use of a standard flexible endoscope on bench-top using a colonoscopy training simulator and silicone colon model. The semi-autonomous routines for targeted and random quadrant biopsy were shown to reduce user workload with comparable times to using a standard flexible endoscope.

Robotic, self-propelled, self-steerable, and disposable colonoscopes: Reality or pipe dream? A state of the art review.

Robotic colonoscopes could potentially provide a comfortable, less painful and safer alternative to standard colonoscopy. Recent exciting developments in this field are pushing the boundaries to what is possible in the future. This article provides a comprehensive review of the current work in robotic colonoscopes including self-propelled, steerable and disposable endoscopes that could be alternatives to standard colonoscopy. We discuss the advantages and disadvantages of these systems currently in development and highlight the technical readiness of each system to help the reader understand where and when such systems may be available for routine clinical use and get an idea of where and in which situation they can best be deployed.

Micro-scale aerosol jet printing of superparamagnetic Fe3O4 nanoparticle patterns.

The opportunity to create different patterns of magnetic nanoparticles on surfaces is highly desirable across many technological and biomedical applications. In this paper, this ability is demonstrated for the first time using a computer-controlled aerosol jet printing (AJP) technology. AJP is an emerging digitally driven, non-contact and mask-less printing process which has distinguishing advantages over other patterning technologies as it offers high-resolution and versatile direct-write deposition of a wide range of materials onto a variety of substrates. This research demonstrates the ability of AJP to reliably print large-area, fine-feature patterns of superparamagnetic iron oxide nanoparticles (SPIONs) onto both rigid material (glass) and soft and flexible materials (polydimethylsiloxane (PDMS) films and poly-L-lactic acid (PLLA) nanofilms). Investigation identified and controlled influential process variables which permitted feature sizes in the region of 20 µm to be realised. This method could be employed for a wide range of applications that require a flexible and responsive process that permits high yield and rapid patterning of magnetic material over large areas. As a first proof of concept, we present patterned magnetic nanofilms with enhanced manipulability under external magnetic field gradient control and which are capable of performing complex movements such as rotation and bending, with applicability to soft robotics and biomedical engineering applications.

Towards Autonomous Robotic Minimally Invasive Ultrasound Scanning and Vessel Reconstruction on Non-Planar Surfaces.

Autonomous robotic Ultrasound (US) scanning has been the subject of research for more than 2 decades. However, little work has been done to apply this concept into a minimally invasive setting, in which accurate force sensing is generally not available and robot kinematics are unreliable due to the tendon-driven, compliant robot structure. As a result, the adequate orientation of the probe towards the tissue surface remains unknown and the anatomy reconstructed from scan may become highly inaccurate. In this work we present solutions to both of these challenges: an attitude sensor fusion scheme for improved kinematic sensing and a visual, deep learning based algorithm to establish and maintain contact between the organ surface and the US probe. We further introduce a novel scheme to estimate and orient the probe perpendicular to the center line of a vascular structure. Our approach enables, for the first time, to autonomously scan across a non-planar surface and navigate along an anatomical structure with a robotically guided minimally invasive US probe. Our experiments on a vessel phantom with a convex surface confirm a significant improvement of the reconstructed curved vessel geometry, with our approach strongly reducing the mean positional error and variance. In the future, our approach could help identify vascular structures more effectively and help pave the way towards semi-autonomous assistance during partial hepatectomy and the potential to reduce procedure length and complication rates.

Traditional Multiwell Plates and Petri Dishes Limit the Evaluation of the Effects of Ultrasound on Cells In Vitro.

Ultrasound accelerates healing in fractured bone; however, the mechanisms responsible are poorly understood. Experimental setups and ultrasound exposures vary or are not adequately characterized across studies, resulting in inter-study variation and difficulty in concluding biological effects. This study investigated experimental variability introduced through the cell culture platform used. Continuous wave ultrasound (45 kHz; 10, 25 or 75 mW/cm2, 5 min/d) was applied, using a Duoson device, to Saos-2 cells seeded in multiwell plates or Petri dishes. Pressure field and vibration quantification and finite-element modelling suggested formation of complex interference patterns, resulting in localized displacement and velocity gradients, more pronounced in multiwell plates. Cell experiments revealed lower metabolic activities in both culture platforms at higher ultrasound intensities and absence of mineralization in certain regions of multiwell plates but not in Petri dishes. Thus, the same transducer produced variable results in different cell culture platforms. Analysis on Petri dishes further revealed that higher intensities reduced vinculin expression and distorted cell morphology, while causing mitochondrial and endoplasmic reticulum damage and accumulation of cells in sub-G1 phase, leading to cell death. More defined experimental setups and reproducible ultrasound exposure systems are required to study the real effect of ultrasound on cells for development of effective ultrasound-based therapies not just limited to bone repair and regeneration.

Active Stabilization of Interventional Tasks Utilizing a Magnetically Manipulated Endoscope.

Magnetically actuated robots have become increasingly popular in medical endoscopy over the past decade. Despite the significant improvements in autonomy and control methods, progress within the field of medical magnetic endoscopes has mainly been in the domain of enhanced navigation. Interventional tasks such as biopsy, polyp removal, and clip placement are a major procedural component of endoscopy. Little advancement has been done in this area due to the problem of adequately controlling and stabilizing magnetically actuated endoscopes for interventional tasks. In the present paper we discuss a novel model-based Linear Parameter Varying (LPV) control approach to provide stability during interventional maneuvers. This method linearizes the non-linear dynamic interaction between the external actuation system and the endoscope in a set of equilibria, associated to different distances between the magnetic source and the endoscope, and computes different controllers for each equilibrium. This approach provides the global stability of the overall system and robustness against external disturbances. The performance of the LPV approach is compared to an intelligent teleoperation control method (based on a Proportional Integral Derivative (PID) controller), on the Magnetic Flexible Endoscope (MFE) platform. Four biopsies in different regions of the colon and at two different system equilibria are performed. Both controllers are asked to stabilize the endoscope in the presence of external disturbances (i.e. the introduction of the biopsy forceps through the working channel of the endoscope). The experiments, performed in a benchtop colon simulator, show a maximum reduction of the mean orientation error of the endoscope of 45.8% with the LPV control compared to the PID controller.

Patient-Specific Magnetic Catheters for Atraumatic Autonomous Endoscopy.

Despite increasing interest in minimally invasive surgical techniques and related developments in flexible endoscopes and catheters, follow-the-leader motion remains elusive. Following the path of least resistance through a tortuous and potentially delicate environment without relying on interaction with the surrounding anatomy requires the control of many degrees of freedom. This typically results in large-diameter instruments. One viable solution to obtain dexterity without increasing size is via multiple-point magnetic actuation over the length of the catheter. The main challenge of this approach is planning magnetic interaction to allow the catheter to adapt to the surrounding anatomy during navigation. We design and manufacture a fully shape-forming, soft magnetic catheter of 80 mm length and 2 mm diameter, capable of navigating a human anatomy in a follow-the-leader fashion. Although this system could be exploited for a range of endoscopic or intravascular applications, here we demonstrate its efficacy for navigational bronchoscopy. From a patient-specific preoperative scan, we optimize the catheters' magnetization profiles and the shape-forming actuating field. To generate the required transient magnetic fields, a dual-robot arm system is employed. We fabricate three separate prototypes to demonstrate minimal contact navigation through a three-dimensional bronchial tree phantom under precomputed robotic control. We also compare a further four separate optimally designed catheters against mechanically equivalent designs with axial magnetization profiles along their length and only at the tip. Using our follow-the-leader approach, we demonstrate up to 50% more accurate tracking, 50% reduction in obstacle contact time during navigation over the state of the art, and an improvement in targeting error of 90%.

Ethical implications of AI in robotic surgical training: A Delphi consensus statement.

CONTEXT: As the role of AI in healthcare continues to expand there is increasing awareness of the potential pitfalls of AI and the need for guidance to avoid them. OBJECTIVES: To provide ethical guidance on developing narrow AI applications for surgical training curricula. We define standardised approaches to developing AI driven applications in surgical training that address current recognised ethical implications of utilising AI on surgical data. We aim to describe an ethical approach based on the current evidence, understanding of AI and available technologies, by seeking consensus from an expert committee. EVIDENCE ACQUISITION: The project was carried out in 3 phases: (1) A steering group was formed to review the literature and summarize current evidence. (2) A larger expert panel convened and discussed the ethical implications of AI application based on the current evidence. A survey was created, with input from panel members. (3) Thirdly, panel-based consensus findings were determined using an online Delphi process to formulate guidance. 30 experts in AI implementation and/or training including clinicians, academics and industry contributed. The Delphi process underwent 3 rounds. Additions to the second and third-round surveys were formulated based on the answers and comments from previous rounds. Consensus opinion was defined as ≥ 80% agreement. EVIDENCE SYNTHESIS: There was 100% response from all 3 rounds. The resulting formulated guidance showed good internal consistency, with a Cronbach alpha of >0.8. There was 100% consensus that there is currently a lack of guidance on the utilisation of AI in the setting of robotic surgical training. Consensus was reached in multiple areas, including: 1. Data protection and privacy; 2. Reproducibility and transparency; 3. Predictive analytics; 4. Inherent biases; 5. Areas of training most likely to benefit from AI. CONCLUSIONS: Using the Delphi methodology, we achieved international consensus among experts to develop and reach content validation for guidance on ethical implications of AI in surgical training. Providing an ethical foundation for launching narrow AI applications in surgical training. This guidance will require further validation. PATIENT SUMMARY: As the role of AI in healthcare continues to expand there is increasing awareness of the potential pitfalls of AI and the need for guidance to avoid them.In this paper we provide guidance on ethical implications of AI in surgical training.

A decade retrospective of medical robotics research from 2010 to 2020.

Robotics is a forward-looking discipline. Attention is focused on identifying the next grand challenges. In an applied field such as medical robotics, however, it is important to plan the future based on a clear understanding of what the research community has recently accomplished and where this work stands with respect to clinical needs and commercialization. This Review article identifies and analyzes the eight key research themes in medical robotics over the past decade. These thematic areas were identified using search criteria that identified the most highly cited papers of the decade. Our goal for this Review article is to provide an accessible way for readers to quickly appreciate some of the most exciting accomplishments in medical robotics over the past decade; for this reason, we have focused only on a small number of seminal papers in each thematic area. We hope that this article serves to foster an entrepreneurial spirit in researchers to reduce the widening gap between research and translation.

Closed-loop control of soft continuum manipulators under tip follower actuation.

Continuum manipulators, inspired by nature, have drawn significant interest within the robotics community. They can facilitate motion within complex environments where traditional rigid robots may be ineffective, while maintaining a reasonable degree of precision. Soft continuum manipulators have emerged as a growing subfield of continuum robotics, with promise for applications requiring high compliance, including certain medical procedures. This has driven demand for new control schemes designed to precisely control these highly flexible manipulators, whose kinematics may be sensitive to external loads, such as gravity. This article presents one such approach, utilizing a rapidly computed kinematic model based on Cosserat rod theory, coupled with sensor feedback to facilitate closed-loop control, for a soft continuum manipulator under tip follower actuation and external loading. This approach is suited to soft manipulators undergoing quasi-static deployment, where actuators apply a follower wrench (i.e., one that is in a constant body frame direction regardless of robot configuration) anywhere along the continuum structure, as can be done in water-jet propulsion. In this article we apply the framework specifically to a tip actuated soft continuum manipulator. The proposed control scheme employs both actuator feedback and pose feedback. The actuator feedback is utilized to both regulate the follower load and to compensate for non-linearities of the actuation system that can introduce kinematic model error. Pose feedback is required to maintain accurate path following. Experimental results demonstrate successful path following with the closed-loop control scheme, with significant performance improvements gained through the use of sensor feedback when compared with the open-loop case.

Feasibility of Fiber Reinforcement Within Magnetically Actuated Soft Continuum Robots.

Soft continuum manipulators have the potential to replace traditional surgical catheters; offering greater dexterity with access to previously unfeasible locations for a wide range of interventions including neurological and cardiovascular. Magnetically actuated catheters are of particular interest due to their potential for miniaturization and remote control. Challenges around the operation of these catheters exist however, and one of these occurs when the angle between the actuating field and the local magnetization vector of the catheter exceeds 90°. In this arrangement, deformation generated by the resultant magnetic moment acts to increase magnetic torque, leading to potential instability. This phenomenon can cause unpredictable responses to actuation, particularly for soft, flexible materials. When coupled with the inherent challenges of sensing and localization inside living tissue, this behavior represents a barrier to progress. In this feasibility study we propose and investigate the use of helical fiber reinforcement within magnetically actuated soft continuum manipulators. Using numerical simulation to explore the design space, we optimize fiber parameters to enhance the ratio of torsional to bending stiffness. Through bespoke fabrication of an optimized helix design we validate a single, prototypical two-segment, 40 mm × 6 mm continuum manipulator demonstrating a reduction of 67% in unwanted twisting under actuation.

An Origami-Based Soft Robotic Actuator for Upper Gastrointestinal Endoscopic Applications.

Soft pneumatic actuators have been explored for endoscopic applications, but challenges in fabricating complex geometry with desirable dimensions and compliance remain. The addition of an endoscopic camera or tool channel is generally not possible without significant change in the diameter of the actuator. Radial expansion and ballooning of actuator walls during bending is undesirable for endoscopic applications. The inclusion of strain limiting methods like, wound fibre, mesh, or multi-material molding have been explored, but the integration of these design approaches with endoscopic requirements drastically increases fabrication complexity, precluding reliable translation into functional endoscopes. For the first time in soft robotics, we present a multi-channel, single material elastomeric actuator with a fully corrugated design (inspired by origami); offering specific functionality for endoscopic applications. The features introduced in this design include i) fabrication of multi-channel monolithic structure of 8.5 mm diameter, ii) incorporation of the benefits of corrugated design in a single material (i.e., limited radial expansion and improved bending efficiency), iii) design scalability (length and diameter), and iv) incorporation of a central hollow channel for the inclusion of an endoscopic camera. Two variants of the actuator are fabricated which have different corrugated or origami length, i.e., 30 mm and 40 mm respectively). Each of the three actuator channels is evaluated under varying volumetric (0.5 mls-1 and 1.5 mls-1 feed rate) and pressurized control to achieve a similar bending profile with the maximum bending angle of 150°. With the intended use for single use upper gastrointestinal endoscopic application, it is desirable to have linear relationships between actuation and angular position in soft pneumatic actuators with high bending response at low pressures; this is where the origami actuator offers contribution. The soft pneumatic actuator has been demonstrated to achieve a maximum bending angle of 200° when integrated with manually driven endoscope. The simple 3-step fabrication technique produces a complex origami pattern in a soft robotic structure, which promotes low pressure bending through the opening of the corrugation while retaining a small diameter and a central lumen, required for successful endoscope integration.

Guidelines for Robotic Flexible Endoscopy at the Time of COVID-19.

Flexible endoscopy involves the insertion of a long narrow flexible tube into the body for diagnostic and therapeutic procedures. In the gastrointestinal (GI) tract, flexible endoscopy plays a major role in cancer screening, surveillance, and treatment programs. As a result of gas insufflation during the procedure, both upper and lower GI endoscopy procedures have been classified as aerosol generating by the guidelines issued by the respective societies during the COVID-19 pandemic-although no quantifiable data on aerosol generation currently exists. Due to the risk of COVID-19 transmission to healthcare workers, most societies halted non-emergency and diagnostic procedures during the lockdown. The long-term implications of stoppage in cancer diagnoses and treatment is predicted to lead to a large increase in preventable deaths. Robotics may play a major role in this field by allowing healthcare operators to control the flexible endoscope from a safe distance and pave a path for protecting healthcare workers through minimizing the risk of virus transmission without reducing diagnostic and therapeutic capacities. This review focuses on the needs and challenges associated with the design of robotic flexible endoscopes for use during a pandemic. The authors propose that a few minor changes to existing platforms or considerations for platforms in development could lead to significant benefits for use during infection control scenarios.

Magnetic flexible endoscope for colonoscopy: an initial learning curve analysis.

Background and study aims Colonoscopy is a technically challenging procedure that requires extensive training to minimize discomfort and avoid trauma due to its drive mechanism. Our academic team developed a magnetic flexible endoscope (MFE) actuated by magnetic coupling under supervisory robotic control to enable a front-pull maneuvering mechanism, with a motion controller user interface, to minimize colon wall stress and potentially reduce the learning curve. We aimed to evaluate this learning curve and understand the user experience. Methods Five novices (no endoscopy experience), five experienced endoscopists, and five experienced MFE users each performed 40 trials on a model colon using 1:1 block randomization between a pediatric colonoscope (PCF) and the MFE. Cecal intubation (CI) success, time to cecum, and user experience (NASA task load index) were measured. Learning curves were determined by the number of trials needed to reach minimum and average proficiency-defined as the slowest average CI time by an experienced user and the average CI time by all experienced users, respectively. Results MFE minimum proficiency was achieved by all five novices (median 3.92 trials) and five experienced endoscopists (median 2.65 trials). MFE average proficiency was achieved by four novices (median 14.21 trials) and four experienced endoscopists (median 7.00 trials). PCF minimum and average proficiency levels were achieved by only one novice. Novices' perceived workload with the MFE significantly improved after obtaining minimum proficiency. Conclusions The MFE has a short learning curve for users with no prior experience-requiring relatively few attempts to reach proficiency and at a reduced perceived workload.

Evolutionary Inverse Material Identification: Bespoke Characterization of Soft Materials Using a Metaheuristic Algorithm.

The growing interest in soft robotics has resulted in an increased demand for accurate and reliable material modelling. As soft robots experience high deformations, highly nonlinear behavior is possible. Several analytical models that are able to capture this nonlinear behavior have been proposed, however, accurately calibrating them for specific materials and applications can be challenging. Multiple experimental testbeds may be required for material characterization which can be expensive and cumbersome. In this work, we propose an alternative framework for parameter fitting established hyperelastic material models, with the aim of improving their utility in the modelling of soft continuum robots. We define a minimization problem to reduce fitting errors between a soft continuum robot deformed experimentally and its equivalent finite element simulation. The soft material is characterized using four commonly employed hyperelastic material models (Neo Hookean; Mooney-Rivlin; Yeoh; and Ogden). To meet the complexity of the defined problem, we use an evolutionary algorithm to navigate the search space and determine optimal parameters for a selected material model and a specific actuation method, naming this approach as Evolutionary Inverse Material Identification (EIMI). We test the proposed approach with a magnetically actuated soft robot by characterizing two polymers often employed in the field: Dragon Skin™ 10 MEDIUM and Ecoflex™ 00-50. To determine the goodness of the FEM simulation for a specific set of model parameters, we define a function that measures the distance between the mesh of the FEM simulation and the experimental data. Our characterization framework showed an improvement greater than 6% compared to conventional model fitting approaches at different strain ranges based on the benchmark defined. Furthermore, the low variability across the different models obtained using our approach demonstrates reduced dependence on model and strain-range selection, making it well suited to application-specific soft robot modelling.