Spatiotemporal Modeling of Grip Forces Captures Proficiency in Manual Robot Control.

New technologies for monitoring grip forces during hand and finger movements in non-standard task contexts have provided unprecedented functional insights into somatosensory cognition. Somatosensory cognition is the basis of our ability to manipulate and transform objects of the physical world and to grasp them with the right amount of force. In previous work, the wireless tracking of grip-force signals recorded from biosensors in the palm of the human hand has permitted us to unravel some of the functional synergies that underlie perceptual and motor learning under conditions of non-standard and essentially unreliable sensory input. This paper builds on this previous work and discusses further, functionally motivated, analyses of individual grip-force data in manual robot control. Grip forces were recorded from various loci in the dominant and non-dominant hands of individuals with wearable wireless sensor technology. Statistical analyses bring to the fore skill-specific temporal variations in thousands of grip forces of a complete novice and a highly proficient expert in manual robot control. A brain-inspired neural network model that uses the output metric of a self-organizing pap with unsupervised winner-take-all learning was run on the sensor output from both hands of each user. The neural network metric expresses the difference between an input representation and its model representation at any given moment in time and reliably captures the differences between novice and expert performance in terms of grip-force variability.Functionally motivated spatiotemporal analysis of individual average grip forces, computed for time windows of constant size in the output of a restricted amount of task-relevant sensors in the dominant (preferred) hand, reveal finger-specific synergies reflecting robotic task skill. The analyses lead the way towards grip-force monitoring in real time. This will permit tracking task skill evolution in trainees, or identify individual proficiency levels in human robot-interaction, which represents unprecedented challenges for perceptual and motor adaptation in environmental contexts of high sensory uncertainty. Cross-disciplinary insights from systems neuroscience and cognitive behavioral science, and the predictive modeling of operator skills using parsimonious Artificial Intelligence (AI), will contribute towards improving the outcome of new types of surgery, in particular the single-port approaches such as NOTES (Natural Orifice Transluminal Endoscopic Surgery) and SILS (Single-Incision Laparoscopic Surgery).

Distortion and instability compensation with deep learning for rotational scanning endoscopic optical coherence tomography.

Optical Coherence Tomography (OCT) is increasingly used in endoluminal procedures since it provides high-speed and high resolution imaging. Distortion and instability of images obtained with a proximal scanning endoscopic OCT system are significant due to the motor rotation irregularity, the friction between the rotating probe and outer sheath and synchronization issues. On-line compensation of artefacts is essential to ensure image quality suitable for real-time assistance during diagnosis or minimally invasive treatment. In this paper, we propose a new online correction method to tackle both B-scan distortion, video stream shaking and drift problem of endoscopic OCT linked to A-line level image shifting. The proposed computational approach for OCT scanning video correction integrates a Convolutional Neural Network (CNN) to improve the estimation of azimuthal shifting of each A-line. To suppress the accumulative error of integral estimation we also introduce another CNN branch to estimate a dynamic overall orientation angle. We train the network with semi-synthetic OCT videos by intentionally adding rotational distortion into real OCT scanning images. The results show that networks trained on this semi-synthetic data generalize to stabilize real OCT videos, and the algorithm efficacy is demonstrated on both ex vivo and in vivo data, where strong scanning artifacts are successfully corrected.

Correlating Grip Force Signals from Multiple Sensors Highlights Prehensile Control Strategies in a Complex Task-User System.

Wearable sensor systems with transmitting capabilities are currently employed for the biometric screening of exercise activities and other performance data. Such technology is generally wireless and enables the non-invasive monitoring of signals to track and trace user behaviors in real time. Examples include signals relative to hand and finger movements or force control reflected by individual grip force data. As will be shown here, these signals directly translate into task, skill, and hand-specific (dominant versus non-dominant hand) grip force profiles for different measurement loci in the fingers and palm of the hand. The present study draws from thousands of such sensor data recorded from multiple spatial locations. The individual grip force profiles of a highly proficient left-hander (expert), a right-handed dominant-hand-trained user, and a right-handed novice performing an image-guided, robot-assisted precision task with the dominant or the non-dominant hand are analyzed. The step-by-step statistical approach follows Tukey's "detective work" principle, guided by explicit functional assumptions relating to somatosensory receptive field organization in the human brain. Correlation analyses (Person's product moment) reveal skill-specific differences in co-variation patterns in the individual grip force profiles. These can be functionally mapped to from-global-to-local coding principles in the brain networks that govern grip force control and its optimization with a specific task expertise. Implications for the real-time monitoring of grip forces and performance training in complex task-user systems are brought forward.

Steerable OCT catheter for real-time assistance during teleoperated endoscopic treatment of colorectal cancer.

When detected early, colorectal cancer can be treated with minimally invasive flexible endoscopy. However, since only specialized experts can delineate margins and perform endoscopic resections of lesions, patients still often undergo colon resections. To better assist in the performance of surgical tasks, a robotized flexible interventional endoscope was previously developed, having two additional side channels for surgical instrument. We propose to enhance the imaging capabilities of this device by combining it with optical coherence tomography (OCT). For this purpose, we have developed a new steerable OCT instrument with an outer diameter of 3.5 mm. The steerable instrument is terminated with a 2 cm long transparent sheath to allow three-dimensional OCT imaging using a side-focusing optical probe with two external scanning actuators. The instrument is connected to an OCT imaging system built around the OCT Axsun engine, with a 1310 nm center wavelength swept source laser and 100 kHz A-line rate. Once inserted in one of the side channels of the robotized endoscope, bending, rotation and translation of the steerable OCT instrument can be controlled by a physician using a joystick. Ex vivo and in vivo tests show that the novel, steerable and teleoperated OCT device enhances dexterity, allowing for inspection of the surgical field without the need for changing the position of the main endoscope.

Sensors for Expert Grip Force Profiling: Towards Benchmarking Manual Control of a Robotic Device for Surgical Tool Movements.

STRAS (Single access Transluminal Robotic Assistant for Surgeons) is a new robotic system based on the Anubis® platform of Karl Storz for application to intra-luminal surgical procedures. Pre-clinical testing of STRAS has recently permitted to demonstrate major advantages of the system in comparison with classic procedures. Benchmark methods permitting to establish objective criteria for 'expertise' need to be worked out now to effectively train surgeons on this new system in the near future. STRAS consists of three cable-driven sub-systems, one endoscope serving as guide, and two flexible instruments. The flexible instruments have three degrees of freedom and can be teleoperated by a single user via two specially designed master interfaces. In this study, small force sensors sewn into a wearable glove to ergonomically fit the master handles of the robotic system were employed for monitoring the forces applied by an expert and a trainee (complete novice) during all the steps of surgical task execution in a simulator task (4-step-pick-and-drop). Analysis of grip-force profiles is performed sensor by sensor to bring to the fore specific differences in handgrip force profiles in specific sensor locations on anatomically relevant parts of the fingers and hand controlling the master/slave system.

A Novel Telemanipulated Robotic Assistant for Surgical Endoscopy: Preclinical Application to ESD.

OBJECTIVE: Minimally invasive surgical interventions in the gastrointestinal tract, such as endoscopic submucosal dissection (ESD), are very difficult for surgeons when performed with standard flexible endoscopes. Robotic flexible systems have been identified as a solution to improve manipulation. However, only a few such systems have been brought to preclinical trials as of now. As a result, novel robotic tools are required. METHODS: We developed a telemanipulated robotic device, called STRAS, which aims to assist surgeons during intraluminal surgical endoscopy. This is a modular system, based on a flexible endoscope and flexible instruments, which provides 10 degrees of freedom (DoFs). The modularity allows the user to easily set up the robot and to navigate toward the operating area. The robot can then be teleoperated using master interfaces specifically designed to intuitively control all available DoFs. STRAS capabilities have been tested in laboratory conditions and during preclinical experiments. RESULTS: We report 12 colorectal ESDs performed in pigs, in which large lesions were successfully removed. Dissection speeds are compared with those obtained in similar conditions with the manual Anubiscope platform from Karl Storz. We show significant improvements ( ). CONCLUSION: These experiments show that STRAS (v2) provides sufficient DoFs, workspace, and force to perform ESD, that it allows a single surgeon to perform all the surgical tasks and those performances are improved with respect to manual systems. SIGNIFICANCE: The concepts developed for STRAS are validated and could bring new tools for surgeons to improve comfort, ease, and performances for intraluminal surgical endoscopy.

Endoluminal surgical triangulation 2.0: A new flexible surgical robot. Preliminary pre-clinical results with colonic submucosal dissection.

BACKGROUND: Complex intraluminal surgical interventions of the gastrointestinal tract are challenging due to the limitation of existing instruments. Our group has developed a master-slave robotic flexible endoscopic platform that provides instrument triangulation in an endoluminal environment. MATERIALS AND METHODS: Colonic endoscopic submucosal dissections (ESD) were carried out in eight pigs. The robot was introduced transanally. A combination of adapted tele-operated instruments was used. Specimens were inspected and measured. RESULTS: Out of 18 ESDs in total, 12 were successfully completed. Among the completed procedures, two perforations and one system failure occurred and were managed intraoperatively. There was no major bleeding. Mean size of the removed specimens was 18.2 ± 9.8 cm2 and mean total procedure time was 73 ± 35.5 min. CONCLUSIONS: Experimental colorectal ESDs using the flexible surgical robot were feasible and reflected a short learning curve. After some technical improvements the system might allow for a wider adoption of complex endoluminal surgical procedures.

An adaptive and fully automatic method for estimating the 3D position of bendable instruments using endoscopic images.

BACKGROUND: Flexible bendable instruments are key tools for performing surgical endoscopy. Being able to measure the 3D position of such instruments can be useful for various tasks, such as controlling automatically robotized instruments and analyzing motions. METHODS: An automatic method is proposed to infer the 3D pose of a single bending section instrument, using only the images provided by a monocular camera embedded at the tip of the endoscope. The proposed method relies on colored markers attached onto the bending section. The image of the instrument is segmented using a graph-based method and the corners of the markers are extracted by detecting the color transitions along Bézier curves fitted on edge points. These features are accurately located and then used to estimate the 3D pose of the instrument using an adaptive model that takes into account the mechanical play between the instrument and its housing channel. RESULTS: The feature extraction method provides good localization of marker corners with images of the in vivo environment despite sensor saturation due to strong lighting. The RMS error on estimation of the tip position of the instrument for laboratory experiments was 2.1, 1.96, and 3.18 mm in the x, y and z directions, respectively. Qualitative analysis in the case of in vivo images shows the ability to correctly estimate the 3D position of the instrument tip during real motions. CONCLUSIONS: The proposed method provides an automatic and accurate estimation of the 3D position of the tip of a bendable instrument in realistic conditions, where standard approaches fail.

Using simulation to design control strategies for robotic no-scar surgery.

No-scar surgery, which aims at performing surgical operations without visible scars, is the vanguard in the field of Minimally Invasive Surgery. No-scar surgery can be performed with flexible instruments, carried by a guide under the vision of an endoscopic camera. This technique brings many benefits for the patient, but also introduces several difficulties for the surgeon. We aim at developing a teleoperated robotic system for assisting surgeons in this kind of operations. In this paper, we present a virtual simulator of the system that allows to assess different control strategies for our robot and to study possible mechanical issues.

Design of a telemanipulated system for transluminal surgery.

Flexible endoscopes have been recently used for new surgical procedures called NOTES, i.e. Natural Orifice Transluminal Endoscopic Surgery. However, the movements of conventional flexible endoscopes are limited and surgeons can only perform basic tasks with these systems. In order to enhance endoscopes possibilities and workspace, several solutions have been proposed to redesign the whole endoscopic system. New devices have been proposed recently which are made up of a classical endoscope basis with two additional arms. However, these mechanical devices are not completely adequate to properly perform NOTES. That is why we are currently developing a robotized system which can be simultaneously teleoperated while performing autonomous motions. This article presents the constraints of transluminal surgery, existing devices and our new system together with its mathematical modeling.

Simultaneous physiological motion cancellation and depth adaptation in flexible endoscopy.

Flexible endoscopes are used in many diagnostic exams and surgical procedures in gastroenterology as well as in natural orifice transluminal endoscopic surgery. In order to assist the surgeon during these difficult procedures, physiological motion cancellation has been successfully applied on a robotized endoscope. However, the stability and performance of the classical controllers were ensured only on a small working area, thus preventing the surgeon to manually move the endoscope during motion rejection. In this paper, we propose original methods to improve the physiological motion rejection while taking into account manual depth changes performed by the surgeon. For this purpose, an adaptive repetitive controller based on depth estimation is proposed. The validity of the approach is demonstrated in in vitro experiments.

A patient-mounted robotic platform for CT-scan guided procedures.

In this paper, we present a novel robotic assistant dedicated to medical interventions under computed tomography scan guidance. This compact and lightweight patient-mounted robot is designed so as to fulfill the requirements of most interventional radiology procedures. It is built from an original 5 DOF parallel structure with a semispherical workspace, particularly well suited to CT-scan interventional procedures. The specifications, the design, and the choice of compatible technological solutions are detailed. A preclinical evaluation is presented, with the registration of the robot in the CT-scan.