Robust tracking of deformable anatomical structures with severe occlusions using deformable geometrical primitives.

BACKGROUND AND OBJECTIVE: Surgical robotics tends to develop cognitive control architectures to provide certain degree of autonomy to improve patient safety and surgery outcomes, while decreasing the required surgeons' cognitive load dedicated to low level decisions. Cognition needs workspace perception, which is an essential step towards automatic decision-making and task planning capabilities. Robust and accurate detection and tracking in minimally invasive surgery suffers from limited visibility, occlusions, anatomy deformations and camera movements. METHOD: This paper develops a robust methodology to detect and track anatomical structures in real time to be used in automatic control of robotic systems and augmented reality. The work focuses on the experimental validation in highly challenging surgery: fetoscopic repair of Open Spina Bifida. The proposed method is based on two sequential steps: first, selection of relevant points (contour) using a Convolutional Neural Network and, second, reconstruction of the anatomical shape by means of deformable geometric primitives. RESULTS: The methodology performance was validated with different scenarios. Synthetic scenario tests, designed for extreme validation conditions, demonstrate the safety margin offered by the methodology with respect to the nominal conditions during surgery. Real scenario experiments have demonstrated the validity of the method in terms of accuracy, robustness and computational efficiency. CONCLUSIONS: This paper presents a robust anatomical structure detection in present of abrupt camera movements, severe occlusions and deformations. Even though the paper focuses on a case study, Open Spina Bifida, the methodology is applicable in all anatomies which contours can be approximated by geometric primitives. The methodology is designed to provide effective inputs to cognitive robotic control and augmented reality systems that require accurate tracking of sensitive anatomies.

Robot assisted Fetoscopic Laser Coagulation: Improvements in navigation, re-location and coagulation.

Fetoscopic Laser Coagulation (FLC) for Twin to Twin Transfusion Syndrome is a challenging intervention due to the working conditions: low quality images acquired from a 3 mm fetoscope inside a turbid liquid environment, local view of the placental surface, unstable surgical field and delicate tissue layers. FLC is based on locating, coagulating and reviewing anastomoses over the placenta's surface. The procedure demands the surgeons to generate a mental map of the placenta with the distribution of the anastomoses, maintaining, at the same time, precision in coagulation and protecting the placenta and amniotic sac from potential damages. This paper describes a teleoperated platform with a cognitive-based control that provides assistance to improve patient safety and surgery performance during fetoscope navigation, target re-location and coagulation processes. A comparative study between manual and teleoperated operation, executed in dry laboratory conditions, analyzes basic fetoscopic skills: fetoscope navigation and laser coagulation. Two exercises are proposed: first, fetoscope guidance and precise coagulation. Second, a resolved placenta (all anastomoses are indicated) to evaluate navigation, re-location and coagulation. The results are analyzed in terms of economy of movement, execution time, coagulation accuracy, amount of coagulated placental surface and risk of placenta puncture. In addition, new metrics, based on navigation and coagulation maps evaluate robotic performance. The results validate the developed platform, showing noticeable improvements in all the metrics.

A Framework for the Evaluation of Human Machine Interfaces of Robot-Assisted Colonoscopy.

The Human Machine Interface (HMI) of intraluminal robots has a crucial impact on the clinician's performance. It increases or decreases the difficulty of the tasks, and is connected to the users' physical and mental stress. OBJECTIVE: This article presents a framework to compare and evaluate different HMIs for robotic colonoscopy, with the objective of identifying the optimal HMI that minimises the clinician's effort and maximises the clinical outcomes. METHODS: The framework comprises a 1) a virtual simulator (clinically validated), 2) wearable sensors measuring the cognitive load, 3) a data collection unit of metrics correlated to the clinical performance, and 4) questionnaires exploring the users' impressions and perceived stress. The framework was tested with 42 clinicians investigating the optimal device for tele-operated control of robotic colonoscopes. Two control devices were selected and compared: a haptic serial-kinematic device and a standard videogame joypad. RESULTS: The haptic device was preferred by the endoscopists, but the joypad enabled better clinical performance and reduced cognitive and physical load. CONCLUSION: The framework can be used to evaluate different aspects of a HMI, both hardware and software, and determine the optimal HMI that can reduce the burden on clinicians while improving the clinical outcome. SIGNIFICANCE: The findings of this study, and of future studies performed with this framework, can inform the design and development of HMIs for intraluminal robots, leading to improved clinical performance, reduced physical and mental stress for clinicians, and ultimately better patient outcomes.

Meshless Simulation of Multi-site Radio Frequency Catheter Ablation through the Fragile Points Method.

Computational models for radio frequency catheter ablation (RFCA) of cardiac arrhythmia have been developed and tested in conditions where a single ablation site is considered. However, in reality arrhythmic events are generated at multiple sites which are ablated during treatment. Under such conditions, heat accumulation from several ablations is expected and models should take this effect into account. Moreover, such models are solved using the Finite Element Method which requires a good quality mesh to ensure numerical accuracy. Therefore, clinical application is limited since heat accumulation effects are neglected and numerical accuracy depends on mesh quality. In this work, we propose a novel meshless computational model where tissue heat accumulation from previously ablated sites is taken into account. In this way, we aim to overcome the mesh quality restriction of the Finite Element Method and enable realistic multi-site ablation simulation. We consider a two ablation sites protocol where tissue temperature at the end of the first ablation is used as initial condition for the second ablation. The effect of the time interval between the ablation of the two sites is evaluated. The proposed method demonstrates that previous models that do not account for heat accumulation between ablations may underestimate the tissue heat distribution.Clinical relevance- The proposed computational model may be used to build and update a heat map for ablation guidance taking into account the contribution from previously ablated sites. Being a meshless model, it does not require significant input from the user during preprocessing. Therefore, it is suitable for application in a clinical setting.

EndoTrainer: a novel hybrid training platform for endoscopic surgery.

PURPOSE: Endoscopy implies high demanding procedures, and their practice requires structured formation curricula supported by adequate training platforms. Physical platforms are the most standardised solution for surgical training, but over the last few years, virtual platforms have been progressively introduced. This research work presents a new hybrid, physic-virtual, endoscopic training platform that exploits the benefits of the two kind of platforms combining realistic tools and phantoms together with the capacity of measuring all relevant parameters along the execution of the exercises and of providing an objective assessment performance. METHODS: The developed platform, EndoTrainer, has been designed to train and assess surgical skills in hysteroscopy and cystoscopy following a structured curricula. The initial development and validation is focused on hysteroscopic exercises proposed in the Gynaecological Endoscopic Surgical Education and Assessment (GESEA) Certification Programme from The Academy and European Society for Gynaecological Endoscopy (ESGE) and analyses the obtained results of an extensive study with 80 gynaecologists executing 30 trials of the standard 30 degree endoscope navigation exercise. RESULTS: The experiments demonstrate the benefits of the presented hybrid platform. Multi-variable statistical analysis points out that all subjects have obtained statistically significant improvement in all relevant parameters: shorter and safer trajectories, improved 30-degree endoscope navigation, accurate positioning over the targets and reduction of the execution time. CONCLUSION: This paper presents a new hybrid approach for training, and evaluating whether it provides an objectivable improvement of camera navigation endoscopic basic skills. The obtained results demonstrate the initial hypothesis: all subjects have improved their camera handling and navigation skills.

Beat-to-beat fetal heart rate analysis using portable medical device and wavelet transformation technique.

Objective: Beat-to-beat tele-fetal monitoring and comparison with clinical data are studied with a wavelet transformation approach. Tele-fetal monitoring is a big progress toward a wearable medical device for pregnant women capable of obtaining prenatal care at home. Study Design: We apply a wavelet transformation algorithm for fetal cardiac monitoring using a portable fetal Doppler medical device. After an investigation of 85 different mother wavelets, a bio-orthogonal 2.2 mother wavelet in level 4 of decomposition is chosen. The efficiency of the proposed method is evaluated using two data sets including public and clinical. Results: From publicly available data on PhysioBank, and simultaneous clinical measurement, we prove that the comparison between obtained fetal heart rate by the algorithm and the baselines yields a promising accuracy beyond 95%. Conclusion: Finally, we conclude that the proposed algorithm would be a robust technique for any similar tele-fetal monitoring approach.

Training Simulators for Gastrointestinal Endoscopy: Current and Future Perspectives.

Gastrointestinal (GI) endoscopy is the gold standard in the detection and treatment of early and advanced GI cancers. However, conventional endoscopic techniques are technically demanding and require visual-spatial skills and significant hands-on experience. GI endoscopy simulators represent a valid solution to allow doctors to practice in a pre-clinical scenario. From the first endoscopy mannequin, developed in 1969, several simulation platforms have been developed, ranging from purely mechanical systems to more complex mechatronic devices and animal-based models. Considering the recent advancement of technologies (e.g., artificial intelligence, augmented reality, robotics), simulation platforms can now reach high levels of realism, representing a valid and smart alternative to standard trainee/mentor learning programs. This is particularly true nowadays, when the current demographic trend and the most recent pandemic demand, more than ever, the ability to cope with many patients. This review offers a broad view of the technology available for GI endoscopy training, including platforms currently in the market and the relevant advancements in this research and application field. Additionally, new training needs and new emerging technologies are discussed to understand where medical education is heading.

Vision Based Robot Assistance in TTTS Fetal Surgery.

This paper presents an accurate and robust tracking vision algorithm for Fetoscopic Laser Photo-coagulation (FLP) surgery for Twin-Twin Transfusion Syndrome (TTTS). The aim of the proposed method is to assist surgeons during anastomosis localization, coagulation and review using a tele-operated robotic system. The algorithm computes the relative position of the fetoscope tool tip with respect to the placenta, via local vascular structure registration. The algorithm uses image features (local superficial vascular structures of the placenta's surface) to automatically match consecutive fetoscopic images. It is composed of three sequential steps: image processing (filtering, binarization and vascular structures segmentation); relevant Points Of Interest (POIs) seletion; and image registration between consecutive images. The algorithm has to deal with the low quality of fetoscopic images, the liquid and dirty environment inside the placenta jointly with the thin diameter of the fetoscope optics and low amount of environment light reduces the image quality. The obtained images are blurred, noisy and with very poor color components. The tracking system has been tested using real video sequences of FLP surgery for TTTS. The computational performance enables real time tracking, locally guiding the robot over the placenta's surface with enough accuracy.

Sliding to predict: vision-based beating heart motion estimation by modeling temporal interactions.

PURPOSE: Technical advancements have been part of modern medical solutions as they promote better surgical alternatives that serve to the benefit of patients. Particularly with cardiovascular surgeries, robotic surgical systems enable surgeons to perform delicate procedures on a beating heart, avoiding the complications of cardiac arrest. This advantage comes with the price of having to deal with a dynamic target which presents technical challenges for the surgical system. In this work, we propose a solution for cardiac motion estimation. METHODS: Our estimation approach uses a variational framework that guarantees preservation of the complex anatomy of the heart. An advantage of our approach is that it takes into account different disturbances, such as specular reflections and occlusion events. This is achieved by performing a preprocessing step that eliminates the specular highlights and a predicting step, based on a conditional restricted Boltzmann machine, that recovers missing information caused by partial occlusions. RESULTS: We carried out exhaustive experimentations on two datasets, one from a phantom and the other from an in vivo procedure. The results show that our visual approach reaches an average minima in the order of magnitude of [Formula: see text] while preserving the heart's anatomical structure and providing stable values for the Jacobian determinant ranging from 0.917 to 1.015. We also show that our specular elimination approach reaches an accuracy of 99% compared to a ground truth. In terms of prediction, our approach compared favorably against two well-known predictors, NARX and EKF, giving the lowest average RMSE of 0.071. CONCLUSION: Our approach avoids the risks of using mechanical stabilizers and can also be effective for acquiring the motion of organs other than the heart, such as the lung or other deformable objects.

Robust cardiac motion estimation using ultrafast ultrasound data: a low-rank topology-preserving approach.

Cardiac motion estimation is an important diagnostic tool for detecting heart diseases and it has been explored with modalities such as MRI and conventional ultrasound (US) sequences. US cardiac motion estimation still presents challenges because of complex motion patterns and the presence of noise. In this work, we propose a novel approach to estimate cardiac motion using ultrafast ultrasound data. Our solution is based on a variational formulation characterized by the L 2-regularized class. Displacement is represented by a lattice of b-splines and we ensure robustness, in the sense of eliminating outliers, by applying a maximum likelihood type estimator. While this is an important part of our solution, the main object of this work is to combine low-rank data representation with topology preservation. Low-rank data representation (achieved by finding the k-dominant singular values of a Casorati matrix arranged from the data sequence) speeds up the global solution and achieves noise reduction. On the other hand, topology preservation (achieved by monitoring the Jacobian determinant) allows one to radically rule out distortions while carefully controlling the size of allowed expansions and contractions. Our variational approach is carried out on a realistic dataset as well as on a simulated one. We demonstrate how our proposed variational solution deals with complex deformations through careful numerical experiments. The low-rank constraint speeds up the convergence of the optimization problem while topology preservation ensures a more accurate displacement. Beyond cardiac motion estimation, our approach is promising for the analysis of other organs that exhibit motion.

Towards Retrieving Force Feedback in Robotic-Assisted Surgery: A Supervised Neuro-Recurrent-Vision Approach.

Robotic-assisted minimally invasive surgeries have gained a lot of popularity over conventional procedures as they offer many benefits to both surgeons and patients. Nonetheless, they still suffer from some limitations that affect their outcome. One of them is the lack of force feedback which restricts the surgeon's sense of touch and might reduce precision during a procedure. To overcome this limitation, we propose a novel force estimation approach that combines a vision based solution with supervised learning to estimate the applied force and provide the surgeon with a suitable representation of it. The proposed solution starts with extracting the geometry of motion of the heart's surface by minimizing an energy functional to recover its 3D deformable structure. A deep network, based on a LSTM-RNN architecture, is then used to learn the relationship between the extracted visual-geometric information and the applied force, and to find accurate mapping between the two. Our proposed force estimation solution avoids the drawbacks usually associated with force sensing devices, such as biocompatibility and integration issues. We evaluate our approach on phantom and realistic tissues in which we report an average root-mean square error of 0.02 N.

Design and development of a microarray processing station (MPS) for automated miniaturized immunoassays.

Here we describe the design and evaluation of a fluidic device for the automatic processing of microarrays, called microarray processing station or MPS. The microarray processing station once installed on a commercial microarrayer allows automating the washing, and drying steps, which are often performed manually. The substrate where the assay occurs remains on place during the microarray printing, incubation and processing steps, therefore the addressing of nL volumes of the distinct immunoassay reagents such as capture and detection antibodies and samples can be performed on the same coordinate of the substrate with a perfect alignment without requiring any additional mechanical or optical re-alignment methods. This allows the performance of independent immunoassays in a single microarray spot.

Towards estimating cardiac motion using low-rank representation and topology preservation for ultrafast ultrasound data.

Estimation of the cardiac motion is very important in order to detect heart diseases. This work presents a cardiac motion estimation approach using ultrafast ultrasound data. We optimize a variational framework which has the benefits of combining low-rank data representation with topology preservation. We show through the analysis of experimental results that this combination offers a radical reduction in computational time and noise while ensuring preservation of the anatomical structure of the heart under complex deformations. Although in this work we use the heart as a study case, our solution is promising to analyze other organs experiencing motion.

Adaptive segmentation and mask-specific Sobolev inpainting of specular highlights for endoscopic images.

Minimally invasive surgical and diagnostic systems rely on endoscopic images of internal organs to assist medical tasks. Specular highlights are common on those images due to the strong reflectivity of the mucus layer on the organs and the relatively high intensity of the light source. This is a significant source of error that can affect the systems' performance. In this paper, we propose a segmentation method of the specular regions based on an automatic color-adaptive threshold and a gradient-based edge detector. The segmented regions are then recovered using a robust mask-specific Sobolev inpainting approach. Experimental results demonstrate the precision and efficiency of the proposed method. In contrast to the existing approaches, the proposed solution does not require manual threshold selection or complex computations to achieve accurate results. Moreover, our method has a real-time performance and can be generalized to various applications.

Compliant gait assistance triggered by user intention.

An automatic gait initialization strategy based on user intention sensing in the context of rehabilitation with a lower-limb wearable robot is proposed and evaluated. The proposed strategy involves monitoring the human-orthosis interaction torques and initial position deviation to determine the gait initiation instant and to modify orthosis operation for gait assistance, when needed. During gait, the compliant control algorithm relies on the adaptation of the joints' stiffness in function of their interaction torques and their deviation from the desired trajectories, while maintaining the dynamic stability. As a reference input, the average of a set of recorded gaits obtained from healthy subjects is used. The algorithm has been tested with five healthy subjects showing its efficient behavior in initiating the gait and maintaining the equilibrium while walking in presence of external forces. The work is performed as a preliminary study to assist patients suffering from incomplete Spinal cord injury and Stroke.

Force-feedback sensory substitution using supervised recurrent learning for robotic-assisted surgery.

The lack of force feedback is considered one of the major limitations in Robot Assisted Minimally Invasive Surgeries. Since add-on sensors are not a practical solution for clinical environments, in this paper we present a force estimation approach that starts with the reconstruction of a 3D deformation structure of the tissue surface by minimizing an energy functional. A Recurrent Neural Network-Long Short Term Memory (RNN-LSTM) based architecture is then presented to accurately estimate the applied forces. According to the results, our solution offers long-term stability and shows a significant percentage of accuracy improvement, ranging from about 54% to 78%, over existing approaches.

Automatic and robust single-camera specular highlight removal in cardiac images.

In computer-assisted beating heart surgeries, accurate tracking of the heart's motion is of huge importance and there is a continuous need to eliminate any source of error that might disturb the tracking process. One source of error is the specular reflection that appears on the glossy surface of the heart. In this paper, we propose a robust solution for the detection and removal of specular highlights. A hybrid color attributes and wavelet based edge projection approach is applied to accurately identify the affected regions. These regions are then recovered using a dynamic search-based inpainting with adaptive windowing. Experimental results demonstrate the precision and efficiency of the proposed method. Moreover, it has a real-time performance and can be generalized to various other applications.

Auto-adaptive robot-aided therapy using machine learning techniques.

This paper presents an application of a classification method to adaptively and dynamically modify the therapy and real-time displays of a virtual reality system in accordance with the specific state of each patient using his/her physiological reactions. First, a theoretical background about several machine learning techniques for classification is presented. Then, nine machine learning techniques are compared in order to select the best candidate in terms of accuracy. Finally, first experimental results are presented to show that the therapy can be modulated in function of the patient state using machine learning classification techniques.

Unconstrained ℓ1-regularized minimization with interpolated transformations for heart motion compensation.

Motion compensation constitutes a challenging issue in minimally invasive beating heart surgery. Since the zone to be repaired has a dynamic behaviour, precision and surgeon's dexterity decrease. In order to solve this problem, various proposals have been presented using ℓ2-norm. However, as they present some limitations in terms of robustness and efficiency, motion compensation is still considered an open problem. In this work, a solution based on the class of ℓ1 Regularized Optimization is proposed. It has been selected due to its mathematical properties and practical benefits. In particular, deformation is characterized by cubic B-splines since they offer an excellent balance between computational cost and accuracy. Moreover, due to the non-differentiability of the functional, the logarithmic barrier function is used for generating a standard optimization problem. Results have provided a very good tradeoff between accuracy and efficiency, indicating the potential of the proposed approach and proving its stability even under complex deformations.