E-Calib: A Fast, Robust, and Accurate Calibration Toolbox for Event Cameras.

Event cameras triggered a paradigm shift in the computer vision community delineated by their asynchronous nature, low latency, and high dynamic range. Calibration of event cameras is always essential to account for the sensor intrinsic parameters and for 3D perception. However, conventional image-based calibration techniques are not applicable due to the asynchronous, binary output of the sensor. The current standard for calibrating event cameras relies on either blinking patterns or event-based image reconstruction algorithms. These approaches are difficult to deploy in factory settings and are affected by noise and artifacts degrading the calibration performance. To bridge these limitations, we present E-Calib, a novel, fast, robust, and accurate calibration toolbox for event cameras utilizing the asymmetric circle grid, for its robustness to out-of-focus scenes. E-Calib introduces an efficient reweighted least squares (eRWLS) method for feature extraction of the calibration pattern circles with sub-pixel accuracy and robustness to noise. In addition, a modified hierarchical clustering algorithm is devised to detect the calibration grid apart from the background clutter. The proposed method is tested in a variety of rigorous experiments for different event camera models, on circle grids with different geometric properties, on varying calibration trajectories and speeds, and under challenging illumination conditions. The results show that our approach outperforms the state-of-the-art in detection success rate, reprojection error, and pose estimation accuracy.

Proprioception and Exteroception of a Soft Robotic Finger Using Neuromorphic Vision-Based Sensing.

Equipping soft robotic grippers with sensing and perception capabilities faces significant challenges due to their high compliance and flexibility, limiting their ability to successfully interact with the environment. In this work, we propose a sensorized soft robotic finger with embedded marker pattern that integrates a high-speed neuromorphic event-based camera to enable finger proprioception and exteroception. A learning-based approach involving a convolutional neural network is developed to process event-based heat maps and achieve specific sensing tasks. The feasibility of the sensing approach for proprioception is demonstrated by showing its ability to predict the two-dimensional deformation of three points located on the finger structure, whereas the exteroception capability is assessed in a slip detection task that can classify slip heat maps at a temporal resolution of 2 ms. Our results show that our proposed approach can enable complete sensorization of the finger for both proprioception and exteroception using a single camera without negatively affecting the finger compliance. Using such sensorized finger in robotic grippers may provide safe, adaptive, and precise grasping for handling a wide category of objects.

Neuromorphic Camera Denoising Using Graph Neural Network-Driven Transformers.

Neuromorphic vision is a bio-inspired technology that has triggered a paradigm shift in the computer vision community and is serving as a key enabler for a wide range of applications. This technology has offered significant advantages, including reduced power consumption, reduced processing needs, and communication speedups. However, neuromorphic cameras suffer from significant amounts of measurement noise. This noise deteriorates the performance of neuromorphic event-based perception and navigation algorithms. In this article, we propose a novel noise filtration algorithm to eliminate events that do not represent real log-intensity variations in the observed scene. We employ a graph neural network (GNN)-driven transformer algorithm, called GNN-Transformer, to classify every active event pixel in the raw stream into real log-intensity variation or noise. Within the GNN, a message-passing framework, referred to as EventConv, is carried out to reflect the spatiotemporal correlation among the events while preserving their asynchronous nature. We also introduce the known-object ground-truth labeling (KoGTL) approach for generating approximate ground-truth labels of event streams under various illumination conditions. KoGTL is used to generate labeled datasets, from experiments recorded in challenging lighting conditions, including moon light. These datasets are used to train and extensively test our proposed algorithm. When tested on unseen datasets, the proposed algorithm outperforms state-of-the-art methods by at least 8.8% in terms of filtration accuracy. Additional tests are also conducted on publicly available datasets (ETH Zürich Color-DAVIS346 datasets) to demonstrate the generalization capabilities of the proposed algorithm in the presence of illumination variations and different motion dynamics. Compared to state-of-the-art solutions, qualitative results verified the superior capability of the proposed algorithm to eliminate noise while preserving meaningful events in the scene.

Hierarchical Spatiotemporal Graph Regularized Discriminative Correlation Filter for Visual Object Tracking.

Visual object tracking is a fundamental and challenging task in many high-level vision and robotics applications. It is typically formulated by estimating the target appearance model between consecutive frames. Discriminative correlation filters (DCFs) and their variants have achieved promising speed and accuracy for visual tracking in many challenging scenarios. However, because of the unwanted boundary effects and lack of geometric constraints, these methods suffer from performance degradation. In the current work, we propose hierarchical spatiotemporal graph-regularized correlation filters for robust object tracking. The target sample is decomposed into a large number of deep channels, which are then used to construct a spatial graph such that each graph node corresponds to a particular target location across all channels. Such a graph effectively captures the spatial structure of the target object. In order to capture the temporal structure of the target object, the information in the deep channels obtained from a temporal window is compressed using the principal component analysis, and then, a temporal graph is constructed such that each graph node corresponds to a particular target location in the temporal dimension. Both spatial and temporal graphs span different subspaces such that the target and the background become linearly separable. The learned correlation filter is constrained to act as an eigenvector of the Laplacian of these spatiotemporal graphs. We propose a novel objective function that incorporates these spatiotemporal constraints into the DCFs framework. We solve the objective function using alternating direction methods of multipliers such that each subproblem has a closed-form solution. We evaluate our proposed algorithm on six challenging benchmark datasets and compare it with 33 existing state-of-the art trackers. Our results demonstrate an excellent performance of the proposed algorithm compared to the existing trackers.

Neuromorphic Vision Based Contact-Level Classification in Robotic Grasping Applications.

In recent years, robotic sorting is widely used in the industry, which is driven by necessity and opportunity. In this paper, a novel neuromorphic vision-based tactile sensing approach for robotic sorting application is proposed. This approach has low latency and low power consumption when compared to conventional vision-based tactile sensing techniques. Two Machine Learning (ML) methods, namely, Support Vector Machine (SVM) and Dynamic Time Warping-K Nearest Neighbor (DTW-KNN), are developed to classify material hardness, object size, and grasping force. An Event-Based Object Grasping (EBOG) experimental setup is developed to acquire datasets, where 243 experiments are produced to train the proposed classifiers. Based on predictions of the classifiers, objects can be automatically sorted. If the prediction accuracy is below a certain threshold, the gripper re-adjusts and re-grasps until reaching a proper grasp. The proposed ML method achieves good prediction accuracy, which shows the effectiveness and the applicability of the proposed approach. The experimental results show that the developed SVM model outperforms the DTW-KNN model in term of accuracy and efficiency for real time contact-level classification.

A Proposed Clinical Evaluation of a Simulation Environment for Magnetically-Driven Active Endoscopic Capsules.

Background A simulation environment for magnetically-driven, active endoscopic capsules (Abu-Kheil Y, Seneviratne L, Dias J, A simulation environment for active endoscopic capsules. 2017 IEEE 30th international symposium on Computer Based Medical Systems (CBMS), Thessaloniki, pp 714-719, 2017), can perform four main operations: capsule tele- operation, tracking of a specific region of interest, haptic feedback for capsule navigation and virtual reality navigation.Methods The main operations of the simulation environment can be clinically evaluated. In this paper, we proposed a clinical evaluation for the main functions of the simulation environment. There main testing procedures for the navigation strategies are proposed; i) vision-based tele-operation, ii) vision/haptic-based navigation without head control, and iii) vision/haptic-based navigation with head control. The navigation ways can be compared with each other in terms of introduction time, visualization and procedure comfort. Human-subject studies are to be conducted in which 20 students and 12 expert gastroenterologists participated.

Modeling, Control, and Numerical Simulations of a Novel Binary-Controlled Variable Stiffness Actuator (BcVSA).

This research work aims at realizing a new compliant robotic actuator for safe human-robotic interaction. In this paper, we present the modeling, control, and numerical simulations of a novel Binary-Controlled Variable Stiffness Actuator (BcVSA) aiming to be used for the development of a novel compliant robotic manipulator. BcVSA is the proof of concept of the active revolute joint with the variable recruitment of series-parallel elastic elements. We briefly recall the basic design principle which is based on a stiffness varying mechanism consisting of a motor, three inline clutches, and three torsional springs with stiffness values ( K 0 , 2 K 0 , 4 K 0 ) connected to the load shaft and the motor shaft through two planetary sun gear trains with ratios (4:1, 4:1 respectively). We present the design concept, stiffness and dynamic modeling, and control of our BcVSA. We implemented three kinds of Multiple Model Predictive Control (MPC) to control our actuator. The main motivation of choosing this controller lies in the fact that working principle of multiple MPC and multiple states space representation (stiffness level) of our actuator share similar interests. In particular, we implemented Multiple MPC, Multiple Explicit MPC, and Approximated Multiple Explicit MPC. Numerical simulations are performed in order to evaluate their effectiveness for the future experiments on the prototype of our actuator. The simulation results showed that the Multiple MPC, and the Multiple Explicit MPC have similar results from the robustness point of view. On the other hand, the robustness performance of Approximated Multiple Explicit MPC is not good as compared to other controllers but it works in the offline framework while having the capability to compute the sub-optimal results. We also performed the comparison of MPC based controllers with the Computed Torque Control (CTC), and Linear Quadratic Regulator (LQR). In future, we are planning to test the presented approach on the hardware prototype of our actuator.

Additive Manufactured Sandwich Composite/ABS Parts for Unmanned Aerial Vehicle Applications.

Fused deposition modelling (FDM) is one of most popular 3D printing techniques of thermoplastic polymers. Nonetheless, the poor mechanical strength of FDM parts restricts the use of this technology in functional parts of many applications such as unmanned aerial vehicles (UAVs) where lightweight, high strength, and stiffness are required. In the present paper, the fabrication process of low-density acrylonitrile butadiene styrenecarbon (ABS) with carbon fibre reinforced polymer (CFRP) sandwich layers for UAV structure is proposed to improve the poor mechanical strength and elastic modulus of printed ABS. The composite sandwich structures retains FDM advantages for rapid making of complex geometries, while only requires simple post-processing steps to improve the mechanical properties. Artificial neural network (ANN) was used to investigate the influence of the core density and number of CFRP layers on the mechanical properties. The results showed an improvement of specific strength and elastic modulus with increasing the number of CFRP. The specific strength of the samples improved from 20 to 145 KN·m/kg while the Young's modulus increased from 0.63 to 10.1 GPa when laminating the samples with CFRP layers. On the other hand, the core density had no significant effect on both specific strength and elastic modulus. A case study was undertaken by applying the CFRP/ABS/CFRP sandwich structure using the proposed method to manufacture improved dual-tilting clamps of a quadcopter UAV.

Ex vivo study of prostate cancer localization using rolling mechanical imaging towards minimally invasive surgery.

Rolling mechanical imaging (RMI) is a novel technique towards the detection and quantification of malignant tissue in locations that are inaccessible to palpation during robotic minimally invasive surgery (MIS); the approach is shown to achieve results of higher precision than is possible using the human hand. Using a passive robotic manipulator, a lightweight and force sensitive wheeled probe is driven across the surface of tissue samples to collect continuous measurements of wheel-tissue dynamics. A color-coded map is then generated to visualize the stiffness distribution within the internal tissue structure. Having developed the RMI device in-house, we aim to compare the accuracy of this technique to commonly used methods of localizing prostate cancer in current practice: digital rectal exam (DRE), magnetic resonance imaging (MRI) and transrectal ultrasound (TRUS) biopsy. Final histology is the gold standard used for comparison. A total of 126 sites from 21 robotic-assisted radical prostatectomy specimens were examined. Analysis was performed for sensitivity, specificity, accuracy, and predictive value across all patient risk profiles (defined by PSA, Gleason score and pathological score). Of all techniques, pre-operative biopsy had the highest sensitivity (76.2%) and accuracy (64.3%) in the localization of tumor in the final specimen. However, RMI had a higher sensitivity (44.4%) and accuracy (57.9%) than both DRE (38.1% and 52.4%, respectively) and MRI (33.3% and 57.9%, respectively). These findings suggest a role for RMI towards MIS, where haptic feedback is lacking. While our approach has focused on urological tumors, RMI has potential applicability to other extirpative oncological procedures and to diagnostics (e.g., breast cancer screening).

Three-Degree-of-Freedom MR-Compatible Multisegment Cardiac Catheter Steering Mechanism.

This paper presents a novel MR-compatible 3-DOF cardiac catheter steering mechanism. The catheter's steerable structure is tendon driven and consists of miniature deflectable, helical segments created by a precise rapid prototyping technique. The created catheter prototype has an outer diameter of 9 Fr (3 mm) and a steerable distal end that can be deflected in a 3-D space via four braided high-tensile Spectra fiber tendons. Any longitudinal twist commonly observed in helical structures is compensated for by employing clockwise (CW) and counter clockwise (CCW) helical segments in an alternating fashion. A 280 µm flexible carbon fiber rod is used as a backbone in a central channel to improve the structure's steering and positioning repeatability. In addition to the backbone, a carbon fiber tube can be inserted into the structure to a varying amount capable of changing the structure's forcibility and, thus, providing a means to change the curvature and to modify the deflectable length of the catheter leading to an extension of reachable points in the catheter-tip workspace. A unique feature of this helical segment structure is that the stiffness can be further adjusted by appropriately tensioning tendons simultaneously. An experimental study has been conducted examining the catheter-tip trajectory in a 3-D space and its positioning repeatability using a 5-DOF magnetic coil tracking system. Furthermore, MRI experiments in a 1.5-T scanner confirmed the MR-compatibility of the catheter prototype. The study shows that the proposed concept for catheter steering has great potential to be employed for robotically steered and MR-guided cardiac catheterization.

Using visual cues to enhance haptic feedback for palpation on virtual model of soft tissue.

This paper explores methods that make use of visual cues aimed at generating actual haptic sensation to the user, namely pseudo-haptics. We propose a new pseudo-haptic feedback-based method capable of conveying 3D haptic information and combining visual haptics with force feedback to enhance the user's haptic experience. We focused on an application related to tumor identification during palpation and evaluated the proposed method in an experimental study where users interacted with a haptic device and graphical interface while exploring a virtual model of soft tissue, which represented stiffness distribution of a silicone phantom tissue with embedded hard inclusions. The performance of hard inclusion detection using force feedback only, pseudo-haptic feedback only, and the combination of the two feedbacks was compared with the direct hand touch. The combination method and direct hand touch had no significant difference in the detection results. Compared with the force feedback alone, our method increased the sensitivity by 5%, the positive predictive value by 4%, and decreased detection time by 48.7%. The proposed methodology has great potential for robot-assisted minimally invasive surgery and in all applications where remote haptic feedback is needed.

Bilateral shared autonomous systems with passive and nonpassive input forces under time varying delay.

In this paper, we address stability and tracking control problem of bilateral shared autonomous systems in the presence of passive and nonpassive input interaction forces. The design comprises delayed position and position-velocity signals with the known and unknown structures of the master and slave manipulator dynamics. Using novel Lyapunov-Krasovskii functional, stability and tracking conditions of the coupled master-slave shared autonomous systems are developed under symmetrical and unsymmetrical time varying data transmission delays. This condition allows the designer to estimate the control design parameters to ensure position, velocity and synchronizing errors of the master and slave manipulators. Finally, evaluation results are presented to demonstrate the validity of the proposed design for real-time teleoperation applications.

Intra-operative tumour localisation in robot-assisted minimally invasive surgery: A review.

Robot-assisted minimally invasive surgery has many advantages compared to conventional open surgery and also certain drawbacks: it causes less operative trauma and faster recovery times but does not allow for direct tumour palpation as is the case in open surgery. This article reviews state-of-the-art intra-operative tumour localisation methods used in robot-assisted minimally invasive surgery and in particular methods that employ force-based sensing, tactile-based sensing, and medical imaging techniques. The limitations and challenges of these methods are discussed and future research directions are proposed.

Inverse finite-element modeling for tissue parameter identification using a rolling indentation probe.

This paper investigates the use of inverse finite-element modeling (IFEM)-based methods for tissue parameter identification using a rolling indentation probe for surgical palpation. An IFEM-based algorithm is proposed for tissue parameter identification through uniaxial indentation. IFEM-based algorithms are also created for locating and identifying the properties of an embedded tumor through rolling indentation of the soft tissue. Two types of parameter identification for the tissue tumor are investigated (1) identifying the stiffness (µ) of a tumor at a known depth and (2) estimating the depth of the tumor (D) with known mechanical properties. The efficiency of proposed methods has been evaluated through silicone and porcine kidney experiments for both uniaxial indentation and rolling indentation. The results show that both of the proposed IFEM methods for uniaxial indentation and rolling indentation have good robustness and can rapidly converge to the correct results. The tissue properties estimated using the developed method are generic and in good agreement with results obtained from standard material tests. The estimation error of µ through uniaxial indentation is below 3 % for both silicone and kidney; the estimation error of µ for the tumor through rolling indentation is 7-9 %. The estimation error of D through rolling indentation is 1-2 mm.

Towards kinematic modeling of a multi-DOF tendon driven robotic catheter.

This paper presents a recent study on kinematic modeling of a remote-controlled active catheter. The tip steering motion of the catheter is actuated in a tendon-driven manner. Two antagonistic groups of tendon actuation realize the distal tip deflecting with two-degree-of-freedom (2-DOF) allowing it to reach a considerable large spatial workspace without catheter shaft rotation. However, when modeling such bending deformation, the sequential rotation approach is easily misapplied. We address this problem and introduce a novel and unified modeling methodology based on the concept of simultaneous rotation and the use of Rodrigues' rotation formula. An accurate model is created for robotic catheters and also can be generalized to common multi-tendon-driven continuum manipulators. It is essential in achieving accurate control and improving autonomous execution of command tracking tasks.

Air-float palpation probe for tissue abnormality identification during minimally invasive surgery.

This paper presents a novel palpation probe based on optical fiber technology. It is designed to measure stiffness distribution of a soft tissue while sliding over the tissue surface in a near frictionless manner. A novelty of the probe is its ability to measure indentation depth for nonplanar tissue profiles which are commonly experienced during surgery. Since tumors are often harder than the surrounding tissue, the proposed probe can intraoperatively aid the surgeon to rapidly identify the presence, location, and size of the tumors through the generation of a tissue stiffness map. The probe can concurrently measure tissue reaction force, indentation depth, and the orientation of the probe with respect to the tissue surface. Hence, it can generate an elasticity model of the tissue with minimum measurement inaccuracies caused by surface profile variations. Further, the probe has a tunable force range and the indentation force can be adjusted externally to match tissue limitations. The performance of the probe developed was validated using simulated soft tissues samples. Our tumor identification experiments showed that the probe can accurately identify the location and size of tumors hidden inside nonflat tissue surfaces. Further, the probe has clearly demonstrated its potential to identify tumors with tumor-tissue stiffness ratios as low as 2.1.

A stiffness probe based on force and vision sensing for soft tissue diagnosis.

this paper introduces a novel approach of stiffness measurement based on force and vision sensing for tissue diagnosis. The developed probe is mainly composed of a force sensor and an image acquisition unit capable of obtaining contact area of probe-soft tissue interaction. By measuring the change of diameter of contact area during indentation test, the indentation depth can be determined. The stiffness of target soft tissue then can be evaluated by measuring indentation force and depth simultaneously. The probe can generalize a mechanical image to visualize the stiffness distribution for localization of abnormalities when sliding over soft tissue. The performance of the developed probe was validated by experiments on multiple materials including silicone phantoms and pork organs. The results show that the probe can perform stiffness measurement effectively when the probe indents or slides on the tissue surface.

Finite-element modeling of soft tissue rolling indentation.

We describe a finite-element (FE) model for simulating wheel-rolling tissue deformations using a rolling FE model (RFEM). A wheeled probe performing rolling tissue indentation has proven to be a promising approach for compensating for the loss of haptic and tactile feedback experienced during robotic-assisted minimally invasive surgery (H. Liu, D. P. Noonan, B. J. Challacombe, P. Dasgupta, L. D. Seneviratne, and K. Althoefer, "Rolling mechanical imaging for tissue abnormality localization during minimally invasive surgery, " IEEE Trans. Biomed. Eng., vol. 57, no. 2, pp. 404-414, Feb. 2010; K. Sangpradit, H. Liu, L. Seneviratne, and K. Althoefer, "Tissue identification using inverse finite element analysis of rolling indentation," in Proc. IEEE Int. Conf. Robot. Autom. , Kobe, Japan, 2009, pp. 1250-1255; H. Liu, D. Noonan, K. Althoefer, and L. Seneviratne, "The rolling approach for soft tissue modeling and mechanical imaging during robot-assisted minimally invasive surgery," in Proc. IEEE Int. Conf. Robot. Autom., May 2008, pp. 845-850; H. Liu, P. Puangmali, D. Zbyszewski, O. Elhage, P. Dasgupta, J. S. Dai, L. Seneviratne, and K. Althoefer, "An indentation depth-force sensing wheeled probe for abnormality identification during minimally invasive surgery," Proc. Inst. Mech. Eng., H, vol. 224, no. 6, pp. 751-63, 2010; D. Noonan, H. Liu, Y. Zweiri, K. Althoefer, and L. Seneviratne, "A dual-function wheeled probe for tissue viscoelastic property identification during minimally invasive surgery," in Proc. IEEE Int. Conf. Robot. Autom. , 2008, pp. 2629-2634; H. Liu, J. Li, Q. I. Poon, L. D. Seneviratne, and K. Althoefer, "Miniaturized force indentation-depth sensor for tissue abnormality identification," IEEE Int. Conf. Robot. Autom., May 2010, pp. 3654-3659). A sound understanding of wheel-tissue rolling interaction dynamics will facilitate the evaluation of signals from rolling indentation. In this paper, we model the dynamic interactions between a wheeled probe and a soft tissue sample using the ABAQUS FE software package. The aim of this work is to more precisely locate abnormalities within soft tissue organs using RFEM and hence aid surgeons to improve diagnostic ability. The soft tissue is modeled as a nonlinear hyperelastic material with geometrical nonlinearity. The proposed RFEM was validated on a silicone phantom and a porcine kidney sample. The results show that the proposed method can predict the wheel-tissue interaction forces of rolling indentation with good accuracy and can also accurately identify the location and depth of simulated tumors.

MRI-compatible intensity-modulated force sensor for cardiac catheterization procedures.

This paper presents a novel, magnetic resonance imaging (MRI)-compatible, force sensor suitable for cardiac catheterization procedures. The miniature, fiber-optic sensor is integrated with the tip of a catheter to allow the detection of interaction forces with the cardiac walls. The optical fiber light intensity is modulated when a force acting at the catheter tip deforms an elastic element, which, in turn, varies the distance between a reflector and the optical fiber. The tip sensor has an external diameter of 9 Fr (3 mm) and can be used during cardiac catheterization procedures. The sensor is able to measure forces in the range of 0-0.85 N, with relatively small hysteresis. A nonlinear method for calibration is used and real-time MRI in vivo experiments are carried out, to prove the feasibility of this low-cost sensor, enabling the detection of catheter-tip contact forces under dynamic conditions.

A comparative study between an improved novel air-cushion sensor and a wheeled probe for minimally invasive surgery.

PURPOSE: We describe a comparative study between an enhanced air-cushion tactile sensor and a wheeled indentation probe. These laparoscopic tools are designed to rapidly locate soft-tissue abnormalities during minimally invasive surgery (MIS). MATERIALS AND METHODS: The air-cushion tactile sensor consists of an optically based sensor with a 7.8 mm sphere "floating" on a cushion of air at the tip of a shaft. The wheeled indentation probe is a 10 mm wide and 5 mm in diameter wheel mounted to a force/torque sensor. A continuous rolling indentation technique is used to pass the sensors over the soft-tissue surfaces. The variations in stiffness of the viscoelastic materials that are detected during the rolling indentations are illustrated by stiffness maps that can be used for tissue diagnosis. The probes were tested by having to detect four embedded nodules in a silicone phantom. Each probe was attached to a robotic manipulator and rolled over the silicone phantom in parallel paths. The readings of each probe collected during the process of rolling indentation were used to achieve the final results. RESULTS: The results show that both sensors reliably detected the areas of variable stiffness by accurately identifying the location of each nodule. These are illustrated in the form of two three-dimensional spatiomechanical maps. CONCLUSIONS: These probes have the potential to be used in MIS because they could provide surgeons with information on the mechanical properties of soft tissue, consequently enhancing the reduction in haptic feedback.