E-Calib: A Fast, Robust, and Accurate Calibration Toolbox for Event Cameras.

Event cameras triggered a paradigm shift in the computer vision community delineated by their asynchronous nature, low latency, and high dynamic range. Calibration of event cameras is always essential to account for the sensor intrinsic parameters and for 3D perception. However, conventional image-based calibration techniques are not applicable due to the asynchronous, binary output of the sensor. The current standard for calibrating event cameras relies on either blinking patterns or event-based image reconstruction algorithms. These approaches are difficult to deploy in factory settings and are affected by noise and artifacts degrading the calibration performance. To bridge these limitations, we present E-Calib, a novel, fast, robust, and accurate calibration toolbox for event cameras utilizing the asymmetric circle grid, for its robustness to out-of-focus scenes. E-Calib introduces an efficient reweighted least squares (eRWLS) method for feature extraction of the calibration pattern circles with sub-pixel accuracy and robustness to noise. In addition, a modified hierarchical clustering algorithm is devised to detect the calibration grid apart from the background clutter. The proposed method is tested in a variety of rigorous experiments for different event camera models, on circle grids with different geometric properties, on varying calibration trajectories and speeds, and under challenging illumination conditions. The results show that our approach outperforms the state-of-the-art in detection success rate, reprojection error, and pose estimation accuracy.

Drone-Person Tracking in Uniform Appearance Crowd: A New Dataset.

Drone-person tracking in uniform appearance crowds poses unique challenges due to the difficulty in distinguishing individuals with similar attire and multi-scale variations. To address this issue and facilitate the development of effective tracking algorithms, we present a novel dataset named D-PTUAC (Drone-Person Tracking in Uniform Appearance Crowd). The dataset comprises 138 sequences comprising over 121 K frames, each manually annotated with bounding boxes and attributes. During dataset creation, we carefully consider 18 challenging attributes encompassing a wide range of viewpoints and scene complexities. These attributes are annotated to facilitate the analysis of performance based on specific attributes. Extensive experiments are conducted using 44 state-of-the-art (SOTA) trackers, and the performance gap between the visual object trackers on existing benchmarks compared to our proposed dataset demonstrate the need for a dedicated end-to-end aerial visual object tracker that accounts the inherent properties of aerial environment.

A neuromorphic dataset for tabletop object segmentation in indoor cluttered environment.

Event-based cameras are commonly leveraged to mitigate issues such as motion blur, low dynamic range, and limited time sampling, which plague conventional cameras. However, a lack of dedicated event-based datasets for benchmarking segmentation algorithms, especially those offering critical depth information for occluded scenes, has been observed. In response, this paper introduces a novel Event-based Segmentation Dataset (ESD), a high-quality event 3D spatial-temporal dataset designed for indoor object segmentation within cluttered environments. ESD encompasses 145 sequences featuring 14,166 manually annotated RGB frames, along with a substantial event count of 21.88 million and 20.80 million events from two stereo-configured event-based cameras. Notably, this densely annotated 3D spatial-temporal event-based segmentation benchmark for tabletop objects represents a pioneering initiative, providing event-wise depth, and annotated instance labels, in addition to corresponding RGBD frames. By releasing ESD, our aim is to offer the research community a challenging segmentation benchmark of exceptional quality.

TactiGraph: An Asynchronous Graph Neural Network for Contact Angle Prediction Using Neuromorphic Vision-Based Tactile Sensing.

Vision-based tactile sensors (VBTSs) have become the de facto method for giving robots the ability to obtain tactile feedback from their environment. Unlike other solutions to tactile sensing, VBTSs offer high spatial resolution feedback without compromising on instrumentation costs or incurring additional maintenance expenses. However, conventional cameras used in VBTS have a fixed update rate and output redundant data, leading to computational overhead.In this work, we present a neuromorphic vision-based tactile sensor (N-VBTS) that employs observations from an event-based camera for contact angle prediction. In particular, we design and develop a novel graph neural network, dubbed TactiGraph, that asynchronously operates on graphs constructed from raw N-VBTS streams exploiting their spatiotemporal correlations to perform predictions. Although conventional VBTSs use an internal illumination source, TactiGraph is reported to perform efficiently in both scenarios (with and without an internal illumination source) thus further reducing instrumentation costs. Rigorous experimental results revealed that TactiGraph achieved a mean absolute error of 0.62∘ in predicting the contact angle and was faster and more efficient than both conventional VBTS and other N-VBTS, with lower instrumentation costs. Specifically, N-VBTS requires only 5.5% of the computing time needed by VBTS when both are tested on the same scenario.

Proprioception and Exteroception of a Soft Robotic Finger Using Neuromorphic Vision-Based Sensing.

Equipping soft robotic grippers with sensing and perception capabilities faces significant challenges due to their high compliance and flexibility, limiting their ability to successfully interact with the environment. In this work, we propose a sensorized soft robotic finger with embedded marker pattern that integrates a high-speed neuromorphic event-based camera to enable finger proprioception and exteroception. A learning-based approach involving a convolutional neural network is developed to process event-based heat maps and achieve specific sensing tasks. The feasibility of the sensing approach for proprioception is demonstrated by showing its ability to predict the two-dimensional deformation of three points located on the finger structure, whereas the exteroception capability is assessed in a slip detection task that can classify slip heat maps at a temporal resolution of 2 ms. Our results show that our proposed approach can enable complete sensorization of the finger for both proprioception and exteroception using a single camera without negatively affecting the finger compliance. Using such sensorized finger in robotic grippers may provide safe, adaptive, and precise grasping for handling a wide category of objects.

Elastomer-Based Visuotactile Sensor for Normality of Robotic Manufacturing Systems.

Modern aircrafts require the assembly of thousands of components with high accuracy and reliability. The normality of drilled holes is a critical geometrical tolerance that is required to be achieved in order to realize an efficient assembly process. Failure to achieve the required tolerance leads to structures prone to fatigue problems and assembly errors. Elastomer-based tactile sensors have been used to support robots in acquiring useful physical interaction information with the environments. However, current tactile sensors have not yet been developed to support robotic machining in achieving the tight tolerances of aerospace structures. In this paper, a novel elastomer-based tactile sensor was developed for cobot machining. Three commercial silicon-based elastomer materials were characterised using mechanical testing in order to select a material with the best deformability. A Finite element model was developed to simulate the deformation of the tactile sensor upon interacting with surfaces with different normalities. Additive manufacturing was employed to fabricate the tactile sensor mould, which was chemically etched to improve the surface quality. The tactile sensor was obtained by directly casting and curing the optimum elastomer material onto the additively manufactured mould. A machine learning approach was used to train the simulated and experimental data obtained from the sensor. The capability of the developed vision tactile sensor was evaluated using real-world experiments with various inclination angles, and achieved a mean perpendicularity tolerance of 0.34°. The developed sensor opens a new perspective on low-cost precision cobot machining.

Neuromorphic Camera Denoising Using Graph Neural Network-Driven Transformers.

Neuromorphic vision is a bio-inspired technology that has triggered a paradigm shift in the computer vision community and is serving as a key enabler for a wide range of applications. This technology has offered significant advantages, including reduced power consumption, reduced processing needs, and communication speedups. However, neuromorphic cameras suffer from significant amounts of measurement noise. This noise deteriorates the performance of neuromorphic event-based perception and navigation algorithms. In this article, we propose a novel noise filtration algorithm to eliminate events that do not represent real log-intensity variations in the observed scene. We employ a graph neural network (GNN)-driven transformer algorithm, called GNN-Transformer, to classify every active event pixel in the raw stream into real log-intensity variation or noise. Within the GNN, a message-passing framework, referred to as EventConv, is carried out to reflect the spatiotemporal correlation among the events while preserving their asynchronous nature. We also introduce the known-object ground-truth labeling (KoGTL) approach for generating approximate ground-truth labels of event streams under various illumination conditions. KoGTL is used to generate labeled datasets, from experiments recorded in challenging lighting conditions, including moon light. These datasets are used to train and extensively test our proposed algorithm. When tested on unseen datasets, the proposed algorithm outperforms state-of-the-art methods by at least 8.8% in terms of filtration accuracy. Additional tests are also conducted on publicly available datasets (ETH Zürich Color-DAVIS346 datasets) to demonstrate the generalization capabilities of the proposed algorithm in the presence of illumination variations and different motion dynamics. Compared to state-of-the-art solutions, qualitative results verified the superior capability of the proposed algorithm to eliminate noise while preserving meaningful events in the scene.

Laser Powder Bed Fusion of Ti-6Al-2Sn-4Zr-6Mo Alloy and Properties Prediction Using Deep Learning Approaches.

Ti-6Al-2Sn-4Zr-6Mo is one of the most important titanium alloys characterised by its high strength, fatigue, and toughness properties, making it a popular material for aerospace and biomedical applications. However, no studies have been reported on processing this alloy using laser powder bed fusion. In this paper, a deep learning neural network (DLNN) was introduced to rationalise and predict the densification and hardness due to Laser Powder Bed Fusion of Ti-6Al-2Sn-4Zr-6Mo alloy. The process optimisation results showed that near-full densification is achieved in Ti-6Al-2Sn-4Zr-6Mo alloy samples fabricated using an energy density of 77-113 J/mm3. Furthermore, the hardness of the builds was found to increase with increasing the laser energy density. Porosity and the hardness measurements were found to be sensitive to the island size, especially at high energy density. Hot isostatic pressing (HIP) was able to eliminate the porosity, increase the hardness, and achieve the desirable α and ß phases. The developed model was validated and used to produce process maps. The trained deep learning neural network model showed the highest accuracy with a mean percentage error of 3% and 0.2% for the porosity and hardness. The results showed that deep learning neural networks could be an efficient tool for predicting materials properties using small data.

Neuromorphic Vision Based Contact-Level Classification in Robotic Grasping Applications.

In recent years, robotic sorting is widely used in the industry, which is driven by necessity and opportunity. In this paper, a novel neuromorphic vision-based tactile sensing approach for robotic sorting application is proposed. This approach has low latency and low power consumption when compared to conventional vision-based tactile sensing techniques. Two Machine Learning (ML) methods, namely, Support Vector Machine (SVM) and Dynamic Time Warping-K Nearest Neighbor (DTW-KNN), are developed to classify material hardness, object size, and grasping force. An Event-Based Object Grasping (EBOG) experimental setup is developed to acquire datasets, where 243 experiments are produced to train the proposed classifiers. Based on predictions of the classifiers, objects can be automatically sorted. If the prediction accuracy is below a certain threshold, the gripper re-adjusts and re-grasps until reaching a proper grasp. The proposed ML method achieves good prediction accuracy, which shows the effectiveness and the applicability of the proposed approach. The experimental results show that the developed SVM model outperforms the DTW-KNN model in term of accuracy and efficiency for real time contact-level classification.

Dynamic-Vision-Based Force Measurements Using Convolutional Recurrent Neural Networks.

In this paper, a novel dynamic Vision-Based Measurement method is proposed to measure contact force independent of the object sizes. A neuromorphic camera (Dynamic Vision Sensor) is utilizused to observe intensity changes within the silicone membrane where the object is in contact. Three deep Long Short-Term Memory neural networks combined with convolutional layers are developed and implemented to estimate the contact force from intensity changes over time. Thirty-five experiments are conducted using three objects with different sizes to validate the proposed approach. We demonstrate that the networks with memory gates are robust against variable contact sizes as the networks learn object sizes in the early stage of a grasp. Moreover, spatial and temporal features enable the sensor to estimate the contact force every 10 ms accurately. The results are promising with Mean Squared Error of less than 0.1 N for grasping and holding contact force using leave-one-out cross-validation method.

A Novel Event-Based Incipient Slip Detection Using Dynamic Active-Pixel Vision Sensor (DAVIS).

In this paper, a novel approach to detect incipient slip based on the contact area between a transparent silicone medium and different objects using a neuromorphic event-based vision sensor (DAVIS) is proposed. Event-based algorithms are developed to detect incipient slip, slip, stress distribution and object vibration. Thirty-seven experiments were performed on five objects with different sizes, shapes, materials and weights to compare precision and response time of the proposed approach. The proposed approach is validated by using a high speed constitutional camera (1000 FPS). The results indicate that the sensor can detect incipient slippage with an average of 44.1 ms latency in unstructured environment for various objects. It is worth mentioning that the experiments were conducted in an uncontrolled experimental environment, therefore adding high noise levels that affected results significantly. However, eleven of the experiments had a detection latency below 10 ms which shows the capability of this method. The results are very promising and show a high potential of the sensor being used for manipulation applications especially in dynamic environments.

Additive Manufactured Sandwich Composite/ABS Parts for Unmanned Aerial Vehicle Applications.

Fused deposition modelling (FDM) is one of most popular 3D printing techniques of thermoplastic polymers. Nonetheless, the poor mechanical strength of FDM parts restricts the use of this technology in functional parts of many applications such as unmanned aerial vehicles (UAVs) where lightweight, high strength, and stiffness are required. In the present paper, the fabrication process of low-density acrylonitrile butadiene styrenecarbon (ABS) with carbon fibre reinforced polymer (CFRP) sandwich layers for UAV structure is proposed to improve the poor mechanical strength and elastic modulus of printed ABS. The composite sandwich structures retains FDM advantages for rapid making of complex geometries, while only requires simple post-processing steps to improve the mechanical properties. Artificial neural network (ANN) was used to investigate the influence of the core density and number of CFRP layers on the mechanical properties. The results showed an improvement of specific strength and elastic modulus with increasing the number of CFRP. The specific strength of the samples improved from 20 to 145 KN·m/kg while the Young's modulus increased from 0.63 to 10.1 GPa when laminating the samples with CFRP layers. On the other hand, the core density had no significant effect on both specific strength and elastic modulus. A case study was undertaken by applying the CFRP/ABS/CFRP sandwich structure using the proposed method to manufacture improved dual-tilting clamps of a quadcopter UAV.

Stability analysis of a three-term backpropagation algorithm.

Efficient learning by the backpropagation (BP) algorithm is required for many practical applications. The BP algorithm calculates the weight changes of artificial neural networks, and a common approach is to use a two-term algorithm consisting of a learning rate (LR) and a momentum factor (MF). The major drawbacks of the two-term BP learning algorithm are the problems of local minima and slow convergence speeds, which limit the scope for real-time applications. Recently the addition of an extra term, called a proportional factor (PF), to the two-term BP algorithm was proposed. The third increases the speed of the BP algorithm. However, the PF term also reduces the convergence of the BP algorithm, and criteria for evaluating convergence are required to facilitate the application of the three terms BP algorithm. This paper analyzes the convergence of the new three-term backpropagation algorithm. If the learning parameters of the three-term BP algorithm satisfy the conditions given in this paper, then it is guaranteed that the system is stable and will converge to a local minimum. It is proved that if at least one of the eigenvalues of matrix F (compose of the Hessian of the cost function and the system Jacobian of the error vector at each iteration) is negative, then the system becomes unstable. Also the paper shows that all the local minima of the three-term BP algorithm cost function are stable. The relationship between the learning parameters are established in this paper such that the stability conditions are met.