Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation.

Segmenting unknown or anomalous object instances is a critical task in autonomous driving applications, and it is approached traditionally as a per-pixel classification problem. However, reasoning individually about each pixel without considering their contextual semantics results in high uncertainty around the objects' boundaries and numerous false positives. We propose a paradigm change by shifting from a per-pixel classification to a mask classification. Our mask-based method, Mask2Anomaly, demonstrates the feasibility of integrating a mask-classification architecture to jointly address anomaly segmentation, open-set semantic segmentation, and open-set panoptic segmentation. Mask2Anomaly includes several technical novelties that are designed to improve the detection of anomalies/unknown objects: i) a global masked attention module to focus individually on the foreground and background regions; ii) a mask contrastive learning that maximizes the margin between an anomaly and known classes; iii) a mask refinement solution to reduce false positives; and iv) a novel approach to mine unknown instances based on the mask- architecture properties. By comprehensive qualitative and qualitative evaluation, we show Mask2Anomaly achieves new state-of-the-art results across the benchmarks of anomaly segmentation, open-set semantic segmentation, and open-set panoptic segmentation. The code and pre-trained models are available: https://github.com/shyam671/Mask2Anomaly-Unmasking-Anomalies-in-Road-Scene-Segmentation/tree/main.

Self-Supervised Learning Across Domains.

Human adaptability relies crucially on learning and merging knowledge from both supervised and unsupervised tasks: the parents point out few important concepts, but then the children fill in the gaps on their own. This is particularly effective, because supervised learning can never be exhaustive and thus learning autonomously allows to discover invariances and regularities that help to generalize. In this paper we propose to apply a similar approach to the problem of object recognition across domains: our model learns the semantic labels in a supervised fashion, and broadens its understanding of the data by learning from self-supervised signals on the same images. This secondary task helps the network to focus on object shapes, learning concepts like spatial orientation and part correlation, while acting as a regularizer for the classification task over multiple visual domains. Extensive experiments confirm our intuition and show that our multi-task method, combining supervised and self-supervised knowledge, provides competitive results with respect to more complex domain generalization and adaptation solutions. It also proves its potential in the novel and challenging predictive and partial domain adaptation scenarios.

Modeling the Background for Incremental and Weakly-Supervised Semantic Segmentation.

Deep neural networks have enabled major progresses in semantic segmentation. However, even the most advanced neural architectures suffer from important limitations. First, they are vulnerable to catastrophic forgetting, i.e., they perform poorly when they are required to incrementally update their model as new classes are available. Second, they rely on large amount of pixel-level annotations to produce accurate segmentation maps. To tackle these issues, we introduce a novel incremental class learning approach for semantic segmentation taking into account a peculiar aspect of this task: since each training step provides annotation only for a subset of all possible classes, pixels of the background class exhibit a semantic shift. Therefore, we revisit the traditional distillation paradigm by designing novel loss terms which explicitly account for the background shift. Additionally, we introduce a novel strategy to initialize classifier's parameters at each step in order to prevent biased predictions toward the background class. Finally, we demonstrate that our approach can be extended to point- and scribble-based weakly supervised segmentation, modeling the partial annotations to create priors for unlabeled pixels. We demonstrate the effectiveness of our approach with an extensive evaluation on the Pascal-VOC, ADE20K, and Cityscapes datasets, significantly outperforming state-of-the-art methods.

Inferring Latent Domains for Unsupervised Deep Domain Adaptation.

Unsupervised Domain Adaptation (UDA) refers to the problem of learning a model in a target domain where labeled data are not available by leveraging information from annotated data in a source domain. Most deep UDA approaches operate in a single-source, single-target scenario, i.e., they assume that the source and the target samples arise from a single distribution. However, in practice most datasets can be regarded as mixtures of multiple domains. In these cases, exploiting traditional single-source, single-target methods for learning classification models may lead to poor results. Furthermore, it is often difficult to provide the domain labels for all data points, i.e. latent domains should be automatically discovered. This paper introduces a novel deep architecture which addresses the problem of UDA by automatically discovering latent domains in visual datasets and exploiting this information to learn robust target classifiers. Specifically, our architecture is based on two main components, i.e. a side branch that automatically computes the assignment of each sample to its latent domain and novel layers that exploit domain membership information to appropriately align the distribution of the CNN internal feature representations to a reference distribution. We evaluate our approach on publicly available benchmarks, showing that it outperforms state-of-the-art domain adaptation methods.

MultiDIAL: Domain Alignment Layers for (Multisource) Unsupervised Domain Adaptation.

One of the main challenges for developing visual recognition systems working in the wild is to devise computational models immune from the domain shift problem, i.e., accurate when test data are drawn from a (slightly) different data distribution than training samples. In the last decade, several research efforts have been devoted to devise algorithmic solutions for this issue. Recent attempts to mitigate domain shift have resulted into deep learning models for domain adaptation which learn domain-invariant representations by introducing appropriate loss terms, by casting the problem within an adversarial learning framework or by embedding into deep network specific domain normalization layers. This paper describes a novel approach for unsupervised domain adaptation. Similarly to previous works we propose to align the learned representations by embedding them into appropriate network feature normalization layers. Opposite to previous works, our Domain Alignment Layers are designed not only to match the source and target feature distributions but also to automatically learn the degree of feature alignment required at different levels of the deep network. Differently from most previous deep domain adaptation methods, our approach is able to operate in a multi-source setting. Thorough experiments on four publicly available benchmarks confirm the effectiveness of our approach.

Publisher Correction: Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics.

A hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person's life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among them the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics.

Adaptive learning to speed-up control of prosthetic hands: A few things everybody should know.

Domain adaptation methods have been proposed to reduce the training efforts needed to control an upper-limb prosthesis by adapting well performing models from previous subjects to the new subject. These studies generally reported impressive reductions in the required number of training samples to achieve a certain level of accuracy for intact subjects. We further investigate two popular methods in this field to verify whether this result also applies to amputees. Our findings show instead that this improvement can largely be attributed to a suboptimal hyperparameter configuration. When hyperparameters are appropriately tuned, the standard approach that does not exploit prior information performs on par with the more complicated transfer learning algorithms. Additionally, earlier studies erroneously assumed that the number of training samples relates proportionally to the efforts required from the subject. However, a repetition of a movement is the atomic unit for subjects and the total number of repetitions should therefore be used as reliable measure for training efforts. Also when correcting for this mistake, we do not find any performance increase due to the use of prior models.

Megane Pro: Myo-electricity, visual and gaze tracking data acquisitions to improve hand prosthetics.

During the past 60 years scientific research proposed many techniques to control robotic hand prostheses with surface electromyography (sEMG). Few of them have been implemented in commercial systems also due to limited robustness that may be improved with multimodal data. This paper presents the first acquisition setup, acquisition protocol and dataset including sEMG, eye tracking and computer vision to study robotic hand control. A data analysis on healthy controls gives a first idea of the capabilities and constraints of the acquisition procedure that will be applied to amputees in a next step. Different data sources are not fused together in the analysis. Nevertheless, the results support the use of the proposed multimodal data acquisition approach for prosthesis control. The sEMG movement classification results confirm that it is possible to classify several grasps with sEMG alone. sEMG can detect the grasp type and also small differences in the grasped object (accuracy: 95%). The simultaneous recording of eye tracking and scene camera data shows that these sensors allow performing object detection for grasp selection and that several neurocognitive parameters need to be taken into account for this. In conclusion, this work on intact subjects presents an innovative acquisition setup and protocol. The first results in terms of data analysis are promising and set the basis for future work on amputees, aiming to improve the robustness of prostheses with multimodal data.

Repeatability of grasp recognition for robotic hand prosthesis control based on sEMG data.

Control methods based on sEMG obtained promising results for hand prosthetics. Control system robustness is still often inadequate and does not allow the amputees to perform a large number of movements useful for everyday life. Only few studies analyzed the repeatability of sEMG classification of hand grasps. The main goals of this paper are to explore repeatability in sEMG data and to release a repeatability database with the recorded experiments. The data are recorded from 10 intact subjects repeating 7 grasps 12 times, twice a day for 5 days. The data are publicly available on the Ninapro web page. The analysis for the repeatability is based on the comparison of movement classification accuracy in several data acquisitions and for different subjects. The analysis is performed using mean absolute value and waveform length features and a Random Forest classifier. The accuracy obtained by training and testing on acquisitions at different times is on average 27.03% lower than training and testing on the same acquisition. The results obtained by training and testing on different acquisitions suggest that previous acquisitions can be used to train the classification algorithms. The inter-subject variability is remarkable, suggesting that specific characteristics of the subjects can affect repeatibility and sEMG classification accuracy. In conclusion, the results of this paper can contribute to develop more robust control systems for hand prostheses, while the presented data allows researchers to test repeatability in further analyses.

Effect of clinical parameters on the control of myoelectric robotic prosthetic hands.

Improving the functionality of prosthetic hands with noninvasive techniques is still a challenge. Surface electromyography (sEMG) currently gives limited control capabilities; however, the application of machine learning to the analysis of sEMG signals is promising and has recently been applied in practice, but many questions still remain. In this study, we recorded the sEMG activity of the forearm of 11 male subjects with transradial amputation who were mentally performing 40 hand and wrist movements. The classification performance and the number of independent movements (defined as the subset of movements that could be distinguished with >90% accuracy) were studied in relationship to clinical parameters related to the amputation. The analysis showed that classification accuracy and the number of independent movements increased significantly with phantom limb sensation intensity, remaining forearm percentage, and temporal distance to the amputation. The classification results suggest the possibility of naturally controlling up to 11 movements of a robotic prosthetic hand with almost no training. Knowledge of the relationship between classification accuracy and clinical parameters adds new information regarding the nature of phantom limb pain as well as other clinical parameters, and it can lay the foundations for future "functional amputation" procedures in surgery.

Characterization of a benchmark database for myoelectric movement classification.

In this paper, we characterize the Ninapro database and its use as a benchmark for hand prosthesis evaluation. The database is a publicly available resource that aims to support research on advanced myoelectric hand prostheses. The database is obtained by jointly recording surface electromyography signals from the forearm and kinematics of the hand and wrist while subjects perform a predefined set of actions and postures. Besides describing the acquisition protocol, overall features of the datasets and the processing procedures in detail, we present benchmark classification results using a variety of feature representations and classifiers. Our comparison shows that simple feature representations such as mean absolute value and waveform length can achieve similar performance to the computationally more demanding marginal discrete wavelet transform. With respect to classification methods, the nonlinear support vector machine was found to be the only method consistently achieving high performance regardless of the type of feature representation. Furthermore, statistical analysis of these results shows that classification accuracy is negatively correlated with the subject's Body Mass Index. The analysis and the results described in this paper aim to be a strong baseline for the Ninapro database. Thanks to the Ninapro database (and the characterization described in this paper), the scientific community has the opportunity to converge to a common position on hand movement recognition by surface electromyography, a field capable to strongly affect hand prosthesis capabilities.

Proceedings of the first workshop on Peripheral Machine Interfaces: going beyond traditional surface electromyography.

One of the hottest topics in rehabilitation robotics is that of proper control of prosthetic devices. Despite decades of research, the state of the art is dramatically behind the expectations. To shed light on this issue, in June, 2013 the first international workshop on Present and future of non-invasive peripheral nervous system (PNS)-Machine Interfaces (MI; PMI) was convened, hosted by the International Conference on Rehabilitation Robotics. The keyword PMI has been selected to denote human-machine interfaces targeted at the limb-deficient, mainly upper-limb amputees, dealing with signals gathered from the PNS in a non-invasive way, that is, from the surface of the residuum. The workshop was intended to provide an overview of the state of the art and future perspectives of such interfaces; this paper represents is a collection of opinions expressed by each and every researcher/group involved in it.

Movement error rate for evaluation of machine learning methods for sEMG-based hand movement classification.

There has been increasing interest in applying learning algorithms to improve the dexterity of myoelectric prostheses. In this work, we present a large-scale benchmark evaluation on the second iteration of the publicly released NinaPro database, which contains surface electromyography data for 6 DOF force activations as well as for 40 discrete hand movements. The evaluation involves a modern kernel method and compares performance of three feature representations and three kernel functions. Both the force regression and movement classification problems can be learned successfully when using a nonlinear kernel function, while the exp- χ(2) kernel outperforms the more popular radial basis function kernel in all cases. Furthermore, combining surface electromyography and accelerometry in a multimodal classifier results in significant increases in accuracy as compared to when either modality is used individually. Since window-based classification accuracy should not be considered in isolation to estimate prosthetic controllability, we also provide results in terms of classification mistakes and prediction delay. To this extent, we propose the movement error rate as an alternative to the standard window-based accuracy. This error rate is insensitive to prediction delays and it allows us therefore to quantify mistakes and delays as independent performance characteristics. This type of analysis confirms that the inclusion of accelerometry is superior, as it results in fewer mistakes while at the same time reducing prediction delay.

Stable myoelectric control of a hand prosthesis using non-linear incremental learning.

Stable myoelectric control of hand prostheses remains an open problem. The only successful human-machine interface is surface electromyography, typically allowing control of a few degrees of freedom. Machine learning techniques may have the potential to remove these limitations, but their performance is thus far inadequate: myoelectric signals change over time under the influence of various factors, deteriorating control performance. It is therefore necessary, in the standard approach, to regularly retrain a new model from scratch. We hereby propose a non-linear incremental learning method in which occasional updates with a modest amount of novel training data allow continual adaptation to the changes in the signals. In particular, Incremental Ridge Regression and an approximation of the Gaussian Kernel known as Random Fourier Features are combined to predict finger forces from myoelectric signals, both finger-by-finger and grouped in grasping patterns. We show that the approach is effective and practically applicable to this problem by first analyzing its performance while predicting single-finger forces. Surface electromyography and finger forces were collected from 10 intact subjects during four sessions spread over two different days; the results of the analysis show that small incremental updates are indeed effective to maintain a stable level of performance. Subsequently, we employed the same method on-line to teleoperate a humanoid robotic arm equipped with a state-of-the-art commercial prosthetic hand. The subject could reliably grasp, carry and release everyday-life objects, enforcing stable grasping irrespective of the signal changes, hand/arm movements and wrist pronation and supination.

Electromyography data for non-invasive naturally-controlled robotic hand prostheses.

Recent advances in rehabilitation robotics suggest that it may be possible for hand-amputated subjects to recover at least a significant part of the lost hand functionality. The control of robotic prosthetic hands using non-invasive techniques is still a challenge in real life: myoelectric prostheses give limited control capabilities, the control is often unnatural and must be learned through long training times. Meanwhile, scientific literature results are promising but they are still far from fulfilling real-life needs. This work aims to close this gap by allowing worldwide research groups to develop and test movement recognition and force control algorithms on a benchmark scientific database. The database is targeted at studying the relationship between surface electromyography, hand kinematics and hand forces, with the final goal of developing non-invasive, naturally controlled, robotic hand prostheses. The validation section verifies that the data are similar to data acquired in real-life conditions, and that recognition of different hand tasks by applying state-of-the-art signal features and machine-learning algorithms is possible.

Classification of hand movements in amputated subjects by sEMG and accelerometers.

Numerous recent studies have aimed to improve myoelectric control of prostheses. However, the majority of these studies is characterized by two problems that could be easily fulfilled with recent resources supplied by the scientific literature. First, the majority of these studies use only intact subjects, with the unproved assumption that the results apply equally to amputees. Second, usually only electromyography data are used, despite other sensors (e.g., accelerometers) being easy to include into a real life prosthesis control system. In this paper we analyze the mentioned problems by the classification of 40 hand movements in 5 amputated and 40 intact subjects, using both sEMG and accelerometry data and applying several different state of the art methods. The datasets come from the NinaPro database, which supplies publicly available sEMG data to develop and test machine learning algorithms for prosthetics. The number of subjects can seem small at first sight, but it is not considering the literature of the field (which has to face the difficulty of recruiting trans-radial hand amputated subjects). Our results indicate that the maximum average classification accuracy for amputated subjects is 61.14%, which is just 15.86% less than intact subjects, and they show that intact subjects results can be used as proxy measure for amputated subjects. Finally, our comparison shows that accelerometry as a modality is less affected by amputation than electromyography, suggesting that real life prosthetics performance may easily be improved by inclusion of accelerometers.

Natural control capabilities of robotic hands by hand amputated subjects.

People with transradial hand amputations who own a myoelectric prosthesis currently have some control capabilities via sEMG. However, the control systems are still limited and not natural. The Ninapro project is aiming at helping the scientific community to overcome these limits through the creation of publicly available electromyography data sources to develop and test machine learning algorithms. In this paper we describe the movement classification results gained from three subjects with an homogeneous level of amputation, and we compare them with the results of 40 intact subjects. The number of considered subjects can seem small at first sight, but it is not considering the literature of the field (which has to face the difficulty of recruiting trans-radial hand amputated subjects). The classification is performed with four different classifiers and the obtained balanced classification rates are up to 58.6% on 50 movements, which is an excellent result compared to the current literature. Successively, for each subject we find a subset of up to 9 highly independent movements, (defined as movements that can be distinguished with more than 90% accuracy), which is a deeply innovative step in literature. The natural control of a robotic hand in so many movements could lead to an immediate progress in robotic hand prosthetics and it could deeply change the quality of life of amputated subjects.

Learning Categories From Few Examples With Multi Model Knowledge Transfer.

Learning a visual object category from few samples is a compelling and challenging problem. In several real-world applications collecting many annotated data is costly and not always possible. However, a small training set does not allow to cover the high intraclass variability typical of visual objects. In this condition, machine learning methods provide very few guarantees. This paper presents a discriminative model adaptation algorithm able to proficiently learn a target object with few examples by relying on other previously learned source categories. The proposed method autonomously chooses from where and how much to transfer information by solving a convex optimization problem which ensures to have the minimal leave-one-out error on the available training set. We analyze several properties of the described approach and perform an extensive experimental comparison with other existing transfer solutions, consistently showing the value of our algorithm.

Exploiting accelerometers to improve movement classification for prosthetics.

Recent studies have explored the integration of additional input modalities to improve myoelectric control of prostheses. Arm dynamics in particular are an interesting option, as these can be measured easily by means of accelerometers. In this work, the benefit of accelerometer signals is demonstrated on a large scale movement classification task, consisting of 40 hand and wrist movements obtained from 20 subjects. The results demonstrate that the accelerometer modality is indeed highly informative and even outperforms surface electromyography in terms of classification accuracy. The highest accuracy, however, is obtained when both modalities are integrated in a multi-modal classifier.