MVibCode: Multi-Channel Vibrotactile Codec Using Hierarchical Perceptual Clustering.

To achieve convincing remote vibrotactile experiences, it is necessary to transmit a large number of signal channels corresponding to dense interaction points on the human skin. This leads to a dramatic increase in the amount of data to be transmitted. In order to cope with these data efficiently, vibrotactile codecs have to be used to reduce the data rate demands. Although first vibrotactile codecs have been introduced in the past, they are mostly single-channel codecs and cannot achieve the required data reduction. Therefore, in this paper, a multi-channel vibrotactile codec is presented that extends a wavelet-based codec for single channel signals. By leveraging interchannel redundancies using channel clustering and differential coding, the presented codec is able to achieve a reduction of 69.1% in data rate compared to the state-of-the-art single-channel codec while maintaining a perceptual ST-SIM quality score of 95%.

A Novel Energy Compensation Scheme for Quality Enhancement in Time-Delayed Teleoperation With Multi-DoF Haptic Data Reduction and Communication.

Efficient and low-delay exchange of hapticinformation plays a key role for the design of high-fidelity teleoperation systems that operate over real-world communication networks. In the presence of communication unreliabilities such as delay and packet loss, a combination of stability-ensuring control schemes and haptic data reduction approaches is essential to address the challenges from both the control and communication perspective. In this paper, we extend our previous one degree-of-freedom (DoF) solution to 3DoF that combines the time-domain passivity approach (TDPA) and the perceptual deadband-based (DB) haptic data reduction approach. We also extend the 1DoF energy compensation (EC) scheme to 3DoF for mitigating the control artifacts introduced by the inter-play of the TDPA and DB approaches. The 3DoF solution is analyzed in detail and verified through a real-world teleoperation setup. The performance of the proposed method and the existing solutions is compared though objective and subjective assessments. Experimental results show that by jointly considering the evaluated objective/subjective quality and the required packet rate, the proposed 3DoF solution with the EC scheme improves system performance significantly compared to the state-of-the-art solutions.

Quality Enhancement of Compressed Vibrotactile Signals Using Recurrent Neural Networks and Residual Learning.

We present a neural network-based compression artifact removal technique for vibrotactile signals. The proposed decoder-side quality enhancement approach is based on recurrent neural networks (RNNs) and the principle of residual learning. We use a total of 8 nonlinear RNN layers trained to first estimate the difference between the original and the compressed signal. The estimated difference signal is then added to the compressed signal, followed by further linear processing steps to construct the enhanced signal. With our approach, we are able to enhance signals at almost all compression ratios by up to $1.25\ \mathrm {dB}$. For the signals in our data set, rougly 86% are enhanced in their quality. Through an ablation study, we show that every block of our network is functioning as intended and contributes to the compression artifact removal. Additionally, we show that the chosen network parameters maximize performance.

Perceptual Quality Assessment of Compressed Vibrotactile Signals Through Comparative Judgment.

In this article, we present a comprehensive scheme for the quality assessment of compressed vibrotactile signals with human assessors. Inspired by the multiple stimulus test with hidden reference and anchors (MUSHRA) from the audio domain, we designed a method in which each compressed signal is compared to its original signal and rated on a numerical scale. For each signal tested, the hidden reference and two anchor signals are used to validate the results and provide assessor screening criteria. Differing from previous approaches, our method is hierarchically structured and strictly timed in a sequential manner to avoid experimental confounds and provide precise psychophysical assessments. We validated our method in an experiment with 20 human participants in which we compared two state-of-the-art lossy codecs. The results show that, with our approach, the performance of different codecs can be compared effectively. Furthermore, the method also provides a measure of subjective quality at different data compression rates. The proposed procedure can be easily adapted to evaluate other vibrotactile codecs.

Adaptive Equalization of Vibrotactile Actuators.

In order to provide convincing artifical touch sensations, humans should be presented with high quality haptic stimuli. In the vibrotactile domain, signals are usually displayed through mechanical actuators. Current high quality actuators exhibit a high dynamic range and have the ability to display a wide range of frequencies. However, fundamentally all actuators introduce distortions into the displayed signals. These distortions are usually nonlinear with additive noise components and they can be detrimental to some vibrotactile application scenarios that require high signal playback precision. To neutralize these distortions, we propose a signal-based equalization setup with adaptive filtering. Such a setup is very general and can be applied to any actuator in a straightforward manner. We introduce a novel adaptive filter based on Volterra and bilinear filter models that is nonlinear and more robust than previous approaches. In simulations and experiments, we show that our filter model is able to consistently outperform existing adaptive filter models and equalize vibrotactile actuators efficiently.

Subjective Evaluation of the Spectral Temporal SIMilarity (ST-SIM) Measure for Vibrotactile Quality Assessment.

Recent standardization efforts for Tactile Internet (TI) and haptic codecs have paved the route for delivering tactile experiences in synchrony with audio and visual interaction components. Since humans are the ultimate consumers of tactile interactions, it is utmost important to develop objective quality assessment measures that are in close agreements with human perception. In this article, we present the results of a large-scale subjective study of a recently proposed objective quality assessment approach for vibrotactile signals called ST-SIM (Spectral Temporal SIMilarity). ST-SIM encompasses two components: perceptual spectral and temporal similarity measures. Two subjective experiments were conducted to validate ST-SIM, and elicited subjective ratings are used to create a VibroTactile Quality Assessment (VTQA) database. The VTQA database together with ST-SIM provide viable means to the development of vibrotactile compression and transmission applications. Our experimental results show that the ST-SIM highly correlates with human opinions in both experiments and significantly outperforms commonly used measures. The VTQA database is made publicly available at https://www.raniahassen.com/RESEARCH/.

Haptic Material Analysis and Classification Inspired by Human Exploratory Procedures.

We present a framework for the acquisition and parametrization of object material properties. The introduced acquisition device, denoted as Texplorer2, is able to extract surface material properties while a human operator is performing exploratory procedures. Using the Texplorer2, we scanned 184 material classes which we labeled according to biological, chemical, and geological naming conventions. Based on these real material recordings, we introduce a novel set of mathematical features which align with corresponding material properties defined in perceptual studies from related work and classify the materials using common machine learning techniques. Validation results of the proposed multi-modal features lead to an overall classification accuracy of 90.2% ± 1.2% and an F[Formula: see text] score of 0.90 ± 0.01 using the random forest classifier. For the sake of comparison, a deep neural network is trained and tested on images of the material surfaces; it outperforms (90.7% ± 1.0%) the hand-crafted feature-based approach yet leads to more critical misclassifications in terms of the proposed taxonomy.

A Tactile Computer Mouse for the Display of Surface Material Properties.

We present a novel input/output device to display the tactile properties of surface materials. The proposed Tactile Computer Mouse (TCM) is equipped with a series of actuators that can create perceptually relevant tactile cues to a user. The display capabilities of our TCM match the major tactile dimensions in human surface material perception, namely, hardness, friction, warmth, microscopic roughness, and macroscopic roughness. The TCM also preserves necessary interaction capabilities of a typical computer mouse. In addition to the TCM design, we introduce data acquisition procedures and concepts that are necessary to derive a parametric representation of a surface material and further demonstrate the corresponding rendering approach on the TCM. We conducted subjective experiments to determine tactile property ratings of real materials, perceived property ratings using the TCM, and how precisely subjects match the real materials to corresponding virtual material representations using the TCM in the absence of visual and audible clues. Our experimental results show that our TCM successfully displays the five fundamental tactile dimensions and that the twenty participants were able to perceive the TCM-produced virtual surface material tactile sensations with a recognition rate of 89.6 percent for ten different materials.

On the Transparency of Client/Server-Based Haptic Interaction with Deformable Objects.

This paper studies the transparency of client/server-based haptic interaction with simulated deformable objects. In the considered remote interaction scenario, the server simulates the computationally expensive finite-element-based object deformation at a low temporal update rate and transmits the result to the clients. There, the received deformation data is applied to the polygon mesh, which is used to locally render force feedback with a penalty-based force rendering algorithm at the required high rate. Based on a one-dimensional deformable object example, we analyze the transparency of this multi-rate architecture for a two-user interaction. Communication delay leads to increased force magnitudes and an increased impedance displayed to the clients that actively interact with the object. We propose a method that adjusts the stiffness used in the local force rendering at the clients to compensate for this effect. The conducted objective and subjective evaluations show that the proposed method successfully compensates for the effect of communication delay in the tested delay range of up to 100 ms.

Multimodal Feature-Based Surface Material Classification.

When a tool is tapped on or dragged over an object surface, vibrations are induced in the tool, which can be captured using acceleration sensors. The tool-surface interaction additionally creates audible sound waves, which can be recorded using microphones. Features extracted from camera images provide additional information about the surfaces. We present an approach for tool-mediated surface classification that combines these signals and demonstrate that the proposed method is robust against variable scan-time parameters. We examine freehand recordings of 69 textured surfaces recorded by different users and propose a classification system that uses perception-related features, such as hardness, roughness, and friction; selected features adapted from speech recognition, such as modified cepstral coefficients applied to our acceleration signals; and surface texture-related image features. We focus on mitigating the effect of variable contact force and exploration velocity conditions on these features as a prerequisite for a robust machine-learning-based approach for surface classification. The proposed system works without explicit scan force and velocity measurements. Experimental results show that our proposed approach allows for successful classification of textured surfaces under variable freehand movement conditions, exerted by different human operators. The proposed subset of six features, selected from the described sound, image, friction force, and acceleration features, leads to a classification accuracy of 74 percent in our experiments when combined with a Naive Bayes classifier.

Energy Prediction for Teleoperation Systems That Combine the Time Domain Passivity Approach with Perceptual Deadband-Based Haptic Data Reduction.

We study the combination of the perceptual deadband (PD)-based haptic packet rate reduction scheme with the time domain passivity approach (TDPA) for time-delayed teleoperation and propose a novel energy prediction (EP) scheme that deals with the conservative behavior of the resulting controller. The PD approach leads to irregular packet transmission, resulting in degraded system transparency and reduced teleoperation quality when the PD approach is combined with the TDPA. The proposed method (PD+TDPA+EP) adaptively predicts the system energy during communication interruptions and allows for larger energy output. This achieves less conservative control and improves the teleoperation quality. Evaluation of the displayed impedance shows that the PD+TDPA+EP method achieves improved system transparency, both objectively and subjectively, when compared with related approaches from literature. According to a subjective user study, the PD+TDPA+EP method allows for a high packet rate reduction (up to 80 percent) without noticeably distorting the perceived interaction quality. We also show that the PD+TDPA+EP method is preferred over related approaches from literature in a direct comparison test. Thus, with the proposed PD+TDPA+EP method, a high data reduction and a high teleoperation quality are simultaneously achieved for time-delayed teleoperation.

RDTC optimized compression of image-based scene representations (Part I): modeling and theoretical analysis.

Rendering of virtual views in interactive streaming of compressed image-based scene representations requires random access to arbitrary parts of the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. In this work, we extend the classical RD optimization approach using hybrid video coding concepts to a tradeoff between the storage rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). We present a theoretical model for this RDTC space with a focus on the decoding complexity and, in addition, the impact of client side caching on the RDTC measures is considered and evaluated. Experimental results qualitatively match those predicted by our theoretical models and show that an adaptation of the encoding process to scenario specific parameters like computational power of the receiver and channel throughput can significantly reduce the user-perceived delay or required storage for RDTC optimized streams compared to RD optimized or independently encoded scene representations.

RDTC optimized compression of image-based scene representations (Part II): practical coding.

Interactive streaming of compressed image-based scene representations requires random access to the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. The transmission data rate and the decoding complexity at the client have received attention in the literature, but their incorporation into the optimization procedure for compression and streaming is missing. If scenario-specific measures are considered, the traditional RD optimization can be extended to a tradeoff between the (storage) rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). In the first part of this sequel of papers, we have theoretically analyzed the RDTC space for the compression of densely sampled image-based scene representations. In this second part, we consider practical RDTC optimization. We propose a modeling and encoding parameter selection procedure that allows us to adapt the compression to scenario-specific properties. The impact of client side caching is considered and evaluated using an experimental testbed. Our results show a significant reduction of the user perceived delay, memory consumption or required minimum channel and storage bitrate for RDTC optimized streams compared to classical RD optimized or independently encoded scene representations.