A methodological framework to assess the accuracy of virtual reality hand-tracking systems: A case study with the Meta Quest 2.

Optical markerless hand-tracking systems incorporated into virtual reality (VR) headsets are transforming the ability to assess fine motor skills in VR. This promises to have far-reaching implications for the increased applicability of VR across scientific, industrial, and clinical settings. However, so far, there are little data regarding the accuracy, delay, and overall performance of these types of hand-tracking systems. Here we present a novel methodological framework based on a fixed grid of targets, which can be easily applied to measure these systems' absolute positional error and delay. We also demonstrate a method to assess finger joint-angle accuracy. We used this framework to evaluate the Meta Quest 2 hand-tracking system. Our results showed an average fingertip positional error of 1.1cm, an average finger joint angle error of 9.6∘ and an average temporal delay of 45.0 ms. This methodological framework provides a powerful tool to ensure the reliability and validity of data originating from VR-based, markerless hand-tracking systems.

Erratum: Touching with the eyes: Oculomotor self-touch induces illusory body ownership.

[This corrects the article DOI: 10.1016/j.isci.2023.106180.].

Touching with the eyes: Oculomotor self-touch induces illusory body ownership.

Self-touch plays a central role in the construction and plasticity of the bodily self. But which mechanisms support this role? Previous accounts emphasize the convergence of proprioceptive and tactile signals from the touching and the touched body parts. Here, we hypothesise that proprioceptive information is not necessary for self-touch modulation of body-ownership. Because eye movements do not rely on proprioceptive signals as limb movements do, we developed a novel oculomotor self-touch paradigm where voluntary eye movements generated corresponding tactile sensations. We then compared the effectiveness of eye versus hand self-touch movements in generating an illusion of owning a rubber hand. Voluntary oculomotor self-touch was as effective as hand-driven self-touch, suggesting that proprioception does not contribute to body ownership during self-touch. Self-touch may contribute to a unified sense of bodily self by binding voluntary actions toward our own body with their tactile consequences.

Skin properties and afferent density in the deterioration of tactile spatial acuity with age.

Tactile sensitivity is affected by age, as shown by the deterioration of spatial acuity assessed with the two-point discrimination task. This is assumed to be partly a result of age-related changes of the peripheral somatosensory system. In particular, in the elderly, the density of mechanoreceptive afferents decreases with age and the skin tends to become drier, less elastic and less stiff. To assess to what degree mechanoreceptor density, skin hydration, elasticity and stiffness can account for the deterioration of tactile spatial sensitivity observed in the elderly, several approaches were combined, including psychophysics, measurements of finger properties, modelling and simulation of the response of first-order tactile neurons. Psychophysics confirmed that the Elderly group has lower tactile acuity than the Young group. Correlation and commonality analysis showed that age was the most important factor in explaining decreases in behavioural performance. Biological elasticity, hydration and finger pad area were also involved. These results were consistent with the outcome of simulations showing that lower afferent density and lower Young's modulus (i.e. lower stiffness) negatively affected the tactile encoding of stimulus information. Simulations revealed that these changes resulted in a lower build-up of task-relevant stimulus information. Importantly, the reduction in discrimination performance with age in the simulation was less than that observed in the psychophysical testing, indicating that there are additional peripheral as well as central factors responsible for age-related changes in tactile discrimination. KEY POINTS: Ageing effects on tactile perception involve the deterioration of spatial sensitivity, although the contribution of central and peripheral factors is not clear. We combined psychophysics, measurements of finger properties, modelling and simulation of the response of first-order tactile neurons to investigate to what extent skin elasticity, stiffness, hydration, finger pad area and afferent density can account for the lower spatial sensitivity observed in the elderly. Correlation and commonality analysis revealed that age was the most important factor to predict behavioural performance. Skin biological elasticity, hydration and finger pad area contributed to a lesser extent. The simulation of first-order tactile neuron responses indicated that reduction in afferent density plays a major role in the deterioration of tactile spatial acuity. Simulations also showed that lower skin stiffness and lower afferent density affect the build-up of stimulus information and the response of SA1 (i.e. type 1 slowly adapting fibres) and RA1 (i.e. type 1 rapidly adapting fibres) afferent fibres.

Skin and Mechanoreceptor Contribution to Tactile Input for Perception: A Review of Simulation Models.

We review four current computational models that simulate the response of mechanoreceptors in the glabrous skin to tactile stimulation. The aim is to inform researchers in psychology, sensorimotor science and robotics who may want to implement this type of quantitative model in their research. This approach proves relevant to understanding of the interaction between skin response and neural activity as it avoids some of the limitations of traditional measurement methods of tribology, for the skin, and neurophysiology, for tactile neurons. The main advantage is to afford new ways of looking at the combined effects of skin properties on the activity of a population of tactile neurons, and to examine different forms of coding by tactile neurons. Here, we provide an overview of selected models from stimulus application to neuronal spiking response, including their evaluation in terms of existing data, and their applicability in relation to human tactile perception.

Causality shifts the perceived temporal order of audiovisual events.

Causality poses explicit constraints to the timing of sensory signals produced by events, as sound travels slower than light, making auditory stimulation to lag visual stimulation. Previous studies show that implied causality between unrelated events can change the tolerance of simultaneity judgments for audiovisual asynchronies. Here, we tested whether apparent causality between audiovisual events may also affect their perceived temporal order. To this aim, we used a disambiguated stream-bounce display, with stimuli either bouncing or streaming upon each other. These two possibilities were accompanied by a sound played around the time of contact between the objects, which could be perceived as causally related to the visual event according to the condition. Participants reported whether the visual contact occurred before or after the sound. Our results show that when the audiovisual stimuli are consistent with a causal interpretation (i.e., the bounce caused the sound), their perceived temporal order is systematically biased. Namely, a stimulus dynamic consistent with a causal relation induces a perceptual delay in the audio component, even if the sound was presented first. We thus conclude that causality can systematically bias the perceived temporal order of events, possibly due to expectations based on the dynamics of events in the real world. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Contact forces in roughness discrimination.

Roughness perception through fingertip contact with a textured surface can involve spatial and temporal cues from skin indentation and vibration respectively. Both types of cue may be affected by contact forces when feeling a surface and we ask whether, on a given trial, discrimination performance relates to contact forces. We examine roughness discrimination performance in a standard psychophysical method (2-interval forced choice, in which the participant identifies which of two spatial textures formed by parallel grooves feels rougher) while continuously measuring the normal and tangential forces applied by the index finger. Fourteen participants discriminated spatial gratings in fine (spatial period of 320-580 micron) and coarse (1520-1920 micron) ranges using static pressing or sliding contact of the index finger. Normal contact force (mean and variability) during pressing or sliding had relatively little impact on accuracy of roughness judgments except when pressing on surfaces in the coarse range. Discrimination was better for sliding than pressing in the fine but not the coarse range. In contrast, tangential force fluctuations during sliding were strongly related to roughness judgment accuracy.

Peri-personal space as a prior in coupling visual and proprioceptive signals.

It has been suggested that the integration of multiple body-related sources of information within the peri-personal space (PPS) scaffolds body ownership. However, a normative computational framework detailing the functional role of PPS is still missing. Here we cast PPS as a visuo-proprioceptive Bayesian inference problem whereby objects we see in our environment are more likely to engender sensations as they come near to the body. We propose that PPS is the reflection of such an increased a priori probability of visuo-proprioceptive coupling that surrounds the body. To test this prediction, we immersed participants in a highly realistic virtual reality (VR) simulation of their right arm and surrounding environment. We asked participants to perform target-directed reaches toward visual, proprioceptive, and visuo-proprioceptive targets while visually displaying their reaching arm (body visible condition) or not (body invisible condition). Reach end-points are analyzed in light of the coupling prior framework, where the extension of PPS is taken to be represented by the spatial dispersion of the coupling prior between visual and proprioceptive estimates of arm location. Results demonstrate that if the body is not visible, the spatial dispersion of the visuo-proprioceptive coupling relaxes, whereas the strength of coupling remains stable. By demonstrating a distance-dependent alteration in visual and proprioceptive localization attractive pull toward one another (stronger pull at small spatial discrepancies) when the body is rendered invisible - an effect that is well accounted for by the visuo-proprioceptive coupling prior - the results suggest that the visible body grounds visuo-proprioceptive coupling preferentially in the near vs. far space.

An experimental setup to test dual-joystick directional responses to vibrotactile stimuli.

In this paper we investigate the influence of the location of vibrotactile stimulation in triggering the response made using two handheld joysticks. In particular, we compare performance with stimuli delivered either using tactors placed on the palm or on the back of the hand and with attractive (move toward the vibration) or repulsive prompts (move away from the vibration). The experimental set-up comprised two joysticks and two gloves, each equipped with four pager motors along the cardinal directions. In different blocks, fifty-three volunteers were asked to move the joysticks as fast as possible either towards or away with respect to the direction specified by a set of vibrating motors. Results indicate that participants performed better with attractive prompts (i.e. responses were faster and with fewer errors in conditions where participants were asked to move the joysticks in the direction of the felt vibration) and that the stimulation delivered on the back of the hand from the gloves gives better results than the stimulation on the palm delivered by the joysticks. Finally, we analyse the laterality, the relation between correct responses and reaction times, the direction patterns for wrong responses and we perform an analysis on the Stimulus-Response Compatibility and on the training effect.

Modality-specific temporal constraints for state-dependent interval timing.

The ability to discriminate temporal intervals in the milliseconds-to-seconds range has been accounted for by proposing that duration is encoded in the dynamic change of a neuronal network state. A critical limitation of such networks is that their activity cannot immediately return to the initial state, a restriction that could hinder the processing of intervals presented in rapid succession. Empirical evidence in the literature consistently shows impaired duration discrimination performance for 100 ms intervals demarked by short auditory stimuli immediately preceded by a similar interval. Here we tested whether a similar interference is present with longer intervals (300 ms) demarked either by auditory or by visual stimuli. Our results show that while temporal estimates of auditory stimuli in this range are not affected by the interval between them, duration discrimination with this duration is significantly impaired with visual intervals presented in rapid succession. The difference in performance between modalities is overall consistent with state-dependent temporal computations, as it suggests that the limits due to slow neuronal dynamics greatly depends on the sensory modality with which the intervals are demarked, in line with the idea of intrinsic, modality-specific neural mechanisms for interval timing.

Musical Scales in Tone Sequences Improve Temporal Accuracy.

Predicting the time of stimulus onset is a key component in perception. Previous investigations of perceived timing have focused on the effect of stimulus properties such as rhythm and temporal irregularity, but the influence of non-temporal properties and their role in predicting stimulus timing has not been exhaustively considered. The present study aims to understand how a non-temporal pattern in a sequence of regularly timed stimuli could improve or bias the detection of temporal deviations. We presented interspersed sequences of 3, 4, 5, and 6 auditory tones where only the timing of the last stimulus could slightly deviate from isochrony. Participants reported whether the last tone was 'earlier' or 'later' relative to the expected regular timing. In two conditions, the tones composing the sequence were either organized into musical scales or they were random tones. In one experiment, all sequences ended with the same tone; in the other experiment, each sequence ended with a different tone. Results indicate higher discriminability of anisochrony with musical scales and with longer sequences, irrespective of the knowledge of the final tone. Such an outcome suggests that the predictability of non-temporal properties, as enabled by the musical scale pattern, can be a factor in determining the sensitivity of time judgments.

Correction: Temporal Regularity of the Environment Drives Time Perception.

[This corrects the article DOI: 10.1371/journal.pone.0159842.].

Temporal Regularity of the Environment Drives Time Perception.

It's reasonable to assume that a regularly paced sequence should be perceived as regular, but here we show that perceived regularity depends on the context in which the sequence is embedded. We presented one group of participants with perceptually regularly paced sequences, and another group of participants with mostly irregularly paced sequences (75% irregular, 25% regular). The timing of the final stimulus in each sequence could be varied. In one experiment, we asked whether the last stimulus was regular or not. We found that participants exposed to an irregular environment frequently reported perfectly regularly paced stimuli to be irregular. In a second experiment, we asked participants to judge whether the final stimulus was presented before or after a flash. In this way, we were able to determine distortions in temporal perception as changes in the timing necessary for the sound and the flash to be perceived synchronous. We found that within a regular context, the perceived timing of deviant last stimuli changed so that the relative anisochrony appeared to be perceptually decreased. In the irregular context, the perceived timing of irregular stimuli following a regular sequence was not affected. These observations suggest that humans use temporal expectations to evaluate the regularity of sequences and that expectations are combined with sensory stimuli to adapt perceived timing to follow the statistics of the environment. Expectations can be seen as a-priori probabilities on which perceived timing of stimuli depend.

Optimal Perceived Timing: Integrating Sensory Information with Dynamically Updated Expectations.

The environment has a temporal structure, and knowing when a stimulus will appear translates into increased perceptual performance. Here we investigated how the human brain exploits temporal regularity in stimulus sequences for perception. We find that the timing of stimuli that occasionally deviate from a regularly paced sequence is perceptually distorted. Stimuli presented earlier than expected are perceptually delayed, whereas stimuli presented on time and later than expected are perceptually accelerated. This result suggests that the brain regularizes slightly deviant stimuli with an asymmetry that leads to the perceptual acceleration of expected stimuli. We present a Bayesian model for the combination of dynamically-updated expectations, in the form of a priori probability of encountering future stimuli, with incoming sensory information. The asymmetries in the results are accounted for by the asymmetries in the distributions involved in the computational process.

The consistency of crossmodal synchrony perception across the visual, auditory, and tactile senses.

Crossmodal judgments of relative timing commonly yield a nonzero point of subjective simultaneity (PSS). Here, we test whether subjective simultaneity is coherent across all pairwise combinations of the visual, auditory, and tactile modalities. To this end, we examine PSS estimates for transitivity: If Stimulus A has to be presented x ms before Stimulus B to result in subjective simultaneity, and B y ms before C, then A and C should appear simultaneous when A precedes C by z ms, where z = x + y. We obtained PSS estimates via 2 different timing judgment tasks-temporal order judgments (TOJs) and synchrony judgments (SJs)-thus allowing us to examine the relationship between TOJ and SJ. We find that (a) SJ estimates do not violate transitivity, and that (b) TOJ and SJ data are linearly related. Together, these findings suggest that both TOJ and SJ access the same perceptual representation of simultaneity and that this representation is globally coherent across the tested modalities. Furthermore, we find that (b) TOJ estimates are intransitive. This is consistent with the proposal that while the perceptual representation of simultaneity is coherent, relative timing judgments that access this representation can at times be incoherent with each other because of postperceptual response biases. (PsycINFO Database Record

Differential processing of binocular and monocular gloss cues in human visual cortex.

The visual impression of an object's surface reflectance ("gloss") relies on a range of visual cues, both monocular and binocular. Whereas previous imaging work has identified processing within ventral visual areas as important for monocular cues, little is known about cortical areas involved in processing binocular cues. Here, we used human functional MRI (fMRI) to test for brain areas selectively involved in the processing of binocular cues. We manipulated stereoscopic information to create four conditions that differed in their disparity structure and in the impression of surface gloss that they evoked. We performed multivoxel pattern analysis to find areas whose fMRI responses allow classes of stimuli to be distinguished based on their depth structure vs. material appearance. We show that higher dorsal areas play a role in processing binocular gloss information, in addition to known ventral areas involved in material processing, with ventral area lateral occipital responding to both object shape and surface material properties. Moreover, we tested for similarities between the representation of gloss from binocular cues and monocular cues. Specifically, we tested for transfer in the decoding performance of an algorithm trained on glossy vs. matte objects defined by either binocular or by monocular cues. We found transfer effects from monocular to binocular cues in dorsal visual area V3B/kinetic occipital (KO), suggesting a shared representation of the two cues in this area. These results indicate the involvement of mid- to high-level visual circuitry in the estimation of surface material properties, with V3B/KO potentially playing a role in integrating monocular and binocular cues.

Perceived time and temporal structure: Neural entrainment to isochronous stimulation increases duration estimates.

Distortions of perceived duration can give crucial insights into the mechanisms that underlie the processing and representation of stimulus timing. One factor that affects duration estimates is the temporal structure of stimuli that fill an interval. For example, regular filling (isochronous interval) leads to an overestimation of perceived duration as compared to irregular filling (anisochronous interval). In the present article, we use electroencephalography (EEG) to investigate the neural basis of this subjective lengthening of perceived duration with isochrony. In a two-interval forced choice task, participants judged which of two intervals lasts longer - one always being isochronous, the other one anisochronous. Response proportions confirm the subjective overestimation of isochronous intervals. At the neural level, isochronous sequences are associated with enhanced pairwise phase consistency (PPC) at the stimulation frequency, reflecting the brain's entrainment to the regular stimulation. The PPC over the entrainment channels is further enhanced for isochronous intervals that are reported to be longer, and the magnitude of this PCC effect correlates with the amount of perceptual bias. Neural entrainment has been proposed as a mechanism of attentional selection, enabling increased neural responsiveness toward stimuli that arrive at an expected point in time. The present results support the proposed relationship between neural response magnitudes and temporal estimates: An increase in neural responsiveness leads to a more pronounced representation of the individual stimuli filling the interval and in turn to a subjective increase in duration.

Look but don't touch: Visual cues to surface structure drive somatosensory cortex.

When planning interactions with nearby objects, our brain uses visual information to estimate shape, material composition, and surface structure before we come into contact with them. Here we analyse brain activations elicited by different types of visual appearance, measuring fMRI responses to objects that are glossy, matte, rough, or textured. In addition to activation in visual areas, we found that fMRI responses are evoked in the secondary somatosensory area (S2) when looking at glossy and rough surfaces. This activity could be reliably discriminated on the basis of tactile-related visual properties (gloss, rough, and matte), but importantly, other visual properties (i.e., coloured texture) did not substantially change fMRI activity. The activity could not be solely due to tactile imagination, as asking explicitly to imagine such surface properties did not lead to the same results. These findings suggest that visual cues to an object's surface properties evoke activity in neural circuits associated with tactile stimulation. This activation may reflect the a-priori probability of the physics of the interaction (i.e., the expectation of upcoming friction) that can be used to plan finger placement and grasp force.

Timing Rhythms: Perceived Duration Increases with a Predictable Temporal Structure of Short Interval Fillers.

Variations in the temporal structure of an interval can lead to remarkable differences in perceived duration. For example, it has previously been shown that isochronous intervals, that is, intervals filled with temporally regular stimuli, are perceived to last longer than intervals left empty or filled with randomly timed stimuli. Characterizing the extent of such distortions is crucial to understanding how duration perception works. One account to explain effects of temporal structure is a non-linear accumulator-counter mechanism reset at the beginning of every subinterval. An alternative explanation based on entrainment to regular stimulation posits that the neural response to each filler stimulus in an isochronous sequence is amplified and a higher neural response may lead to an overestimation of duration. If entrainment is the key that generates response amplification and the distortions in perceived duration, then any form of predictability in the temporal structure of interval fillers should lead to the perception of an interval that lasts longer than a randomly filled one. The present experiments confirm that intervals filled with fully predictable rhythmically grouped stimuli lead to longer perceived duration than anisochronous intervals. No general over- or underestimation is registered for rhythmically grouped compared to isochronous intervals. However, we find that the number of stimuli in each group composing the rhythm also influences perceived duration. Implications of these findings for a non-linear clock model as well as a neural response magnitude account of perceived duration are discussed.

Speed/accuracy tradeoff in force perception.

There is a well-known tradeoff between speed and accuracy in judgments made under uncertainty. Diffusion models have been proposed to capture the increase in response time for more uncertain decisions and the change in performance due to a prioritization of speed or accuracy in the responses. Experimental paradigms have been confined to the visual modality and model analysis have mostly used quantile-probability (QP) plots--response probability as a function of quantized RTs. Here, we extend diffusion modeling to haptics and test a novel type of analysis for judging model fitting. Participants classified force stimuli applied to the hand as "high" or "low." Data in QP plots indicate that the diffusion model captures well the overall pattern of responses in conditions where either speed or accuracy has been prioritized. To further the analysis, we compute just noticeable difference (JND) values separately for responses delivered with different RTs--we define these plots as JND quantile. The pattern of results evidences that slower responses lead to better force discrimination up to a plateau that is unaffected by prioritization instructions. Instead, the diffusion model predicts two well-separated plateaus depending on the condition. We propose that analyzing the relation between JNDs and response time should be considered in the evaluation of the diffusion model beyond the haptic modality, thus including vision.