Different time scales of common-cause evidence shape multisensory integration, recalibration and motor adaptation.

Perceptual coherence in the face of discrepant multisensory signals is achieved via the processes of multisensory integration, recalibration and sometimes motor adaptation. These supposedly operate on different time scales, with integration reducing immediate sensory discrepancies and recalibration and motor adaptation reflecting the cumulative influence of their recent history. Importantly, whether discrepant signals are bound during perception is guided by the brains' inference of whether they originate from a common cause. When combined, these two notions lead to the hypothesis that the time scales on which integration and recalibration (or motor adaptation) operate are associated with different time scales of evidence about a common cause underlying two signals. We tested this prediction in a well-established visuo-motor paradigm, in which human participants performed visually guided hand movements. The kinematic correlation between hand and cursor movements indicates their common origin, which allowed us to manipulate the common-cause evidence by titrating this correlation. Specifically, we dissociated hand and cursor signals during individual movements while preserving their correlation across the series of movement endpoints. Following our hypothesis, this manipulation reduced integration compared with a condition in which visual and proprioceptive signals were perfectly correlated. In contrast, recalibration and motor adaption were not affected by this manipulation. This supports the notion that multisensory integration and recalibration deal with sensory discrepancies on different time scales guided by common-cause evidence: Integration is prompted by local common-cause evidence and reduces immediate discrepancies, whereas recalibration and motor adaptation are prompted by global common-cause evidence and reduce persistent discrepancies.

Short-term effects of visuomotor discrepancies on multisensory integration, proprioceptive recalibration, and motor adaptation.

Information about the position of our hand is provided by multisensory signals that are often not perfectly aligned. Discrepancies between the seen and felt hand position or its movement trajectory engage the processes of 1) multisensory integration, 2) sensory recalibration, and 3) motor adaptation, which adjust perception and behavioral responses to apparently discrepant signals. To foster our understanding of the coemergence of these three processes, we probed their short-term dependence on multisensory discrepancies in a visuomotor task that has served as a model for multisensory perception and motor control previously. We found that the well-established integration of discrepant visual and proprioceptive signals is tied to the immediate discrepancy and independent of the outcome of the integration of discrepant signals in immediately preceding trials. However, the strength of integration was context dependent, being stronger in an experiment featuring stimuli that covered a smaller range of visuomotor discrepancies (±15°) compared with one covering a larger range (±30°). Both sensory recalibration and motor adaptation for nonrepeated movement directions were absent after two bimodal trials with same or opposite visuomotor discrepancies. Hence our results suggest that short-term sensory recalibration and motor adaptation are not an obligatory consequence of the integration of preceding discrepant multisensory signals.NEW & NOTEWORTHY The functional relation between multisensory integration and recalibration remains debated. We here refute the notion that they coemerge in an obligatory manner and support the hypothesis that they serve distinct goals of perception.

Visuo-proprioceptive integration and recalibration with multiple visual stimuli.

To organize the plethora of sensory signals from our environment into a coherent percept, our brain relies on the processes of multisensory integration and sensory recalibration. We here asked how visuo-proprioceptive integration and recalibration are shaped by the presence of more than one visual stimulus, hence paving the way to study multisensory perception under more naturalistic settings with multiple signals per sensory modality. We used a cursor-control task in which proprioceptive information on the endpoint of a reaching movement was complemented by two visual stimuli providing additional information on the movement endpoint. The visual stimuli were briefly shown, one synchronously with the hand reaching the movement endpoint, the other delayed. In Experiment 1, the judgments of hand movement endpoint revealed integration and recalibration biases oriented towards the position of the synchronous stimulus and away from the delayed one. In Experiment 2 we contrasted two alternative accounts: that only the temporally more proximal visual stimulus enters integration similar to a winner-takes-all process, or that the influences of both stimuli superpose. The proprioceptive biases revealed that integration-and likely also recalibration-are shaped by the superposed contributions of multiple stimuli rather than by only the most powerful individual one.

Exploring the time window for causal inference and the multisensory integration of actions and their visual effects.

Successful computer use requires the operator to link the movement of the cursor to that of his or her hand. Previous studies suggest that the brain establishes this perceptual link through multisensory integration, whereby the causality evidence that drives the integration is provided by the correlated hand and cursor movement trajectories. Here, we explored the temporal window during which this causality evidence is effective. We used a basic cursor-control task, in which participants performed out-and-back reaching movements with their hand on a digitizer tablet. A corresponding cursor movement could be shown on a monitor, yet slightly rotated by an angle that varied from trial to trial. Upon completion of the backward movement, participants judged the endpoint of the outward hand or cursor movement. The mutually biased judgements that typically result reflect the integration of the proprioceptive information on hand endpoint with the visual information on cursor endpoint. We here manipulated the time period during which the cursor was visible, thereby selectively providing causality evidence either before or after sensory information regarding the to-be-judged movement endpoint was available. Specifically, the cursor was visible either during the outward or backward hand movement (conditions Out and Back, respectively). Our data revealed reduced integration in the condition Back compared with the condition Out, suggesting that causality evidence available before the to-be-judged movement endpoint is more powerful than later evidence in determining how strongly the brain integrates the endpoint information. This finding further suggests that sensory integration is not delayed until a judgement is requested.

Explicit knowledge of sensory non-redundancy can reduce the strength of multisensory integration.

The brain integrates incoming sensory signals to a degree that depends on the signals' redundancy. Redundancy-which is commonly high when signals originate from a common physical object or event-is estimated by the brain from the signals' spatial and/or temporal correspondence. Here we tested whether verbally instructed knowledge of non-redundancy can also be used to reduce the strength of the sensory integration. We used a cursor-control task in which cursor motions in the frontoparallel plane were controlled by hand movements in the horizontal plane, yet with a small and randomly varying visuomotor rotation that created spatial discrepancies between hand and cursor positions. Consistent with previous studies, we found mutual biases in the hand and cursor position judgments, indicating partial sensory integration. The integration was reduced in strength, but not eliminated, after participants were verbally informed about the non-redundancy (i.e., the spatial discrepancies) in the hand and cursor positions. Comparisons with model predictions excluded confounding bottom-up effects of the non-redundancy instruction. Our findings thus show that participants have top-down control over the degree to which they integrate sensory information. Additionally, we found that the magnitude of this top-down modulatory capability is a reliable individual trait. A comparison between participants with and without video-gaming experience tentatively suggested a relation between top-down modulation of integration strength and attentional control.

Sensory integration of movements and their visual effects is not enhanced by spatial proximity.

Spatial proximity enhances the sensory integration of exafferent position information, likely because it indicates whether the information comes from a single physical source. Does spatial proximity also affect the integration of position information regarding an action (here a hand movement) with that of its visual effect (here a cursor motion), that is, when the sensory information comes from physically distinct objects? In this study, participants made out-and-back hand movements whereby the outward movements were accompanied by corresponding cursor motions on a monitor. Their subsequent judgments of hand or cursor movement endpoints are typically biased toward each other, consistent with an underlying optimal integration mechanism. To study the effect of spatial proximity, we presented the hand and cursor either in orthogonal planes (horizontal and frontal, respectively) or we aligned them in the horizontal plane. We did not find the expected enhanced integration strength in the latter spatial condition. As a secondary question we asked whether spatial transformations required for the position judgments (i.e., horizontal to frontal or vice versa) could be the origin of previously observed suboptimal variances of the integrated hand and cursor position judgments. We found, however, that the suboptimality persisted when spatial transformations were omitted (i.e., with the hand and cursor in the same plane). Our findings thus clearly show that the integration of actions with their visual effects is, at least for cursor control, independent of spatial proximity.

Optimal integration of actions and their visual effects is based on both online and prior causality evidence.

The brain needs to identify redundant sensory signals in order to integrate them optimally. The identification process, referred to as causal inference, depends on the spatial and temporal correspondence of the incoming sensory signals ('online sensory causality evidence') as well as on prior expectations regarding their causal relation. We here examine whether the same causal inference process underlies spatial integration of actions and their visual consequences. We used a basic cursor-control task for which online sensory causality evidence is provided by the correlated hand and cursor movements, and prior expectations are formed by everyday experience of such correlated movements. Participants made out-and-back movements and subsequently judged the hand or cursor movement endpoints. In one condition, we omitted the online sensory causality evidence by showing the cursor only at the movement endpoint. The integration strength was lower than in conditions where the cursor was visible during the outward movement, but a substantial level of integration persisted. These findings support the hypothesis that the binding of actions and their visual consequences is based on the general mechanism of optimal integration, and they specifically show that such binding can occur even if it is previous experience only that identifies the action consequence.

Kinematic cross-correlation induces sensory integration across separate objects.

In a basic cursor-control task, the perceived positions of the hand and the cursor are biased towards each other. We recently found that this phenomenon conforms to the reliability-based weighting mechanism of optimal multisensory integration. This indicates that optimal integration is not restricted to sensory signals originating from a single source, as is the prevailing view, but that it also applies to separate objects that are connected by a kinematic relation (i.e. hand and cursor). In the current study, we examined which aspects of the kinematic relation are crucial for eliciting the sensory integration: (i) the cross-correlation between kinematic variables of the hand and cursor trajectories, and/or (ii) an internal model of the hand-cursor kinematic transformation. Participants made out-and-back movements from the centre of a semicircular workspace to its boundary, after which they judged the position where either their hand or the cursor hit the boundary. We analysed the position biases and found that the integration was strong in a condition with high kinematic correlations (a straight hand trajectory was mapped to a straight cursor trajectory), that it was significantly reduced for reduced kinematic correlations (a straight hand trajectory was transformed into a curved cursor trajectory) and that it was not affected by the inability to acquire an internal model of the kinematic transformation (i.e. by the trial-to-trial variability of the cursor curvature). These findings support the idea that correlations play a crucial role in multisensory integration irrespective of the number of sensory sources involved.

Perceptual attraction in tool use: evidence for a reliability-based weighting mechanism.

Humans are well able to operate tools whereby their hand movement is linked, via a kinematic transformation, to a spatially distant object moving in a separate plane of motion. An everyday example is controlling a cursor on a computer monitor. Despite these separate reference frames, the perceived positions of the hand and the object were found to be biased toward each other. We propose that this perceptual attraction is based on the principles by which the brain integrates redundant sensory information of single objects or events, known as optimal multisensory integration. That is, 1) sensory information about the hand and the tool are weighted according to their relative reliability (i.e., inverse variances), and 2) the unisensory reliabilities sum up in the integrated estimate. We assessed whether perceptual attraction is consistent with optimal multisensory integration model predictions. We used a cursor-control tool-use task in which we manipulated the relative reliability of the unisensory hand and cursor position estimates. The perceptual biases shifted according to these relative reliabilities, with an additional bias due to contextual factors that were present in experiment 1 but not in experiment 2 The biased position judgments' variances were, however, systematically larger than the predicted optimal variances. Our findings suggest that the perceptual attraction in tool use results from a reliability-based weighting mechanism similar to optimal multisensory integration, but that certain boundary conditions for optimality might not be satisfied.NEW & NOTEWORTHY Kinematic tool use is associated with a perceptual attraction between the spatially separated hand and the effective part of the tool. We provide a formal account for this phenomenon, thereby showing that the process behind it is similar to optimal integration of sensory information relating to single objects.

Biases in rhythmic sensorimotor coordination: effects of modality and intentionality.

Sensorimotor biases were examined for intentional (tracking task) and unintentional (distractor task) rhythmic coordination. The tracking task involved unimanual tracking of either an oscillating visual signal or the passive movements of the contralateral hand (proprioceptive signal). In both conditions the required coordination patterns (isodirectional and mirror-symmetric) were defined relative to the body midline and the hands were not visible. For proprioceptive tracking the two patterns did not differ in stability, whereas for visual tracking the isodirectional pattern was performed more stably than the mirror-symmetric pattern. However, when visual feedback about the unimanual hand movements was provided during visual tracking, the isodirectional pattern ceased to be dominant. Together these results indicated that the stability of the coordination patterns did not depend on the modality of the target signal per se, but on the combination of sensory signals that needed to be processed (unimodal vs. cross-modal). The distractor task entailed rhythmic unimanual movements during which a rhythmic visual or proprioceptive distractor signal had to be ignored. The observed biases were similar as for intentional coordination, suggesting that intentionality did not affect the underlying sensorimotor processes qualitatively. Intentional tracking was characterized by active sensory pursuit, through muscle activity in the passively moved arm (proprioceptive tracking task) and rhythmic eye movements (visual tracking task). Presumably this pursuit afforded predictive information serving the coordination process.

Moving the weber fraction: the perceptual precision for moment of inertia increases with exploration force.

How does the magnitude of the exploration force influence the precision of haptic perceptual estimates? To address this question, we examined the perceptual precision for moment of inertia (i.e., an object's "angular mass") under different force conditions, using the Weber fraction to quantify perceptual precision. Participants rotated a rod around a fixed axis and judged its moment of inertia in a two-alternative forced-choice task. We instructed different levels of exploration force, thereby manipulating the magnitude of both the exploration force and the angular acceleration. These are the two signals that are needed by the nervous system to estimate moment of inertia. Importantly, one can assume that the absolute noise on both signals increases with an increase in the signals' magnitudes, while the relative noise (i.e., noise/signal) decreases with an increase in signal magnitude. We examined how the perceptual precision for moment of inertia was affected by this neural noise. In a first experiment we found that a low exploration force caused a higher Weber fraction (22%) than a high exploration force (13%), which suggested that the perceptual precision was constrained by the relative noise. This hypothesis was supported by the result of a second experiment, in which we found that the relationship between exploration force and Weber fraction had a similar shape as the theoretical relationship between signal magnitude and relative noise. The present study thus demonstrated that the amount of force used to explore an object can profoundly influence the precision by which its properties are perceived.

Exploratory movements determine cue weighting in haptic length perception of handheld rods.

In the present study, we sought to unravel how exploratory movements affect length perception of rods that are held in and wielded by hand. We manipulated the mechanical rod properties--mass (m), first moment of mass distribution (M), major principal moment of inertia (I(1))--individually, allowing us to assess the relative contribution of each of these mechanical variables to the perceptual judgment. Furthermore we developed a method to quantify the force components of the mechanical variables in the total of forces acting at the hand-rod interface, and we calculated each component's relative contribution. The laws of mechanics dictate that these relative force contributions depend on the characteristics of the exploratory movements performed. We found a clear relationship between the relative force contribution of the mechanical variables and their contribution to perceived rod length. This finding is the first quantitative demonstration that exploration style determines how much each mechanical variable influences length perception. Moreover, this finding suggested a cue weighting mechanism in which exploratory movements determine cue reliability (and thus cue weighting). We developed a cue combination model for which we first identified three length cues in the form of ratios between the mechanical variables. Second, we calculated the weights of these cues from the recorded rod movements. The model provided a remarkably good prediction of the experimental data. This strongly suggests that rod length perception by wielding is achieved through a weighted combination of three specific length cues, whereby the weighting depends on the characteristics of the exploratory movements.