The vision of Hsiao on somatosensation.

The goal of this review is to start to consolidate and distill the substantial body of research that comprises the published work of the late Professor Steven S. Hsiao. The studies of Hsiao began by demonstrating the receptive field properties of somatosensory neurons, progressed to describing cortical feature selectivity, and then eventually elevated the field to hopes of tapping into natural neural codes with artificial somatosensory feedback. With ongoing analogies to contemporaneous studies in visual neuroscience, the research results and writings of Hsiao have provided the fields of haptics and somatosensory neurophysiology with the conceptual tools needed to allow profound progress. Specifically, Hsiao suggested that slowly adapting tactile form perception could be restored with cortical microstimulation, rapidly adapting slip reflexes should be relegated to low-level, hard-wired prosthetic components, and Pacinian-corpuscle spatiotemporal population responses could potentially be decoded/encoded to provide information about interactions of hands and hand-held instruments with external objects. Future studies will be guided by these insightful reports from Hsiao.

Speed invariance of independent control of finger movements in pianists.

Independent control of finger movements characterizes skilled motor behaviors such as tool use and musical performance. The purpose of the present study was to identify the effect of movement frequency (tempo) on individuated finger movements in piano playing. Joint motion at the digits was recorded while 5 expert pianists were playing 30 excerpts from musical pieces with different fingering and key locations either at a predetermined normal tempo or as fast as possible. Principal component analysis and cluster analysis using an expectation-maximization algorithm determined three distinct patterns of finger movement coordination for a keypress with each of the index, middle, ring, and little fingers at each of the two tempi. The finger kinematics of each coordination pattern was overall similar across the tempi. Tone sequences assigned into each cluster were also similar for both tempi. A linear regression analysis determined no apparent difference in the amount of movement covariation between the striking and nonstriking fingers at both metacarpo-phalangeal and proximal-interphalangeal joints across the two tempi, which indicated no effect of tempo on independent finger movements in piano playing. In addition, the standard deviation of interkeystroke interval across strokes did not differ between the two tempi, indicating maintenance of rhythmic accuracy of keystrokes. Strong temporal constraints on finger movements during piano playing may underlie the maintained independent control of fingers over a wider range of tempi, a feature being likely to be specific to skilled pianists.

Hand kinematics of piano playing.

Dexterous use of the hand represents a sophisticated sensorimotor function. In behaviors such as playing the piano, it can involve strong temporal and spatial constraints. The purpose of this study was to determine fundamental patterns of covariation of motion across joints and digits of the human hand. Joint motion was recorded while 5 expert pianists played 30 excerpts from musical pieces, which featured ∼50 different tone sequences and fingering. Principal component analysis and cluster analysis using an expectation-maximization algorithm revealed that joint velocities could be categorized into several patterns, which help to simplify the description of the movements of the multiple degrees of freedom of the hand. For the thumb keystroke, two distinct patterns of joint movement covariation emerged and they depended on the spatiotemporal patterns of the task. For example, the thumb-under maneuver was clearly separated into two clusters based on the direction of hand translation along the keyboard. While the pattern of the thumb joint velocities differed between these clusters, the motions at the metacarpo-phalangeal and proximal-phalangeal joints of the four fingers were more consistent. For a keystroke executed with one of the fingers, there were three distinct patterns of joint rotations, across which motion at the striking finger was fairly consistent, but motion of the other fingers was more variable. Furthermore, the amount of movement spillover of the striking finger to the adjacent fingers was small irrespective of the finger used for the keystroke. These findings describe an unparalleled amount of independent motion of the fingers.

Spatial and temporal aspects of cognitive influences on smooth pursuit.

It is well known that prediction is used to overcome processing delays within the motor system and ocular control is no exception. Motion extrapolation is one mechanism that can be used to overcome the visual processing delay. Expectations based on previous experience or cognitive cues are also capable of overcoming this delay. The present experiment was designed to examine how smooth pursuit is altered by cognitive information about the time and/or direction of an upcoming change in target direction. Subjects visually tracked a cursor as it moved at a constant velocity on a computer screen. The target initially moved from left to right and then abruptly reversed horizontal direction and traveled along one of seven possible oblique paths. In half of the trials, a cue was present throughout the trial to signal the position (as well as the time), and/or the direction of the upcoming change. Whenever a position cue (which will be referred to as a timing cue throughout the paper) was present, there were clear anticipatory adjustments to the horizontal velocity component of smooth pursuit. In the presence of a timing cue, a directional cue also led to anticipatory adjustments in the vertical velocity, and hence the direction of smooth pursuit. However, without the timing cue, a directional cue alone produced no anticipation. Thus, in this task, a cognitive spatial cue about the new direction could not be used unless it was made explicit in the time domain.

Multiple Factors Underlying Haptic Perception of Length and Orientation.

Information about the shape and spatial orientation of an object can be gathered during exploratory hand and arm movements, and then must be synthesized into a unified percept. During the robotically guided exploration of virtual polygons or triangles, the perception of the lengths of two adjoining segments is not always geometrically consistent with the perception of the internal angles between these segments. The present study further characterized this established inconsistency, and also found that subjects' internal angle judgments were influenced by the spatial orientations of the segments, especially the segment that was explored last in the sequence. Internal angle judgments were also biased by the subjects' own active forces, applied in the direction perpendicular to the programmed handle motion. For the last segment, but not for the earlier segments, subjects produced more outward force when they reported larger angles and more inward force when they reported smaller angles. Thus, the haptic synthesis of object shape is influenced by multiple geometric, spatial, and self-produced factors.

Incorporating prediction in models for two-dimensional smooth pursuit.

A predictive component can contribute to the command signal for smooth pursuit. This is readily demonstrated by the fact that low frequency sinusoidal target motion can be tracked with zero time delay or even with a small lead. The objective of this study was to characterize the predictive contributions to pursuit tracking more precisely by developing analytical models for predictive smooth pursuit. Subjects tracked a small target moving in two dimensions. In the simplest case, the periodic target motion was composed of the sums of two sinusoidal motions (SS), along both the horizontal and the vertical axes. Motions following the same or similar paths, but having a richer spectral composition, were produced by having the target follow the same path but at a constant speed (CS), and by combining the horizontal SS velocity with the vertical CS velocity and vice versa. Several different quantitative models were evaluated. The predictive contribution to the eye tracking command signal could be modeled as a low-pass filtered target acceleration signal with a time delay. This predictive signal, when combined with retinal image velocity at the same time delay, as in classical models for the initiation of pursuit, gave a good fit to the data. The weighting of the predictive acceleration component was different in different experimental conditions, being largest when target motion was simplest, following the SS velocity profiles.

Role of auditory feedback in the control of successive keystrokes during piano playing.

The purpose of this study was to elucidate the role of auditory feedback derived from one keystroke in the control of the rhythmicity and velocity of successive keystrokes during piano playing. We examined the effects of transient auditory perturbations with respect to the pitch, loudness, and timing of one tone on subsequent keystrokes while six pianists played short excerpts from three simple musical pieces having different tempi ("event rates"). Immediately after a delay in tone production, the inter-keystroke interval became shorter. This compensatory action depended on the tempo, being most prominent at the medium tempo. This indicates that temporal information provided by auditory feedback is utilized to regulate the timing of movement elements produced in a sequence. We also found that the keystroke velocity changed after the timing, pitch, or loudness of a tone was altered, although the response differed depending on the type of perturbation. While delaying the timing or altering the pitch led to an increase in the velocity, altering the loudness changed the velocity in an inconsistent manner. Furthermore, perturbing a tone elicited by the right hand also affected the rhythmicity and velocity of keystrokes with the left hand, indicating that bimanual coordination of tone production was maintained. Finally, altering the pitch sometimes resulted in striking an incorrect key, mostly in the slow piece, emphasizing the importance of pitch information for accurate planning and execution of sequential piano keystrokes.

Does temporal asynchrony affect multimodal curvature detection?

Multiple sensory modalities gather information about our surroundings to plan appropriate movements based on the properties of the environment and the objects within it. This study was designed to examine the sensitivity of visual and haptic information alone and together for detecting curvature. When both visual and haptic information were present, temporal delays in signal onset were used to determine the effect of asynchronous sensory information on the interference of vision on the haptic estimate of curvature. Even under the largest temporal delays where visual and haptic information were clearly disparate, the presentation of visual information influenced the haptic perception of curvature. The uncertainty associated with the unimodal vision condition was smaller than that in the unimodal haptic condition, regardless of whether the haptic information was procured actively or under robot assistance for curvature detection. When both visual and haptic information were available, the uncertainty was not reduced; it was equal to that of the unimodal haptic condition. The weighting of the visual and haptic information was highly variable across subjects with some subjects making judgments based largely on haptic information, while others tended to rely on visual information equally or to a larger extent than the haptic information.

Models for the extrapolation of target motion for manual interception.

Intercepting a moving target requires a prediction of the target's future motion. This extrapolation could be achieved using sensed parameters of the target motion, e.g., its position and velocity. However, the accuracy of the prediction would be improved if subjects were also able to incorporate the statistical properties of the target's motion, accumulated as they watched the target move. The present experiments were designed to test for this possibility. Subjects intercepted a target moving on the screen of a computer monitor by sliding their extended finger along the monitor's surface. Along any of the six possible target paths, target speed could be governed by one of three possible rules: constant speed, a power law relation between speed and curvature, or the trajectory resulting from a sum of sinusoids. A go signal was given to initiate interception and was always presented when the target had the same speed, irrespective of the law of motion. The dependence of the initial direction of finger motion on the target's law of motion was examined. This direction did not depend on the speed profile of the target, contrary to the hypothesis. However, finger direction could be well predicted by assuming that target location was extrapolated using target velocity and that the amount of extrapolation depended on the distance from the finger to the target. Subsequent analysis showed that the same model of target motion was also used for on-line, visually mediated corrections of finger movement when the motion was initially misdirected.

Effects of object compliance on three-digit grasping.

Compared with rigid objects, grasping and lifting compliant objects presents additional uncertainties. For any static grasp, forces at the fingertips depend on factors including the locations of the contact points and the contact forces must be coordinated to maintain equilibrium. For compliant objects, the locations and orientations of the contact surfaces change in a force-dependent manner, thus changing the force requirements. Furthermore, every force adjustment then results in additional changes in object shape. This study characterized force and muscle activation patterns in this situation. Fingertip forces were measured as subjects grasped and lifted a 200-g object using their thumb, index, and ring fingers. A spring was sometimes placed under the index and/or ring finger contact surface. Surface electromyographic activity was recorded from ten hand muscles and one proximal arm muscle. The patterns of grip (normal) force and muscle activity were similar across conditions during the load and lift phases, but their amplitude depended on whether the contact surface was compliant. Specifically, the grip force increased smoothly during the load phase of the task under all conditions. To the contrary, the tangential contact (load) force did not increase monotonically when one or more of the contact surfaces were compliant, resulting in a decoupling of the grip and load forces.

Sensorimotor control of contact force.

Interacting with objects in the environment introduces several new challenges for motor control: the potential for instability, external constraints on possible motions and novel dynamics. Grasping and manipulating objects provide the most elaborate examples of such motor tasks. We review each of these topics and suggest that when sensory feedback is reliable, it is used to adapt the motion to the requirements imposed by the object. When sensory feedback is unreliable, subjects adapt the stiffness of muscles and joints to the task's requirements. One of the simplifications introduced in the control of such movements is a reduction in the effective number of degrees of freedom (sensorimotor axes and muscle synergies) and recent findings and methodological considerations relevant to this topic are discussed.

Extrapolation of visual motion for manual interception.

A frequent goal of hand movement is to touch a moving target or to make contact with a stationary object that is in motion relative to the moving head and body. This process requires a prediction of the target's motion, since the initial direction of the hand movement anticipates target motion. This experiment was designed to define the visual motion parameters that are incorporated in this prediction of target motion. On seeing a go signal (a change in target color), human subjects slid the right index finger along a touch-sensitive computer monitor to intercept a target moving along an unseen circular or oval path. The analysis focused on the initial direction of the interception movement, which was found to be influenced by the time required to intercept the target and the target's distance from the finger's starting location. Initial direction also depended on the curvature of the target's trajectory in a manner that suggested that this parameter was underestimated during the process of extrapolation. The pattern of smooth pursuit eye movements suggests that the extrapolation of visual target motion was based on local motion cues around the time of the onset of hand movement, rather than on a cognitive synthesis of the target's pattern of motion.

Multi-digit control of contact forces during rotation of a hand-held object.

Rotation of an object held with three fingers is produced by modulation of force amplitude and direction at one or more contact points. Changes in the moment arm through which these forces act can also contribute to the modulation of the rotational moment. Therefore force amplitude and direction as well as the center of pressure on each contact surface must be carefully coordinated to produce a rotation. Because there is not a single solution, this study sought to describe consistent strategies for simple position-to-position rotations in the pitch, roll, and yaw axes. Force amplitude and direction, and center of pressure on the contact surfaces (and thus the moment arm), were measured as human subjects rotated a 420 g force-transducer instrumented object, grasped with the thumb, index and ring fingers (average movement time: 500 ms). Electromyographic (EMG) activity was recorded from five intrinsic and three extrinsic hand muscles and two wrist muscles. Principal components analysis of force and EMG revealed just two main temporal patterns: the main one followed rotational position and the secondary one had a time course that resembled that of rotational velocity. Although the task could have been accomplished by dynamic modulation of the activity of wrist muscles alone, these two main dynamic EMG patterns were seen in intrinsic hand muscles as well. In contrast to previous reports of shifting in time of the phasic activity bursts of various muscles, in this task, all EMG records were well described by just two temporal patterns, resembling the position and velocity traces.

Factors Influencing Haptic Perception of Complex Shapes.

Exploration of an object by arm movement and somatosensation is a serial process that relies on memories and expectations. The present experiments tested the hypothesis that this process involves breaking the object into component shapes (primitives). This was tested by having human subjects explore shapes composed of semicircular arcs, as well as quarter circles or quarter ellipses. The subjects' perception was reported using a visual display. In the first experiment, in which a series of semicircular arcs was presented, with offsets that differed from trial to trial, performance was consistent with the perception of two (left and right) semicircles. In the second experiment, subjects often failed to detect the quarter circles or quarter ellipses and again behaved as if the object was composed of two (top and bottom) semicircles. The results suggest that the synthesis of haptically sensed shapes is biased toward simple geometric objects and that it can be strongly influenced by expectations.

Target interception: hand-eye coordination and strategies.

This study was designed to define the characteristics of eye-hand coordination in a task requiring the interception of a moving target. It also assessed the extent to which the motion of the target was predicted and the strategies subjects used to determine when to initiate target interception. Target trajectories were constructed from sums of sines in the horizontal and vertical dimensions. Subjects intercepted these trajectories by moving their index finger along the surface of a display monitor. They were free to initiate the interception at any time, and on successful interception, the target disappeared. Although they were not explicitly instructed to do so, subjects tracked target motion with normal, high-gain smooth-pursuit eye movements right up until the target was intercepted. However, the probability of catch-up saccades was substantially depressed shortly after the onset of manual interception. The initial direction of the finger movement anticipated the motion of the target by approximately 150 ms. For any given trajectory, subjects tended to initiate interception at predictable times that depended on the characteristics of the target trajectories [i.e., when the curvature (or angular velocity) of the target was small and when the target was moving toward the finger]. The relative weighting of various parameters that influenced the decision to initiate interception varied from subject to subject and was not accounted for by a model based on the short-range predictability of target motion.

Multidigit control of contact forces during transport of handheld objects.

When an object is lifted vertically, the normal force increases and decreases in tandem with tangential (load) force to safely avoid slips. For horizontal object transport, horizontal forces at the contact surfaces can be decomposed into manipulation forces (producing acceleration/deceleration) and grasping forces. Although the grasping forces must satisfy equilibrium constraints, it is not clear what determines their modulation across time, nor the extent to which they result from active muscle contraction or mechanical interactions of the digits with the moving object. Grasping force was found to increase in an experimental condition where the center of mass was below the contact plane, compared with when it was in the contact plane. This increase may be aimed at stabilizing object orientation during translation. In another experimental condition, more complex moments were introduced by allowing the low center of mass to swing around a pivot point. Electromyographic (EMG) activity recorded from several intrinsic and extrinsic hand muscles failed to reveal active feedback regulation of contact force in this situation. Instead, in all experimental conditions, EMG data revealed a strategy of feedforward stiffness modulation. Multiple regression analysis revealed that muscle activity at remote digits (e.g., the index and ring fingers) was highly correlated with the contact force measured at another digit (e.g., the thumb). The data suggest that to maintain grasp stability during horizontal translation, predictable as well as somewhat unpredictable inertial forces are compensated for by controlling the stiffness of the hand through cocontraction and modulation of hand muscle activity.

Anisotropies in the gain of smooth pursuit during two-dimensional tracking as probed by brief perturbations.

Previous investigations suggest the gain of smooth pursuit is directionally anisotropic and is regulated in a task-dependent manner. Smooth pursuit is also known to be influenced by expectations concerning the target's motion, but the role of such expectations in modulating feedback gain is not known. In the present work, the gain of smooth pursuit was probed by applying brief perturbations to quasi-predictable two-dimensional target motion at multiple time points. The target initially moved in a straight line, then followed the circumference of a circle for distances ranging between 180 degrees and 270 degrees . Finally, the path reverted to linear motion. Perturbations consisted of a pulse of velocity 50 or 100 ms in duration, applied in one of eight possible directions. They were applied at the onset of the curve or after the target had traversed an arc of 45 degrees or 90 degrees . Pursuit gain was measured by computing the average amplitude of the response in smooth pursuit velocity over a 100 ms interval. To do so we used a coordinate system defined by the motion of the target at the onset of the perturbation, with directions tangential and normal to the path. Responses to the perturbations had two components: one that was modulated with the direction of the perturbation and one that was directionally nonspecific. For the directional response, on average the gain in the normal direction was slightly larger than the gain in the tangential direction, with a ratio ranging from 1.0 to 1.3. The directionally nonspecific response, which was more prominent for perturbations at curve onset or at 90 degrees , consisted of a transient decrease in pursuit speed. Perturbations applied at curve onset also delayed the tracking of the curved target motion.

Factors influencing the radial-tangential illusion in haptic perception.

According to the radial-tangential illusion, in the horizontal plane, arm movements executed in directions radial to the trunk are sensed to be longer than movements of the same length in the orthogonal direction. It has been suggested that the illusion arises from the fact that radially directed movements are executed more slowly and require more effort. These suggestions were tested in a series of experiments, using a robotically controlled manipulandum. In all of the experiments subjects grasped the handle of the manipulandum, in some cases exploring the virtual boundary of a rectangle, while in others being guided along a rectangular contour by the robot. In a two-alternative forced choice design, subjects reported whether the rectangle was wide or narrow. In a control experiment, subjects manifested the radial-tangential illusion. Contrary to the hypothesis, the magnitude of this illusion was not altered when a resistive force was added in the tangential direction or when the ratio of movement times in the tangential and radial directions was changed. However, when the contour was explored in the counterclockwise direction, the illusion was much smaller than when it was explored in the clockwise direction. A second series of experiments, in which subjects only explored two sides (i.e., an L-shape), demonstrated that this effect arose from distortions induced by the serial ordering of the exploratory movements. The illusion was much smaller when the radial segment was explored first. We suggest that this distortion arises from the serial nature of haptic exploration, in which the length of the initial segment decreases as it is stored in working memory for subsequent comparison.

Predicting curvilinear target motion through an occlusion.

When a tracked target is occluded transiently, extraretinal signals are known to maintain smooth pursuit, albeit with a reduced gain. The extent to which extraretinal signals incorporate predictions of time-varying behavior, such as gradual changes in target direction, is not known. Three experiments were conducted to examine this question. In the experiments, subjects tracked a target that initially moved along a straight path, then (briefly) followed the arc of a circle, before it disappeared behind a visible occlusion. In the first experiment, the target did not emerge from the occlusion and subjects were asked to point to the location where they thought the target would have emerged. Gaze and pointing behaviors demonstrated that most of the subjects predicted that the target would follow a linear path through the occlusion. The direction of this extrapolated path was the same as the final visible target direction. In the second set of experiments, the target did emerge after following a curvilinear path through the occlusion, and subjects were asked to track the target with their eyes. Gaze behaviors indicated that, in this experimental condition, the subjects predicted curvilinear target motion while the target was occluded. Saccades were directed to the unseen curvilinear path and pursuit continued to follow this same path at a reduced speed in the occlusion. Importantly, the direction of smooth pursuit continued to change throughout the occlusion. Smooth pursuit angular velocity was maintained for approximately 200 ms following target disappearance. The results of the experiments indicate that extraretinal signals indeed incorporate cognitive expectations about the time-varying behavior of target motion.

Intrinsic Hand Muscle Activation for Grasp and Horizontal Transport.

During object manipulation, the hand and arm muscles produce internal forces on the object (grasping forces) and forces that result in external translation or rotation of the object in space (transport forces). The present study tested whether the intrinsic hand muscles are actively involved in transport as well as grasping. Intrinsic hand muscle activity increased with increasing demands for grasp stability, but also showed the timing and directional tuning patterns appropriate for actively transmitting external forces to the object, during the translational acceleration and deceleration of object transport.