The vision of Hsiao on somatosensation.

The goal of this review is to start to consolidate and distill the substantial body of research that comprises the published work of the late Professor Steven S. Hsiao. The studies of Hsiao began by demonstrating the receptive field properties of somatosensory neurons, progressed to describing cortical feature selectivity, and then eventually elevated the field to hopes of tapping into natural neural codes with artificial somatosensory feedback. With ongoing analogies to contemporaneous studies in visual neuroscience, the research results and writings of Hsiao have provided the fields of haptics and somatosensory neurophysiology with the conceptual tools needed to allow profound progress. Specifically, Hsiao suggested that slowly adapting tactile form perception could be restored with cortical microstimulation, rapidly adapting slip reflexes should be relegated to low-level, hard-wired prosthetic components, and Pacinian-corpuscle spatiotemporal population responses could potentially be decoded/encoded to provide information about interactions of hands and hand-held instruments with external objects. Future studies will be guided by these insightful reports from Hsiao.

Predictive mechanisms in the control of contour following.

In haptic exploration, when running a fingertip along a surface, the control system may attempt to anticipate upcoming changes in curvature in order to maintain a consistent level of contact force. Such predictive mechanisms are well known in the visual system, but have yet to be studied in the somatosensory system. Thus, the present experiment was designed to reveal human capabilities for different types of haptic prediction. A robot arm with a large 3D workspace was attached to the index fingertip and was programmed to produce virtual surfaces with curvatures that varied within and across trials. With eyes closed, subjects moved the fingertip around elliptical hoops with flattened regions or Limaçon shapes, where the curvature varied continuously. Subjects anticipated the corner of the flattened region rather poorly, but for the Limaçon shapes, they varied finger speed with upcoming curvature according to the two-thirds power law. Furthermore, although the Limaçon shapes were randomly presented in various 3D orientations, modulation of contact force also indicated good anticipation of upcoming changes in curvature. The results demonstrate that it is difficult to haptically anticipate the spatial location of an abrupt change in curvature, but smooth changes in curvature may be facilitated by anticipatory predictions.

Patterns of muscle activity for digital coarticulation.

Although piano playing is a highly skilled task, basic features of motor pattern generation may be shared across tasks involving fine movements, such as handling coins, fingering food, or using a touch screen. The scripted and sequential nature of piano playing offered the opportunity to quantify the neuromuscular basis of coarticulation, i.e., the manner in which the muscle activation for one sequential element is altered to facilitate production of the preceding and subsequent elements. Ten pianists were asked to play selected pieces with the right hand at a uniform tempo. Key-press times were recorded along with the electromyographic (EMG) activity from seven channels: thumb flexor and abductor muscles, a flexor for each finger, and the four-finger extensor muscle. For the thumb and index finger, principal components of EMG waveforms revealed highly consistent variations in the shape of the flexor bursts, depending on the type of sequence in which a particular central key press was embedded. For all digits, the duration of the central EMG burst scaled, along with slight variations across subjects in the duration of the interkeystroke intervals. Even within a narrow time frame (about 100 ms) centered on the central EMG burst, the exact balance of EMG amplitudes across multiple muscles depended on the nature of the preceding and subsequent key presses. This fails to support the idea of fixed burst patterns executed in sequential phases and instead provides evidence for neuromuscular coarticulation throughout the time course of a hand movement sequence.

Hand kinematics of piano playing.

Dexterous use of the hand represents a sophisticated sensorimotor function. In behaviors such as playing the piano, it can involve strong temporal and spatial constraints. The purpose of this study was to determine fundamental patterns of covariation of motion across joints and digits of the human hand. Joint motion was recorded while 5 expert pianists played 30 excerpts from musical pieces, which featured ∼50 different tone sequences and fingering. Principal component analysis and cluster analysis using an expectation-maximization algorithm revealed that joint velocities could be categorized into several patterns, which help to simplify the description of the movements of the multiple degrees of freedom of the hand. For the thumb keystroke, two distinct patterns of joint movement covariation emerged and they depended on the spatiotemporal patterns of the task. For example, the thumb-under maneuver was clearly separated into two clusters based on the direction of hand translation along the keyboard. While the pattern of the thumb joint velocities differed between these clusters, the motions at the metacarpo-phalangeal and proximal-phalangeal joints of the four fingers were more consistent. For a keystroke executed with one of the fingers, there were three distinct patterns of joint rotations, across which motion at the striking finger was fairly consistent, but motion of the other fingers was more variable. Furthermore, the amount of movement spillover of the striking finger to the adjacent fingers was small irrespective of the finger used for the keystroke. These findings describe an unparalleled amount of independent motion of the fingers.

Coordination of hand shape.

The neural control of hand movement involves coordination of the sensory, motor, and memory systems. Recent studies have documented the motor coordinates for hand shape, but less is known about the corresponding patterns of somatosensory activity. To initiate this line of investigation, the present study characterized the sense of hand shape by evaluating the influence of differences in the amount of grasping or twisting force, and differences in forearm orientation. Human subjects were asked to use the left hand to report the perceived shape of the right hand. In the first experiment, six commonly grasped items were arranged on the table in front of the subject: bottle, doorknob, egg, notebook, carton, and pan. With eyes closed, subjects used the right hand to lightly touch, forcefully support, or imagine holding each object, while 15 joint angles were measured in each hand with a pair of wired gloves. The forces introduced by supporting or twisting did not influence the perceptual report of hand shape, but for most objects, the report was distorted in a consistent manner by differences in forearm orientation. Subjects appeared to adjust the intrinsic joint angles of the left hand, as well as the left wrist posture, so as to maintain the imagined object in its proper spatial orientation. In a second experiment, this result was largely replicated with unfamiliar objects. Thus, somatosensory and motor information appear to be coordinated in an object-based, spatial-coordinate system, sensitive to orientation relative to gravitational forces, but invariant to grasp forcefulness.

What is the biological basis of sensorimotor integration?

This Prospects presents the problems that must be solved by the vertebrate nervous system in the process of sensorimotor integration and motor control. The concepts of efference copy and inverse model are defined, and multiple biological mechanisms are described, including those that form the basis of integration, extrapolation, and comparison/cancellation operations. Open questions for future research include the biological basis of continuous and distributed versus modular control, and somatosensory-motor coordination.

Somatosensory comparison during haptic tracing.

Active sensing involves memory retrieval and updating as well as mechanisms that trigger corrections to the ongoing exploratory movement. The present study examined this process in a task where human subjects moved the index fingertip clockwise around the circumference of a virtual sphere created by a robotic device. The fingertip pressed into the sphere during the movement, and the subjects were to report slight differences in sphere size (or surface curvature), which occurred from trial to trial. During each 2- to 3-s trial, subjects gradually adjusted their speed and pressure according to the current surface curvature, achieving a consistent level of contact force in the last half of the exploration. The results demonstrate that subjects were gradually accumulating haptic information about curvature and, at the same time, gradually changing the motor commands for the movement. When subjects encountered an unexpected transition in curvature (from circular to flat), they reacted by abruptly decreasing contact force at a latency of about 50 ms. This short latency indicates that spinally mediated corrections are engaged during this task. The results support the hypothesis that during haptic exploration, the neural comparison between expected and actual somatosensory feedback takes places at multiple levels, including the spinal cord.

Multiple Factors Underlying Haptic Perception of Length and Orientation.

Information about the shape and spatial orientation of an object can be gathered during exploratory hand and arm movements, and then must be synthesized into a unified percept. During the robotically guided exploration of virtual polygons or triangles, the perception of the lengths of two adjoining segments is not always geometrically consistent with the perception of the internal angles between these segments. The present study further characterized this established inconsistency, and also found that subjects' internal angle judgments were influenced by the spatial orientations of the segments, especially the segment that was explored last in the sequence. Internal angle judgments were also biased by the subjects' own active forces, applied in the direction perpendicular to the programmed handle motion. For the last segment, but not for the earlier segments, subjects produced more outward force when they reported larger angles and more inward force when they reported smaller angles. Thus, the haptic synthesis of object shape is influenced by multiple geometric, spatial, and self-produced factors.

Effects of object compliance on three-digit grasping.

Compared with rigid objects, grasping and lifting compliant objects presents additional uncertainties. For any static grasp, forces at the fingertips depend on factors including the locations of the contact points and the contact forces must be coordinated to maintain equilibrium. For compliant objects, the locations and orientations of the contact surfaces change in a force-dependent manner, thus changing the force requirements. Furthermore, every force adjustment then results in additional changes in object shape. This study characterized force and muscle activation patterns in this situation. Fingertip forces were measured as subjects grasped and lifted a 200-g object using their thumb, index, and ring fingers. A spring was sometimes placed under the index and/or ring finger contact surface. Surface electromyographic activity was recorded from ten hand muscles and one proximal arm muscle. The patterns of grip (normal) force and muscle activity were similar across conditions during the load and lift phases, but their amplitude depended on whether the contact surface was compliant. Specifically, the grip force increased smoothly during the load phase of the task under all conditions. To the contrary, the tangential contact (load) force did not increase monotonically when one or more of the contact surfaces were compliant, resulting in a decoupling of the grip and load forces.

Sensorimotor control of contact force.

Interacting with objects in the environment introduces several new challenges for motor control: the potential for instability, external constraints on possible motions and novel dynamics. Grasping and manipulating objects provide the most elaborate examples of such motor tasks. We review each of these topics and suggest that when sensory feedback is reliable, it is used to adapt the motion to the requirements imposed by the object. When sensory feedback is unreliable, subjects adapt the stiffness of muscles and joints to the task's requirements. One of the simplifications introduced in the control of such movements is a reduction in the effective number of degrees of freedom (sensorimotor axes and muscle synergies) and recent findings and methodological considerations relevant to this topic are discussed.

Extrapolation of visual motion for manual interception.

A frequent goal of hand movement is to touch a moving target or to make contact with a stationary object that is in motion relative to the moving head and body. This process requires a prediction of the target's motion, since the initial direction of the hand movement anticipates target motion. This experiment was designed to define the visual motion parameters that are incorporated in this prediction of target motion. On seeing a go signal (a change in target color), human subjects slid the right index finger along a touch-sensitive computer monitor to intercept a target moving along an unseen circular or oval path. The analysis focused on the initial direction of the interception movement, which was found to be influenced by the time required to intercept the target and the target's distance from the finger's starting location. Initial direction also depended on the curvature of the target's trajectory in a manner that suggested that this parameter was underestimated during the process of extrapolation. The pattern of smooth pursuit eye movements suggests that the extrapolation of visual target motion was based on local motion cues around the time of the onset of hand movement, rather than on a cognitive synthesis of the target's pattern of motion.

Multi-digit control of contact forces during rotation of a hand-held object.

Rotation of an object held with three fingers is produced by modulation of force amplitude and direction at one or more contact points. Changes in the moment arm through which these forces act can also contribute to the modulation of the rotational moment. Therefore force amplitude and direction as well as the center of pressure on each contact surface must be carefully coordinated to produce a rotation. Because there is not a single solution, this study sought to describe consistent strategies for simple position-to-position rotations in the pitch, roll, and yaw axes. Force amplitude and direction, and center of pressure on the contact surfaces (and thus the moment arm), were measured as human subjects rotated a 420 g force-transducer instrumented object, grasped with the thumb, index and ring fingers (average movement time: 500 ms). Electromyographic (EMG) activity was recorded from five intrinsic and three extrinsic hand muscles and two wrist muscles. Principal components analysis of force and EMG revealed just two main temporal patterns: the main one followed rotational position and the secondary one had a time course that resembled that of rotational velocity. Although the task could have been accomplished by dynamic modulation of the activity of wrist muscles alone, these two main dynamic EMG patterns were seen in intrinsic hand muscles as well. In contrast to previous reports of shifting in time of the phasic activity bursts of various muscles, in this task, all EMG records were well described by just two temporal patterns, resembling the position and velocity traces.

Factors Influencing Haptic Perception of Complex Shapes.

Exploration of an object by arm movement and somatosensation is a serial process that relies on memories and expectations. The present experiments tested the hypothesis that this process involves breaking the object into component shapes (primitives). This was tested by having human subjects explore shapes composed of semicircular arcs, as well as quarter circles or quarter ellipses. The subjects' perception was reported using a visual display. In the first experiment, in which a series of semicircular arcs was presented, with offsets that differed from trial to trial, performance was consistent with the perception of two (left and right) semicircles. In the second experiment, subjects often failed to detect the quarter circles or quarter ellipses and again behaved as if the object was composed of two (top and bottom) semicircles. The results suggest that the synthesis of haptically sensed shapes is biased toward simple geometric objects and that it can be strongly influenced by expectations.

Multidigit control of contact forces during transport of handheld objects.

When an object is lifted vertically, the normal force increases and decreases in tandem with tangential (load) force to safely avoid slips. For horizontal object transport, horizontal forces at the contact surfaces can be decomposed into manipulation forces (producing acceleration/deceleration) and grasping forces. Although the grasping forces must satisfy equilibrium constraints, it is not clear what determines their modulation across time, nor the extent to which they result from active muscle contraction or mechanical interactions of the digits with the moving object. Grasping force was found to increase in an experimental condition where the center of mass was below the contact plane, compared with when it was in the contact plane. This increase may be aimed at stabilizing object orientation during translation. In another experimental condition, more complex moments were introduced by allowing the low center of mass to swing around a pivot point. Electromyographic (EMG) activity recorded from several intrinsic and extrinsic hand muscles failed to reveal active feedback regulation of contact force in this situation. Instead, in all experimental conditions, EMG data revealed a strategy of feedforward stiffness modulation. Multiple regression analysis revealed that muscle activity at remote digits (e.g., the index and ring fingers) was highly correlated with the contact force measured at another digit (e.g., the thumb). The data suggest that to maintain grasp stability during horizontal translation, predictable as well as somewhat unpredictable inertial forces are compensated for by controlling the stiffness of the hand through cocontraction and modulation of hand muscle activity.

Timing of muscle activation in a hand movement sequence.

Recent studies have described muscle synergies as overlapping, multimuscle groups defined by synchronous covariation in activation intensity. A different approach regards a synergy as a fixed temporal sequence of bursts of activity across groups of motoneurons. To pursue this latter definition, the present study used a principal component (PC) analysis tailored to reveal the across-muscle temporal synergies of human hand movement. Electromyographic (EMG) activity was recorded as subjects used a manual alphabet to spell a list of words. The analysis was applied to the EMG waveforms from 27 letter-to-letter transitions of equal duration. The first PC (of 27) represented the main temporal synergy; after practice, it began to account for more of the EMG variance (up to 40%). This main synergy began with a burst in the 4-finger extensor and a silent period in the flexors. There were then progressively later and shorter bursts in the thumb abductor, thumb flexor, little finger abductor, and finally the finger flexors. The results suggest that hand movements may be generated by activity waves unfolding in time. Because finger muscles are under relatively direct cortical control, this suggests a specific form of cortical pattern generation.

Intrinsic Hand Muscle Activation for Grasp and Horizontal Transport.

During object manipulation, the hand and arm muscles produce internal forces on the object (grasping forces) and forces that result in external translation or rotation of the object in space (transport forces). The present study tested whether the intrinsic hand muscles are actively involved in transport as well as grasping. Intrinsic hand muscle activity increased with increasing demands for grasp stability, but also showed the timing and directional tuning patterns appropriate for actively transmitting external forces to the object, during the translational acceleration and deceleration of object transport.

Oculomotor responses to gradual changes in target direction.

Smooth pursuit tracking of targets moving linearly (in one dimension) is well characterized by a model where retinal image motion drives eye acceleration. However, previous findings suggest that this model cannot be simply extended to two-dimensional (2D) tracking. To examine 2D pursuit, in the present study, human subjects tracked a target that moved linearly and then followed the arc of a circle. The subjects' gaze angular velocity accurately matched target angular velocity, but the direction of smooth pursuit always lagged behind the current target direction. Pursuit speed slowly declined after the onset of the curve (for about 500 ms), even though the target speed was constant. In a second experiment, brief perturbations were presented immediately prior to the beginning of the change in direction. The subjects' responses to these perturbations consisted of two components: (1) a response specific to the parameters of the perturbation and (2) a nonspecific response that always consisted of a transient decrease in gaze velocity. With the exception of this nonspecific response, pursuit behavior in response to the gradual changes in direction and to the perturbations could be explained by using retinal slip (image velocity) as the input signal. The retinal slip was parallel and perpendicular to the instantaneous direction of pursuit ultimately resulted in changes in gaze velocity (via gaze acceleration). Perhaps due to the subjects' expectations that the target will curve, the sensitivity to the image motion in the direction of pursuit was not as strong as the sensitivity to image motion perpendicular to gaze velocity.

Haptic feature extraction.

This study examined the process by which the shape of a haptically explored object is synthesized from the geometric characteristics of simpler constituent elements, such as arcs and ellipses. Subjects traced the outlines of virtual objects by means of whole arm movements. Each object consisted of the union of a large central ellipse and two smaller circles, extending upward and outward from the top left and right sides of the base. The sizes of the two circles and the eccentricity of the elliptical base were varied. After exploring the object's contour in the absence of vision, subjects reproduced the sensed shape by means of freehand drawing. Speed and force were modulated during the exploratory phase in a manner that suggested that subjects reacted to rather than predicted changes in curvature. Also, subjects typically devoted more time to exploring the part of the contour encompassing the two smaller circles. During drawing, individual features of the explored shape were reproduced with varying degrees of fidelity. Aspects related to the size and location of the smaller circles were reproduced better than was the eccentricity of the ellipse forming the base. Since subjects spent proportionally less time exploring the base, these results suggest that subjects selectively focused attention to regions of high spatial contrast and that the exploratory strategy introduced distortions in the haptically sensed shapes.

Manual tracking in three dimensions.

Little is known about the manual tracking of targets that move in three dimensions. In the present study, human subjects followed, with the tip of a hand-held pen, a virtual target moving four times (period 5 s) around a novel, unseen path. Two basic types of target paths were used: a peanut-shaped Cassini ellipse and a quasi-spherical shape where four connected semicircles lay in orthogonal planes. The quasi-spherical shape was presented in three different sizes, and the Cassini shape was varied in spatial orientation and by folding it along one of the three bend axes. During the first cycle of Cassini shapes, the hand lagged behind the target by about 150 ms on average, which decreased to 100 ms during the last three cycles. Tracking performance gradually improved during the first 3 s of the first cycle and then stabilized. Tracking was especially good during the smooth, planar sections of the shapes, and time lag was significantly shorter when the tracking of a low-frequency component was compared to performance at a higher frequency (-88 ms at 0.2 Hz vs. -101 ms at 0.6 Hz). Even after the appropriate adjustment of the virtual target path to a virtual shape tracing condition, tracking in depth was poor compared to tracking in the frontal plane, resulting in a flattening of the hand path. In contrast to previous studies where target trajectories were linear or sinusoidal, these complex trajectories may have involved estimation of the overall shape, as well as prediction of target velocity.

Planning and drawing complex shapes.

Arm and hand movements are generally controlled using a combination of sensory-based and memory-based guidance mechanisms. This study examined similarities and differences in visually-guided and memory-guided arm movements, and sought to determine as to what extent certain control principles apply to each type of movement. In particular, the 2/3 power law is a principle that appears to govern the formation of complex, curved hand trajectories; it specifies that the tangential velocity should be proportional to the radius of curvature raised to an exponent of 1/3. A virtual reality system was used to project complex target paths in three-dimensional (3D) space. Human subjects first tracked (with the tip of a handheld pen) a single target moving along an unseen path. The entire target path then became visible and the subject traced the shape. Finally, the target shape disappeared and the subject was to draw it, in the same 3D space, from memory. Most aspects of the movements (speed, path size, shape and arm postures) were very similar across the three conditions. However, subjects adhered to the 2/3 power law most closely in the tracing condition, when the entire target path was visible. Also, only within the tracing condition, there were significant differences in the value of the exponent depending on the size and the spatial orientation of the trajectory. In the tracking and drawing conditions, the exponent was greater than 1/3, indicating that subjects spent more time in areas of tight curvature. This may represent a strategy for learning and remembering the complex shape.