Audiovisual Speech Web-Lab: an Internet teaching and research laboratory.

Internet resources now enable laboratories to make full-length experiments available on line. A handful of existing web sites offer users the ability to participate in experiments and generate usable data. We have integrated this technology into a web site that also provides full discussion of the theoretical and methodological aspects of the experiments using text and simple interactive demonstrations. The content of the web site (http://www.psych.ucr.edu/avspeech/lab) concerns audiovisual speech perception and its relation to face perception. The site is designed to be useful for users of multiple interests and levels of expertise.

Face and mouth inversion effects on visual and audiovisual speech perception.

Three experiments examined whether image manipulations known to disrupt face perception also disrupt visual speech perception. Research has shown that an upright face with an inverted mouth looks strikingly grotesque whereas an inverted face and an inverted face containing an upright mouth look relatively normal. The current study examined whether a similar sensitivity to upright facial context plays a role in visual speech perception. Visual and audiovisual syllable identification tasks were tested under 4 presentation conditions: upright face-upright mouth, inverted face-inverted mouth, inverted face-upright mouth, and upright face-inverted mouth. Results revealed that for some visual syllables only the upright face-inverted mouth image disrupted identification. These results suggest that upright facial context can play a role in visual speech perception. A follow-up experiment testing isolated mouths supported this conclusion.

Effects of talker variability on speechreading.

The effects of talker variability on visual speech perception were tested by having subjects speechread sentences from either single-talker or mixed-talker sentence lists. Results revealed that changes in talker from trial to trial decreased speechreading performance. To help determine whether this decrement was due to talker change--and not a change in superficial characteristics of the stimuli--Experiment 2 tested speechreading from visual stimuli whose images were tinted by a single color, or mixed colors. Results revealed that the mixed-color lists did not inhibit speechreading performance relative to the single-color lists. These results are analogous to findings in the auditory speech literature and suggest that, like auditory speech, visual speech operations include a resource-demanding component that is influenced by talker variability.

The McGurk effect in infants.

In the McGurk effect, perceptual identification of auditory speech syllables is influenced by simultaneous presentation of discrepant visible speech syllables. This effect has been found in subjects of different ages and with various native language backgrounds. But no McGurk tests have been conducted with prelinguistic infants. In the present series of experiments, 5-month-old English-exposed infants were tested for the McGurk effect. Infants were first gaze-habituated to an audiovisual /va/. Two different dishabituation stimuli were then presented: audio /ba/-visual /va/ (perceived by adults as /va/), and audio /da/-visual /va/ (perceived by adults as /da/). The infants showed generalization from the audiovisual /va/ to the audio /ba/-visual /va/ stimulus but not to the audio /da/-visual /va/ stimulus. Follow-up experiments revealed that these generalization differences were not due to a general preference for the audio /da/-visual /va/ stimulus or to the auditory similarity of /ba/ to /va/ relative to /da/. These results suggest that the infants were visually influenced in the same way as English-speaking adults are visually influenced.

Point-light facial displays enhance comprehension of speech in noise.

Seeing a talker's face can improve the perception of speech in noise. There is little known about which characteristics of the face are useful for enhancing the degraded signal. In this study, a point-light technique was employed to help isolate the salient kinematic aspects of a visible articulating face. In this technique, fluorescent dots were arranged on the lips, teeth, tongue, cheeks, and jaw of an actor. The actor was videotaped speaking in the dark, so that when shown to observers, only the moving dots were seen. To test whether these reduced images could contribute to the perception of degraded speech, noise-embedded sentences were dubbed with the point-light images at various signal-to-noise ratios. It was found that these images could significantly improve comprehension for adults with normal hearing and that the images became more effective as participants gained experience with the stimuli. These results have implications for uncovering salient visual speech information as well as in the development of telecommunication systems for listeners who are hearing impaired.

An audiovisual test of kinematic primitives for visual speech perception.

Isolated kinematic properties of visible speech can provide information for lip reading. Kinematic facial information is isolated by darkening an actor's face and attaching dots to various articulators so that only moving dots can be seen with no facial features present. To test the salience of these images, the authors conducted experiments to determine whether the images could visually influence the perception of discrepant auditory syllables. Results showed that these images can influence auditory speech independently of the participant's knowledge of the stimuli. In other experiments, single frozen frames of visible syllables were presented with discrepant auditory syllables to test the salience of static facial features. Although the influence of the kinematic stimuli was perceptual, any influence of the static featural stimuli was likely based on participant's misunderstanding or postperceptual response bias.

Dynamics and the orientation of kinematic forms in visual event recognition.

The authors investigated event dynamics as a determinant of the perceptual significance of forms of motion. Patch-light displays were recorded for 9 simple events selected to represent rigid-body dynamics, biodynamics, hydrodynamics, and aerodynamics. Observers described events in a free-response task or by circling properties on a list. Cluster analyses performed on descriptor frequencies reflected the dynamics. Observers discriminated hydro- versus aerodynamic events and animate versus inanimate events. The latter result was confirmed by using a forced-choice task. Dynamical models of the events led us to consider energy flows as a determinant of kinematic properties that allowed animacy to be distinguished. Orientation was manipulated in 3 viewing conditions. Descriptions varied with absolute display orientation rather than the relative orientation of display and observer.

Selective adaptation in speech perception using a compelling audiovisual adaptor.

A replication of the audiovisual test of speech selective adaptation performed by Roberts and Summerfield [Percept. Psychophys. 30, 309-314 (1981)] was conducted. The audiovisual methodology allows for the dissociation of acoustic and phonetic components of an adapting stimulus. Roberts and Summerfield's results have been interpreted to support an auditory basis for selective adaptation. However, their subjects did not consistently report hearing the adaptor as a visually influenced syllable making this interpretation questionable. In the present experiment, a more compelling audiovisual adaptor was implemented resulting in a visually influenced percept 99% of the time. Still, systematic adaptation occurred only for the auditory component.

Visual influences on auditory pluck and bow judgments.

In the McGurk effect, visual information specifying a speaker's articulatory movements can influence auditory judgments of speech. In the present study, we attempted to find an analogue of the McGurk effect by using nonspeech stimuli--the discrepant audiovisual tokens of plucks and bows on a cello. The results of an initial experiment revealed that subjects' auditory judgments were influenced significantly by the visual pluck and bow stimuli. However, a second experiment in which speech syllables were used demonstrated that the visual influence on consonants was significantly greater than the visual influence observed for pluck-bow stimuli. This result could be interpreted to suggest that the nonspeech visual influence was not a true McGurk effect. In a third experiment, visual stimuli consisting of the words pluck and bow were found to have no influence over auditory pluck and bow judgments. This result could suggest that the nonspeech effects found in Experiment 1 were based on the audio and visual information's having an ostensive lawful relation to the specified event. These results are discussed in terms of motor-theory, ecological, and FLMP approaches to speech perception.

Dynamical constraints on pictorial action lines.

Pictorial action lines are an effective way of portraying movement in a static drawing. When such lines emanate from the backs of characters, they can give a sense of the path or style of movement. Seven experiments assessed whether photographic streak lines-lines that depict actual movement paths-can, in and of themselves, be informative about the act that produced them. Lines were produced as a darkly clad actor with point-lights attached to his major joints performed a number of actions in front of an open-lens camera. Completely naive Ss had little success identifying events in these photographs. Once Ss were told of the photographic technique, however, striking proficiency was achieved. Subtle distinctions (e.g., whether the movement was forward or backward or performed while wearing weights) were made for some of the events. Results are discussed in terms of various treatments of pictorial information.

Auditory looming perception: influences on anticipatory judgments.

Several studies in the auditory-perception literature hint that listeners may be able to anticipate the time of arrival of an approaching sound source. Two experiments are reported in which listeners judged the time of arrival of an approaching car on the basis of various portions of its auditory signal. Subjects pressed a computer key to indicate when the car would have just passed them, assuming that the car maintained a constant approach velocity. A number of variables were tested including (a) the time between the offset of the signal and the virtual time of passage, (b) duration of the signal, and (c) feedback concerning judgment accuracy. Results indicate that increasing the time between signal offset and virtual time of passage decreases judgment accuracy whereas the actual duration of the signal had no significant effect. Feedback significantly improved performance overall.

Discrimination tests of visually influenced syllables.

In the McGurk effect, perception of audiovisually discrepant syllables can depend on auditory, visual, or a combination of audiovisual information. Under some conditions, visual information can override auditory information to the extent that identification judgments of a visually influenced syllable can be as consistent as for an analogous audiovisually compatible syllable. This might indicate that visually influenced and analogous audiovisually compatible syllables are phonetically equivalent. Experiments were designed to test this issue using a compelling visually influenced syllable in an AXB matching paradigm. Subjects were asked to match an audio syllable/va/either to an audiovisually consistent syllable (audio/va/-video/fa/) or an audiovisually discrepant syllable (audio/ba/-video/fa/). It was hypothesized that if the two audiovisual syllables were phonetically equivalent, then subjects should choose them equally often in the matching task. Results show, however, that subjects are more likely to match the audio/va/ to the audiovisually consistent/va/, suggesting differences in phonetic convincingness. Additional experiments further suggest that this preference is not based on a phonetically extraneous dimension or on noticeable relative audiovisual discrepancies.

Audiovisual investigation of the loudness-effort effect for speech and nonspeech events.

There is some evidence that loudness judgments of speech are more closely related to the degree of vocal effort induced in speech production than to the speech signal's surface-acoustic properties such as intensity. Other researchers have claimed that speech loudness can be rationalized simply by considering the acoustic complexity of the signal. Because vocal effort can be specified optically as well as acoustically, a study to test the effort-loudness hypothesis was conducted that used conflicting audiovisual presentations of a speaker that produced consonant-vowel syllables with different efforts. It was predicted that if loudness judgments are constrained by effort perception rather than by simple acoustic parameters, then judgments ought to be affected by visual as well as auditory information. It is shown that loudness judgments are affected significantly by visual information even when subjects are instructed to base their judgments only on what they hear. A similar (though less pronounced) patterning of results is shown for a nonspeech "clapping" event, which attests to the generality of the loudness-effort effect previously thought to be special to speech. Results are discussed in terms of auditory, fuzzy logical, motor, and ecological theories of speech perception.

Task dynamics and resource dynamics in the assembly of a coordinated rhythmic activity.

Task dynamics corresponding to rhythmic movements emerge from interactions among dynamical resources composed of the musculature, the link segments, and the nervous and circulatory systems. This article investigated whether perturbations of interlimb coordination might be effect over circulatory and nervous elements. Stiffness of wrist-pendulums oscillated at a common tempo and at 180 degrees relative phase was perturbed through the use of tonic activity about an ankle. Left and right stiffnesses, the common period, and the phase relation all changed. Stiffnesses increased with ankle torque in proportion to the wrist's inertial load. Despite different changes in stiffness at the two wrists, isochrony was preserved. The stability was shown to be consistent with the proportionality of changes in stiffness to the inertial loads. The phase departed from antiphase in proportion to the asymmetry of inertial loads. The size of departures decreased with increasing ankle torque. An account was developed in terms of muscular, circulatory, and nervous functions.

Duplex perception: a comparison of monosyllables and slamming doors.

Duplex perception has been interpreted as revealing distinct systems for general auditory perception and speech perception. The systems yield distinct experiences of the same acoustic signal, the one conforming to the acoustic structure itself and the other to its source in vocal-tract activity. However, this interpretation has not been tested by examining whether duplex perception can be obtained for nonspeech sounds that are not plausibly perceived by a specialized system. In five experiments, we replicate some of the phenomena associated with duplex perception of speech using the sound of a slamming door. Similarities between subjects' responses to syllables and door sounds are striking enough to suggest that some conclusions in the speech literature should be tempered that (a) duplex perception is special to sounds for which there are perceptual modules and (b) duplex perception occurs because distinct systems have rendered different percepts of the same acoustic signal.

Hefting for a maximum distance throw: a smart perceptual mechanism.

Objects for throwing to a maximum distance were selected by hefting objects varying in size and weight. Preferred weights increased with size reproducing size-weight illusion scaling between weight and volume. In maximum distance throws, preferred objects were thrown the farthest. Throwing was related to hefting as a smart perceptual mechanism. Two strategies for conveying high kinetic energy to projectiles were investigated by studying the kinematics of hefting light, preferred, and heavy objects. Changes in tendon lengths occurring when objects of varying size were grasped corresponded to changes in stiffness at the wrist. Hefting with preferred objects produced an invariant phase between the wrist and elbow. This result corresponded to an optimal relation at peak kinetic energy for the hefting. A paradigm for the study of perceptual properties was compared to size-weight illusion methodology.

'Clock' and 'motor' components in absolute coordination of rhythmic movements.

Human subjects swung, through motions at the two wrists, hand-held pendulums of variable mass and length. Within a pair, the two pendulums could be of the same or different magnitude. The subjects were required to produce a coordinated state in which the two rhythmic units oscillated at a single common period. Fifty-four conditions of absolute coordination, that is, 54 different pairs of wrist-pendulum systems, were investigated for each of three subjects in the course of six sessions. Each condition of absolute coordination was conducted in the out of phase mode and at the single most comfortable period. The period variances of the right and left systems in the 162 instances of absolute coordination were analysed according to a method that assumes that a timekeeper function and a motor implementation function contribute independently to the variance in the periodic timing of a rhythmic movement. The major findings were that in absolute coordination: (a) a system's 'motor' variance, but not its 'clock' variance, depended on the deviation of the period of absolute coordination tau from the system's characteristic period; (b) right and left 'clock' variances were related and (c) neither the 'motor' variances nor the 'clock' variances were affected by deviations in the mean phase relation from 180 degrees. The results were discussed in terms of their implications for interpreting von Holst's notions of maintenance tendency and magnetic effect and, more generally, the neural and dynamical basis of absolute coordination.

Maintenance tendency in co-ordinated rhythmic movements: relative fluctuations and phase.

Evidence from the oscillatory behavior of fish fins and the crayfish swimmeret system suggests that local rhythmic-pattern generators preserve their characteristic properties over the various locomotory co-ordinations in which they participate. This maintenance tendency, as von Holst termed it, was investigated in an experiment in which human subjects swung, through motions at the wrists, hand-held pendulums of variable mass and length. In the experiment (comprising six sessions over 21 months with the same three subjects) the context for the maintenance tendency was steady-state absolute co-ordination: two rhythmic units oscillating at a single, common period and at a bounded phase relation. The experimental methodology permitted systematic control of (a) the characteristic periods of the individual rhythmic units and (b) the deviations from these periods. Relative fluctuations in periodic timing and amplitude were least when a rhythmic unit's period in absolute co-ordination approximated its characteristic period and increased with departures from the characteristic period. Rates of increase in timing fluctuations were approximately the same for deviations on either side of the characteristic period; the rate of increase in spacing fluctuations was substantially greater for the range in which periods were less than the characteristic period. The phase relation between two co-ordinated rhythmic movement units in absolute co-ordination depended on the difference between their characteristic periods. The intended phase relation of 180 degrees was attained only when the characteristic periods were identical. When the characteristic periods differed, the departure from 180 degrees increased systematically with the difference. The fluctuation results are discussed in terms of the relation between relaxation and harmonic dynamics in producing rhythmic movements, with particular emphasis on the harmonic tuning of relaxation oscillations. The phase results are discussed in terms of whether or not the very many stable phase relations in absolute co-ordination are reflective of the nervous system or of differences in response latencies in left and right muscle systems induced by different degrees of inertial compensation.

On the time allometry of co-ordinated rhythmic movements.

The focus is the power formulae relating periodic time in terrestrial locomotion and flight to mass and length. The periodic timing of limbs and wings oscillating comfortably in absolute co-ordination is viewed as the characteristic period tau 0 of a system in which the free, undamped oscillatory motion of a point mass m at a distance l from a fixed axis does work against two conservative forces. These forces are in the form of gravity g acting on the point mass and a spring of stiffness k acting at a distance b from the axis. The system's characteristic period can be expressed most simply as: tau 0 = 2 pi [ml2/(mlg + kb2)]1/2. In the biological instantiation of this hybrid mass-spring/simple pendulum system, muscular and other tissues function as the spring that elastically stores and releases mechanical energy. Regular oscillations are brought about and sustained by a muscular driving force that ordinarily is close to resonance. The resultant dynamical regime--basically, raising and lowering a mass at regular intervals with respect to gravity--is referred to as the pendular clocking mode of movement organization. The mode is investigated comfortably at a common period and a fixed phase. In absolute co-ordination, two wrist-pendulum systems can be interpreted physically as a virtual single system. The evidence suggests that the scalings of the periodic times of such systems to mass and to length follow directly from the dynamical properties inherent in the resonance equation of the pendular clocking mode. Recourse to biological constants to rationalize the time scale is unnecessary. Experiments on human wrist-pendular activity and detailed analyses of the mass and length dependencies of the locomotory cycle times of quadrupeds, large birds, small passerines, hummingbirds, and insects are performed with respect to the dynamical properties predicted for systems in the pendular clocking mode. The major conclusion is that all the time scales of terrestrial in locomotory time allometries follow systematically from differences in the length scale and differences in the relation of mass to length.

Relative effectiveness of three stimulus variables for locating a moving sound source.

A study is reported in which it is shown that observers can use at least three types of acoustic variables that indicate reliably when a moving sound source is passing: interaural temporal differences, the Doppler effect, and amplitude change. Each of these variables was presented in isolation and each was successful in indicating when a (stimulated) moving sound source passed an observer. These three variables were put into competition (with each indicating that closest passage occurred at a different time) in an effort to determine their relative importance. It was found that amplitude change dominated interaural temporal differences which, in turn, dominated the Doppler effect stimulus variable. The results are discussed in terms of two interpretations. First, it is possible that subjects based their judgements on the potential discriminability of each stimulus variable. However, because the stimuli used involved easily discriminable changes, subjects may instead have based their judgements on the independence of a stimulus variable from different environmental situation conditions. The dominance ordering obtained supports the second interpretation.