On the nature of action-sentence compatibility effects.

Strong versions of the embodied account of language processing propose that comprehension depends on the mental simulation of sensorimotor experiences conveyed by linguistic meaning. Primary support in favor of this view is based on demonstrations of processing advantages for compatibility between an action implied by sentence content and concurrent sensorimotor processing. Although these effects have been reported across a variety of contexts, various attempts to reproduce these results, both through direct replication and conceptual extension, have not been successful. We present a series of experiments that examine the viability of previous methods used to obtain compatibility effects and the validity of the typical interpretation of such effects as evidence for mental simulation of described actions. Our findings add to the growing body of evidence that compatibility between sentence content and sensorimotor processing does not produce robust compatibility effects. Further, our findings suggest the data obtained from some studies that have been successful in generating compatibility effects can be accounted for without appealing to the notion that these effects are due to the simulation of actions implied by the meaning of a sentence. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

A pre-registered, multi-lab non-replication of the action-sentence compatibility effect (ACE).

The Action-sentence Compatibility Effect (ACE) is a well-known demonstration of the role of motor activity in the comprehension of language. Participants are asked to make sensibility judgments on sentences by producing movements toward the body or away from the body. The ACE is the finding that movements are faster when the direction of the movement (e.g., toward) matches the direction of the action in the to-be-judged sentence (e.g., Art gave you the pen describes action toward you). We report on a pre-registered, multi-lab replication of one version of the ACE. The results show that none of the 18 labs involved in the study observed a reliable ACE, and that the meta-analytic estimate of the size of the ACE was essentially zero.

The role of cognitive control and top-down processes in object affordances.

A widely held though debatable claim is that the picture of an object like a frying pan automatically elicits features of a left/right-handed grasp action even in perceptual tasks that make no demands on the observer to consider the graspable properties of the depicted object. Here, we sought to further elucidate this claim by relying on a methodology that allowed us to distinguish between the influence of motor versus spatial codes on the selection of a left/right-handed response while electroencephalographic data were recorded. In our experiment, participants classified images of frying pans as upright or inverted using a left/right key press or by making a left/right-handed reach-and-grasp action towards a centrally located response element while we recorded electroencephalographic (EEG) data. In line with previous evidence (Bub, Masson, & van Noordenne, Journal of Experiment Psychology: Human Perception and Performance, 47(1), 53-80, 2021), these two modes of responding generated distinct correspondence effects on performance induced by the same set of images. In terms of our EEG data, we found that neither motor (the lateralized readiness potential) nor visual (N100 and P100) potentials were sensitive to handle-response hand correspondence. However, an exploratory theta analysis revealed that changes in frontal theta power mirrored the different correspondence effects evoked by the image on key press responses versus reach and grasp actions. Importantly, our results provide a link between these disparate effects and the engagement of cognitive control, highlighting a possible role of top-down control processes in separating motor features from the task-irrelevant features of an object, and thus in claims regarding object affordances more generally.

Motor representations evoked by objects under varying action intentions.

In an extension of Gibson's (1979) concept of object affordance, it has been proposed that motor representations are automatically evoked by pictures of graspable objects. A variety of effects on left/right-handed keypress responses to the perceptual attributes of such images have been taken as evidence that features of actions, including the hand best suited to grasp an object, contribute to the effect of the handle's left/right location on response selection. We present an argument against this claim by establishing that all of these effects are based on spatial codes, including effects mistakenly interpreted to reflect the influence of limb-specific features of a grasp action. We also present 6 experiments showing that under certain task conditions, limb-specific effects on response selection are indeed automatically generated by the task-irrelevant image of a graspable object. These effects are found either when the observer makes keypress or reach-and-grasp responses to the laterality of a pictured hand superimposed on a depicted object. Both tasks recruit control processes that determine how the hand is selected and configured to grasp an object. We infer that processes implicated in the planning of a reach-and-grasp action themselves determine whether the task-irrelevant picture of an object triggers motor-based rather than spatial features. Our results have deep implications for the widely used concept of an affordance for action furnished by an object. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Switching between lift and use grasp actions.

Switching between competing grasp postures incurs costs on speeded performance. We examined switch costs between lift versus use actions under task conditions that required subjects to identify familiar objects. There were no asymmetrical interference effects, though reliable costs occurred when the same object required a different action on consecutive trials. In addition, lift actions were faster to objects targeted for a prospective use action than objects irrelevant to this intended goal. The benefit of a lift-then-use action sequence was not merely due to the production of two different actions in short order on the same object; use actions to an object marked for the distal goal of a lift action were not faster than use actions applied to another object. We propose that the intention to use an object facilitates the prior action of lifting it because the motor sequence lift-then-use is habitually conscripted to enact the proper function of an object.

Time course of motor affordances evoked by pictured objects and words.

Correspondence effects based on the relationship between the left/right position of a pictured object's handle and the hand used to make a response, or on the size of the object and the nature of a grip response (power/precision), have been attributed to motor affordances evoked by the object. Effects of this nature, however, are readily explained by the similarity in the abstract spatial coding of the features that define the stimulus and response, without recourse to object-based affordances. We propose that in the task context of making reach-and-grasp actions, pictured objects may evoke genuine, limb-specific action constituents. We demonstrate that when subjects make reach-and-grasp responses, there is a qualitative difference in the time course of correspondence effects induced by pictures of objects versus the names of those objects. For word primes, this time course was consistent with the abstract spatial coding account, in which effects should emerge slowly and become apparent only among longer response times. In contrast, correspondence effects attributable to object primes were apparent even among the shortest response times and were invariant across the entire response-time distribution. Using rotated versions of object primes provided evidence for a short-lived competition between canonical and depicted orientations of an object with respect to eliciting components of associated actions. These results suggest that under task conditions requiring reach-and-grasp responses, pictured objects rapidly trigger constituents of real-world actions. (PsycINFO Database Record

Grasp representations depend on knowledge and attention.

Seeing pictures of objects activates the motor cortex and can have an influence on subsequent grasping actions. However, the exact nature of the motor representations evoked by these pictures is unclear. For example, action plans engaged by pictures could be most affected by direct visual input and computed online based on object shape. Alternatively, action plans could be influenced by experience seeing and grasping these objects. We provide evidence for a dual-route theory of action representations evoked by pictures of objects, suggesting that these representations are influenced by both direct visual input and stored knowledge. We find that that familiarity with objects has a facilitative effect on grasping actions, with knowledge about the object's canonical orientation or its name speeding grasping actions for familiar objects compared to novel objects. Furthermore, the strength of contributions from each route to action can be modulated by the manner in which the objects are attended. Thus, evocation of grasping representations depends on an interaction between one's familiarity with perceived objects and how those objects are attended while making grasp actions. (PsycINFO Database Record

Components of action representations evoked when identifying manipulable objects.

We examined the influence of holding planned hand actions in working memory on the time taken to visually identify objects with handles. Features of the hand actions and position of the object's handle were congruent or incongruent on two dimensions: alignment (left vs. right) and orientation (horizontal vs. vertical). When an object was depicted in an upright view, subjects were slower to name it when its handle was congruent with the planned hand actions on one dimension but incongruent on the other, relative to when the object handle and actions were congruent on both or neither dimension. This pattern is consistent with many other experiments demonstrating that a cost occurs when there is partial feature overlap between a planned action and a perceived target. An opposite pattern of results was obtained when the depicted object appeared in a 90° rotated view (e.g., a beer mug on its side), suggesting that the functional goal associated with the object (e.g., drinking from an upright beer mug) was taken into account during object perception and that this knowledge superseded the influence of the action afforded by the depicted view of the object. These results have implications for the relationship between object perception and action representations, and for the mechanisms that support the identification of rotated objects.

Time course of action representations evoked during sentence comprehension.

The nature of hand-action representations evoked during language comprehension was investigated using a variant of the visual-world paradigm in which eye fixations were monitored while subjects viewed a screen displaying four hand postures and listened to sentences describing an actor using or lifting a manipulable object. Displayed postures were related to either a functional (using) or volumetric (lifting) interaction with an object that matched or did not match the object mentioned in the sentence. Subjects were instructed to select the hand posture that matched the action described in the sentence. Even before the manipulable object was mentioned in the sentence, some sentence contexts allowed subjects to infer the object's identity and the type of action performed with it and eye fixations immediately favored the corresponding hand posture. This effect was assumed to be the result of ongoing motor or perceptual imagery in which the action described in the sentence was mentally simulated. In addition, the hand posture related to the manipulable object mentioned in a sentence, but not related to the described action (e.g., a writing posture in the context of a sentence that describes lifting, but not using, a pencil), was favored over other hand postures not related to the object. This effect was attributed to motor resonance arising from conceptual processing of the manipulable object, without regard to the remainder of the sentence context.

Embodied effects of conceptual knowledge continuously perturb the hand in flight.

Attending to a manipulable object evokes a mental representation of hand actions associated with the object's form and function. In one view, these representations are sufficiently abstract that their competing influence on an unrelated action is confined to the planning stages of movement and does not affect its on-line control. Alternatively, an object may evoke action representations that affect the entire trajectory of an unrelated grasping action. We developed a new methodology to statistically analyze the forward motion and rotation of the hand and fingers under different task conditions. Using this novel approach, we established that a grasping action executed after seeing a photograph of an object is systematically perturbed even into the late stages of its trajectory by the competing influence of the grasping posture associated with the object. Our results show that embodied effects of conceptual knowledge continuously modulate the hand in flight.

Features of planned hand actions influence identification of graspable objects.

We demonstrate that constituents of motor actions associated with handled objects play a role in identifying such objects. Subjects held in working memory action plans for reaching movements; these action plans specified both the hand to be used (left or right) and a wrist orientation (vertical or horizontal). Speeded object identification was impaired when a pictured object matched the action on only one of these two categorical dimensions (e.g., a beer mug with its handle facing left, an action plan involving the right hand and vertical wrist orientation), relative to when the object matched the action on both dimensions or neither dimension. This result implies that identification of a manipulable object leads to automatic retrieval of matching features of a planned action along with nonmatching features to which they are bound. These discrepant features conflict with those of the target object, which results in delayed identification.

Dynamic evocation of hand action representations during sentence comprehension.

When listening to a sentence describing an interaction with a manipulable object, understanding the actor's intentions is shown to have a striking influence on action representations evoked during comprehension. Subjects performed a cued reach and grasp response while listening to a context sentence. Responses were primed when they were consistent with the proximal intention of an actor ("John lifted the cell phone..."), but this effect was evanescent and appeared only when sentences mentioned the proximal intention first. When the sentence structure was changed to mention the distal intention first ("To clear the shelf..."), priming effects were no longer context specific and actions pertaining to the function of an object were clearly favored. These results are not compatible with a straightforward mental-simulation account of sentence comprehension but instead reflect a hierarchy of intentions distinguishing how and why actions are performed.

Does acquisition of Greeble expertise in prosopagnosia rule out a domain-general deficit?

According to the expertise account of face specialization, a deficit that affects general expertise mechanisms should similarly impair the expert individuation of both faces and other visually homogeneous object classes. To test this possibility, we attempted to train a prosopagnosic patient, LR, to become a Greeble expert using the standard Greeble expertise-training paradigm (Gauthier & Tarr, 2002). Previous research demonstrated that LR's prosopagnosia was related to an inability to simultaneously use multiple features in a speeded face recognition task (Bukach, Bub, Gauthier, & Tarr, 2006). We hypothesized that LR's inability to use multiple face features would manifest in his acquisition of Greeble expertise, even though his basic object recognition is unimpaired according to standard neuropsychological testing. Although LR was eventually able to reach expertise criterion, he took many more training sessions than controls, suggesting use of an abnormal strategy. To further explore LR's Greeble processing strategies, we assessed his ability to use multiple Greeble features both before and after Greeble training. LR's performance in two versions of this task demonstrates that, even after training, he relies heavily on a single feature to identify Greebles. This correspondence between LR's face recognition and post-training Greeble recognition supports the idea that impaired face recognition is simply the most visible symptom of a more general object recognition impairment in acquired prosopagnosia.

On the dynamics of action representations evoked by names of manipulable objects.

Two classes of hand action representations are shown to be activated by listening to the name of a manipulable object (e.g., cellphone). The functional action associated with the proper use of an object is evoked soon after the onset of its name, as indicated by primed execution of that action. Priming is sustained throughout the duration of the word's enunciation. Volumetric actions (those used to simply lift an object) show a negative priming effect at the onset of a word, followed by a short-lived positive priming effect. This time-course pattern is explained by a dual-process mechanism involving frontal and parietal lobes for resolving conflict between candidate motor responses. Both types of action representations are proposed to be part of the conceptual knowledge recruited when the name of a manipulable object is encountered, although functional actions play a more central role in the representation of lexical concepts.

Priming of reach and grasp actions by handled objects.

Pictures of handled objects such as a beer mug or frying pan are shown to prime speeded reach and grasp actions that are compatible with the object. To determine whether the evocation of motor affordances implied by this result is driven merely by the physical orientation of the object's handle as opposed to higher-level properties of the object, including its function, prime objects were presented either in an upright orientation or rotated 90° from upright. Rotated objects successfully primed hand actions that fit the object's new orientation (e.g., a frying pan rotated 90° so that its handle pointed downward primed a vertically oriented power grasp), but only when the required grasp was commensurate with the object's proper function. This constraint suggests that rotated objects evoke motor representations only when they afford the potential to be readily positioned for functional action.

On the nature of hand-action representations evoked during written sentence comprehension.

We examine the nature of motor representations evoked during comprehension of written sentences describing hand actions. We distinguish between two kinds of hand actions: a functional action, applied when using the object for its intended purpose, and a volumetric action, applied when picking up or holding the object. In Experiment 1, initial activation of both action representations was followed by selection of the functional action, regardless of sentence context. Experiment 2 showed that when the sentence was followed by a picture of the object, clear context-specific effects on evoked action representations were obtained. Experiment 3 established that when a picture of an object was presented alone, the time course of both functional and volumetric actions was the same. These results provide evidence that representations of object-related hand actions are evoked as part of sentence processing. In addition, we discuss the conditions that elicit context-specific evocation of motor representations.

Grasping beer mugs: on the dynamics of alignment effects induced by handled objects.

We examined automatic spatial alignment effects evoked by handled objects. Using color as the relevant cue carried by an irrelevant handled object aligned or misaligned with the response hand, responses to color were faster when the handle aligned with the response hand. Alignment effects were observed only when the task was to make a reach and grasp response. No alignment effects occurred if the response involved a left-right key press. Alignment effects emerged over time, becoming more apparent either when the color cue was delayed or when relatively long, rather than short, response times were analyzed. These results are consistent with neurophysiological evidence indicating that the cued goal state has a modulatory influence on sensorimotor representations, and that handled objects initially generate competition between neural populations coding for a left- or right-handed action that must be resolved before a particular hand is favored.

Does face inversion change spatial frequency tuning?

The authors examined spatial frequency (SF) tuning of upright and inverted face identification using an SF variant of the Bubbles technique (F. Gosselin & P. G. Schyns, 2001). In Experiment 1, they validated the SF Bubbles technique in a plaid detection task. In Experiments 2a-c, the SFs used for identifying upright and inverted inner facial features were investigated. Although a clear inversion effect was present (mean accuracy was 24% higher and response times 455 ms shorter for upright faces), SF tunings were remarkably similar in both orientation conditions (mean r = .98; an SF band of 1.9 octaves centered at 9.8 cycles per face width for faces of about 6 degrees ). In Experiments 3a and b, the authors demonstrated that their technique is sensitive to both subtle bottom-up and top-down induced changes in SF tuning, suggesting that the null results of Experiments 2a-c are real. The most parsimonious explanation of the findings is provided by the quantitative account of the face inversion effect: The same information is used for identifying upright and inverted inner facial features, but processing has greater sensitivity with the former.

Language-based access to gestural components of conceptual knowledge.

We report two experiments in which production of articulated hand gestures was used to reveal the nature of gestural knowledge evoked by sentences referring to manipulable objects. Two gesture types were examined: functional gestures (executed when using an object for its intended purpose) and volumetric gestures (used when picking up an object simply to move it). Participants read aloud a sentence that referred to an object but did not mention any form of manual interaction (e.g., Jane forgot the calculator) and were cued after a delay of 300 or 750 ms to produce the functional or volumetric gesture associated with the object, or a gesture that was unrelated to the object. At both cue delays, functional gestures were primed relative to unrelated gestures, but no significant priming was found for volumetric gestures. Our findings elucidate the types of motor representations that are directly linked to the meaning of words referring to manipulable objects in sentences.

Preservation of mouth region processing in two cases of prosopagnosia.

Although most adults are considered experts in face recognition, brain trauma can produce a selective loss in this ability, a condition referred to as prosopagnosia. This study examined the processing strategies of prosopagnosic patients LR and HH using the Face Dimensions Test. In this test, featural and configural information in the upper and lower halves of the face was parametrically varied and sensitivity to these changes measured. We found that relative to age-matched control participants, LR and HH exhibited an impaired ability to discriminate differences in the eye region, but a preserved ability to detect featural and configural differences in the mouth region. This pattern of impairment and sparing was demonstrated in tests of direct perception and immediate memory. The obtained findings demonstrate that prosopagnosia does not necessarily cause a global impairment to face perception, but a selective impairment to the perception of information in the upper half of the face.