Perceiving fingerspelling via point-light displays: The stimulus and the perceiver both matter.

Signed languages such as American Sign Language (ASL) rely on visuospatial information that combines hand and bodily movements, facial expressions, and fingerspelling. Signers communicate in a wide array of sub-optimal environments, such as in dim lighting or from a distance. While fingerspelling is a common and essential part of signed languages, the perception of fingerspelling in difficult visual environments is not well understood. The movement and spatial patterns of ASL are well-suited to representation by dynamic Point Light Display (PLD) stimuli in which human movement is shown as an array of moving dots affixed to joints on the body. We created PLD videos of fingerspelled location names. The location names were either Real (e.g., KUWAIT) or Pseudo-names (e.g., CLARTAND), and the PLDs showed either a High or a Low number of markers. In an online study, Deaf and Hearing ASL users (total N = 283) watched 27 PLD stimulus videos that varied by Word Type and Number of Markers. Participants watched the videos and typed the names they saw, along with how confident they were in their response. We predicted that when signers see ASL fingerspelling PLDs, language experience in ASL will be positively correlated with accuracy and self-rated confidence scores. We also predicted that Real location names would be understood better than Pseudo names. Our findings supported those predictions. We also discovered a significant interaction between Age and Word Type, which suggests that as people age, they use outside world knowledge to inform their fingerspelling success. Finally, we examined the accuracy and confidence in fingerspelling perception in early ASL users. Studying the relationship between language experience with PLD fingerspelling perception allows us to explore how hearing status, ASL fluency levels, and age of language acquisition affect the core abilities of understanding fingerspelling.

Different Language Modalities Yet Similar Cognitive Processes in Arithmetic Fact Retrieval.

Does experience with signed language impact the neurocognitive processes recruited by adults solving arithmetic problems? We used event-related potentials (ERPs) to identify the components that are modulated by operation type and problem size in Deaf American Sign Language (ASL) native signers and in hearing English-speaking participants. Participants were presented with single-digit subtraction and multiplication problems in a delayed verification task. Problem size was manipulated in small and large problems with an additional extra-large subtraction condition to equate the overall magnitude of large multiplication problems. Results show comparable behavioral results and similar ERP dissociations across groups. First, an early operation type effect is observed around 200 ms post-problem onset, suggesting that both groups have a similar attentional differentiation for processing subtraction and multiplication problems. Second, for the posterior-occipital component between 240 ms and 300 ms, subtraction problems show a similar modulation with problem size in both groups, suggesting that only subtraction problems recruit quantity-related processes. Control analyses exclude possible perceptual and cross-operation magnitude-related effects. These results are the first evidence that the two operation types rely on distinct cognitive processes within the ASL native signing population and that they are equivalent to those observed in the English-speaking population.

Attitudes Toward Signing Avatars Vary Depending on Hearing Status, Age of Signed Language Acquisition, and Avatar Type.

The use of virtual humans (i.e., avatars) holds the potential for interactive, automated interaction in domains such as remote communication, customer service, or public announcements. For signed language users, signing avatars could potentially provide accessible content by sharing information in the signer's preferred or native language. As the development of signing avatars has gained traction in recent years, researchers have come up with many different methods of creating signing avatars. The resulting avatars vary widely in their appearance, the naturalness of their movements, and facial expressions-all of which may potentially impact users' acceptance of the avatars. We designed a study to test the effects of these intrinsic properties of different signing avatars while also examining the extent to which people's own language experiences change their responses to signing avatars. We created video stimuli showing individual signs produced by (1) a live human signer (Human), (2) an avatar made using computer-synthesized animation (CS Avatar), and (3) an avatar made using high-fidelity motion capture (Mocap avatar). We surveyed 191 American Sign Language users, including Deaf (N = 83), Hard-of-Hearing (N = 34), and Hearing (N = 67) groups. Participants rated the three signers on multiple dimensions, which were then combined to form ratings of Attitudes, Impressions, Comprehension, and Naturalness. Analyses demonstrated that the Mocap avatar was rated significantly more positively than the CS avatar on all primary variables. Correlations revealed that signers who acquire sign language later in life are more accepting of and likely to have positive impressions of signing avatars. Finally, those who learned ASL earlier were more likely to give lower, more negative ratings to the CS avatar, but we did not see this association for the Mocap avatar or the Human signer. Together, these findings suggest that movement quality and appearance significantly impact users' ratings of signing avatars and show that signed language users with earlier age of ASL acquisition are the most sensitive to movement quality issues seen in computer-generated avatars. We suggest that future efforts to develop signing avatars consider retaining the fluid movement qualities integral to signed languages.

Enhanced biological motion perception in deaf native signers.

We conducted two studies to test how deaf signed language users perceive biological motions. We created 18 Biological Motion point-light displays (PLDs) depicting everyday human actions, and 18 Scrambled control PLDs. First, we conducted an online behavioral rating survey, in which deaf and hearing raters identified the biological motion PLDs and rated how easy it was for them to identify the actions. Then, we conducted an EEG study in which Deaf Signers and Hearing Non-Signers watched both the Biological Motion PLDs and the Scrambled PLDs, and we computed the time-frequency responses within the theta, alpha, and beta EEG rhythms. From the behavioral rating task, we show that the deaf raters reported significantly less effort required for identifying the Biological motion PLDs, across all stimuli. The EEG results showed that the Deaf Signers showed theta, mu, and beta differentiation between Scrambled and Biological PLDs earlier and more consistently than Hearing Non-Signers. We conclude that native ASL users exhibit experience-dependent neuroplasticity in the domain of biological human motion perception.

A Positive Relationship Between Sign Language Comprehension and Mental Rotation Abilities.

Past work investigating spatial cognition suggests better mental rotation abilities for those who are fluent in a signed language. However, no prior work has assessed whether fluency is needed to achieve this performance benefit or what it may look like on the neurobiological level. We conducted an electroencephalography experiment and assessed accuracy on a classic mental rotation task given to deaf fluent signers, hearing fluent signers, hearing non-fluent signers, and hearing non-signers. Two of the main findings of the study are as follows: (1) Sign language comprehension and mental rotation abilities are positively correlated and (2) Behavioral performance differences between signers and non-signers are not clearly reflected in brain activity typically associated with mental rotation. In addition, we propose that the robust impact sign language appears to have on mental rotation abilities strongly suggests that "sign language use" should be added to future measures of spatial experiences.

Sensorimotor system engagement during ASL sign perception: An EEG study in deaf signers and hearing non-signers.

When a person observes someone else performing an action, the observer's sensorimotor cortex activates as if the observer is the one performing the action, a phenomenon known as action simulation. While this process has been well-established for basic (e.g., grasping) and complex (e.g., dancing) actions, it remains unknown if the framework of action simulation is applicable to visual languages such as American Sign Language (ASL). We conducted an EEG experiment with deaf signers and hearing non-signers to compare overall sensorimotor EEG between groups, and to test whether sensorimotor systems are differentially sensitive to signs that are produced with one hand ("1H") or two hands ("2H"). We predicted greater alpha and beta event-related desynchronization (previously correlated with action simulation) during the perception of 2H ASL signs compared to 1H ASL signs, due to greater demands on sensorimotor processing systems required for producing two-handed actions. We recorded EEG from both groups as they observed videos of ASL signs, half 1H and half 2H. Event-related spectral perturbations (ERSPs) in the alpha and beta ranges were computed for the two conditions at central electrode sites overlying the sensorimotor cortex. Sensorimotor EEG responses in both Hearing and Deaf groups were sensitive to the observed gross motor characteristics of the observed signs. We show for the first time that despite hearing non-signers showing overall more sensorimotor cortex involvement during sign observation, mirroring-related processes are in fact involved when deaf signers observe signs.

Sensorimotor characteristics of sign translations modulate EEG when deaf signers read English.

Bilingual individuals automatically translate written words from one language to another. While this process is established in spoken-language bilinguals, there is less known about its occurrence in deaf bilinguals who know signed and spoken languages. Since sign language uses motion and space to convey linguistic content, it is possible that action simulation in the brain's sensorimotor system plays a role in this process. We recorded EEG from deaf participants fluent in ASL as they read individual English words and found significant differences in alpha and beta EEG at central electrode sites during the reading of English words whose ASL translations use two hands, compared to English words whose ASL translations use one hand. Hearing non-signers did not show any differences between conditions. These results demonstrate the involvement of the sensorimotor system in cross-linguistic, cross-modal translation, and suggest that covert action simulation processes are involved when deaf signers read.

Neural bases of action abstraction.

There has been recent debate over whether actions are processed primarily by means of motor simulation or cognitive semantics. The current study investigated how abstract action concepts are processed in the brain, independent of the format in which they are presented. Eighteen healthy adult participants viewed different actions (e.g., diving, boxing) in the form of verbs and schematic action pictograms while functional magnetic resonance imaging (fMRI) was collected. We predicted that sensorimotor and semantic brain regions would show similar patterns of neural activity for different instances of the same action (e.g., diving pictogram and the word 'diving'). A representational similarity analysis revealed posterior temporal and sensorimotor regions where specific action concepts were encoded, independent of the format of presentation. These results reveal the neural instantiations of abstract action concepts, and demonstrate that both sensorimotor and semantic systems are involved in processing actions.

Fronto-temporal regions encode the manner of motion in spatial language.

When describing spatial events, dynamic actions can be decomposed into the path of motion (where the object moves), and the manner of motion (how the object moves). These components may be instantiated in two processing streams in the human brain, wherein dorsal parietal areas process path-related information, while ventral temporal regions process manner information. Previous research showed this pattern during the observation of videos showing animate characters in motion [15]. It is unknown whether reading language describing path and manner information - a level of abstraction beyond the perception of visual motion - relies on similar mechanisms. Here, we use functional neuroimaging to show that the left pMTG processes the manner of motion during reading. We also demonstrate the involvement of other ventral fronto-temporal regions in the understanding of manner of motion in spatial language.

Rethinking actions: implementation and association.

Action processing allows us to move through and interact with the world, as well as understand the movements performed by other people. In recent years, there has been increasing interest in the semantics of actions as differentiated from the semantics of objects. However, as the understanding of action semantics has evolved, it is evident that the existing literature conflates two senses of the word 'action'-one that stems from studies of tool use and the other from event representation. In this paper, we suggest that this issue can be clarified by closely examining differences in how the human parietal and temporal cortices of the brain process action-related stimuli. By contrasting the posterior parietal cortex to the posterolateral temporal cortex, we characterize two complementary action systems in the human brain, each with its own specialization of function. We suggest that these two systems be referred to as the parietal Action Implementation System, and the posterolateral temporal Action Association System. While the frontoparietal system is concerned primarily with how we perform actions, and simulate others' actions, the temporal action system is more involved with processing actions from a third-person, conceptual standpoint. Recent work in cognitive neuroscience of perception and language, as well as the neuroanatomical organization of these brain regions support this distinction. We will discuss the implications of this work for cognition-, language-, and neuroscience-based action research.

Influence of action-effect associations acquired by ideomotor learning on imitation.

According to the ideomotor theory, actions are represented in terms of their perceptual effects, offering a solution for the correspondence problem of imitation (how to translate the observed action into a corresponding motor output). This effect-based coding of action is assumed to be acquired through action-effect learning. Accordingly, performing an action leads to the integration of the perceptual codes of the action effects with the motor commands that brought them about. While ideomotor theory is invoked to account for imitation, the influence of action-effect learning on imitative behavior remains unexplored. In two experiments, imitative performance was measured in a reaction time task following a phase of action-effect acquisition. During action-effect acquisition, participants freely executed a finger movement (index or little finger lifting), and then observed a similar (compatible learning) or a different (incompatible learning) movement. In Experiment 1, finger movements of left and right hands were presented as action-effects during acquisition. In Experiment 2, only right-hand finger movements were presented during action-effect acquisition and in the imitation task the observed hands were oriented orthogonally to participants' hands in order to avoid spatial congruency effects. Experiments 1 and 2 showed that imitative performance was improved after compatible learning, compared to incompatible learning. In Experiment 2, although action-effect learning involved perception of finger movements of right hand only, imitative capabilities of right- and left-hand finger movements were equally affected. These results indicate that an observed movement stimulus processed as the effect of an action can later prime execution of that action, confirming the ideomotor approach to imitation. We further discuss these findings in relation to previous studies of action-effect learning and in the framework of current ideomotor approaches to imitation.

Visual influences on sensorimotor EEG responses during observation of hand actions.

There is growing interest within the field of social-cognitive neuroscience in the dynamics of sensorimotor EEG rhythms during the observation of actions performed by others. However, there remain important gaps in the literature regarding the effects of perceptual aspects of the observed hand movements. This study investigated two visual influences on the EEG response to hand actions. Specifically, the perspective of the action in relation to the participant (egocentric/allocentric) was varied and the effect of the hand used to carry out the action (left/right) was also assessed. While EEG was recorded, 28 undergraduate participants observed video clips showing an actor's hand reaching for, grasping, and lifting a cylindrical object across four conditions (right-hand egocentric, left-hand egocentric, right-hand allocentric, and left-hand allocentric). For actions viewed from an egocentric perspective, significantly greater event-related desynchronization (ERD) was present in the 7-9 Hz range over right mid-frontal, right central, and bilateral mid-parietal sites for right-handed actions compared to left-handed actions. In addition, greater ERD was observed within the 7-9 Hz band during the observation of right-handed egocentric actions compared to actions viewed from the allocentric perspective. This finding was present at bilateral central and mid-parietal sites, and emerged as an anticipatory effect prior to the onset of the observed hand movements.

The effect of action experience on sensorimotor EEG rhythms during action observation.

A recent line of inquiry has examined how an observer׳s experience with action changes the neural processing of similar actions when they are subsequently observed. The current study used electroencephalography (EEG) to test the hypothesis that giving participants different types and amounts of experience with specific objects would lead to differential patterns of sensorimotor rhythms during the observation of similar actions on those objects. While EEG was recorded, three groups of participants (n=20 in each group; mean age=22.0 years, SD=2.7) watched video clips of an actor reaching, grasping, and lifting two objects. Participants then received information about differences in weight between the two objects. One group gained this information through extended sensorimotor experience with the objects, a second group received much briefer sensorimotor experience with the objects, and the third group read written information about the objects׳ weights. Participants then viewed the action sequences again. For participants who had sensorimotor experience with the objects, the EEG response to viewing the actions was differentially sensitive to the anticipated weight of the objects. We conclude that this sensitivity was based on the participant׳s prior sensorimotor experience with the objects. The participants who only received semantic information about the objects showed no such effects. The primary conclusion is that even brief experience with actions affects sensorimotor cortex activity during the subsequent observation of similar actions.

Somatosensory experiences with action modulate alpha and beta power during subsequent action observation.

How does prior experience with action change how we perceive a similar action performed by someone else? Previous research has examined the role of sensorimotor and visual experiences in action mirroring during subsequent observation, but the contribution of somatosensory experiences to this effect has not been adequately examined. The current study tests whether prior somatosensory stimulation experienced during action production modulates brain activity during observation of similar actions being performed by others. Specifically, changes in alpha- and beta-range oscillations in the electroencephalogram (EEG) during observation of reaching actions were examined in relation to the observer's own prior experience of somatosensory stimulation while carrying out similar actions. Analyses revealed that alpha power over central electrodes was significantly decreased during observation of an action expected to result in somatosensory stimulation. Conversely, beta power was increased when an observed action was expected to result in somatosensory stimulation. These results suggest that somatosensory experiences may uniquely contribute to the way in which we process other people's actions.

Sensitivity of alpha and beta oscillations to sensorimotor characteristics of action: an EEG study of action production and gesture observation.

The sensorimotor experiences we gain when performing an action have been found to influence how our own motor systems are activated when we observe others performing that same action. Here we asked whether this phenomenon applies to the observation of gesture. Would the sensorimotor experiences we gain when performing an action on an object influence activation in our own motor systems when we observe others performing a gesture for that object? Participants were given sensorimotor experience with objects that varied in weight, and then observed video clips of an actor producing gestures for those objects. Electroencephalography (EEG) was recorded while participants first observed either an iconic gesture (pantomiming lifting an object) or a deictic gesture (pointing to an object) for an object, and then grasped and lifted the object indicated by the gesture. We analyzed EEG during gesture observation to determine whether oscillatory activity was affected by the observer's sensorimotor experiences with the object represented in the gesture. Seeing a gesture for an object previously experienced as light was associated with a suppression of power in alpha and beta frequency bands, particularly at posterior electrodes. A similar pattern was found when participants lifted the light object, but over more diffuse electrodes. Moreover, alpha and beta bands at right parieto-occipital electrodes were sensitive to the type of gesture observed (iconic vs. deictic). These results demonstrate that sensorimotor experience with an object affects how a gesture for that object is processed, as measured by the gesture-observer's EEG, and suggest that different types of gestures recruit the observer's own motor system in different ways.

Experience with novel actions modulates frontal α EEG desynchronization.

There is growing interest in the effects of experience on the neural processes linking action execution and action perception. We tested whether experience with unfamiliar actions can alter desynchronization of alpha-range power in the EEG upon re-observation of those actions. In a training session, participants (N=21) watched videos of novel drawing movements. Half of the movements were imitated after each viewing, and half of the movements were seen but not imitated, thus forming two training conditions: visual plus motor experience (V+M), and visual experience only (VO). In a testing session the next day, participants were shown the same videos of both sets of movements, and were also shown a third, completely novel, set of movements. Imitative performance was better for both training conditions than for novel actions. Event-related EEG desynchronization in the upper alpha band during action observation differed between conditions at frontal electrode sites, with novel actions being associated with less frontal desynchronization compared to V+M and VO actions. Differences between conditions were not noted over other regions. This suggests that moderate amounts of initial experience with novel actions can alter the neural processing of these actions when viewed again, particularly over frontal regions.

Abnormal brain activation during working memory in children with prenatal exposure to drugs of abuse: the effects of methamphetamine, alcohol, and polydrug exposure.

Structural and metabolic abnormalities in fronto-striatal structures have been reported in children with prenatal methamphetamine (MA) exposure. The current study was designed to quantify functional alterations to the fronto-striatal circuit in children with prenatal MA exposure using functional magnetic resonance imaging (fMRI). Because many women who use MA during pregnancy also use alcohol, a known teratogen, we examined 50 children (age range 7-15), 19 with prenatal MA exposure, 15 of whom had concomitant prenatal alcohol exposure (the MAA group), 13 with heavy prenatal alcohol but no MA exposure (ALC group), and 18 unexposed controls (CON group). We hypothesized that MA exposed children would demonstrate abnormal brain activation during a visuospatial working memory (WM) "N-Back" task. As predicted, the MAA group showed less activation than the CON group in many brain areas, including the striatum and frontal lobe in the left hemisphere. The ALC group showed less activation than the MAA group in several regions, including the right striatum. We found an inverse correlation between performance and activity in the striatum in both the CON and MAA groups. However, this relationship was significant in the caudate of the CON group but not the MAA group, and in the putamen of the MAA group but not the CON group. These findings suggest that structural damage in the fronto-striatal circuit after prenatal MA exposure leads to decreased recruitment of this circuit during a WM challenge, and raise the possibility that a rewiring of cortico-striatal networks may occur in children with prenatal MA exposure.

Perceived similarity and neural mirroring: evidence from vicarious error processing.

This study examined the effects of interpersonal similarity on vicarious error processing. We predicted that high similarity between self and other would predict increased neural responsiveness to the other's errors, based on the assumption that experience is more strongly shared when it involves similar others. Participants observed a confederate performing a flanker task while event-related brain potentials were recorded from the observer. Physiological data revealed two error-related potentials, the observational error-related negativity (oERN) and positivity (oPe). Self-reports of perceived similarity toward the confederate predicted both components. Participants reporting higher interpersonal similarity showed a larger oPe response to the other's errors, suggesting increased salience of errors committed by similar others. Unexpectedly, higher similarity also predicted a decreased oERN response. Divergent results for oERN and oPe may reflect the different functional roles of the two components. Together the results demonstrate that vicarious error monitoring is sensitive to social factors.

Trouble crossing the bridge: altered interhemispheric communication of emotional images in anxiety.

Worry is thought to involve a strategy of cognitive avoidance, in which internal verbalization acts to suppress threatening emotional imagery. This study tested the hypothesis that worry-prone individuals would exhibit patterns of between-hemisphere communication that reflect cognitive avoidance. Specifically, the hypothesis predicted slower transfer of threatening images from the left to the right hemisphere among worriers. Event-related potential (ERP) measures of interhemispheric transfer time supported this prediction. Left-to-right hemisphere transfer times for angry faces were relatively slower for individuals scoring high in self-reported worry compared with those scoring low, whereas transfer of happy and neutral faces did not differ between groups. These results suggest that altered interhemispheric communication may constitute one mechanism of cognitive avoidance in worry.

Error-monitoring ability predicts daily stress regulation.

This study examined whether individual differences in error-related self-regulation predict emotion regulation in daily life, as suggested by a common-systems view of cognitive and emotional self-regulation. Participants (N= 47) completed a Stroop task, from which error-related brain potentials and behavioral measures of error correction were computed. Participants subsequently reported on daily stressors and anxiety over a 2-week period. As predicted by the common-systems view, a physiological marker of error monitoring and a behavioral measure of error correction predicted emotion regulation in daily life. Specifically, participants higher in cognitive control, as assessed neurally and behaviorally, were less reactive to stress in daily life. The results support the notion that cognitive control and emotion regulation depend on common or interacting systems.