Memory representations in a cross-modal matching task: evidence for a verbal component.

In everyday tasks, one often uses touch to find what has been seen. Recent research has identified that when individuals view or touch an object, they may create a verbal memory representation; however, this research involved object naming, which may have prompted the use of verbal strategies. Research has also identified variability in memory representations for objects, which may indicate individual differences. To investigate memory representations and their associations with individual differences in cognitive styles, we measured the cognitive styles of 127 participants and had them complete a non-verbal matching task without distractors, or with verbal or visual distractors. In the task, they viewed an object and then touched an object - or vice versa - and indicated whether the objects were the same or different. On trials where different objects were presented, participants responded consistently more slowly and made more matching errors for similar objects compared to distinct objects. Importantly, higher scores on the verbalizer cognitive style predicted faster reaction times on the matching task across all trial types and distraction conditions. Overall, this indicates that cross-modal object processing in short-term memory may be facilitated by a verbal code.

The impact of attentional demands on audiovisual integration depends on task-specific components.

Research investigating how attentional demands impacts audiovisual (AV) integration has used a variety of multisensory tasks and procedures to manipulate attentional demands, leading to very differing results. Also, the secondary tasks used to increase attentional demands draw on the sensory modalities already being investigated; for example, a visual distracter task may be used to increase attentional demands in an audiovisual integration task. It is therefore not clear whether the additional task interfered with sensory processing or with audiovisual integration. We used a Colavita task where participants are asked to report the modality of auditory, visual, and audiovisual stimuli to investigate whether increasing attentional demands would impact audiovisual integration. In Experiments 1 and 2, we used a concurrent foot-tapping task to show that increasing attentional demands by having participants completing a secondary task in a different modality interfered with sensory processing but did not affect audiovisual integration. In Experiments 3 and 4, we manipulated attentional demands by having participants respond to all stimuli or only to target stimuli and showed that audiovisual integration was only impacted when targets were infrequent: When participants responded to specific targets amongst five different distracters, they no longer produced more "visual-only" responses than "auditory-only" responses. Whether attentional demands can impact audiovisual integration does not seem unitary and instead seems to depend on task-specific components. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Developing a memory representation: Do we visualize or do we "verbalize" objects?

According to encoding specificity, participants perform better when testing conditions match learning conditions. It is interesting that recent findings in visuo-haptic object identification violate this principle: Participants who learned to recognise objects haptically performed just as well when asked to identify objects by sight and by touch. One possible explanation is that participants who explore objects haptically visualize the objects they explore, creating a multisensory memory trace equally accessible to vision and touch. We evaluated this possibility by asking undergraduate participants to learn to recognise novel objects either by sight or by touch. Participants completed sequences of learning trials where they explored each object and test trials where they recalled the name of each object. During learning trials, some participants were presented with a visual distractor (either verbal or nonverbal characters) that they had to recognise later, whereas other participants completed a distractorless control condition. Consistent with past findings, our results violated encoding specificity for participants who learned to recognise objects haptically-this was not modified by the addition of a secondary task. It is interesting, however, that only the verbal distractors interfered with learning. These results suggest that the creation of memory representations for novel objects involves a verbal code rather than visualization, independently of how objects are initially explored. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Visuo-haptic integration in object identification using novel objects.

Although some studies have shown that haptic and visual identification seem to rely on similar processes, few studies have directly compared the two. We investigated haptic and visual object identification by asking participants to learn to recognize (Experiments 1, and 3), or to match (Experiment 2) novel objects that varied only in shape. Participants explored objects haptically, visually, or bimodally, and were then asked to identify objects haptically and/or visually. We demonstrated that patterns of identification errors were similar across identification modality, independently of learning and testing condition, suggesting that the haptic and visual representations in memory were similar. We also demonstrated that identification performance depended on both learning and testing conditions: visual identification surpassed haptic identification only when participants explored the objects visually or bimodally. When participants explored the objects haptically, haptic and visual identification were equivalent. Interestingly, when participants were simultaneously presented with two objects (one was presented haptically, and one was presented visually), object similarity only influenced performance when participants were asked to indicate whether the two objects were the same, or when participants had learned about the objects visually-without any haptic input. The results suggest that haptic and visual object representations rely on similar processes, that they may be shared, and that visual processing may not always lead to the best performance.

Impact of semantic similarity in novel associations: Direct and indirect routes to action.

Actions produced in response to familiar objects are predominantly mediated by the visual structure of objects, and less so by their semantic associations. Choosing an action in response to an object tends to be faster than choosing the object's name, leading to the suggestion that there are direct links between the visual representations of objects and their actions. The relative contribution of semantics, however, is unclear when actions are produced in response to novel objects. To investigate the role of semantics when object-action associations are novel, we had participants learn to use and name novel objects and rehearse the object, action, and name associations over one week. Each object-action pair was associated with a label that was either semantically similar or semantically distinct. We found that semantic similarity only affected action and name production when the object associations were novel, suggesting that semantic information is recruited when actions are produced in response to novel objects. We also observed that the advantage to producing an action was absent when associations were novel, suggesting that practice is necessary for these direct links to develop.

Influences of visual and action information on object identification and action production.

To evaluate the impact of semantic information elicited by labels, participants learned to identify or use novel graspable objects associated with novel actions. We identified each object/action pair with labels that elicited visual form or action semantics and varied the congruence between the label's information and the visual form or action of novel objects. In Experiment 1, participants named objects, and in Experiment 2 they produced the action associated with objects. Generally, congruent labels facilitated performance. Furthermore, for participants who learned incongruent associations, the visual form and semantic information elicited by labels influenced performance in opposite patterns. These findings support the notion that naming may be required before actions are produced when object/action associations are novel. Our findings further support the notion that links between the structural properties of objects and their actions may already be stronger than the links between verbal labels and actions in novel object/action associations.

A role for action knowledge in visual object identification.

We evaluated the impact of visual similarity and action similarity on visual object identification. We taught participants to associate novel objects with nonword labels and verified that in memory visually similar objects were confused more often than visually dissimilar objects. We then taught participants to associate novel actions with nonword labels and verified that similar actions were confused moreoften th an dissimilaractions. We then paired specific objects with specific actions. Visually similar objects paired with similar actions were confused more often in memory than when these same objects were paired with dissimilar actions. Hence the actions associated with objects served to increase or decrease their separation in memory space, and influenced the ease with which these objects could be identified. These experiments ultimately demonstrated that when identifying stationary objects, the memory of how these object were used dramatically influenced the ability to identify these objects.

The importance of object similarity in the production and identification of actions associated with objects.

Past research suggests that the similarity between the objects associated with actions impacts visual action identification and action production. Indeed, people often confuse actions that are visually similar, as well as actions that are associated with visually similar objects. However, because the action errors often involve actions that are visually similar and are associated with visually similar objects, it is difficult to disambiguate between the influences of object similarity and action similarity. In our experiments, healthy participants were asked to learn to associate nonword names and actions with novel objects. Participants were first shown each object and its action and were then asked to visually identify each object. In Experiment 1, participants were then asked to produce the action associated with each object, and in Experiment 2, they were asked to visually identify the action associated with each object. Actions were confused more often when they were associated with similar objects than when they were associated with dissimilar objects. Furthermore, following an object naming error, participants were more likely to produce the action associated with the erroneous name than any other erroneous action. The results suggest that the visual characteristics of the objects influenced action production and action identification.

Understanding the structural determinants of object confusion in memory: an assessment of psychophysical approaches to estimating visual similarity.

Computer-generated shapes varying on visual dimensions such as curvature, tapering, and thickness have been used to investigate identification deficits in the category-specific visual agnosia (CSVA) Patient E.L.M.. However, whether the implemented variations on each of these dimensions were perceived by novice observers as "similar amounts of change" is unknown. To estimate distance in psychophysical shape space, sets of shapes were developed using two different scaling methods--an objective method based on visual search, and a subjective method based on judgments of similarity--and a third approach that did not involve scaling. How well each method estimated psychophysical shape space was assessed by measuring the confusions within memory among the shapes. The results suggested that, although neither of the approaches perfectly reflected psychophysical shape space, subjective scaling was a better estimator of distance in psychophysical shape space than were other approaches. The number of confusions produced on each set of shapes was used to develop a new set of shapes that accurately estimated distance in psychophysical shape space. These results suggest that a combination of approaches is preferable in order to accurately estimate distance in psychophysical shape space.

Not all triads are created equal: further support for the importance of visual and semantic proximity in object identification.

Identification deficits were investigated in ELM, a temporal lobe stroke patient with category-specific deficits. We replicated previous work done on FS, a patient with category specific deficits as a result of herpes viral encephalitis. ELM was tested using novel, computer generated shapes that were paired with artifact labels. We paired semantically close or disparate labels to shapes and ELM attempted to learn these pairings. Overall, ELM's shape-label confusions were most detrimentally affected when we used labels that referred to objects that were visually and semantically close. However, as with FS, ELM had as many errors when shapes were paired with the labels "donut," "tire," and "washer" as he did when they were paired with visually and semantically close artifact labels. Two explanations are put forth to account for the anomalous performance by both patients on the triad of donut-tire-washer.

The role of premorbid expertise on object identification in a patient with category-specific visual agnosia.

The role of premorbid expertise in object identification was investigated in the category-specific visual agnosia patient ELM. For several years prior to his stroke ELM played bugle in a military band. We surmised that band membership would lead to preferential exposure to, and expertise for, brass instruments relative to other musical-instrument families. To test this hypothesis we assessed ELM's musical instrument identification capability for brass and stringed instruments. Testing was conducted 14 years post-stroke. ELM listed significantly more correct attributes for five brass instruments than for five stringed instruments. On a picture-word matching task ELM showed significantly better identification of brass, relative to stringed, musical instruments. Finally, when ELM was required to pair novel shapes with labels denoting brass or stringed instruments, he made significantly more errors in the stringed-instrument condition than in the brass-instrument condition. We conclude that the elevated attribute knowledge accompanying expertise serves to increase the visual and semantic distance between objects within a category, thereby protecting them against identification deficits in the context of category-specific visual agnosia.