Integration of visual context in early and late bilingual language processing: evidence from eye-tracking.

Previous research on the processing of language embedded in a rich visual context has revealed the strong effect that a recently viewed action event has on language comprehension. It has been shown that listeners are more likely to view the target object of a recently performed event than look at the target object of a plausible future event during sentence utterance, regardless of the tense cue. In the current visual-world eye-tracking experiments, we tested the strength of the recently observed visual context with a group of English monolingual and two groups of English-French early and late bilingual speakers. By comparing these different groups, we examined whether bilingual speakers, as a consequence of greater cognitive flexibility when integrating visual context and language information, show early anticipatory eye-movements toward the target object. We further asked whether early and late bilinguals show differences in their processing. The findings of the three eye-tracking experiments revealed an overall preference for the recently seen event. However, as a result of the early provision of tense cue, this preference was quickly diminished in all three groups. Moreover, the bilingual groups showed an earlier decrease in reliance on the recently seen event compared to monolingual speakers and the early bilinguals showed anticipatory eye-movements toward the plausible future event target. Furthermore, a post-experimental memory test revealed that the bilingual groups recalled the future events marginally better than the recent events, whereas the reverse was found in the monolingual groups.

Influence of Actor's Congruent and Incongruent Gaze on Language Processing.

In interpreting spoken sentences in event contexts, comprehenders both integrate their current interpretation of language with the recent past (e.g., events they have witnessed) and develop expectations about future event possibilities. Tense cues can disambiguate this linking but temporary ambiguity in their interpretation may lead comprehenders to also rely on further, situation-specific cues (e.g., an actor's gaze as a cue to his future actions). How comprehenders reconcile these different cues in real time is an open issue that we must address to accommodate comprehension. It has been suggested that relating a referential expression (e.g., a verb) to a referent (e.g., a recent event) is preferred over relying on other cues that refer to the future and are not yet referentially grounded ("recent-event preference"). Two visual-world eye-tracking experiments compared this recent-event preference with effects of an actor's gaze and of tense/temporal adverbs as cues to a future action event. The results revealed that people overall preferred to focus on the recent (vs. future) event target in their interpretation, suggesting that while a congruent and incongruent actor gaze can jointly with futuric linguistic cues neutralize the recent-event preference late in the sentence, the latter still plays a key role in shaping participants' initial verb-based event interpretation. Additional post-experimental memory tests provided insight into the longevity of the gaze effects.

Anticipating a future versus integrating a recent event? Evidence from eye-tracking.

When comprehending a spoken sentence that refers to a visually-presented event, comprehenders both integrate their current interpretation of language with the recent event and develop expectations about future event possibilities. Tense cues can disambiguate this linking, but temporary ambiguity in these cues may lead comprehenders to also rely on further, experience-based (e.g., frequency or an actor's gaze) cues. How comprehenders reconcile these different cues in real time is an open issue. Extant results suggest that comprehenders preferentially relate their unfolding interpretation to a recent event by inspecting its target object. We investigated to what extent this recent-event preference could be overridden by short-term experiential and situation-specific cues. In Experiments 1-2 participants saw substantially more future than recent events and listened to more sentences about future-events (75% in Experiment 1 and 88% in Experiment 2). Experiment 3 cued future target objects and event possibilities via an actor's gaze. The event frequency increase yielded a reduction in the recent event inspection preference early during sentence processing in Experiments 1-2 compared with Experiment 3 (where event frequency and utterance tense were balanced) but did not eliminate the overall recent-event preference. Actor gaze also modulated the recent-event preference, and jointly with future tense led to its reversal in Experiment 3. However, our results showed that people overall preferred to focus on recent (vs. future) events in their interpretation, suggesting that while two cues (actor gaze and short-term event frequency) can partially override the recent-event preference, the latter still plays a key role in shaping participants' interpretation.

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects.

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing and the perception of the visual world. The recording of eye-gaze patterns has provided good evidence for the contribution of both the visual context and linguistic/world knowledge to language comprehension. Initial research assessed object-context effects to test theories of modularity in language processing. In the introduction, we describe how subsequent investigations have taken the role of the wider visual context in language processing as a research topic in its own right, asking questions such as how our visual perception of events and of speakers contributes to comprehension informed by comprehenders' experience. Among the examined aspects of the visual context are actions, events, a speaker's gaze, and emotional facial expressions, as well as spatial object configurations. Following an overview of the eye-tracking method and its different applications, we list the key steps of the methodology in the protocol, illustrating how to successfully use it to study visually-situated language comprehension. A final section presents three sets of representative results and illustrates the benefits and limitations of eye tracking for investigating the interplay between the perception of the visual world and language comprehension.

Preferential Inspection of Recent Real-World Events Over Future Events: Evidence from Eye Tracking during Spoken Sentence Comprehension.

Eye-tracking findings suggest people prefer to ground their spoken language comprehension by focusing on recently seen events more than anticipating future events: When the verb in NP1-VERB-ADV-NP2 sentences was referentially ambiguous between a recently depicted and an equally plausible future clipart action, listeners fixated the target of the recent action more often at the verb than the object that hadn't yet been acted upon. We examined whether this inspection preference generalizes to real-world events, and whether it is (vs. isn't) modulated by how often people see recent and future events acted out. In a first eye-tracking study, the experimenter performed an action (e.g., sugaring pancakes), and then a spoken sentence either referred to that action or to an equally plausible future action (e.g., sugaring strawberries). At the verb, people more often inspected the pancakes (the recent target) than the strawberries (the future target), thus replicating the recent-event preference with these real-world actions. Adverb tense, indicating a future versus past event, had no effect on participants' visual attention. In a second study we increased the frequency of future actions such that participants saw 50/50 future and recent actions. During the verb people mostly inspected the recent action target, but subsequently they began to rely on tense, and anticipated the future target more often for future than past tense adverbs. A corpus study showed that the verbs and adverbs indicating past versus future actions were equally frequent, suggesting long-term frequency biases did not cause the recent-event preference. Thus, (a) recent real-world actions can rapidly influence comprehension (as indexed by eye gaze to objects), and (b) people prefer to first inspect a recent action target (vs. an object that will soon be acted upon), even when past and future actions occur with equal frequency. A simple frequency-of-experience account cannot accommodate these findings.