Natural discourse reference generation reduces cognitive load in spoken systems.

The generation of referring expressions is a central topic in computational linguistics. Natural referring expressions - both definite references like 'the baseball cap' and pronouns like 'it' - are dependent on discourse context. We examine the practical implications of context-dependent referring expression generation for the design of spoken systems. Currently, not all spoken systems have the goal of generating natural referring expressions. Many researchers believe that the context-dependency of natural referring expressions actually makes systems less usable. Using the dual-task paradigm, we demonstrate that generating natural referring expressions that are dependent on discourse context reduces cognitive load. Somewhat surprisingly, we also demonstrate that practice does not improve cognitive load in systems that generate consistent (context-independent) referring expressions. We discuss practical implications for spoken systems as well as other areas of referring expression generation.

Linguistically mediated visual search.

During an individual's normal interaction with the environment and other humans, visual and linguistic signals often coincide and can be integrated very quickly. This has been clearly demonstrated in recent eye tracking studies showing that visual perception constrains on-line comprehension of spoken language. In a modified visual search task, we found the inverse, that real-time language comprehension can also constrain visual perception. In standard visual search tasks, the number of distractors in the display strongly affects search time for a target defined by a conjunction of features, but not for a target defined by a single feature. However we found that when a conjunction target was identified by a spoken instruction presented concurrently with the visual display, the incremental processing of spoken language allowed the search process to proceed in a manner considerably less affected by the number of distractors. These results suggest that perceptual systems specializedfor language and for vision interact more fluidly than previously thought.

Time course of frequency effects in spoken-word recognition: evidence from eye movements.

In two experiments, eye movements were monitored as participants followed spoken instructions to click on and move pictures with a computer mouse. In Experiment 1, a referent picture (e.g., the picture of a bench) was presented along with three pictures, two of which had names that shared the same initial phonemes as the name of the referent (e.g., bed and bell). Participants were more likely to fixate the picture with the higher frequency name (bed) than the picture with the lower frequency name (bell). In Experiment 2, referent pictures were presented with three unrelated distractors. Fixation latencies to referents with high-frequency names were shorter than those to referents with low-frequency names. The proportion of fixations to the referents and distractors were analyzed in 33-ms time slices to provide fine-grained information about the time course of frequency effects. These analyses established that frequency affects the earliest moments of lexical access and rule out a late-acting, decision-bias locus for frequency. Simulations using models in which frequency operates on resting-activation levels, on connection strengths, and as a postactivation decision bias provided further constraints on the locus of frequency effects.

Eye movements and lexical access in spoken-language comprehension: evaluating a linking hypothesis between fixations and linguistic processing.

A growing number of researchers in the sentence processing community are using eye movements to address issues in spoken language comprehension. Experiments using this paradigm have shown that visually presented referential information, including properties of referents relevant to specific actions, influences even the earliest moments of syntactic processing. Methodological concerns about task-specific strategies and the linking hypothesis between eye movements and linguistic processing are identified and discussed. These concerns are addressed in a review of recent studies of spoken word recognition which introduce and evaluate a detailed linking hypothesis between eye movements and lexical access. The results provide evidence about the time course of lexical activation that resolves some important theoretical issues in spoken-word recognition. They also demonstrate that fixations are sensitive to properties of the normal language-processing system that cannot be attributed to task-specific strategies.

Achieving incremental semantic interpretation through contextual representation.

While much work has been done investigating the role of context in the incremental processing of syntactic indeterminacies, relatively little is known about online semantic interpretation. The experiments in this article made use of the eye-tracking paradigm with spoken language and visual contexts in order to examine how, and when listeners make use of contextually-defined contrast in interpreting simple prenominal adjectives. Experiment 1 focused on intersective adjectives. Experiment 1A provided further evidence that intersective adjectives are processed incrementally. Experiment 1B compared response times to follow instructions such as 'Pick up the blue comb' under conditions where there were two blue objects (e.g. a blue pen and a blue comb), but only one of these objects had a contrasting member in the display. Responses were faster to objects with a contrasting member, establishing that the listeners initially assume a contrastive interpretation for intersective adjectives. Experiments 2 and 3 focused on vague scalar adjectives examining the time course with which listeners establish contrast for scalar adjectives such as tall using information provided by the head noun (e.g. glass) and information provided by the visual context. Use of head-based information was examined by manipulating the typicality of the target object (e.g. whether it was a good or poor example of a tall glass. Use of context-dependent contrast was examined by either having only a single glass in the display (the no contrast condition) or a contrasting object (e.g. a smaller glass). The pattern of results indicated that listeners interpreted the scalar adjective incrementally taking into account context-specific contrast prior to encountering the head. Moreover, the presence of a contrasting object, sharply reduced, and in some conditions completely eliminated, typicality effects. The results suggest a language processing system in which semantic interpretation, as well as syntactic processing, is conducted incrementally, with early integration of contextual information.

Syntactic ambiguity resolution in discourse: modeling the effects of referential context and lexical frequency.

Sentences with temporarily ambiguous reduced relative clauses (e.g., The actress selected by the director believed that...) were preceded by discourse contexts biasing a main clause or a relative clause. Eye movements in the disambiguating region (by the director) revealed that, in the relative clause biasing contexts, ambiguous reduced relatives were no more difficult to process than unambiguous reduced relatives or full (unreduced) relatives. Regression analyses demonstrated that the effects of discourse context at the point of ambiguity (e.g., selected) interacted with the past participle frequency of the ambiguous verb. Reading times were modeled using a constraint-based competition framework in which multiple constraints are immediately integrated during parsing and interpretation. Simulations suggested that this framework reconciles the superficially conflicting results in the literature on referential context effects on syntactic ambiguity resolution.

Eye movements as a window into real-time spoken language comprehension in natural contexts.

When listeners follow spoken instructions to manipulate real objects, their eye movements to the objects are closely time locked to the referring words. We review five experiments showing that this time-locked characteristic of eye movements provides a detailed profile of the processes that underlie real-time spoken language comprehension. Together, the first four experiments showed that listeners immediately integrated lexical, sublexical, and prosodic information in the spoken input with information from the visual context to reduce the set of referents to the intended one. The fifth experiment demonstrated that a visual referential context affected the initial structuring of the linguistic input, eliminating even strong syntactic preferences that result in clear garden paths when the referential context is introduced linguistically. We argue that context affected the earliest moments of language processing because it was highly accessible and relevant to the behavioral goals of the listener.

Integration of visual and linguistic information in spoken language comprehension.

Psycholinguists have commonly assumed that as a spoken linguistic message unfolds over time, it is initially structured by a syntactic processing module that is encapsulated from information provided by other perceptual and cognitive systems. To test the effects of relevant visual context on the rapid mental processes that accompany spoken language comprehension, eye movements were recorded with a head-mounted eye-tracking system while subjects followed instructions to manipulate real objects. Visual context influenced spoken word recognition and mediated syntactic processing, even during the earliest moments of language processing.

A constraint-based lexicalist account of the subject/object attachment preference.

When a noun phrase could either be the object of the preceding verb or the subject of a new clause or a sentence complement, readers and listeners show a strong preference to parse the noun phrase as the object of the verb. This can result in clear garden paths for sentences such as The student read the book was stolen and While the student read the book was stolen. Even when the verb does not permit a noun phrase complement, some processing difficulty is still found. This has led some theorists to propose models in which initial attachments are lexically blind, with lexical information subsequently used as a filter to evaluate and revise initial analyses. In contrast, we show that these results emerge naturally from constraint-based lexicalist models. We present a modeling experiment with a simple recurrent network that was trained to predict upcoming complements for a sample of verbs taken from the Penn Treebank corpus. The model exhibits an object bias and it also shows effects of verb frequency which are similar to those found in the psycholinguistic literature.

Context effects in syntactic ambiguity resolution: discourse and semantic influences in parsing reduced relative clauses.

This article examines how certain types of semantic and discourse context affect the processing of relative clauses which are temporarily ambiguous between a relative clause and a main clause (e.g., "The actress selected by the director ..."). We review recent results investigating local semantic context and temporal context, and we present some new data investigating referential contexts. The set of studies demonstrate that, contrary to many recent claims in the literature, all of these types of context can have early effects on syntactic, ambiguity resolution during on-line reading comprehension. These results are discussed within a "constraint-based" framework for ambiguity resolution in which effects of context are determined by the strength and relevance of the contextual constraint and by the availability of the syntactic alternatives.

Verb-specific constraints in sentence processing: separating effects of lexical preference from garden-paths.

Immediate effects of verb-specific syntactic (subcategorization) information were found in a cross-modal naming experiment, a self-paced reading experiment, and an experiment in which eye movements were monitored. In the reading studies, syntactic misanalysis effects in sentence complements (e.g., "The student forgot the solution was...") occurred at the verb in the complement (e.g., was) for matrix verbs typically used with noun phrase complements but not for verbs typically used with sentence complements. In addition, a complementizer effect for sentence-complement-biased verbs was not due to syntactic misanalysis but was correlated with how strongly a particular verb prefers to be followed by the complementizer that. The results support models that make immediate use of lexically specific constraints, especially constraint-based models, but are problematic for lexical filtering models.

Levels of representation in the interpretation of anaphoric reference and instrument inference.

The level of representation accessed when inferences are made during sentence comprehension was examined. The inferences investigated included antecedent assignment for both definite noun phrase anaphors and pronouns and also instrument inferences. In making these inferences, a listener must access the inferred element, whether an antecedent or an instrument, in either a linguistic form representation or a discourse model. The level of representation involved in these inferences was determined by exploiting differences in the lexical decision and naming tasks, which were argued to exhibit differential sensitivity to representational levels. In three experiments, the priming of antecedent and instrument targets in the lexical decision task was compared with priming of the same targets in the naming task. Differences in the patterns of activation across the two tasks indicated that all three types of inferences required accessing elements in a discourse model. Three control experiments ruled out simple context or congruity checking as an explanation for our results. The following conclusions were also supported by these studies: (1) Antecedent assignment occurs immediately after processing an anaphor; (2) antecedent assignment involves inhibition for the inappropriate antecedent rather than facilitation for the appropriate antecedent; (3) although subjects do not make instrument inferences when they hear isolated sentences containing verbs that strongly imply certain instruments, the inferences are made when sentences are preceded by a context that mentions the instrument.

Lexical projection and the interaction of syntax and semantics in parsing.

A series of self-paced reading studies utilized an embedded anomaly technique to investigate long-distance dependencies with dative verbs. Previous research in our lab demonstrated that argument structure influences the gap-filling process. Experiment 1 extended that work by demonstrating that dative verbs pattern with other complex transitive verbs (i.e., a fronted filler that is implausible as the direct object will not be interpreted as the direct object until the absence of a noun phrase after the verb forces the postulation of a direct object gap. This pattern contrasts with that of transitive verbs that subcategorize for a single internal argument position, where fronted fillers are obligatorily interpreted as the direct object). Experiments 2 and 3 investigate the prediction that semantic analyses precede syntactic analyses in dative questions. It is argued that the lexical information about argument structure and thematic roles can guide semantic interpretation.

Context and lexical access: implications of nonword interference for lexical ambiguity resolution.

To eliminate potential "backward" priming effects, Glucksberg, Kreuz, and Rho (1986) introduced a variant of the cross-modal lexical priming task in which subjects made lexical decisions to nonword targets that were modeled on a word related to either the contextually biased or unbiased sense of an ambiguous word. Lexical decisions to nonwords were longer than controls only when the nonword was related to the contextually biased sense of the ambiguous word, leading Glucksberg et al. to conclude that context does constrain lexical access and that the multiple access pattern observed in previous studies was probably an artifact of backward priming. We did not find nonword interference when the nonword targets used by Glucksberg et al. were preceded by semantically related ambiguous or unambiguous word primes. However, we did replicate their sentence context results when the ambiguous words were removed from the sentences. We conclude that the interference obtained by Glucksberg et al. is due to postlexical judgements of the congruence of the sentence context and the target, not to context constraining lexical access.

Lexical structure in Parsing long-distance dependencies.

We review a series of experiments investigating lexical influences in parsing sentences with long-distance dependencies. We report three primary results. First, gaps are posited and filled immediately following verbs that are typically used transitively, even when the filler is an implausible object of the verb. However, gaps are not posited after verbs that are typically used intransitively. Second, plausibility determines whether or not a filler is treated as the object of a verb when the verb is typically used with both a direct object and an infinitive complement. Finally, verb control information is used immediately in determining which noun phrase will be interpreted as the "understood" subject of an infinitive complement.

Evoked potentials and the study of sentence comprehension.

Evoked brain potentials were used to monitor moment-by-moment decisions during language comprehension. Subjects read sentences containing temporary syntactic ambiguities for which one of the possible interpretations was semantically implausible. The N400 component of the evoked potential, which is sensitive to implausibility, was used to discover when during a sentence subjects made a decision about the ambiguity. The results demonstrate that readers try to interpret a syntactic ambiguity early in a sentence rather than waiting for disambiguating information. This introduces a new way to use brain activity to study sentence comprehension processes.

Lateralization of lexical codes in auditory word recognition.

Three experiments examined the lateralization of lexical codes in auditory word recognition. In Experiment 1 a word rhyming with a binaurally presented cue word was detected faster when the cue and target were spelled similarly than when they were spelled differently. This orthography effect was larger when the target was presented to the right ear than when it was presented to the left ear. Experiment 2 replicated the interaction between ear of presentation and orthography effect when the cue and target were spoken in different voices. In Experiment 3, subjects made lexical decisions to pairs of stimuli presented to the left or the right ear. Lexical decision times and the amount of facilitation which obtained when the target stimuli were semantically related words did not differ as a function of ear of presentation. The results suggest that the semantic, phonological, and orthographic codes for a word are represented in each hemisphere; however, orthographic and phonological representations are integrated only in the left hemisphere.

Phonological code activation during listening.

Rhyme priming to visually dissimilar rhymes (e.g., eight-late) was used in a lexical decision task to investigate the access and maintenance of speech-based codes in sentence comprehension. One member of the rhyme pair was embedded in a sentence and the other was presented visually for lexical decision. Rhyme priming obtained when the prime and target were separated by four but not by seven intervening words, suggesting that the phonological code for the word was initially accessed and then rapidly decayed.

Subvocal motor activity and contextual processing.

Twenty male and female subjects listened for mispronounced words while minimizing either subvocal or frontalis electromyographic activity. Stimuli were varied on size of the distortion, lexical constraint, and contextual constraint, all known to influence detections. Analysis of both the reaction time and detection data indicated that the minimization of subvocal EMG activity reduced or eliminated the effect of contextual constraint, effect. Results indicate that subvocal activity is related to contextual processing. Additionally, reaction time data are reported that indicate that although low contextual constraint greatly slows the decision process, detectability is actually superior. A possible underlying mechanism for this reversal of the speed-accuracy trade-off is discussed.