Continuous integration of phonetic and semantic constraints on spoken-word recognition

D. Dahan* and M. K. Tanenhaus+
* Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
+ University of Rochester, Rochester, U.S.A.

The interpretation of spoken words is generally viewed--implicitly or explicitly--as a staged process. First lexical candidates are activated based on their phonetic match with the signal; then a selection is made based on candidates' semantic fit with the context [1,2]. Little attention has been devoted to determining how these stages interact. Here, we
demonstrate immediate and continuous integration of phonetic and contextual information, using an eyetracking procedure. Fixations provide a sensitive measure of the time course of lexical activation and are affected by significant factors in word recognition, such as lexical frequency and candidates' goodness of fit with the input [3-5]. Here, the
influence of thematic constraints on word recognition was measured using Dutch constructions in which the verb precedes its subject.

Dutch participants were asked to determine which of four displayed objects was mentioned in a sentence. Fixations were monitored using an eyetracker. The object's name (the sentence's subject) was preceded by either a semantically neutral auxiliary verb, or a semantically constraining verb:

target: "bok" (goat)

Neutral:
Nog nooit is een bok zo hoog geklommen.
[never before had a goat so high climbed.]

Constraining:
Nog nooit klom een bok zo hoog.
[never before climbed a goat so high.]

Among the pictured objects were the target (e.g., "bok"), a cohort competitor (e.g. "bot" [bone]), a semantically plausible, but phonetically unrelated, distractor (e.g., "spin" [spider]), and an unrelated distractor (e.g., "eiland" [island]). For neutral sentences, as participants heard the onset of the target's name, fixations were directed equally to the target and cohort-competitor pictures, but not other distractors, revealing standard cohort-activation effects [3,4]. For constraining sentences, the cohort effect disappeared: Fixations to "bot" and the distractor picture were entirely equivalent. The thematic incompatibility of a bone as the subject of the preceding verb "to climb" eliminated the lexical alternative "bot" despite its phonetic overlap with the signal. Thus, semantic information can immediately constrain the interpretation of a spoken word. The presence of the constraining verb did not cause the semantically plausible distractor (e.g., the spider) to receive more fixations, indicating that both semantic fit and phonetic match contributed to establishing the referent.

A second study extended this result. Here, the vowel of the target's name was altered to contain (misleading) coarticulatory cues temporarily supporting the competitor over the target (e.g., bo[t]k). Interpretation of altered target names was initially biased toward target pictures after constraining verbs, but as the coarticulated vowel was heard, fixations to cohort competitors increased significantly. Thus, participants' interpretation of the target's name was modulated by short-lived, yet constraining acoustic cues, even when the preceding context rendered the best-matching candidate semantically implausible.

These results give powerful support to models where phonetic and semantic information are continuously merged and weighted according to the degree of constraint they each impose on the recognition of the current word.

References

[1] Marslen-Wilson, W. D. (1987). Functional parallelism in spoken word-recognition. Cognition, 25, 71-102.

[2] Zwitserlood, P. (1989). The locus of effects of sentential-semantic context in spoken-word processing. Cognition, 32, 25-64.

[3] Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: evidence for continuous mapping models. Journal of Memory and Language, 38, 419-439.

[4] Dahan, D., Magnuson, J. S., & Tanenhaus, M. K. (2001). Time course of frequency effects in spoken-word recognition: Evidence from eye movements. Cognitive Psychology, 42, 317-367.

[5] Dahan, D., Magnuson, J. S., Tanenhaus, M. K., & Hogan, E. M. (2001). Subcategorical mismatches and the time course of lexical access: Evidence for lexical competition. Language and Cognitive Processes, 16, 507-534.