the role of vision and attention in language processing antje s. meyer university of birmingham and...
TRANSCRIPT
![Page 1: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/1.jpg)
The role of vision and attention in language processing
Antje S. Meyer
University of Birminghamand Max Planck Institute for Psycholinguistics Nijmegen
![Page 2: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/2.jpg)
Introduction
• Psycholinguists (quite sensibly) tend to focus on language processing.
• Exception (also sensible): Reading research – Implemented models, e.g. SWIFT (Kliegl et
al., 2006), E-Z Reader (Reichle et al., 2003)
![Page 3: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/3.jpg)
Language is situated
• We see things when we listen and speak.• We often try to ignore the visual
information. This requires cognitive control.
• We often use the visual information. This requires cognitive control, capacity sharing and cross-modal integration.
• Historical background: Modularity
![Page 4: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/4.jpg)
Plan of the talk
• The visual world paradigm of speech comprehension
• The multiple object naming paradigm of production
• Why do the paradigms work?
![Page 5: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/5.jpg)
Cooper, 1974
“ a practical new research tool for the real-time investigation of perceptual and cognitive processes and, in particular, for the detailed study of speech perception, memory, and language processing" (p. 84).
![Page 6: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/6.jpg)
Modern versions of the visual world paradigm
• Acting out “Look at the dog” (e.g., Dahan & Gaskell, 2007)
![Page 7: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/7.jpg)
Modern versions of the visual world paradigm
• Passive listening (e.g. Altmann & Kamide, 2007)
“The man will drink the wine”
![Page 8: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/8.jpg)
SR Eyelink II
![Page 9: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/9.jpg)
Analysis of fixation proportions
(Dahan & Tanenhaus, 2004)
![Page 10: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/10.jpg)
Applications of visual world paradigm
• Processing of low-level phonetic cues, e.g., Salverda et al., 2002– Utterance “Look at the hamster” triggers
earlier looks to target when there is a hamster + three unrelated objects than when there is a hamster in the display as well.
– Hamster is looked at earlier when first syllable is excised from “hamster” than from “hammer”.
![Page 11: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/11.jpg)
Applications of visual world paradigm
• Syntactic analysis, e.g. Tanenhaus, 1995– Participants acted out “Put the apple on
the towel in the box”. – Display includes 1 or 2 apples (one on towel)– 2-apple display leads to earlier looks to box (=
faster understanding) than 1-apple display. – In two-apple condition, listeners immediately
understand that “on the towel” specifies which apple to move.
![Page 12: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/12.jpg)
Applications of visual world paradigm
• Predictive listening, e.g. Altmann & Kamide, 2009
“The boy will eat the cake” leads to earlier fixations on cake than “The boy will move the cake”. This shows rapid use of semantic knowledge.
“The girl/man will ride .. “ lead to early looks to age-appropriate vehicles. This shows rapid use of world knowledge.
![Page 13: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/13.jpg)
Main results of visual world research
• Listeners use various types of information to understand spoken utterances.
• Listeners use the information as soon as possible.
• The listeners’ use of information conveyed by speech and by visual information and their use of semantic and world knowledge is reflected in their eye gaze.
![Page 14: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/14.jpg)
Use of eye tracking in speech production research
• Description of scenes and events (Bock & Griffin, 1998;Gleitman et al., 2007)
• Can we distinguish between pre-linguistic scene apprehension and linguistic formulation?
![Page 15: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/15.jpg)
Multiple-object naming
4
1
2
3
Gaze
4
5
“Globe, kite, doll.”
Speakers look at each object they mention. They look at the objects in the order of mention
![Page 16: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/16.jpg)
![Page 17: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/17.jpg)
The moon…..
![Page 18: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/18.jpg)
The moon next to the square…..
![Page 19: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/19.jpg)
The moon next to the square is pale.
(Meyer, van der Meulen, & Brooks, 2004)
![Page 20: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/20.jpg)
Three core findings
1. Speakers look at each object they mention.
2. They look at the objects in the order of mention.
3. They look at each object until they have generated the phonological representation of the corresponding expression.
![Page 21: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/21.jpg)
Determinants of gaze durations in object naming
• Visual complexity (Meyer et al., 1998)• Ease of lexical selection (Belke et al., 2005)• Name frequency & length (Korvorst et al., 2006; Meyer et
al., 2003, 2005)
![Page 22: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/22.jpg)
Coordination of word planning an eye gaze
Lemma access
Morphological encoding
Phonological encoding
Phonetic encoding
Articulation
Visual-conceptual analysis
Saccade to next object
Gaze onset
(Levelt, Roelofs, & Meyer, 1999)
![Page 23: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/23.jpg)
The Serial Processing Hypothesis
![Page 24: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/24.jpg)
The Gradient Hypothesis
(e.g., Cave & Bichot, 1999)
![Page 25: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/25.jpg)
Extrafoveal object processing
Interloper
(Morgan & Meyer, 2005)
![Page 26: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/26.jpg)
Extrafoveal object processing
Target
![Page 27: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/27.jpg)
Interloper-target relationship
Identical (bat-bat)
Homophonous (bat-bat)
Unrelated (flower-bat)
Dependent measure:
Gaze duration for target
interloper target
![Page 28: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/28.jpg)
400
500
600
700
800
Interloper-target relationship
(ms)
identical homophonous unrelated
• Interlopers affected the target gaze durations.
• Speakers began to process interlopers prior to fixation.
• Interlopers were processed to the level of word form.
Target gaze durations
![Page 29: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/29.jpg)
Effects of left-object difficulty
500
550
600
650
700
750
800
Easy Difficult
Left Object
ms
Identical Interloper Homophonous Interloper Unrelated Interloper
(Meyer, Ouellet, & Häcker, 2008; see also Meyer, subm.,a,b)
Bigger effects!
![Page 30: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/30.jpg)
Gradient Hypothesis
![Page 31: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/31.jpg)
Conditions for parallel object processing
• The foveated object must be easy to process (Malpass & Meyer, 2010).
• Objects must be close together (Konopka & Meyer, subm.).
• Syntactic structure must be constant (Meyer, subm., a, b).
![Page 32: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/32.jpg)
Summary of production studies
• Speakers look at the objects they name.• They fixate upon the objects until they
have retrieved a suitable referring expression.
• They sometimes process two objects in parallel.
• Object recognition and speech planning processes are reflected in the speakers’ eye gaze.
![Page 33: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/33.jpg)
Linking eye gaze and speech planning
• Eye gaze reflects on visual attention (e.g. Irwin, 2004).
• Attended objects are processed with priority (fastest, most accurately).
• The easiest way of attending to an objects it to look at it.
![Page 34: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/34.jpg)
Why do speakers look at and attend to the objects they name?
• Fixation is necessary for identification.• Retrieval of information from the visual
environment is prefered over memory retrieval.
• Visual attention facilitates object recognition and the retrieval of associated information, including the name (e.g. Remington & Folk, 2001;Malpass & Meyer, 2010).
![Page 35: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/35.jpg)
Why do speakers attend to the objects they name?
• Lexical access requires processing capacity (e.g. Ferreira & Pashler, 2002; Cook & Meyer, 2007).
• Speech planning is constrained by the speed of articulation (Wheeldon & Meyer, subm., Frisson & Meyer, in prep). – Eye movements in reading aloud and picture
naming are very similar.
![Page 36: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/36.jpg)
Eye gaze reflects speech plan
• Speakers process the objects largely sequentially.
• This fits well with other results showing that speech planning is highly incremental (e.g., Meyer, 1996; Dell et al., 1997).
• Visual attention and speech planning are governed by a common plan. Executive control processes govern the implementation of the plan.
![Page 37: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/37.jpg)
Prioritised visual-conceptual processing of left object
Dual lemma access
Morphological encoding
Phonological encoding
Phonetic encoding
Articulation
Direct visual attention and eye gaze to appropriate location
WM representation of taskName left object first, then right object
Engage linguistic encoding routines(build two-slot syntactic frame,enable lexical access)
A working model of multiple object naming
![Page 38: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/38.jpg)
Prioritised visual-conceptual processing of left object
Dual lemma access
Morphological encoding
Phonological encoding
Phonetic encoding
Articulation
Direct visual attention and eye gaze to appropriate location
WM representation of task• General: Name objects on screen• Specific: Name left object first, then right object
Engage linguistic encoding routines(build two-slot syntactic frame,enable lexical access)
A working model of multiple object naming
Monitoring• Compare phonological representation to task description• Instruct articulators or initiate repair• Update task description
![Page 39: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/39.jpg)
Linking eye gaze to speech comprehension
• Listeners look at task-relevant objects because this facilitates carrying out the task.
• The task can be explicit (“look at the tree”) or implicit, mapping visual and auditory representations (see Altmann & Kamide, 2009).
• Objects become relevant when they are mentioned or implied.
![Page 40: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/40.jpg)
Direct visual attention to target, prioritised processing of target
Phonetic encoding
Target recognition
Semantic Integration
Create and maintain WM representation of object locations
WM representation of task• General: Map visual and auditory information• Specific: Find the objects mentioned the utterance
Engage linguistic encoding routines
A working model of situated listening
•
Monitoring
![Page 41: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/41.jpg)
Looks to incorrect locations
• Listeners look at – a hammer instead of a hamster (Salverda et
al., 2003)– a rope instead of a snake (Huettig & Altmann,
2007)– a strawberry instead of lips (Huettig &
Altmann, in press).
• Active working memory representations of objects attract attention (usually to task-relevant object, sometimes to a distractor)
![Page 42: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/42.jpg)
Summary
• A key issue in current psycholinguistics is how language processing is related to visual processing, attention and executive control.
• This can be studied in eye tracking experiments.• The results suggest straightforward eye-speech links. • However, these links are mediated by attentional
processes– visual attention governs eye movements– executive control processes govern visual attention
• We are working towards comprehensive yet detailed models of the language-attention-vision interplay.
![Page 43: The role of vision and attention in language processing Antje S. Meyer University of Birmingham and Max Planck Institute for Psycholinguistics Nijmegen](https://reader036.vdocuments.site/reader036/viewer/2022062515/56649c885503460f94940bc6/html5/thumbnails/43.jpg)
Collaborators and sponsorsCollaborators:
Eva Belke Steven Frisson Christine Häcker
Falk Huettig Glyn Humphreys Agnieskza Konopka
Debra Malpass Jane Morgan Linda Mortensen
Marc Ouellet Joost Rommers Anna Telling
Linda Wheeldon
Sponsors:
British Academy BBSRC ESRC
Nuffield Foundation