Anaphoric dependencies: A window into the architecture of the language system Eye tracking experiments

Eric ReulandEric ReulandFrank WijnenFrank Wijnen

Arnout KoornneefArnout Koornneef

EYE-TRACKING

BY

ARNOUT KOORNNEEF

OVERVIEW OF LECTURE

• Part 1: discussion of method

- eye-tracking while reading

• Part 2: discussion of current research

- eye-tracking experiments

THE EYE-TRACKER

THE EYE-TRACKER

THE EYE-TRACKER

• monitors eye-movements from millisecond to millisecond

• provides information about where people look and for how long

TWO MAIN EYE-TRACKING PARADIGMS

• eye-tracking while reading

• eye-tracking while listening

(not discussed)

EYE-TRACKING WHILE READING

SOME FACTS ABOUT READING

• people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next

• a jump (or saccade) covers 7-9 letter spaces• during a saccade visual input is reduced• readers skip short words and words that are highly

predictable (these words are identified in the parafoveal region)

• readers regress (look back)• readers often undershoot on return sweeps (going from

the end of a line to the next line)• the perceptual span is asymmetrical to the right (to the

left for languages like Hebrew)

SOME FACTS ABOUT READING

• people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next

• a jump (or saccade) covers 7-9 letter spaces• during a saccade visual input is reduced• readers skip short words and words that are highly predictable

(these words are identified in the parafoveal region)• readers regress (look back)• readers often undershoot on return sweeps (going from the end of a

line to the next line)• the perceptual span is asymmetrical to the right (to the left for

languages like Hebrew)

SOME FACTS ABOUT READING

• people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next

• a jump (or saccade) covers 7-9 letter spaces• during a saccade visual input is reduced• readers skip short words and words that are highly predictable

(these words are identified in the parafoveal region)• readers regress (look back)• readers often undershoot on return sweeps (going from the end of a

line to the next line)• the perceptual span is asymmetrical to the right (to the left for

languages like Hebrew)

SOME FACTS ABOUT READING

• people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next

• a jump (or saccade) covers 7-9 letter spaces• during a saccade visual input is reduced• readers skip short words and words that are highly predictable

(these words are identified in the parafoveal region)• readers regress (look back)• readers often undershoot on return sweeps (going from the end of a

line to the next line)• the perceptual span is asymmetrical to the right (to the left for

languages like Hebrew)

SOME FACTS ABOUT READING

• people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next

• a jump (or saccade) covers 7-9 letter spaces• during a saccade visual input is reduced• readers skip short words and words that are highly predictable

(these words are identified in the parafoveal region)• readers regress (look back)• readers often undershoot on return sweeps (going from the end of a

line to the next line)• the perceptual span is asymmetrical to the right (to the left for

languages like Hebrew)

SOME FACTS ABOUT READING

• people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next

• a jump (or saccade) covers 7-9 letter spaces• during a saccade visual input is reduced• readers skip short words and words that are highly predictable

(these words are identified in the parafoveal region)• readers regress (look back)• readers often undershoot on return sweeps (going from the end of a

line to the next line)• the perceptual span is asymmetrical to the right (to the left for

languages like Hebrew)

SOME FACTS ABOUT READING

• people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next

• a jump (or saccade) covers 7-9 letter spaces• during a saccade visual input is reduced• readers skip short words and words that are highly predictable

(these words are identified in the parafoveal region)• readers regress (look back)• readers often undershoot on return sweeps (going from the end of a

line to the next line)• the perceptual span is asymmetrical to the right (to the left for

languages like Hebrew)

SOME FACTS ABOUT READING

• people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next

• a jump (or saccade) covers 7-9 letter spaces• during a saccade visual input is reduced• readers skip short words and words that are highly predictable

(these words are identified in the parafoveal region)• readers regress (look back)• readers often undershoot on return sweeps (going from the end of a

line to the next line)• the perceptual span is asymmetrical to the right (to the left for

languages like Hebrew)

A TYPICAL READING EXPERIMENT

Garden-path sentence

Since Jay always jogs a mile seems like a short distance to him.

Control sentence

Since Jay always jogs a mile this seems like a short distance to him.

READING PATTERN (Garden-path sentence)

Since Jay always jogs a mile seems

like a short distance to him.

READING PATTERN (Garden-path sentence)

Since Jay always jogs a mile

like a short distance to him.

= fixation after progressive saccade (first-pass)

= fixation after regressive saccade

= fixation after progressive saccade (second-pass)

seems

READING PATTERN

• If readers experience some sort of trouble they may fixate the difficult region longer and the may even regress to earlier parts of the sentence/text.

HOW DO WE INTERPRET THE

READING PATTERNS?

DIFFERENT MEASURES

• First fixation duration: duration of first fixation in a region

• First-pass duration: time spent in a region before moving on or looking back

• Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time

• Second-pass duration: duration of re-fixations• Total duration: the sum of all fixations in a region• Probability of a regression: the percentage of

regressive eye-movements out of a region

DIFFERENT MEASURES

• First fixation duration: duration of first fixation in a region

• First-pass duration: time spent in a region before moving on or looking back

• Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time

• Second-pass duration: duration of re-fixations• Total duration: the sum of all fixations in a region• Probability of a regression: the percentage of

regressive eye-movements out of a region

DIFFERENT MEASURES

• First fixation duration: duration of first fixation in a region

• First-pass duration: time spent in a region before moving on or looking back

• Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time

• Second-pass duration: duration of re-fixations• Total duration: the sum of all fixations in a region• Probability of a regression: the percentage of

regressive eye-movements out of a region

DIFFERENT MEASURES

• First fixation duration: duration of first fixation in a region

• First-pass duration: time spent in a region before moving on or looking back

• Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time

• Second-pass duration: duration of re-fixations• Total duration: the sum of all fixations in a region• Probability of a regression: the percentage of

regressive eye-movements out of a region

DIFFERENT MEASURES

• First fixation duration: duration of first fixation in a region

• First-pass duration: time spent in a region before moving on or looking back

• Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time

• Second-pass duration: duration of re-fixations• Total duration: the sum of all fixations in a region• Probability of a regression: the percentage of

regressive eye-movements out of a region

DIFFERENT MEASURES

• First fixation duration: duration of first fixation in a region

• First-pass duration: time spent in a region before moving on or looking back

• Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time

• Second-pass duration: duration of re-fixations• Total duration: the sum of all fixations in a region• Probability of a regression: the percentage of

regressive eye-movements out of a region

DIFFERENT MEASURES

• First fixation duration: duration of first fixation in a region

• First-pass duration: time spent in a region before moving on or looking back

• Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time

• Second-pass duration: duration of re-fixations• Total duration: the sum of all fixations in a region• Probability of a regression: the percentage of

regressive eye-movements out of a region

EXPLANATION OF DIFFERENT MEASURES

Bart annoyed Homer because…1 2 3 45 7

Reading Times for word 3 (Homer)

First fixation duration = 3

First-pass duration = 3 + 4

Regression Path duration = 3 + 4 + 5

Second-pass duration = 6

Total duration = 3 + 4 + 6

6

DIFFERENT FIRST-PASS MEASURES

FIRST FIXATION DURATION

FIRST-PASS DURATION

REGRESSION PATH DURATION

TIME

HOW DO WE INTERPRET THE

READING TIMES?

THE LINKING PROBLEM

eye mind

EYE-MIND ASSUMPTION(JUST & CARPENTER, 1980)

• Readers retain fixation on a word until processing is completed

• This includes processes like word recognition, syntactic parsing, semantic integration, referential integration

AN IDEAL WORLD(VAN BERKUM, 2004)

Snowwhite kissed a dwarf

W P S R

W P S R

W P S R

W P S R

TIME

W = word recognition; P = parsing; S = semantic integration; R = referential integration

THE REAL WORLDIS A REAL MESS

(VAN BERKUM, 2004)

Snowwhite kissed a dwarf

W P S R

W P S R

W P S R

W P S R

TIME

W = word recognition; P = parsing; S = semantic integration; R = referential integration

THE REAL WORLDIS A REAL MESS

• In the real world the processing of word X continues while fixating word X + 1 (and possibly while fixating word X + 2 etc.)

• Thus, the effects of a manipulation are often visible more downstream (i.e. after the critical word or region). This is called spill-over.

THE REAL WORLDIS A REAL MESS

• In the real world the processing of word X continues while fixating word X + 1 (and possibly while fixating word X + 2 etc.)

• Thus, the effects of a manipulation are often visible more downstream (i.e. after the critical word or region). This is called spill-over.

LINKING ASSUMPTION(BOLAND, 2004)

• The eyes do not leave a word until it has been structurally integrated (tree building). Therefore, constraints that control structure-building influence first-pass reading time.

other measures (e.g., regression path duration) are sensitive for higher level processes

(semantic integration, discourse processes)

SOLUTION LINKING PROBLEM?

• Perhaps the different measures can provide information about what is happening? (this is an empirical question)

A DISADVANTAGE OF THE READING PARADIGM

• You can only use skilled readers (no children or language-disordered populations).

• This is possible in spoken language paradigms.

CAVEAT:

• the eyes tell us that something is happening at a specific point in time, but not what that something is!

TO CONCLUDE

• Eye-tracking (while reading and listening) excels in the “when” question.

• Not really suited for “what” question (use EEG/MEG instead).

QUESTIONS?

PART 2: THEORY

CONSTRUCTING ANAPHORIC DEPENDENCIES:

VARIABLE BINDING vs. CO-REFERENCE

BASIC ARCHITECTURE PROCESSING SYSTEM

• distinct modules syntax, semantics, discourse

• economy principle: cross-modular operations carry a cost (Reuland 2001)

• syntax first in time-course-model of processing phases (Friederici 1995-)

syntax is cheaper than semantics semantics is cheaper than discourse

CROSS-MODULAR OPERATIONS

Discourse storage (values) a a

C-I objects (variables) x1 x2

Syntactic objects (chains) C1 C2

Basic expressions α β

Discourse storage (values) a

C-I objects (variables) x1 x1

Syntactic objects (chains) C1 C2

Basic expressions α β

Discourse storage (values) a

C-I objects (variables) x1

Syntactic objects (chains) C1 C1

Basic expressions α β

TIME-COURSE SIMPLIFIED

SYNTAX

SEMANTICS

DISCOURSE

SEMANTICS VS. DISCOURSE

• variable binding (semantic dependency)

Every clown thinks that he is not funny.

• co-reference (discourse dependency)

The clown has a big problem.

He is not funny.

PREDICTIONS OF THEORY

• variable binding is cheaper/faster than co-reference

• in an ambiguous situation variable binding has precedence over co-reference

END OF INTRO