Situation Models and Embodied Language Processes

Franz Schmalhofer

University of Osnabrück / Germany

1) Memory and Situation Models

2) Computational Modeling of Inferences

3) What Memory and Language are for

4) Neural Correlates

5) Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

Acknowledgements

Charles A. Perfetti, Michal Balass, Jessica Nelson, Chin-Lung Yang &Edward Wlotko

University of Pittsburgh

funded by Alexander von Humboldt Foundation

Uwe Friese, Markus Raabe, Karin Pietruska

Ho-Ming Chow

Niki Vavatzanidis

Anke Karabanov

University of Osnabrück

Mark GreenleeRoland Rutschmann

University of Oldenburg

Outline

• Behavioral Data and Computational Modeling on Inferencing in Text Comprehension

• Event-related potentials (ERP) on the reading of inference-related words

• An fMRI-experiment on inference processes and the verification of inference related statements

Text comprehension and Inferencing

• Mary heard the ice-cream van coming.

• She remembered the pocket money.

• She rushed into the house.

A blueprint of the reader (Perfetti, 1999)

Visual Input

Mapping to phonology

Linguistic System

Meaning andForm Selection

Comprehension Processes

Situation Model

Parser

Text Representation

Infe

renc

es

Word Identification

PhonologicalUnits

OrthographicUnits

WordRepresentation

LexiconMeaningMorphologySyntax -argument structure

Writing System

Orthography

General Knowledge

Phonology Syntax Morphology

The KiWi-model (Schmalhofer, 1998)

related domain knowledge

situation model

Sensory encodingtext repres.

common sense

direct experiencetext

which emptiedthe glass.

.

which spilled thewine.

The flight attendant didnot see whathappened.

The stains were difficultto remove.

The passengers werehappy with the drink.

After turbulence was completely over, the flightattendant served each passenger a glass of red wine.

wine spilled

explicit

bridging

control

A mapping of mental and brain processes

• Beeman, Bowden & Gernsbacher (2000)

– Information supporting predictive inferences initially activated in RH

• Long & Baines (2002)

– Hemispheric differences in representations

• Tapiero & Fillon: Emotional inferences and hemispheric differences

• Mason & Just (2004)

– Inferencing by right hemisphere language network and reasoning network (dorso-lateral prefrontal cortex)

• Ferstl (2003, HC); Ferstl & von Cramon (2001)

– Shift from local to global aspects precuneus

– Inferencing fronto-median cortex (FMC)

– Situation model (dep. on content) vFMC

Predictions from the KIWi-model (explicit and paraphrase)

...e xting uishe d the b o nfire. ...p ut the b o nfire o ut.

The e xting u ishe d b o nfire ... The e xting u ishe d b o nfire ...

Exp lic it Pa ra p hra se

Situa tio na l

Pro p o sitio na l

Surfa c e

Predictions from the KIWi-model (inference and control conditions)

...p o ure d wa te r o n the b o nfire . ...b ro ug ht wo o d to the b o nfire .

The e xting u ishe d b o nfire ... The e xting u ishe d b o nfire ...

Infe re nc e C o ntro l

Surfa c e

Pro p o sitio na l

Situa tio na l

How many nodes and links had to be newly constructed

Nodes Links

Explicit 1 1

Paraphrase 1 1

Inference 2 2

Control 3 3

which emptiedthe glass.

.

which spilled thewine.

After turbulence was completely over, the flightattendant served each passenger a glass of red wine.

Text Materials in ERP experiment

ERP study of inferencing

• Use ERPs to examine inference processes.

• Vary the accessibility of referents required by the integration (new construction versus preexisting traces) referent previously introduced, possibly inferred or control condition

• Measurements taken on a single word that occurs early in the second sentence

The N400 component in ERP-studies

• N400: A negativity shift around 400 ms is associated with an incongruent meaning.

• “He smeared the bread with socks.”• A marker for semantic processing; sensitive to semantic

congruence effects in sentence contexts (Van Petten & Kutas, 1990)

• Sensitive to global comprehensibility effects in text (St. George, Mannes, & Hoffman, 1994)

General Procedure of ERP-study

Participants:• 32 adult readers

Procedure: • Slow-SVP presentation• 600 ms SOA word-to-word• 300ms exposure duration• 300ms interval• Last word of sentence followed by additional 300ms interval

Materials: • 120 two-sentence passages for each participant; 30 in each

condition; Comprehension Probes (T-F) at random after 25% of trials

Summary (ERP)

• A common set of assumptions about construction and integration processes in text comprehension is useful for accounting for behavioral data as well as neural correlates (e.g. ERP)

• Inference possibilities yield a cognitive preparation for target concept, but different from explicit and control conditions

• Distinction between text and situation representation accounts for behavioral data (priming) and ERP-data quite well

which emptiedthe glass.

and beganshaking badly.

which spilled thewine.

After turbulence was completely over, the flightattendant served each passenger a glass of red wine.

wine spilled

Text Materials of fMRI Experiment

fMRI-Lab

fMRI Lab

• 1.5 T Siemens Sonata fMRI-Scanner (Siemens Medical Solutions)

• 8 Channel Head Coil

• Lumitouch optical response device

• Visual Stimulation and recording of responses were controlled by E-Prime and PC

Behavioral Results

• Results in Milliseconds• Differences are significant except the following pairings:

– Explicit – Paraphrase– Filler – Predictive

fMRI: General Procedure

• 13 adult skilled readers

• 108 (72 + 36) trials; 18 trials per condition

• Experiment is divided into 3 Sessions à 36 trials (6 per condition)

• Pseudorandom order of trials

• Conditions and domains (themes) were counterbalanced by a Latin Square Design

• Different response delays were equalized at the end of each trial to receive a constant trial length

Time sequence of one trial

fMRI - Procedure

BOLD sensitive T2 weighted functional sequence (TR = 3s, TE = 50 ms, FOV = 192mm)

Rotated app. 10° relating to AC-PC line, covering prefrontal, parietal and temporal regions in full and the majority of the occipital cortex

3 Sessions à 324 continuously acquired scans (app. 48 Minutes)

T1 weighted structural sequence(TR= 1900 ms, TE=3.93ms, FOV = 256 mm)

Acquisition at the end of the experiment (app. 10 Minutes)

fMRI – Data Analysis

• Data were preprocessed and analyzed using The SPM2 Software Package (Functional Imaging Laboratory, Welcome Department of Imaging Neuroscience, London)

• Statistical Analysis was conducted by calculating a Random Effects Analysis

• Referring to the problem of multiple comparisons only those clusters with a corrected p < 0.05 were considered as significantly activated

fMRI – Data Analysis

• The last third of reading

– modeled as a block

• Statement Verification Task

– modeled as a block

Reading

• Sentence versus Pseudoword Reading

• Sentence Reading in Inference Condition versus Sentence Reading in Explicit Condition

Statement Verification

• Inference vs Explicit

• Inference vs Paraphrase

• Control vs Explicit

Word-Reading vs Pseudoword-Reading

Summary (fMRI-Experiment)

• fronto-median wall:– reasoning, inferencing, situation model integrations (Ferstl,

Mason & Just; Robertson & Gernsbacher)

• Precuneus, Cuneus, occipital lobe:– Imagery, perception tasks, memory related imagery

operations

• Posterior Cingulate: – Memory retrieval

• Prefrontal: Inferencing• BA9/10: theory of mind

Integration of Fields and Methods

How the brain enables the mind?

Cognitive Neuroscience

Cognitive

Science

How the mind works?

Brain Research

Components of the brain,

How does it work?

Philosophy

Linguistics

Computer Science

Psychology

Language sytems

Electronic machines

Human Behavior

Medicine - Physiology

What the course was about: Sth. Old and Std. New

• Memory and Situation Models• Computational Modeling of Inferences• What Memory and Language are for• Neural Correlates (ERP, fMRI)• Integration of Behavioral Experiments and Neural

Correlates (ERP; fMRI) by Formal Models