what does language do? “harry walked to the cafe.” “harry walked into the cafe.” a sentence...

What does language do?

“Harry walked to the cafe.” “Harry walked into the cafe.”

A sentence can evoke an imagined scene and resulting inferences:

CAFE CAFE

– Goal of action = at cafe– Source = away from cafe– cafe = point-like location

– Goal of action = inside cafe– Source = outside cafe– cafe = containing location

Language understanding

Interpretation

(Utterance, Situation)

Linguistic knowledge

Conceptual knowledge

Analysis

Language understanding: analysis & simulation

“Harry walked to the cafe.”

Schema Trajector Goalwalk Harry cafe

Cafe

Lexicon

Constructicon

General Knowledge

Belief State

Analysis Process

SemanticSpecification

Utterance

Simulation

Interpretation: x-schema simulation

Constructions can• specify which

schemas and entities are involved in an event, and how they are related

• profile particular stages of an event

• set parameters of an event

energy

walker at goal

walker=Harry goal=home

Harry is walking home.

Phonetics

Semantics

Pragmatics

Morphology

Syntax

Traditional Levels of Analysis

Phonetics

Semantics

Pragmatics

Morphology

Syntax

“Harry walked into the cafe.”

Utterance

Construction Grammar

to

block

walk

Form Meaning

A construction is a form-meaning pair whose properties may not be strictly predictable from other constructions.

(Construction Grammar, Goldberg 1995)

Source

Path

GoalTrajector

Form-meaning mappings for language

Formphonological

cuesword orderintonationinflection

Meaningevent structure

sensorimotor control

attention/perspective

social goals...

Linguistic knowledge consists of form-meaning mappings:

Cafe

Constructions as maps between relations

Mover + Motion + Directionbefore(Motion, Direction)before(Mover, Motion)

“is” + Action + “ing”before(“is”, Action)suffix(Action, “ing”)

Mover + Motionbefore(Mover, Motion)

Form Meaning

ProgressiveActionaspect(Action, ongoing)

MotionEventmover(Motion, Mover)

DirectedMotionEventdirection(Motion, Direction)mover(Motion, Mover)

Complex constructions are mappings between relations in form and relations in meaning.

Embodied Construction Grammar

• Embodied representations– active perceptual and motor schemas

– situational and discourse context

• Construction Grammar– Linguistic units relate form and meaning/function.

– Both constituency and (lexical) dependencies allowed.

• Constraint-based (Unification)– based on feature structures (as in HPSG)

– Diverse factors can flexibly interact.

schema Containerroles

interiorexteriorportalboundary

Representing image schemas

Interior

Exterior

Boundary

PortalSource

Path

GoalTrajector

These are abstractions over sensorimotor experiences.

schema Source-Path-Goalroles

sourcepathgoaltrajector

schema name

role name

Inference and Conceptual Schemas

• Hypothesis: – Linguistic input is converted into a mental simulation based on bodily-

grounded structures.

• Components:– Semantic schemas

• image schemas and executing schemas are abstractions over neurally grounded perceptual and motor representations

– Linguistic units • lexical and phrasal construction representations invoke schemas, in part

through metaphor

• Inference links these structures and provides parameters for a simulation engine

Embodied Construction GrammarECG

(Formalizing Cognitive Linguisitcs)

1. Linguistic Analysis

2. Computational Implementationa. Test Grammars

b. Applied Projects – Question Answering

3. Map to Connectionist Models, Brain

4. Models of Grammar Acquisition

ECG Structures

• Schemas– image schemas, force-dynamic schemas, executing

schemas, frames…

• Constructions– lexical, grammatical, morphological, gestural…

• Maps– metaphor, metonymy, mental space maps…

• Spaces– discourse, hypothetical, counterfactual…

ECG Schemas

schema <name> subcase of <schema> evokes <schema> as

<local name> roles < local role >: <role restriction> constraints <role> ↔ <role> <role> <value> <predicate>

schema Hypotenuse subcase of Line-Segment

evokes Right-Tri as rt

roles

{lower-left: Point}

{upper-right: Point}

constraints

self ↔ rt.long-side

Source-Path-Goal; Container

schema SPG

subcase of TrajLandmark

roles

source: Place

path: Directed–Curve

goal: Place

{trajector: Entity}

{landmark: Bounded-

Region}

schema Container

roles

interior: Bounded-Region boundary: Curve portal: Bounded-Region

Referent Descriptor Schemas

schema RD

roles

category

gender

count

specificty

resolved Ref

modifications

schema RD5 // Eve

roles

HumanSchema

Female

one

Known

Eve Sweetser

none

ECG Constructions

construction <name> subcase of <construction> constituents <name>:<construction> form constraints <name> before/meets

<name> meaning: constraints // same as for schemas

construction SpatialPP constituents prep: SpatialPreposition lm: NP form constraints prep meets lm meaning:

TrajectorLandmark constraints

selfm ↔ prep landmark ↔ lm.category

Into and The CXNsconstruction Into subcase

of SpatialPreposition

form: WordForm constraints

orth "into"

meaning: SPG

evokes Container as c constraints

landmark ↔ c

goal ↔ c.interior

construction The subcase of Determiner form:WordForm

constraints

orth "the"

meaning

evokes RD as rd

constraints rd.specificity “known”

Two Grammatical CXNsconstruction DetNoun

subcase of NP constituents

d:Determiner

n:Noun

form constraints

d before n

meaning constraints

selfm ↔ d.rd

category ↔ n

construction NPVP subcase of S constituents

subj: NP

vp: VP

form constraints

subj before vp

meaning constraints

profiled-participant ↔

subj

construction ActiveSelfMotionPath subcase of ActiveMotionPath constituents

{v: verb}

{pp:SpatialPP}

form constraints

{v before pp}

meaning:SelfMotionPathEvent

constraints {spg ↔ pp} {profiled-participant ↔ mover} {profiled-process ↔ motion} {profiled-process ↔ v}

Construction WalkedVerb

subcase of PastPerfectiveVerb form constraints orth "walked" meaning:WalkAction

Combined score determines best-fit

• Syntactic Fit:– Constituency relations– Combine with preferences on non-local elements– Conditioned on syntactic context

• Antecedent Fit:– Ability to find referents in the context– Conditioned on syntax match, feature agreement

• Semantic Fit:– Semantic bindings for frame roles– Frame roles’ fillers are scored

0Eve1walked2into3the4house5

Constructs--------------NPVP[0] (0,5)Eve[3] (0,1)ActiveSelfMotionPath

[2] (1,5)WalkedVerb[57] (1,2)SpatialPP[56] (2,5)Into[174] (2,3)DetNoun[173] (3,5)The[204] (3,4)House[205] (4,5)

Schema Instances

-------------------

SelfMotionPathEvent[1]

HouseSchema[66]

WalkAction[60]

Person[4]

SPG[58]

RD[177] ~ house

RD[5]~ Eve

Unification chains and their fillers

SelfMotionPathEvent[1].mover

SPG[58].trajector

WalkAction[60].walker

RD[5].resolved-ref

RD[5].category

Filler: Person4

SpatialPP[56].m

Into[174].m

SelfMotionPathEvent[1].spg

Filler: SPG58


.landmark

House[205].m

RD[177].category

SPG[58].landmark

Filler:HouseSchema66

WalkedVerb[57].m

WalkAction[60].routine

WalkAction[60].gait


.motion

Filler:WalkAction60

Summary: ECG

• Linguistic constructions are tied to a model of simulated action and perception

• Embedded in a theory of language processing– Constrains theory to be usable– Frees structures to be just structures, used in processing

• Precise, computationally usable formalism– Practical computational applications, like MT and NLU– Testing of functionality, e.g. language learning

• A shared theory and formalism for different cognitive mechanisms– Constructions, metaphor, mental spaces, etc.

Embodied Compositional Semantics

after

Ellen [email protected]

Questions

• What is the nature of compositionality in the Neural Theory of Language?

• How can it be best represented using Embodied Construction Grammar?

Examples

• He bit the apple

• He was bitten (by a toddler)

• He bit into the apple

• His white teeth bit into the apple.

• He shattered the window

• The window was shattered

• The window shattered

Compositionality

• Put the parts together to create the meaning of the whole.

• Questions:– what is the nature of the parts?– How and why do they combine with one another?– What meaning is associated with this composition?

Short answers

• Parts = constructions, schemas

• Combination = binding, unification

• Meaning of the whole = simulation of unified parts

Simulation parameters

• Constructions unify to create semantic specification that supports a simulation

• Two types of simulation parameters for event descriptions:– Event content– Event construal

Summary

• Parts = constructions, schemas

• Combination = binding, unification

• Meaning of the whole = simulation of the combined parts

First example

• He bit the apple.

schema MotorControl subcase of Process roles Actor ↔ Protagonist

Effector Effort

Routine constraints Actor ← animate

Schemas

schema ForceApplication subcase of MotorControl evokes ForceTransfer as FT roles

Actor ↔ FT.Supplier ↔ Protagonist Acted Upon↔ FT.Recipient Effector

Routine Effort ↔ FT.Force.amount

schema ForceTransfer evokes Conact as C roles

Supplier ↔ C.entity1 Recipient ↔ C.entity2 Force

schema MotorControl subcase of Process roles Actor ↔ Protagonist

Effector Effort

Routine constraints Actor ← animate

schema Contact subcase of SpatialRelation roles Entity1: entity Entity2: entity

Schema networks

MotorControl

Motion

SPG

EffectorMotion

EffectorMotionPath

ForceTransfer

ForceApplication

ContactSpatiallyDirectedAction

CauseEffect

Contact

Agentive Impact

SelfMotion

SelfMotionPath

MotionPath

Construction BITE1 subcase of Verb form: bite meaning: ForceApplication constraints: Effector ← teeth Routine ← bite // close mouth

Verb Constructions

schema ForceApplication subcase of MotorControl evokes ForceTransfer as FT roles Actor ↔ FT.Supplier ↔ Protagonist Acted Upon ↔ FT.Recipient Effector Routine Effort ↔ FT.Force.amount

Verb Constructions

schema ForceApplication subcase of MotorControl

schema Agentive Impact subcase of ForceApplication

cxn BITE meaning: ForceApplication

schema MotorControl

cxn GRASP meaning: ForceApplicationcxn PUSH meaning: ForceApplicationcxn SLAP meaning: AgentiveImpactcxn KICK meaning: AgentiveImpactcxn HIT meaning: AgentiveImpact

Argument Structure Construction

construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF

meaning: CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType}

{Vm ↔ ED.ProfiledProcess}

Causer ↔ ED.ProfiledParticipant

FA ↔ Vm

Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm

CauseEffect schema

schema CauseEffect subcase of ForceApplication; Process roles

Causer ↔ Actor Affected ↔ ActedUpon ↔ Process.Protagonist Instrument ↔ Effector

MotorControl

Motion

SPG

EffectorMotion

EffectorMotionPath

ForceTransfer

ForceApplication


CauseEffect

Contact

SelfMotion

SelfMotionPath

MotionPath

Agentive Impact

Process

Schema Network



meaning: CauseEffect evokes: EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType}



FA ↔ Vm


MotorControl

Motion

SPG

EffectorMotion

EffectorMotionPath

ForceTransfer

ForceApplication


CauseEffect

Contact

SelfMotion

SelfMotionPath

MotionPath

Agentive Impact

Process

Schema Network

Important points

Compositionality does not require that each component contain different information.

Shared semantic structure is not viewed as an undesirable redundancy





Causer ↔ ED.ProfiledParticipant FA ↔ Vm


schema EventDescriptor

roles

EventType: Process

ProfiledProcess: Process

ProfiledParticipant: Entity

ProfiledState(s): State

SpatialSetting

TemporalSetting

Event Descriptor schema


Construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF



Causer ↔ ED.ProfiledParticipant FA ↔ Vm


construction NPVP1 constituents: Subj: NP VP : VPform Constraints Subj f before VPf

meaning: EventDescriptor ProfiledParticipant ↔ Subjm

Bindings with other cxnsconstruction ActiveTransitiveAction2 subcase of VP constituents: V ; NP form: VF before NPF

meaning: CauseEffect evokes; EventDescriptor as ED constraints: {Selfm ↔ ED.EventType}



Affected ↔ NPm

Construction NPVP1 constituents: Subj: NP VP : VPform constraints Subj f before VPf


Bindings with other cxnsconstruction ActiveTransitiveAction2 subcase of VP constituents: V ; NP form: VF before NPF




Affected ↔ NPm

schema EventDescriptor roles EventType ProfiledProcess ProfiledParticipant ProfiledState(s) SpatialSetting TemporalSetting

Bindings with other cxns

schema EventDescriptor roles EventType ProfiledProcess ProfiledParticipant ProfiledState(s) SpatialSetting TemporalSetting

construction NPVP1 constituents: Subj: NP VP : VPform Constraints Subj f before VPf


construction ActiveTransitiveAction2 subcase of VP constituents: V ; NP form: VF before NPF




Affected ↔ NPm

Unification

CauseEffect causer affected

ForceApplication actor actedupon

EventDescriptor EventType ProfiledProcess ProfiledParticipant

BITE

TransitiveAction2

HE

NP1

NPVP1

THE APPLE

NP2ReferentDescriptor

ReferentDescriptor

Meaning Constructions

Unification



EventDescriptor EventType ProfiledProcess ProfiledParticipant

BITE

TransitiveAction2

HE

NP1

NPVP1

THE APPLE


ReferentDescriptor resolved referent


Unification



EventDescriptor eventtype ProfiledProcess ProfiledParticipant

BITE

TransitiveAction2 Verb

HE

NP1

NPVP1

THE APPLE


ReferentDescriptor resolved referent


Unification




BITE

TransitiveAction2

HE

NP1

NPVP1 subj

THE APPLE


ReferentDescriptor


Unification




BITE

TransitiveAction2 NP

HE

NP1

NPVP1

THE APPLE


ReferentDescriptor


Semantic SpecificationHe bit the apple



ForceApplication actor actedupon routine bite effector teeth

RD55category

Person

Apple

RD27category

Process

Simulation - He bit the apple

CauseEffect

ForceApplication

Protagonist = Causer ↔ Actor

Protagonist = Affected ↔ ActedUpon

Passive voice

He was bitten (by a toddler)

Argument Structure ConstructionHe was bitten (by a toddler)

construction PassiveTransitiveAction2 subcase of VP constituents: V : PassiveVerb (PP: agentivePP) form constraints: VF before PPF

meaning: CauseEffectAction evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType}


Affected ↔ ED.ProfiledParticipant FA ↔ Vm

Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Causer ↔ PP.NPm

Semantic SpecificationHe was bitten (by a toddler)



ForceApplication actor actedupon routine bite effector teeth

RD48category

Person

Person

RD27category

Effect = Process

Simulation - He was bitten (by a toddler)

CauseEffect

Action = BiteProtagonist = Causer ↔ Actor

Protagonist = Affected ↔ ActedUpon

Variations on a theme

• He shattered the window

• The window was shattered

• The window shattered

Construction SHATTER1 subcase of Verb form: shatter meaning: StateChange constraints: Initial :: Undergoer.state ← whole Final :: Undergoer.state ← shards

Verb Construction -- shatter

schema StateChange subcase of Process roles Undergoer ↔ Protagonist

Argument Structure ConstructionHe shattered the window


meaning: CauseEffect evokes: EventDescriptor as ED; StateChange as SC constraints: {Selfm ↔ ED.EventType}



SC ↔ Vm

Affected ↔ SC.Undergoer Affected ↔ NPm

Semantic SpecificationHe shattered the window



StateChange Undergoer state “wholeness”

RD189category

Person

Window

RD27category

Process

Simulation - He shattered the window

CauseEffect

Action Protagonist = Causer

Protagonist = Affected ↔ Undergoer

Argument Structure ConstructionThe window was shattered

construction PassiveTransitiveAction3 subcase of VP constituents: V : PassiveVerb (PP: agentivePP) form constraints: VF before NPF

meaning: CauseEffect evokes: EventDescriptor as ED; StateChange as SC constraints: {Selfm ↔ ED.EventType}


Affected ↔ ED.ProfiledParticipant

SC ↔ Vm

Affected ↔ SC.Undergoer Causer ↔ PP.NPm

Semantic SpecificationThe window was shattered




RD175category

Window

Process

Simulation - The window was shattered

CauseEffect

Action Protagonist = Causer

Protagonist = Affected ↔ Undergoer

Argument Structure ConstructionThe window shattered

construction ActiveIntransitiveAction1 subcase of VP constituents: V : verb form meaning: Process evokes: EventDescriptor as ED; StateChange as SC constraints: {Selfm ↔ ED.EventType}


Protagonist ↔ ED.ProfiledParticipant

SC ↔ Vm

Protagonist ↔ SC.Undergoer

Semantic SpecificationThe window shattered


Process protagonist


RD177categoryWindow

Process

Simulation - The window shattered

Process

Protagonist = Undergoer

Some more variations on a theme

• He bit the apple

• He bit into the apple

• His white teeth bit into the apple.

Argument Structure ConstructionHe bit into the apple

construction ActiveEffectorMotionPath2 subcase of VP constituents: V : verb PP: Spatial-PP form constraints:

VF before PPF

meaning: EffectorMotionPath evokes; EventDescriptor as ED; ForceApplication as FA constraints:

{Selfm ↔ ED.EventType}


Actor ↔ ED.ProfiledParticipant FA ↔ Vm

Actor ↔ FA.ActorEffector ↔ FA.Effector // INITarget ↔ FA.ActedUpon SPG ↔ PPm

Target ↔ PPm .Prep.LM

Schema

schema EffectorMotionPath subcase of EffectorMotion subcase of SPG // or evokes SPG roles Actor ↔ MotorControl.protagonist

Effector ↔ SPG.Tr ↔ M.Mover ↔ Motion.protagonistTarget ↔ SPG.Lm

MotorControl

Motion

SPG

EffectorMotion

EffectorMotionPath

ForceTransfer

ForceApplication


CauseEffect

Contact

SelfMotion

SelfMotionPath

MotionPath

Agentive Impact

Process

Schema Network

Argument Structure Construction He bit into the apple


VF before PPF

meaning: EffectorMotionPath evokes: EventDescriptor as ED; ForceApplication as FA constraints:






EffectorMotionPath

Action

Source

Path Goal

Effector Motion

Protagonist = Actor

Protagonist = Effector

Argument Structure Construction He bit into the apple


VF before PPF







Simulation: He bit into the apple

Action

Source

Path Goal

Effector Motion

Protagonist = Actor


Argument Structure ConstructionHis white teeth bit into the apple


VF before PPF




Effector ↔ ED.ProfiledParticipant FA ↔ Vm

Actor ↔ FA.Actor // INIEffector ↔ FA.EffectorTarget ↔ FA.ActedUpon SPG ↔ PPm


Simulation: His white teeth bit into the apple

Action

Source

Path Goal

Effector Motion

Protagonist = Actor


Non-agentive biting

• He landed on his feet, hitting the narrow pavement outside the yard with such jarring impact that his teeth bit into the edge of his tongue. [BNC]

• The studs bit into Trent's hand. [BNC]

• His chest burned savagely as the ropes bit into his skin. [BNC]

MotorControl

Motion

SPG

EffectorMotion

EffectorMotionPath

ForceTransfer

ForceApplication


CauseEffect

Contact

SelfMotion

SelfMotionPath

MotionPath

Agentive Impact

Process

Schema Network

Simulation: His teeth bit his tongue

Source

Path Goal

MotionProtagonist = Mover

Summary

• Small set of constructions and schemas

• Composed in different ways

• Unification produces specification of parameters of simulation

• Sentence understanding is simulation

• Different meanings = different simulations

Concluding Remarks

• Complexity

• Simulation

Concluding Remarks

• Complexity• Simulation• Language understanding is simulation• Simulation involves activation of

conceptual structures• Simulation specifications should include:

– which conceptual structures to activate– how these structures should be activated

Extra slides follow:

Prototypes and extensions?

CauseMotion Path:

• He threw the ball across the room

• He kicked the ball over the table

• He sneezed the napkin off the table

• [He coughed the water out of his lungs]

Key points

• In prototypical verb-argument structure construction combinations, verb meaning is very similar to argument structure meaning.

• Verbs whose meaning partially overlaps that of a given argument structure constructions may also co-occur with that argument structure construction

• These less prototypical combinations may motivate extensions to the central argument structure constructions

what does language do? “harry walked to the cafe.” “harry walked into the cafe.” a sentence...

Documents

semantic schemas image

forcedynamic schemas

direction movermotion

action suffixaction

locationgoal of action

direction beforemover

bodilygrounded structures

movercomplex constructions