probabilistic context free grammars · 2018. 11. 28. · slides courtesy rebecca knowles. outline...

Probabilistic Context Free Grammars

CMSC 473/673

UMBC

Outline

Recap: MT word alignment

Structure in Language: Constituency

(Probabilistic) Context Free GrammarsDefinitionsHigh-level tasks: Generating and ParsingSome uses for PCFGs

CKY Algorithm: Parsing with a (P)CFG

Machine Translation as aNoisy Channel Model

Decode Rerank

written in (clean) English

observed Russian (noisy)

text

translation/decode model

(clean) language model

English

language

язы́к

speak

text

word

language

speak

text

word

language

Slides courtesy Rebecca Knowles

?

Idea: Learn Word-to-Word Translation via Word Alignment

The cat is on the chair.

Le chat est sur la chaise.

The cat is on the chair.

Le chat est sur la chaise.


Assumption: Parallel TextsWhereas recognition of the inherent dignity and of the equal and inalienable rights of all members of the human family is the foundation of freedom, justice and peace in the world,

Whereas disregard and contempt for human rights have resulted in barbarous acts which have outraged the conscience of mankind, and the advent of a world in which human beings shall enjoy freedom of speech and belief and freedom from fear and want has been proclaimed as the highest aspiration of the common people,

Whereas it is essential, if man is not to be compelled to have recourse, as a last resort, to rebellion against tyranny and oppression, that human rights should be protected by the rule of law,

Whereas it is essential to promote the development of friendly relations between nations,…http://www.un.org/en/universal-declaration-human-rights/

Yolki, pampa ni tlatepanitalotl, ni tlasenkauajkayotl iuan ni kualinemilistli ipan ni tlalpan, yaya ni moneki moixmatis uan monemilis, ijkinoj nochi kuali tiitstosej ika touampoyouaj.

Pampa tlaj amo tikixmatij tlatepanitalistli uan tlen kuali nemilistli ipanni tlalpan, yeka onkatok kualantli, onkatok tlateuilistli, onkatokmajmajtli uan sekinok tlamantli teixpanolistli; yeka moneki ma kualitimouikakaj ika nochi touampoyouaj, ma amo onkaj majmajyotl uanteixpanolistli; moneki ma onkaj yejyektlalistli, ma titlajtlajtokaj uan ma tijneltokakaj tlen tojuantij tijnekij tijneltokasej uan amo tlen ma topanti, kenke, pampa tijnekij ma onkaj tlatepanitalistli.

Pampa ni tlatepanitalotl moneki ma tiyejyekokaj, ma tijchiuakaj uanma tijmanauikaj; ma nojkia kiixmatikaj tekiuajtinij, uejueyij tekiuajtinij, ijkinoj amo onkas nopeka se akajya touampoj san tlen ueli kinekistechchiuilis, technauatis, kinekis technauatis ma tijchiuakaj se tlamantli tlen amo kuali; yeka ni tlatepanitalotl tlauel moneki ipantonemilis ni tlalpan.

Pampa nojkia tlauel moneki ma kuali timouikakaj, ma tielikaj keuaktiiknimej, nochi tlen tlakamej uan siuamej tlen tiitstokej ni tlalpan.…

http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=nhn


• Bitext/parallel texts: sentences with their (human provided) translations• Sentences are aligned, words are not• Commonly used bitext: Europarl (http://www.statmt.org/europarl/)

http://www.un.org/en/universal-declaration-human-rights/

http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=nhn

http://www.statmt.org/europarl/

Alignments

If we had word-aligned text, we could easily estimate P(f|e).

But we don’t usually have word alignments, and they are expensive to produce by hand…

If we had P(f|e) we could produce alignments automatically.


Join

t m

od

el

un

ob

serv

edIBM Model 1

(1993)

f: vector of French words

(visualization of alignment)

e: vector of English words

a: vector of alignment indices

Le chat est sur la chaise verte

The cat is on the green chair

0 1 2 3 4 6 5


Lexical Translation ModelWord Alignment ModelFor all IBM models, see the original paper (Brown et al,

1993): http://www.aclweb.org/anthology/J93-2003

ob

serv

ed

t(fj|ei) : translation probability of the word fj given the word ei

http://www.aclweb.org/anthology/J93-2003

Expectation Maximization (EM)0. Assume some value for and compute other parameter values

Two step, iterative algorithm

1. E-step: count alignments and translations under uncertainty, assuming these parameters

2. M-step: maximize log-likelihood (update parameters), using uncertain counts

estimated counts

P( | “the cat”)

P( | “the cat”)le chat

le chat


Outline





Parts of Speech

Adapted from Luke Zettlemoyer

Closed class words

Open class words

Nouns

milk cat

cats

UMBC

Baltimorebread

speak

give

can

do

may

Verbs

Adjectives

would-be

wettestlargehappy

red

fake

Kamp & Partee (1995)

Adverbs

recentlyhappily

thenthere (location)

intransitive

run

ditransitive

transitive

subsective

non-subsective

modals, auxiliaries

Numbers

I you one

1,324

Determiners

PrepositionsConjunctions

Pronouns

and or if

athe

every

what

inunder

topParticles

(set) up

so (far)

not

(call) off

becausebecause

Constituency

spans of words that act (syntactically) as a group

“X phrase” (noun phrase)

Constituency



Baltimore is a great place to be.

This house is a great place to be.

This red house is a great place to be.

This red house on the hill is a great place to be.

noun phrase (NP)

Constituency






This red house on the hill is a great place to be. noun phrase (NP) noun phrase (NP)

Constituency







Is this house a great place to be?

noun phrase (NP) noun phrase (NP)

Constituency







*This is house a great place to be.

noun phrase (NP) noun phrase (NP)

Constituents Help Form Grammars

constituent: spans of words that act (syntactically) as a group


Baltimore is a great place to be.This house is a great place to be.

This red house is a great place to be.This red house on the hill is a great place to be.

S NP V NP






S NP V NPNP Det Noun

NP Noun







NP NounNP Det Adj Noun






The hill is a great place to be.










NP NounNP Det Adj NounNP NP Prep NP


constituent: spans of words that act (syntactically) as a group“X phrase” (noun phrase)



This red house near the hill is a great place to be.This red house atop the hill is a great place to be.




NP NP PPPP P NP








NP NounNP Det AdjP

NP NP PPPP P NP

AdjP Adj Noun







S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PPPP P NP

AdjP Adj NounVP V NP







S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP

PP P NPAdjP Adj Noun

VP V NPNoun Baltimore

Outline





Context Free Grammar

Set of rewrite rules, comprised of terminals and non-terminals

Terminals: the words in the language (the lexicon), e.g., Baltimore

Non-terminals: symbols that can trigger rewrite rules, e.g., S, NP, Noun

(Sometimes) Pre-terminals: symbols that can only trigger lexical rewrites, e.g., Noun

S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP




Set of rewrite rules, comprised of terminals and non-terminals




Applications: Learn more

in CMSC 331, 431

Theory: Learn

more in CMSC 451

S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



How Do We Robustly Handle Ambiguities?

How Do We Robustly Handle Ambiguities?

Add probabilities (to what?)

Probabilistic Context Free Grammar

Set of weighted (probabilistic) rewrite rules, comprised of terminals and non-terminals




S NP VPNP Det NounNP NounNP Det AdjPNP NP PP

PP P NPAdjP Adj NounVP V NPNoun Baltimore…






Q: What are the distributions? What must sum to 1?

S NP VPNP Det NounNP NounNP Det AdjPNP NP PP

PP P NPAdjP Adj NounVP V NPNoun Baltimore…






1.0 S NP VP.4 NP Det Noun.3 NP Noun.2 NP Det AdjP.1 NP NP PP

1.0 PP P NP.34 AdjP Adj Noun.26 VP V NP.0003 Noun Baltimore…

Q: What are the distributions? What must sum to 1?

A: P(X Y Z | X)


p(S

NP VP

Noun

Baltimore

Verb NP

is a great city

)= product of probabilities of individual rules used in the

derivation


p(S

NP VP

Noun

Baltimore

Verb NP

is a great city

)=

p(S

NP VP

) *

product of probabilities of individual rules used in the

derivation


p(S

NP VP

Noun

Baltimore

Verb NP

is a great city

)=

p(S

NP VP

) *

p( ) *NP

Noun

p( ) *Noun

Baltimore


derivation


p(S

NP VP

Noun

Baltimore

Verb NP

is a great city

)=

p(S

NP VP

) *

p( ) *

p( ) *

NP

Noun

p( ) *Noun

Baltimore

VP

Verb NP

p( ) *Verb

is

p( )NP

a great city


derivation

Log Probabilistic Context Free Grammar

lp(S

NP VP

Noun

Baltimore

Verb NP

is a great city

)=

lp(S

NP VP

) +

lp( ) +

lp( ) +

NP

Noun

lp( ) +Noun

Baltimore

VP

Verb NP

lp( ) +Verb

is

lp( )NP

a great city

sum of log probabilities of individual rules used in the

derivation

Estimating PCFGs

Attempt 1:

• Get access to a treebank (corpus of syntactically annotated sentences), e.g., the English Penn Treebank

• Count productions

• Smooth these counts

• This gets ~75 F1

Probabilistic Context Free Grammar (PCFG) Tasks

Find the most likely parse (for an observed sequence)

Calculate the (log) likelihood of an observed sequence w1, …, wN

Learn the grammar parameters

Outline






Generate: Iteratively create a string (or tree 1.derivation) using the rewrite rules

Parse: Assign a tree (if possible) to an input string2.

Generate from a Context Free Grammar

S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



S

S


S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



NP VP

S

NP VP


S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



Noun VP

S

NP VP

Noun


S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



Baltimore VP

S

NP VP

Noun

Baltimore


S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



Baltimore V NP

S

NP VP

Noun

Baltimore

Verb NP


S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



…

Baltimore is a great city

S

NP VP

Noun

Baltimore

Verb NP

is a great city

Assign Structure (Parse) with aContext Free Grammar

S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



…

Baltimore is a great city

S

NP VP

Noun

Baltimore

Verb NP

is a great city


S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



…

[S [NP [Noun Baltimore] ] [VP [Verb is] [NP a great city]]]

S

NP VP

Noun

Baltimore

Verb NP

is a great city

bracket notation


S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



…

(S (NP (Noun Baltimore))(VP (V is)

(NP a great city)))

S

NP VP

Noun

Baltimore

Verb NP

is a great city

S-expression

Some CFG Terminology:Derivation/Parse Tree

S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP


VP V `PNoun Baltimore

…

S

NP VP

Noun

Baltimore

Verb NP

is a great city

derivation,parse tree

Some CFG Terminology:Start Symbol

S NP VPNP Det Noun

NP NounNP Det AdjP

NP NP PP



…

S

NP VP

Noun

Baltimore

Verb NP

is a great city

start symbol

Some CFG Terminology:Chomsky Normal Form (CNF)

non-terminal non-terminal non-terminal

non-terminal terminal

X Y Z

X a

binary rules can only involve non-terminals

unary rules can only involve terminals

Restricted binary and unary rules only

No ternary rules (or above)

Outline





What are some benefits to CFGs?

Why should you care about syntax?

Some Uses of CFGs

Clearly disambiguate certain ambiguities

Morphological derivations

Identify “grammatical” sentences

…

Clearly Show Ambiguity

I ate the meal with friends


I ate the meal with friendssalt



NP VP

VP NP PP

S



NP VP

VP NP PP

S

NP VP

S

VP NP

PPNP



NP VP

VP NP PP

S

NP VP

S

VP NP

PPNP

PP Attachment(a common source of errors,

even still today)

Clearly Show Ambiguity…But Not Necessarily All Ambiguity


NP VP

VP NP PP

S

I ate the meal with gusto

I ate the meal with a fork

Other Attachment Ambiguity

We invited the students, Chris and Pat.

Coordination Ambiguity

old men womenand

Grammars Aren’t Just for Syntax

overgeneralization

general -izeA AV

generalizeV

-tionVN

generalizationN

over-NN

overgeneralizationN

Clearly Show Grammaticality (?)

The old man the boats

S

NP VP



S

NP VP

S

NP NP



S

NP VP

S

NP NP

Idea: define grammatical sentences as those that can be parsed by a grammar



S

NP VP

S

NP NP


Issue 1: Which grammar?



S

NP VP

S

NP NP


Issue 1: Which grammar?

Issue 2: Discourse demands flexibility

Q: What do you see?

A: [I see] The old man [and] the boats.

Outline





Parsing with a CFG

Top-down backtracking (brute force)

CKY Algorithm: dynamic bottom-up

Earley’s Algorithm: dynamic top-down

not covered due to time

CKY Precondition

Grammar must be in Chomsky Normal Form (CNF)

non-terminal non-terminal non-terminal

non-terminal terminal

S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det aExample from Jason Eisner

Entire grammarAssume uniform weights

“Papa ate the caviar with a spoon”

S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7

Example from Jason Eisner



S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7



Goal:

(S, 0, 7)


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7



Check 1: What are the non-terminals?


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7




SNPVPPP

NVPDet

Check 2: What are the terminals?


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7




SNPVPPP

NVPDet


Papacaviarspoonate

withthea

Check 3: What are the pre-terminals?


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7




SNPVPPP

NVPDet


Papacaviarspoonate

withthea


NV

PDet

Check 4: Is this in CNF?


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7




SNPVPPP

NVPDet


Papacaviarspoonate

withthea


NV

PDet

Check 4: Is this in CNF?Yes


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7



First: Let’s find all NPs


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7




(NP, 0, 1): Papa(NP, 2, 4): the caviar(NP, 5, 7): a spoon(NP, 2, 7): the caviar with a spoon


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7





Second: Let’s find all VPs

(VP, 1, 7): ate the caviar with a spoon(VP, 1, 4): ate the caviar


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7







Third: Let’s find all Ss

(S, 0, 7): Papa ate the caviar with a spoon(S, 0, 4): Papa ate the caviar


S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7


Entire grammarAssume uniform

weights







(NP, 0, 1) (VP, 1, 7) (S, 0, 7)

“Papa ate the caviar with a spoon”S NP VP

NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7



weights







0

6

1

2

3

4

5

1 2 3 4 5 6 7

(NP, 0, 1) (VP, 1, 7) (S, 0, 7)

start

end


NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7



weights







NP0

6

1

2

3

4

5

1 2 3 4 5 6 7

(NP, 0, 1) (VP, 1, 7) (S, 0, 7)

start

end


NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7



weights







NP

VP

0

6

1

2

3

4

5

1 2 3 4 5 6 7

(NP, 0, 1) (VP, 1, 7) (S, 0, 7)

start

end


NP Det N

NP NP PP

VP V NP

VP VP PP

PP P NP

NP Papa

N caviar

N spoon

V spoon

V ate

P with

Det the

Det a

0 1 2 3 4 5 6 7



weights







NP

VP

0

6

1

2

3

4

5

1 2 3 4 5 6 7

(NP, 0, 1) (VP, 1, 7) (S, 0, 7)

start

end

S

CKY Recognizer

Input: * string of N words

* grammar in CNF

Output: True (with parse)/False

Data structure: N*N table T

Rows indicate span start (0 to N-1)

Columns indicate span end (1 to N)

T[i][j] lists constituents spanning i j

CKY Recognizer

Input: * string of N words* grammar in CNF

Output: True (with parse)/False

Data structure: N*N table TRows indicate span

start (0 to N-1)Columns indicate span

end (1 to N)

T[i][j] lists constituents spanning i j

For Viterbi in HMMs:build table left-to-right

For CKY in trees:1. build smallest-to-largest &2. left-to-right

CKY RecognizerT = Cell[N][N+1]


for(j = 1; j ≤ N; ++j) {

T[j-1][j].add(X for non-terminal X in G if X wordj)

}


for(j = 1; j ≤ N; ++j) {T[j-1][j].add(X for non-terminal X in G if X wordj)

}

for(width = 2; width ≤ N; ++width) {

}



}

for(width = 2; width ≤ N; ++width) {for(start = 0; start < N - width; ++start) {

end = start + width

}}



}


end = start + widthfor(mid = start+1; mid < end; ++mid) {

}}

}



}



for(non-terminal Y : T[start][mid]) {for(non-terminal Z : T[mid][end]) {

T[start][end].add(X for rule X Y Z : G)}

}}

}}

X

Y Z

Y Z



}





}}

}}

Q: What do we return?



}





}}

}}

Q: What do we return?

A: S in T[0][N]



}





}}

}}

Q: How do we get the parse?



}





}}

}}

Q: How do we get the parse?

A: Follow backpointers (stored where?)



}





}}

}}


for(j = 1; j ≤ N; ++j) {

T[j-1][j].add(X for non-terminal X in G if X wordj)

}

for(width = 2; width ≤ N; ++width) {

for(start = 0; start < N - width; ++start) {

end = start + width

for(mid = start+1; mid < end; ++mid) {

for(rule X Y Z : G) {

T[start][end].add(X if Y in T[start][mid] & Z in T[mid][end])

}

}

}

}

CKY RecognizerT = bool[K][N][N+1]

for(j = 1; j ≤ N; ++j) {for(non-terminal X in G if X wordj) {

T[X][j-1][j] = True}

}



for(rule X Y Z : G) {for rule X Y Z : G) {

T[X][start][end] = T[Y][start][mid] & T[Z][mid][end]}

}}

}}

Another PCFG Task: Likelihood of the Observed Words

p(S + w1 w2 w3 … wN)

p(w1 w2 w3 … wN)likelihood of word sequence w1w2…wN

p( )

S

w1 w2 w3 w4

p( )

S

w1

w2 w3

w4

p( )

S

w1

w2

w3 w4

…

likelihood of word sequence w1w2…wN

based on starting at S

“syntactic language model”

CKY is Versatile: PCFG TasksTask PCFG algorithm name HMM analog

Find any parse CKY recognizer none

Find the most likely parse (for an observed sequence)

CKY weighted Viterbi Viterbi

Calculate the (log) likelihood of an observed sequence w1, …, wN Inside algorithm Forward algorithm

Learn the grammar parametersInside-outside algorithm (EM)

Forward-backward/Baum-Welch (EM)

CKY Algorithms

Weights ⓪ ①

RecognizerBoolean

(True/False)or and False True

Viterbi [0,1] max * 0 1

Inside [0,1] + * 0 1

Outside?Not really (“Semiring Parsing,” Goodman, 1998). But there is a

connection between inside-outside and backprop! (“Inside-Outside and Forward-Backward Algorithms are Just Backprop,” Eisner, 2016)

Adapted from Jason Eisner

Outline





probabilistic context free grammars · 2018. 11. 28. · slides courtesy rebecca knowles. outline...

Documents