how to train your multi bottom-up tree transducer

How to train yourmulti bottom-up tree transducer

Andreas Maletti

Universität StuttgartInstitute for Natural Language Processing

Stuttgart, Germany

[email protected]

Portland, OR — June 22, 2011

How to train your multi bottom-up tree transducer A. Maletti · 1

Synchronous Tree SubstitutionGrammar

q q

Used rule Next rule

q q—

S

CONJ

wa

q



q S

CONJ

wa

q

Used rule

q q—

S

CONJ

wa

q

Next rule

S

q1 VP

p q2

q—S

p q1 q2



S

q1 VP

p q2

S

CONJ

wa

S

p q1 q2

Used rule

S

q1 VP

p q2

q—S

p q1 q2

Next rule

V

sawp—

V

ra’aa



S

q1 VP

V

saw

q2

S

CONJ

wa

S

V

ra’aa

q1 q2

Used rule

V

sawp—

V

ra’aa

Next rule

NP

DT

the

r q1—NP

r



S

NP

DT

the

r

VP

V

saw

q2

S

CONJ

wa

S

V

ra’aa

NP

r

q2

Used rule

NP

DT

the

r q1—NP

r

Next rule

N

boyr—

N

atefl



S

NP

DT

the

N

boy

VP

V

saw

q2

S

CONJ

wa

S

V

ra’aa

NP

N

atefl

q2

Used rule

N

boyr—

N

atefl

Next rule

NP

DT

the

r q2—NP

r



S

NP

DT

the

N

boy

VP

V

saw

NP

DT

the

r

S

CONJ

wa

S

V

ra’aa

NP

N

atefl

NP

r

Used rule

NP

DT

the

r q2—NP

r

Next rule

N

doorr—

N

albab



S

NP

DT

the

N

boy

VP

V

saw

NP

DT

the

N

door

S

CONJ

wa

S

V

ra’aa

NP

N

atefl

NP

N

albab

Used rule

N

doorr—

N

albab

Next rule



Rule shape

• states can occur only once in lhs and rhs• states in rhs = states in lhs• exactly one lhs and one rhs→ bijective synchronization relation

S

q1 VP

p q2

q—S

p q1 q2



Rule shape

• states can occur only once in lhs and rhs• states in rhs = states in lhs• exactly one lhs and one rhs→ bijective synchronization relation

Notes

• version with states (instead of locality tests)• equivalent to linear nondeleting extended tree transducers• implemented in TIBURON [May, Knight 2006]


Multi Bottom-up Tree Transducer

q q

Used rule Next rule

S

q pq—

S

wa VP

p p q



S

q p

S

wa VP

p p q

Used rule

S

q pq—

S

wa VP

p p q

Next rule

VP

signed rp— twlY

NP-OBJ

NP

AltwqyE

r



S

q VP

signed r

S

wa VP

twlY NP-OBJ

NP

AltwqyE

r

q

Used rule

VP

signed rp— twlY

NP-OBJ

NP

AltwqyE

r

Next rule

NP-SBJ

p′ Voislavq—

NP-SBJ

p′ NP

fwyslAf



S

NP-SBJ

p′ Voislav

VP

signed r

S

wa VP

twlY NP-OBJ

NP

AltwqyE

r

NP-SBJ

p′ NP

fwyslAf

Used rule

NP-SBJ

p′ Voislavq—

NP-SBJ

p′ NP

fwyslAf

Next rule

NML

Yugoslav Presidentp′—

NP

Alr}ys AlywgwslAfy



S

NP-SBJ

NML

Yugoslav President

Voislav

VP

signed r

S

wa VP

twlY NP-OBJ

NP

AltwqyE

r

NP-SBJ

NP

Alr}ys AlywgwslAfy

. . .

Used rule

NML

Yugoslav Presidentp′—

NP

Alr}ys AlywgwslAfy

Next rule

PP

for NP

Serbia

r—

PP

en NP

SrbyA



S

NP-SBJ

NML

Yugoslav President

Voislav

VP

signed PP

for NP

Serbia

S

wa VP

twlY NP-OBJ

NP

AltwqyE

PP

en NP

SrbyA

. . .

Used rule

PP

for NP

Serbia

r—

PP

en NP

SrbyA

Next rule



Rule shape

• STSG: states can occur only once in lhs and rhsMBOT: states can occur only once in lhs

• STSG: states in rhs = states in lhsMBOT: states in rhs ⊆ states in lhs

• STSG: exactly one lhs and one rhsMBOT: exactly one lhs

→ STSG: bijective synchronization relationMBOT: inverse synchronization relation is functional

S

q pq—

S

wa VP

p p q



Rule shape





VP

signed rp— twlY

NP-OBJ

NP

AltwqyE

r



Rule shape





S

q pq—

S

wa VP

p p q


STSG vs. MBOT

property STSG MBOTsimple and natural 3 ?

symmetric 3 7

preserves regularity 3 7

inverse pres. regularity 3 3

binarizable 7 3

closed under composition 7 3

input parsing O(|M|nx) O(|M|n3)output parsing O(|M|nx) O(|M|ny )

where x = 2 rk(M) + 5 andy = 2 rk(M) + 2


Additional Expressive Power

Finite copying

{〈t , σ(t , t)〉 | t ∈ TΣ}

3 desirable [ATANLP participants, 2010]e.g., for cross-serial dependencies

7 harms preservation of regularity

Non-contiguous rules

3 +0.64 BLEU [Sun, Zhang, Tan, 2009] 25.92→ 26.563 do not harm preservation of regularity


Approach

Old approach

1 extract STSG rules2 train STSG3 convert to MBOT4 work with MBOT

New approach

1 extract (restricted) MBOT rules2 train MBOT3 work with MBOT


Roadmap

1 Motivation

2 Relation to STSSG

3 Rule Extraction and Training

4 Preservation of Recognizability


Synchronous Tree-SequenceSubstitution Grammar

Rule shape

• STSG: states can occur only once in lhs and rhsMBOT: states can occur only once in lhsSTSSG: (no restriction)

• STSG: states in rhs = states in lhsMBOT: states in rhs ⊆ states in lhsSTSSG: (no restriction)

• STSG: exactly one lhs and one rhsMBOT: exactly one lhsSTSSG: (no restriction)


Expressive Power

Corollary

STSG ⊆ MBOT ⊆ STSSG

Theorem

STSG ⊂ MBOT ⊂ STSSG

Claim

MBOT−1 ; MBOT = STSSG


STSG vs. MBOT vs. STSSG

property STSG MBOT STSSGsimple and natural 3 ? ??

symmetric 3 7 3

preserves regularity 3 7 7

inverse pres. regularity 3 3 7

binarizable 7 3 3

closed under composition 7 3 7

input parsing O(|M|nx) O(|M|n3) O(|M|ny )output parsing O(|M|nx) O(|M|ny ) O(|M|ny )

where x = 2 rk(M) + 5 andy = 2 rk(M) + 2


Roadmap

1 Motivation

2 Relation to STSSG




Rule Extraction

Input

• parallel bi-text• parses for input and output• word alignment

Output

• MBOT rules


S

NP-SBJ

NML

JJ

Yugoslav

NNP

President

NNP

Voislav

VP

VBD

signed

PP

IN

for

NP

NNP

Serbia

S

CONJ

w

VP

PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

PREP

En

NP

NN-PROP

SrbyA

NP-SBJ

NP

DET-NN

Alr}ys

DET-ADJ

AlywgwslAfy

NP

NN-PROP

fwyslAf


Rule Extraction

S

NP-SBJ

NML

JJ

Yugoslav

NNP

President

NNP

Voislav

VP

VBD

signed

PP

IN

for

NP

NNP

Serbia

S

CONJ

w

VP

PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

PREP

En

NP

NN-PROP

SrbyA

NP-SBJ

NP

DET-NN

Alr}ys

DET-ADJ

AlywgwslAfy

NP

NN-PROP

fwyslAf

S

NP V P

S—

S

CONJ

w

VP

V P V P NP


Rule Extraction

S

NP-SBJ

NML

JJ

Yugoslav

NNP

President

NNP

Voislav

VP

VBD

signed

PP

IN

for

NP

NNP

Serbia

S

CONJ

w

VP

PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

PREP

En

NP

NN-PROP

SrbyA

NP-SBJ

NP

DET-NN

Alr}ys

DET-ADJ

AlywgwslAfy

NP

NN-PROP

fwyslAf

S

NP V P

S—

S

CONJ

w

VP

V P V P NP

VP

VBD

signed

PP V P—PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP


Extracted Rules

STSG rule

S

NP VP

VBD

signed

PP

S—

VP

PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

NP

MBOT rules STSSG rule


Extracted Rules

STSG rule

S

NP VP

VBD

signed

PP

S—

VP

PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

NP

MBOT rulesS

NP V P

S—VP

V P V P NP

VP

VBD

signed

PP V P—PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

STSSG rule


Extracted Rules

STSG rule

S

NP VP

VBD

signed

PP

S—

VP

PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

NP

MBOT rulesS

NP V P

S—VP

V P V P NP

VP

VBD

signed

PP V P—PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

STSSG rule

VBD

signed

IN

forq—

PV

twlY

NP

DET-NN

AltwqyE

PREP

En


Training

DefinitionA tree language is regular if it can be represented by a TSG(with states).

Example

{σ(t , t) | t ∈ TΣ}

is not regular.


Training

TheoremThe set of derivations of an MBOT is regular.

ConclusionMinimal adaptation of [Graehl, Knight, May 2008] can be used.


Roadmap

1 Motivation

2 Relation to STSSG




Preservation of Regularity

Input

• MBOT M• regular tree language L

QuestionIs M(L) = {t | ∃s ∈ L : 〈s, t〉 ∈ M} regular?

Example

• M = {〈t , σ(t , t)〉 | t ∈ TΣ}• infinite, but regular tree language L ⊆ TΣ

Then M(L) = {σ(t , t) | t ∈ L} is not regular.


Preservation of Regularity

Why desired?

• easy and efficient representation of output languages• allows intersection with (regular) language model• allows bucket-brigade algorithms

[May, Knight, Vogler, 2010]


Rule Composition

Input rules

S

NP V P

S—VP

V P V P NP

VP

VBD

signed

PP V P—PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

Output rule

S

NP VP

VBD

signed

PP

S—

VP

PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

NP


Rule Composition

Power MBOT

Mk = {R1 ◦ · · · ◦ Rk | R1, . . . ,Rk ∈ M}

Explanation

• Mk contains k -fold composed rules• composition fails if states do not match• composition succeeds (with no effect) if no matchable

states present


Finitely Collapsing

DefinitionM is finitely collapsing if there exists k ∈ N such that• every state occurs at most once in rhs of every rule of Mk .

Example

S

NP V P

S—VP

V P V P NP

VP

VBD

signed

PP V P—PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

S

NP VP

VBD

signed

PP

S—

VP

PV

twlY

NP-OBJ

NP

DET-NN

AltwqyE

PP

NP


Finite Synchronization

DefinitionM has finite synchronization if there exists k ∈ N such that• all occurrences of a state occur in the same tree in the rhs

of every rule of Mk .

Theorem (Raoult, 1997)MBOT with finite synchronization and finitely collapsing MBOTpreserve regularity.


Copy-free

DefinitionM is copy-free if there exists k ∈ N such that for every ruleof Mk and all occurrences w of a state:• w is the root of a tree in the rhs or• w belongs to a fixed tree in the rhs.

Example (not copy-free)

X

xq—

X

c

X

c

X

qq—

X

a q

X

a q

X

qq—

X

b q

X

b qq S—

σ

q q


Copy-free

DefinitionM is copy-free if there exists k ∈ N such that for every ruleof Mk and all occurrences w of a state:• w is the root of a tree in the rhs or• w belongs to a fixed tree in the rhs.

Example (copy-free)

Y

X SS—

Z

X X S

X

XX— X X

X

Y

S1 S2

X— S1 S2


Main Result

TheoremCopy-free MBOT preserve regularity.

Theoremfinitely collapsing ⊂ finite synchronization ⊂ copy-free

NoteAll 3 restrictions can be guaranteed during rule extraction.


Thank you for your attention!


References1 ARNOLD, DAUCHET: Morphismes et bimorphismes d’arbres.

Theoret. Comput. Sci., 1982

2 ATANLP participants: Desired properties of syntax-based MT systems.ACL workshop, 2010

3 GRAEHL, KNIGHT, MAY: Training tree transducers.Computational Linguistics, 2008

4 MAY, KNIGHT: TIBURON — A weighted tree automata toolkit.In CIAA, 2006

5 MAY, KNIGHT, VOGLER: Efficient Inference through Cascades ofWeighted Tree Transducers.In ACL, 2010

6 RAOULT: Rational tree relations.Bull. Belg. Math. Soc. Simon Stevin, 1997

7 SUN, ZHANG, TAN: A non-contiguous tree sequence alignment-basedmodel for statistical machine translation.In ACL, 2009


how to train your multi bottom-up tree transducer

Documents