topic models for dynamic translation model adaptation · doc1 doc1 out out in dev w test ....

Topic Models for Dynamic Translation Model Adaptation

Vladimir Eidelman

Jordan Boyd-Graber

Philip Resnik

(Typical) Domain Adaptation

doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

Training Corpus


doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

Newswire


doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

Newswire Web


doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

Newswire Web Europarl


doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

Newswire Web Europarl

dev


doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

out out in

dev


doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

out out in

dev w test

Motivation

doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

Motivation

doc4

doc3 doc2

doc4 doc3

doc2 doc1

doc4 doc3

doc2 doc1 doc1

test dev w test

Aims

• Model Domain

– Induce soft unsupervised domains

• Latent Topics

• Apply to MT

– Bias translation model

• Introduce topic-dependent lexical weighting

Lexical Weighting

• Estimate phrase pair quality word-by-word

粉丝很多 fěnsī hěnduō noodles a lot of

Lexical Weighting

• Estimate phrase pair quality word-by-word

粉丝很多 fěnsī hěnduō noodles a lot of fans a lot of

Topic Models

•Used MALLET (McCallum, 2002) •Latent Dirichlet Allocation (Blei, Ng, Jordan 2003) •Only on source •Topic distribution the same for every sentence in document

Standard Lexical Weighting

粉丝很多

粉丝很多


Source Target P(e|f)

粉丝很多 lots of noodles .45

粉丝很多 lots of fans .33

粉丝很多

粉丝很多

Translation Table


Translation Table

粉丝很多

粉丝很多




Domain Lexical Weighting (Chiang 2011)

粉丝很多

粉丝很多

Domain Lexical Weighting

Translation Table: nw

(Chiang 2011)

粉丝很多

粉丝很多




Domain Lexical Weighting

Translation Table: nw

Translation Table: Web

(Chiang 2011)

粉丝很多

粉丝很多

Source Target Ps=nw(e|f)



Source Target Ps=wb(e|f)



Lexical Weighting with Topic Models

粉丝很多

粉丝很多


粉丝很多

粉丝很多

Translation Table: Topic 1

Source Target Ptopic=1(e|f)





粉丝很多

粉丝很多








Lexical Weighting Adaptation Features

Source Target Ptopic(e|f)












test sentence


ƒ1(e|f) = 0.71 * 0.65














ƒ1(e|f) = 0.15 * 0.65















粉丝很多 lots of noodles .71 0.46

粉丝很多 lots of fans .15 0.09











粉丝很多 ||| lots of fans ||| ƒ1(e|f)=.46 ƒ2(e|f)=.09 ƒ3(e|f)=.02 ƒ1(f|e) ƒ2(f|e) ƒ3(f|e) …













Experiments

• Chinese-English

• Two settings – Small (FBIS)

• 300k sentence pairs

• Document boundaries

– Large (~NIST) • 1.6m sentence pairs

• No documents

• NIST MT06 tune, MT03 & 05 test

• MIRA optimizer

Unsupervised Domain Induction

• What is a document (for topic modeling)?

• Only some MT data have document boundaries

• Treat each sentence as document

Document v. Sentence Results

FBIS Document v. Sentence Results

Large Setting

Future Work

• Improve Topic Model

– Multilingual Topic Modeling

– More (mono,multi)-lingual data

– Hierarchical models

• Other languages

Conclusions

• Extend domain adaptation

– No reliance on collection/genre annotation

– Finer-grained topic distributions

• Bias transation toward topic

– Lexical weighting adaptation with soft membership

• Add Ptopic(e|f) and Ptopic(f|e) features to every rule

• Thank You!

• Question?

Feature Representation

• Topic Identity

– Probability under topic 1, topic 2?

– Cross-domain

• Topic Distribution

– Probability under most probable topic? Second most?

– Dynamic

Global vs. Local Topic Model

Large Corpus

topic models for dynamic translation model adaptation · doc1 doc1 out out in dev w test ....

Documents