generation-heavy hybrid machine translation nizar habash postdoctoral researcher center for...

Generation-Heavy Hybrid Machine Translation

Nizar Habash Postdoctoral Researcher

Center for Computational Learning SystemsColumbia University

Columbia University

NLP Colloquium

October 28, 2004

The IntuitionGeneration-Heavy Machine Translation

Español ‚ عربي ‚

English

Dictionary

gist

gist

E

IntroductionResearch Contributions

• A general reusable and extensible Machine Translation (MT) model that transcends the need for large amounts of deep symmetric knowledge

• Development of reusable large-scale resources for English

• A large-scale Spanish-English MT system: Matador; Matador is more robust across genre and produce more grammatical output than simple statistical or symbolic techniques

Roadmap

Introduction Generation-Heavy Machine Translation Evaluation Conclusion Future Work

IntroductionMT Pyramid

Source word

Source syntax

Source meaning Target meaning

Target syntax

Target word

Analysis Generation

Interlingua

Gisting

Transfer


Source word

Source syntax


Target syntax

Target word

Analysis Generation

Interlingual Lexicons

Dictionaries/Parallel Corpora

Transfer Lexicons


Gisting

Transfer

Source word

Source syntax


Target syntax

Target word

IntroductionWhy gisting is not enough

Sobre la base de dichas experiencias se estableció en 1988 una metodología.

Envelope her basis out speak experiences them settle at 1988 one methodology.

On the basis of these experiences, a methodology was arrived at in 1988.

IntroductionTranslation Divergences

• 35% of sentences in TREC El Norte Corpus (Dorr et al 2002)

• Divergence Types– Categorial (X tener hambre X be hungry)

– Conflational (X dar puñaladas a Z X stab Z)

– Structural (X entrar en Y X enter Y)

– Head Swapping (X cruzar Y nadando X swim across Y)

– Thematic (X gustar a Y Y like X)

Roadmap


Generation-Heavy Hybrid Machine Translation

• Problem: asymmetric resources– High quality, broad coverage, semantic resources

for target language– Low quality resources for source language– Low quality (many-to-many) translation lexicon

• Thesis: we can approximate interlingual MT without the use of symmetric interlingual resources

Relevant Background Work

• Hybrid Natural Language GenerationConstrained Overgeneration Statistical Ranking

Nitrogen (Langkilde and Knight 1998), Halogen (Langkilde 2002)

FERGUS (Rambow and Bangalore 2000)

• Lexical Conceptual Structure (LCS) based MT

(Jackendoff 1983), (Dorr 1993)

LCS-based MTExample

(Dorr, 1993)

Generation-Heavy HybridMachine Translation

Analy

sis

Tra

nsla

tion

Th

eta

Lin

king

Expansio

n

Assig

nm

ent

Pru

nin

g

Lineariza

tion

Rankin

g

…

Generation

MatadorSpanish-English GHMT

Analy

sis

Tra

nsla

tion

Spanish

English

Th

eta

Lin

king

Expansio

n

Assig

nm

ent

Pru

nin

g

Lineariza

tion

Rankin

g

Generation

Expansive Rich Generation for English

EXERGE

GHMTAnalysis

• Source language syntactic dependency

• Example: Yo le di puñaladas a Juan.

• Features of representation– Approximation of

predicate-argument structure

– Long-distance dependencies

dar

Yo puñalada a

Juan

:obj

:obj:mod:subj

GHMTTranslation

• Lexical transfer but NO structural change

• Translation Lexicon (tener V) ((have V) (own V) (possess V) (be V))(deber V) ((owe V) (should AUX) (must AUX))(soler V) ((tend V) (usually AV))

ADMINISTER,CONFER, DELIVER, EXTEND, GIVE,

GRANT, HAND, LAND, RENDER

I, MY, MINE

STAB, KNIFE_WOUND

AT, BY, INTO, THROUGH, TO

JOHN

:obj

:obj :mod:subj

dar

Yo puñalada a

Juan

:obj

:obj:mod:subj

GHMTThematic Linking

• Syntactic Dependency Thematic Dependency

•Which divergence

Goal

EXTEND, GIVE, GRANT, RENDER

I, MY, MINE

STAB, KNIFE_WOUND

JOHN

ThemeAgent

ADMINISTER,CONFER, DELIVER, EXTEND, GIVE, GRANT, HAND,

LAND, RENDER

I, MY, MINE

STAB, KNIFE_WOUND


JOHN

:obj

:obj :mod:subj


Resources

• Word Class Lexicon :NUMBER "V.13.1.a.ii" :NAME "Give - No Exchange” :POS V

:THETA_ROLES (((ag obl) (th obl) (goal obl to)) ((ag obl) (goal obl) (th obl))) :LCS_PRIMS (cause go)

:WORDS (feed give pass pay peddle refund render repay serve))

• Syntactic-Thematic Linking Map (:subj ag instr th exp loc src goal perc mod-poss poss)

(:obj2 goal src th perc ben)

(across goal loc)

(in loc mod-poss perc goal poss prop)

(to prop goal ben info th exp perc pred loc time)



((ADMINISTER V.13.2 ((AG OBL) (TH OBL) (GOAL OPT TO))) (CONFER V.37.6.b ((EXP OBL))) (DELIVER V.11.1 ((AG OBL) (GOAL OBL) (TH OBL) (SRC OPT FROM))) (EXTEND V.47.1 ((TH OBL) (MOD-LOC OPT . T))) (EXTEND V.13.3 ((AG OBL) (TH OBL) (GOAL OPT TO))) (EXTEND V.13.3 ((AG OBL) (GOAL OBL) (TH OBL))) (EXTEND V.13.2 ((AG OBL) (TH OBL) (GOAL OPT TO))) (GIVE V.13.1.a.ii ((AG OBL) (TH OBL) (GOAL OBL TO))) (GIVE V.13.1.a.ii ((AG OBL) (GOAL OBL) (TH OBL))) (GRANT V.29.5.e ((AG OBL) (INFO OBL THAT))) (GRANT V.29.5.d ((AG OBL) (TH OBL) (PROP OBL TO))) (GRANT V.13.3 ((AG OBL) (TH OBL) (GOAL OPT TO))) (GRANT V.13.3 ((AG OBL) (GOAL OBL) (TH OBL))) (HAND V.11.1 ((AG OBL) (TH OBL) (GOAL OPT TO) (SRC OPT FROM))) (HAND V.11.1 ((AG OBL) (GOAL OBL) (TH OBL) (SRC OPT FROM))) (LAND V.9.10 ((AG OBL) (TH OBL))) (RENDER V.13.1.a.ii ((AG OBL) (TH OBL) (GOAL OBL TO))) (RENDER V.13.1.a.ii ((AG OBL) (GOAL OBL) (TH OBL))) (RENDER V.10.6.a ((AG OBL) (TH OBL) (MOD-POSS OPT OF))) (RENDER V.10.6.a.LOCATIVE ((AG OPT) (SRC OBL) (TH OPT OF))))


LAND, RENDER

I, MY, MINE

STAB, KNIFE_WOUND


JOHN

:obj

:obj :mod:subj



Goal

EXTEND, GIVE, GRANT, RENDER

I, MY, MINE

STAB, KNIFE_WOUND

JOHN

ThemeAgent


LAND, RENDER

I, MY, MINE

STAB, KNIFE_WOUND


JOHN

:obj

:obj :mod:subj

Interlingua Approximationthrough Expansion Operations

obj

enter

John room

subj in

enter

John room

subj

in

go

John room

subj

developmentNdevelopV

Categorial Variation

putV

butterN

butterV

Node Conflation / Inflation

RelationConflation / Inflation

Relation Variation

Interlingua Approximation2nd Degree Expansion

obj

cross

John river

subj mod

swimming

across

go

John river

subj mod

swimming

Relation Inflation

across

swim

John river

subj

Node Conflation

GHMTStructural Expansion

• Conflation Example

,

Goal

STABV

I JOHN

Agent Goal

GIVEV

I STABN JOHN

ThemeAgent


• Conflation and Inflation• Structural Expansion Resources

– Word Class Lexicon :NUMBER "V.42.2" :NAME “Poison Verbs” :POS V

:THETA_ROLES (((ag obl)(goal obl)))

:LCS_PRIMS (cause go)

:WORDS (crucify electrocute garrotte hang knife poison shoot smother stab strangle)

– Categorial Variation Database (Habash and Dorr 2003)

(:V (hunger) :N (hunger hungriness) :AJ (hungry))

(:V (validate) :N (validation validity) :AJ (valid))

(:V (cross) :N (crossing cross) :P (across))

(:V (stab) :N (stab))



Goal

GIVEV

I STABN JOHN

ThemeAgent

STABV



Goal

STABV

* *

Agent

[CAUSE GO][CAUSE GO]

Goal

GIVEV

I STABN JOHN

ThemeAgent



,

Goal

STABV

I JOHN

Agent Goal

GIVEV

I STABN JOHN

ThemeAgent

Goal

STABV

I JOHN

Agent Goal

GIVEV

I STABN JOHN

ThemeAgent

GHMT Syntactic Assignment

• Thematic Syntactic Mapping

Object

STABV

I, MY …

JOHN

Subject

IObject

GIVEV

I, MY …

STABN, KNIFE_ WOUNDN

JOHN

ObjectSubject

Object

Mod

GIVEV

I, MY …


TO, AT, …

ObjectSubject

JOHN

GHMT Structural N-gram Pruning

• Statistical lexical selection

Object

STABV

I, MY …

JOHN

Subject

IObject

GIVEV

I, MY …


JOHN

ObjectSubject

Object

Mod

GIVEV

I, MY …


TO, AT, …

ObjectSubject

JOHN

Object

STABV

I JOHN

Subject

IObject

GIVEV

I STABN JOHN

ObjectSubject

Object

Mod

GIVEV

I STABN v TO

ObjectSubject

JOHN

• Structural N-gram Model– Long-distance – Lexemes

• Surface N-gram Model– Local – Surface-forms

GHMTTarget Statistical Resources

cloudevery liningahas silver

a

lining

silver

have

cloud

every

GHMTLinearization &Ranking

• Oxygen Linearization (Habash 2000)

• Halogen Statistical Ranking (Langkilde 2002)---------------------------------------------------------

I stabbed John . [-1.670270 ]I gave a stab at John . [-2.175831]I gave the stab at John . [-3.969686]I gave an stab at John . [-4.489933]I gave a stab by John . [-4.803054]I gave a stab to John . [-5.045810]I gave a stab into John . [-5.810673]I gave a stab through John . [-5.836419]I gave a knife wound by John . [-6.041891]

Roadmap

Introduction Generation-Heavy Machine Translation Evaluation

Overall Evaluation Component Evaluation

Conclusion Future Work

Overall EvaluationSystems

Gisting(GIST)

Systran(SYST)

IBM Model 4(IBM4)

Matador(MTDR)

Approach SymbolicWord-based

SymbolicTransfer-based

StatisticalWord-based

HybridGeneration-Heavy

TranslationModel

400Ksurface-lexeme

pairs 120Klexeme-lexeme

pairsand

large transferlexicon

Model 4Giza Trained

50K UN sentence pairs

50Klexeme-lexeme

pairs

LanguageModel

UnigramsBrown Corpus

1M words

Bigrams3M words (UN)

Bigrams3M words (UN)

andStructural Bigrams1.5M words (UN)

DevelopmentTime

1 person-month Hundreds ofperson-years

1 person-month 1 person-year

(Brown et al 1990)(Al-Onaizan et al 1999)

(Germann and Marcu 2000)

(Resnik 1997)

Overall EvaluationBleu Metric

• Bleu – BiLingual Evaluation Understudy (Papineni et al 2001)

– Modified n-gram precision with length penalty

– Quick, inexpensive and language independent

– Correlates highly with human evaluation

– Bias against synonyms and inflectional variations

Overall EvaluationTest Sets

UN FBIS Bible

GenreUnited Nations

documentsNews broadcast Religious

Spanish-EnglishSentence pairs

2,000 2,000 1,000

Sentence Length(words)

15.39 19.27 16.38

Overall EvaluationResults

0

5

10

15

20

25

30

UN FBIS BIBLE

Corpus

Ble

u S

core

SYST IBM4 MTDR GIST

Overall EvaluationResults

• Systran is overall best

• Gist is overall worst

• Matador is more robust than IBM4

• Matador is more grammatical than IBM4

• Matador has less information loss than IBM4

Overall Evaluation Grammaticality

• Example– SP: Ademàs dijo que solamente una inyecciòn masiva de capital extranjero ...

– EN: Further, he said that only a massive injection of foreign capital ...

– IBM4: further stated that only a massive inyecciòn of capital abroad ...

– MTDR: Also he spoke only a massive injection of foreign capital ...

• Parsed all sentences (Spanish, English reference and English output)– Can we find main verb?

– Pro Drop Restoration

Overall Evaluation Grammaticality: Verb Determination

0

0.5

1

1.5

2

2.5

Verb

s p

er

Sen

ten

ce

English Spanish SYST IBM4 MTDR GIST

Overall Evaluation Grammaticality: Subject Realization

-

10

20

30

40

50

60

70

80

90

100

Perc

en

tag

e o

f R

ealize

d S

ub

ject

s

English Spanish SYST IBM4 MTDR GIST

Overall Evaluation Loss of Information

• Example– SP: El daño causado al pueblo de Sudáfrica jamás debe subestimarse.

– EN: The damage caused to the people of his country should never be underestimated.

– IBM4: the damage * the people of south * must never underestimated .

– MTDR: Never the causado damage to the people of South Africa should be underestimated.

Gisting(GIST)

Systran(SYST)

IBM Model 4(IBM4)

Matador(MTDR)

Reference length

109% 109% 94% 104%

Component Evaluation

• Conducted several component evaluations– Parser

• ~75% correct (labeled dependency links)

– Categorial Variation Database • 81% Precision-Recall

– Structural Expansion– Structural N-grams

Component EvaluationStructural Expansion

• Insignificant increase in Bleu score• 40% of divergences pragmatic• LCS lexicon coverage issues • Minimal handling of nominal divergences• Over-expansion

– Además, destruyó totalmente sus cultivos de subsistencia …– EN: It had totally destroyed Samoa's staple crops ... – MTDR: Furthermore, it totaled their cultivations of subsistence …

– SP: Dicha adición se publica sólo en años impares.– EN: That addendum is issued in odd-numbered years only.– MTDR: concerned addendum is excluded in odd years.

Component EvaluationStructural N-grams

0

5000

10000

15000

20000

25000

30000

Parsing Translation Expansion Linearization Ranking

Module

Tim

e (

seconds)

with Structural N-grams without Structural N-grams

• 60% speed-up with no effect on quality

Roadmap


ConclusionResearch Contributions

• A general reusable and extensible MT model that transcends the need for large amounts of symmetric knowledge

• A systematic non-interlingual/non-transfer framework for handling translation divergences

• Extending the concept of symbolic overgeneration to include conflation and head-swapping of structural variations.

• A model for language-independent syntactic-to-thematic linking

ConclusionResearch Contributions

• Development of reusable large-scale modules and resources: Exerge, Categorial Variation Database, etc.

• A large-scale Spanish-English GHMT implementation

•An evaluation of Matador against four models of machine translation found it to be robust across genre and to produce more grammatical output.

Ongoing Work• Retargetability to new languages

– Chinese, Arabic• Extending system to use bi-texts

– Phrase dictionary– Weighted translation pairs

• Generation-Heavy parsing– Small dependency grammar for foreign language– English structural n-grams to rank parses

• Extending system with new optional modules– Cross-lingual headline generation

DepTrimmer (work with Bonnie Dorr) extending Trimmer (Dorr, et al. 2003) to dependency representation

Future Work

• Categorial Variation Database– Improving word-cluster correctness

• Structural Expansion– Extending to nominal divergences– Improving thematic linking with a statistical

model

• Structural N-grams– Enriching with syntactic/thematic relations

Thank you!

Questions?


Test Sentence

colorless green ideas sleep furiously

Gold Standard References

all dull jade ideas sleep iratelydrab emerald concepts sleep furiously

colorless immature thoughts nap angrily


Test Sentence

colorless green ideas sleep furiously




Unigram precision = 4/5


Test Sentence

colorless green ideas sleep furiouslycolorless green ideas sleep furiouslycolorless green ideas sleep furiouslycolorless green ideas sleep furiously




Unigram precision = 4 / 5 = 0.8Bigram precision = 2 / 4 = 0.5

Bleu Score = (a1 a2 …an)1/n

= (0.8 ╳ 0.5)½ = 0.6325 63.25

Overall Evaluation

• Investigating BLEU’s bias towards inflectional variants– SP: Los programas de ajuste estructural se han aplicado rigurosamente.

– EN: Structural adjustment programmes had been rigorously implemented.

– IBM4: structural adjustment programmes have been applied strictly.– MTDR: programmes of structural adjustment have been added

rigurosament.

Overall Evaluation Inflectional Normalization

0

5

10

15

20

25

30

35

40

45

Raw Normalized

Ble

u Sc

ore

SYST IBM4 MTDR GIST

generation-heavy hybrid machine translation nizar habash postdoctoral researcher center for...

Documents

x slide

english exerge slide

interlingual mt

source language low

x cruzar y nadando x

y thematic x gustar

mt jackendoff

translation lexicon