wrapper syntax for example-based machine translation karolina owczarzak, bart mellebeek, declan...

Wrapper Syntax for Example-Based Machine Translation

Karolina Owczarzak, Bart Mellebeek, Declan Groves, Josef Van Genabith, Andy Way

National Centre for Language TechnologySchool of Computing

Dublin City University

Overview

• TransBooster – wrapper technology for MT– motivation– decomposition process– variables and template contexts– recomposition

• Example-Based Machine Translation– marker-based EBMT

• Experiment– English-Spanish– Europarl, Wall Street Journal section of Penn II Treebank– automatic and manual evaluation

• Comparison with previous experiments

TransBooster – wrapper technology for MT

• Assumption:

MT systems perform better at translating short sentences than long ones.

• Decompose long sentences into shorter and syntactically simpler chunks, send to translation, recompose on output

• Decomposition linguistically guided by syntactic parse of the sentence

TransBooster – wrapper technology for MT

• TransBooster technology is universal and can be applied to any MT system

• Experiments to date:– TB and Rule-Based MT (Mellebeek et al.,

2005a,b)– TB and Statistical MT (Mellebeek et al., 2006a)– TB and Multi-Engine MT (Mellebeek et al.,

2006b)

• TransBooster outperforms baseline MT systems

TransBooster – decomposition

• Input – syntactically parsed sentence (Penn II format)• Decompose into pivot and satellites

– pivot: usually main predicate (plus additional material)– satellites: arguments and adjuncts

• Recursively decompose satellites if longer than x leaves

• Replace satellites around pivot with variables– static: simple same-type phrases with known translation– dynamic: simplified version of original satellites– send off to translation

• Insert each satellite into a template context– static: simple predicate with known translation– dynamic: simpler version of original clause (pivot +

simplified arguments, no adjuncts)– send off to translation

TransBooster – decomposition example

(S (NP (NP (DT the) (NN chairman)) (, ,) (NP (NP (DT a) (JJ long-time) (NN rival)) (PP (IN of) (NP (NNP Bill) (NNP Gates)))) (, ,)) (VP (VBZ likes) (NP (ADJP (JJ fast) (CC and) (JJ confidential)) (NNS deals))) (. .))

[The chairman, a long-time rival of Bill Gates,]ARG1 [likes]pivot [fast and confidential deals]ARG2.

[The chairman]V1 [likes]pivot [deals]V2.

[The chairman, a long-time rival of Bill Gates,]ARG1 [likes deals]V1.

[The chairman likes]V1 [fast and confidential deals]ARG2.

[The man]V1 [likes]pivot [cars]V2.

[The chairman, a long-time rival of Bill Gates,]ARG1 [is sleeping]V1.

[The man sees]V1 [fast and confidential deals]ARG2.

MT engine

TransBooster – recomposition

• MT output: a set of translations with dynamic and static variables and contexts for a sentence S

• Remove translations of dynamic variables and contexts from translation of S

• If unsuccessful, back off to translation with static variables and contexts, remove those

• Recombine translated pivot and satellites into output sentence

TransBooster – recomposition example

[The chairman]V1 [likes]pivot [deals]V2. ->El presidente tiene gusto de repartos.

[The chairman, a long-time rival of Bill Gates,]ARG1 [likes deals]V1. ->

El presidente, un rival de largo plazo de Bill Gates, tiene gusto de repartos.

[The chairman likes]V1 [fast and confidential deals]ARG2. ->

El presidente tiene gusto de repartos rápidos y confidenciales.

[The man]V1 [likes]pivot [cars]V2. ->

El hombre tiene gusto de automóviles.

[The chairman, a long-time rival of Bill Gates,]ARG1 [is sleeping]V1. ->

El presidente, un rival de largo plazo de Bill Gates, está durmiendo.

[The man sees]V1 [fast and confidential deals]ARG2. ->

El hombre ve repartos rápidos y confidenciales.

[El presidente, un rival de largo plazo de Bill Gates,] [tiene gusto de] [repartos rápidos y confidenciales].Original translation:El presidente, rival de largo plazo de Bill Gates, gustos ayuna y los repartos confidenciales.

The chairman, a long-time rival of Bill Gates, likes fast and confidential deals.

EBMT – Overview

• An aligned bilingual corpus

• Input text is matched against this corpus

• The best match is found and a translation is produced

French

F1

F2

F3

F4

EX (input)

search

F2 F4

FX (output)

English

E1

E2

E3

E4

Given in corpus

John went to school Jean est allé à l’école

The butcher’s is next to the baker’s

La boucherie est à côté de la boulangerie

Isolate useful fragments

John went to Jean est allé à

the baker’s la boulangerie

We can now translate

John went to the baker’s

Jean est allé à la boulangerie

EBMT – Marker-Based Chunking

<DET> = {the,a,these……} <DET> = {le,la,l’,une,un,ces…..}

<PREP> = {on, of …} <PREP> = {sur, d’ ..}

English phrase : on virtually all uses of asbestos

French translation: sur virtuellement tous usages d’asbeste

<PREP> on virtually <DET> all uses <PREP> of asbestos

<PREP> sur virtuellement <DET> tous usages <PREP> d’ asbeste

Marker Chunks:

<PREP> on virtually : sur virtuellement

<DET> all uses : tous usages

<PREP> of asbestos : d’asbeste

Lexical Chunks:

<LEX> on : sur <LEX> virtually : virtuellement

<LEX> all : tous <LEX> uses : usages

<LEX> of : d’ <LEX> asbestos : asbeste

EBMT – System Overview

Experiment

• English -> Spanish

• Two test sets:– Wall Street Journal section of Penn II Treebank

800 sentences– Europarl 800 sentences

• “Out-of-domain” factor:– TransBooster developed on perfect Penn II

trees– EBMT trained on 958K English-Spanish

Europarl sentences

Experiment – Results

Results for EBMT vs TransBooster on 741-sentence test set from Europarl.

Europarl BLEU NIST

EBMT 0.2111 5.9243

TransBooster 0.2134 5.9342

Percent of Baseline 101% 100.2%

Wall Street Journal

BLEU NIST

EBMT 0.1098 4.9081


Percent of Baseline 103.8% 100.5%

Results for EBMT vs TransBooster on 800-sentence test set from Penn II Treebank.

Automatic evaluation

Experiment - Results

Manual evaluation• 100 randomly selected sentences from EP test set:

– source English sentence– EBMT translation– EBMT + TransBooster translation

• 3 judges, native speakers of Spanish fluent in English• Accuracy and fluency: relative scale for comparing the two translations

Inter-judge agreement (Kappa): Fluency > 0.948, Accuracy > 0.926

Fluency Accuracy

TB > EBMT 35.33% 35%

EBMT > TB 16% 19.33%

Absolute quality gain when using TransBooster: Fluency 19.33% of sentences Accuracy 15.67% of sentences

Experiment – Results

TB improvements:

Example 1

Source: women have decided that they wish to work, that they wish to make their work

compatible with their family life.

EBMT: hemos decidido su deseo de trabajar, su deseo de hacer su trabajo compatible con su vida familiar. empresarias

TB: mujeres han decidido su deseo de trabajar, su deseo de hacer su trabajo

compatible con su vida familiar.Example 2

Source: if this global warming continues, then part of the territory of the eu member states will become sea or desert.

EBMT: si esto continúa calentamiento global, tanto dentro del territorio de los estados

miembros tendrán tornarse altamar o desértico

TB: si esto calentamiento global perdurará, entonces parte del territorio de los

estados miembros de la unión europea tendrán tornarse altamar o desértico

Previous experiments

TransBooster vs. SMT on 800-sentence test set from Europarl.

TB vs. SMT: EP BLEU NISTSMT 0.198

65.8393TransBooster 0.205

25.8766% of Baseline 103.3

%100.6%

TB vs. RBMT: WSJ

BLEU NISTRule-Based MT 0.310


37.3901% of Baseline 101.7

%100.6%Results for TransBooster vs. Rule-Based MT on 800-

sentence test set from Penn II Treebank.

TB vs. SMT: WSJ

BLEU NISTSMT 0.134

35.1432TransBooster 0.

13795.1259% of Baseline 102.7

%99.7%

TransBooster vs. SMT on 800-sentence test set from Penn II Treebank.

TB vs. EBMT: EP

BLEU NISTEBMT 0.211


45.9342% of Baseline 101% 100.2%TransBooster vs. EBMT on 800-sentence

test set

from Europarl.

TB vs. EBMT: WSJ

BLEU NISTEBMT 0.109


04.9321% of Baseline 103.8

%100.5%TransBooster vs. EBMT on 800-sentence

test set

from Penn II Treebank.


TransBooster vs. SMT on 800-sentence test set from Europarl.

TB vs. SMT: EP BLEU NIST

SMT 0.1986 5.8393


% of Baseline 103.3% 100.6%

TB vs. EBMT: EP BLEU NIST

EBMT 0.2111 5.9243


% of Baseline 101% 100.2%

TransBooster vs. EBMT on 800-sentence test set from Europarl.


TB vs. RBMT: WSJ BLEU NIST

Rule-Based MT 0.3108 7.3428


% of Baseline 101.7% 100.6%TransBooster vs. Rule-Based MT on 800-sentence test set from Penn II Treebank.

TB vs. SMT: WSJ BLEU NIST

SMT 0.1343 5.1432

TransBooster 0. 1379 5.1259

% of Baseline 102.7% 99.7%

TransBooster vs. SMT on 800-sentence test set from Penn II Treebank.

TB vs. EBMT: WSJ BLEU NISTEBMT 0.1098 4.9081TransBooster 0.1140 4.9321% of Baseline 103.8% 100.5%

TransBooster vs. EBMT on 800-sentence test set from Penn II Treebank.

Summary

• TransBooster is a universal technology to decompose and recompose MT text

• Net improvement in translation quality against EBMT:

Fluency 19.33% of sentences Accuracy 15.67% of sentences

• Successful experiments to date: rule-based MT, phrase-based SMT, multi-engine MT, EBMT

• Journal article in preparation

Thank You

wrapper syntax for example-based machine translation karolina owczarzak, bart mellebeek, declan...

Documents

chairman v1

translation slide

mt transbooster technology

satellites pivot

mt engine slide

v1 fast

translation of s

mt assumption