automating post-editing to improve mt systems ariadna font llitjós and jaime carbonell ape...
TRANSCRIPT
![Page 1: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/1.jpg)
Automating Post-Editing To Improve MT Systems
Ariadna Font Llitjós and Jaime Carbonell
APE Workshop, AMTA – Boston
August 12, 2006
![Page 2: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/2.jpg)
Outline• Introduction
– Problem and Motivation– Goal and Solution– Related Work
• Theoretical Space
• Online Error Elicitation
• Rule Refinement Framework
• MT Evaluation Framework
• Conclusions
![Page 3: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/3.jpg)
Problem and Motivation
MT output: Assassinated a diplomat Russian and kidnapped other four in Bagdad
![Page 4: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/4.jpg)
Problem and Motivation
MT output: Assassinated a diplomat Russian and kidnapped other four in Bagdad
• Could hire post-editors to correct machine translations
![Page 5: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/5.jpg)
Problem and Motivation
MT output: Assassinated a diplomat Russian and kidnapped other four in Bagdad
• Could hire post-editors to correct machine translations
• Expensive + time consuming
![Page 6: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/6.jpg)
Problem and Motivation
MT output: Assassinated a diplomat Russian and kidnapped other four in Bagdad
• Not feasible for large amounts of data (google, yahoo, etc.)
• Does not generalize to new sentences
![Page 7: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/7.jpg)
Ultimate Goal
Automatically Improve MT Quality
![Page 8: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/8.jpg)
Two Alternatives
• Automatic learning of post-editing rules (APE)
+ system independent
- several thousands of sentences might need to be corrected for the same error
• Automatic Refinement of MT System+ attacks the core of the problem- system dependent
![Page 9: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/9.jpg)
Our Approach
Automatically Improve MT Quality
by Recycling Post-Editing Information
Back to MT Systems
![Page 10: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/10.jpg)
Our Solution
• Get non-expert bilingual speakers to provide correction feedback online – Make correcting translations easy and fun – 5-10 minutes a day Large amounts of correction
data
SL: Asesinado un diplomático ruso y secuestrados otros cuatro en Bagdad
TL: Assassinated a diplomat Russian and kidnapped other four in Bagdad
![Page 11: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/11.jpg)
Our Solution
• Get non-expert bilingual speakers to provide correction feedback online – Make correcting translations easy and fun – 5-10 minutes a day Large amounts of correction
data
• Feed corrections back into the MT system, so that they can be generalized
System will translate new sentences better
SL: Asesinado un diplomático ruso y secuestrados otros cuatro en Bagdad
TL: Assassinated a diplomat Russian and kidnapped other four in Bagdad
![Page 12: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/12.jpg)
Bilingual Post-editing
• In addition to:– TL sentence, and– Context information, when available
• It also provides post-editors with:– Source Language sentence– Alignment information
Traditional post-editing
Allegedly a harder task, but we can now get much more data for free
![Page 13: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/13.jpg)
MT Approaches
Interlingua
Syntactic Parsing
Semantic Analysis
Sentence Planning
Text Generation
Source (e.g. English)
Target(e.g. Spanish)
Transfer Rules
Direct: SMT, EBMT
![Page 14: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/14.jpg)
[Nishida et al. 1988]
[Corston-Oliver & Gammon, 2003]
[Imamura et al. 2003]
[Menezes & Richardson, 2001]
Related Work
[Su et al. 1995]
[Brill, 1993]
[Gavaldà, 2000]
Post-editing
Rule AdaptationFixingMachine Translation
[Callison-Burch, 2004]
[Allen 2003]
[Allen & Hogan, 2000]
[Knight & Chander, 1994]
Our ApproachNon-expert user feedback
Provides relevant reference translations
Generalizes over unseen data
![Page 15: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/15.jpg)
System ArchitectureINPUT TEXT
OUTPUT TEXT
Rule Learning
and other
Resources
Run-Time MT
System
Online
Translation
Correction
Tool
Rule Refinement
![Page 16: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/16.jpg)
Main Technical Challenge
Mapping between
Simple user edits to MT output
Improved Translation Rules
Blame assignment
Rule Modifications
Lexical Expansions
![Page 17: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/17.jpg)
Outline• Introduction
• Theoretical Space– Error classification– Limitations
• Online Error Elicitation
• Rule Refinement Framework
• MT Evaluation Framework
• Conclusions
![Page 18: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/18.jpg)
Error Typology for Automatic Rule Refinement (simplified)
Missing word
Extra word
Wrong word order
Incorrect word
Local vs Long distance
Word vs. phrase
+ Word change
Sense
Form
Selectional restrictions
Idiom
Missing constraint
Extra constraint
![Page 19: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/19.jpg)
Error Typology for Automatic Rule Refinement (simplified)
Missing word
Extra word
Wrong word order
Incorrect word
Local vs Long distance
Word vs. phrase
+ Word change
Sense
Form
Selectional restrictions
Idiom
Missing constraint
Extra constraint
![Page 20: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/20.jpg)
Limitations of Approach
• The SL sentence needs to be fully parsed by the translation grammar.– current approach needs rules to refine
• Depend on bilingual speakers’ ability to detect MT errors.
![Page 21: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/21.jpg)
Outline
• Introduction
• Theoretical Space
• Online Error Elicitation– Translation Correction Tool
• Rule Refinement Framework
• MT Evaluation Framework
• Conclusions
![Page 22: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/22.jpg)
TCTool (Demo)• Add a word• Delete a word• Modify a word• Change word order
Actions:
Interactive elicitation of error information
![Page 23: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/23.jpg)
Expanding the lexicon
1. OOV words/idoms Hamas cabinet calls a truce to avoid Israeli retaliation
El gabinete de Hamas llama un *TRUCE* para evitar la venganza israelí …acuerda una tregua…
2. New Word FormThe children fell *los niños cayeron
los niños se cayeron
3. New Word SenseThe girl plays the guitar *la muchacha juega la guitarra
la muchacha toca la guitarra
![Page 24: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/24.jpg)
Refining the grammar
4. Add Agreement constraints I see the red car *veo el coche roja veo el coche rojo
5. Add Word or ConstituentYou saw the woman viste la mujer viste a la mujer
6. Move Constituent Order I saw you yo vi te yo te vi
![Page 25: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/25.jpg)
Eng2Spa User Study [LREC 2004]
• MT error classification 9 linguistically-motivated classes [Flanagan, 1994], [White et al. 1994]:
word order, sense, agreement error (number, person, gender, tense), form, incorrect word and no translation
Interactive elicitation of error information
precision recall
error detection 90% 89%
error classification 72% 71%
![Page 26: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/26.jpg)
Outline• Introduction
• Theoretical Space
• Online Error Elicitation
• Rule Refinement Framework– RR operations– Formalizing Error information– Refinement Steps (Example)
• MT Evaluation Framework
• Conclusions
![Page 27: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/27.jpg)
1. Refine a translation rule:R0 R1 (change R0 to make it more
specific or more general)
Types of Refinement Operations
Automatic Rule Adaptation
R0:
R1:
NP
DET N ADJ
NP
DET ADJ N
a nice house
una casa bonito
NP
DET N ADJ
NP
DET ADJ N
a nice house
una casa bonita
N gender = ADJ gender
![Page 28: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/28.jpg)
1. Refine a translation rule:R0 R1 (change R0 to make it more
specific or more general)
Types of Refinement Operations
Automatic Rule Adaptation
R0:
R1:
NP
DET N ADJ
NP
DET ADJ N
a nice house
una casa bonito
NP
DET N ADJ
NP
DET ADJ N
a nice house
una casa bonita
N gender = ADJ gender
![Page 29: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/29.jpg)
2. Bifurcate a translation rule:R0 R0 (same, general rule)
Types of RefinementOperations (2)
Automatic Rule Adaptation
R0: NP
DET N ADJ
NP
DET ADJ N
a nice house una casa bonita
NP
DET ADJ N
NP
DET ADJ N
R1:
a great artist un gran artista
ADJ type: pre-nominal
![Page 30: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/30.jpg)
2. Bifurcate a translation rule:R0 R0 (same, general rule)
R1 (add a new more specific rule)
Types of RefinementOperations (2)
Automatic Rule Adaptation
R0: NP
DET N ADJ
NP
DET ADJ N
a nice house una casa bonita
NP
DET ADJ N
NP
DET ADJ N
R1:
a great artist un gran artista
ADJ type: pre-nominal
![Page 31: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/31.jpg)
Wi = error
Wi’ = correction
Wc = clue word
Automatic Rule Adaptation
NP
DET N ADJ
NP
DET ADJ N
a nice house
una casa bonito
NP
DET N ADJ
NP
DET ADJ N
a nice house
una casa bonita
N gender = ADJ gender
Wi = bonito
Wi’ = bonita
Wc = casa
Formalizing Error Information
![Page 32: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/32.jpg)
Triggering Feature DetectionComparison at the feature level to detect
triggering feature(s)
Delta function: (Wi,Wi’)
Examples:(bonito,bonita) = {gender}
(comiamos,comia) = {person,number}(mujer,guitarra) = {}
If set is empty, need to postulate a new binary feature (feat_i = {+,−})
Automatic Rule Adaptation
gen = masc gen = fem
![Page 33: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/33.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Features
Blame Assignment
Rule Refinement
![Page 34: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/34.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Features
Blame Assignment
Rule Refinement
![Page 35: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/35.jpg)
SL: Gaudí was a great artist
TL: Gaudí era un artista grande
1. Error Correction Elicitation
Wi = grande Edit Word
Change Word Order
Wi’ = gran
CTL: Gaudí era un gran artista
![Page 36: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/36.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Feature
Blame Assignment
Rule Refinement
1. Edit: Wi = grande Wi’ = gran
2. Change Word Order: artista gran gran artista
![Page 37: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/37.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Features
Blame Assignment
Rule Refinement
1. Edit: Wi = grande Wi’ = gran
2. Change Word Order: artista gran gran artista
![Page 38: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/38.jpg)
ADJ
[great] [grande]
agr num = sg
agr gen = masc
ADJ
[great] [gran]
agr num = sg
agr gen = masc
Delta function difference at the feature level?
(grande, gran) =
2. Finding Triggering Features
![Page 39: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/39.jpg)
ADJ
[great] [grande]
agr num = sg
agr gen = masc
ADJ
[great] [gran]
agr num = sg
agr gen = masc
Delta function difference at the feature level?
(grande, gran) =
need to postulate a new binary feature: feat1
2. Finding Triggering Features
![Page 40: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/40.jpg)
ADJ
[great] [grande]
agr num = sg
agr gen = masc
ADJ
[great] [gran]
agr num = sg
agr gen = masc
Delta function difference at the feature level?
(grande, gran) =
need to postulate a new binary feature: feat1
[feat1 = +] = [type = pre-nominal]
[feat1 = −] = [type = post-nominal]
2. Finding Triggering Features
![Page 41: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/41.jpg)
2. Finding Triggering Features
ADJ
[great] [grande]
agr num = sg
agr gen = masc
ADJ
[great] [gran]
agr num = sg
agr gen = masc
Delta function difference at the feature level?
(grande, gran) = new binary feature: feat1
ADJ
[great] [grande]
agr num = sg
agr gen = masc
feat1 = −
ADJ
[great] [gran]
agr num = sg
agr gen = masc
feat1 = +
REFINE
![Page 42: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/42.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Features
Blame Assignment
Rule Refinement
1. Edit: Wi = grande Wi’ = gran
2. Change Word Order: artista gran gran artista
grande [feat1 = −]
gran [feat1 = +]
![Page 43: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/43.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Features
Blame Assignment
Rule Refinement
1. Edit: Wi = grande Wi’ = gran
2. Change Word Order: artista gran gran artista
grande [feat1 = −]
gran [feat1 = +]
![Page 44: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/44.jpg)
(from transfer and generation tree)
tree: <( S,1 (NP,2 (N,5:1 "GAUDI") )
(VP,3 (VB,2 (AUX,17:2 "ERA") )
(NP,8 (DET,0:3 "UN")
(N,4:5 "ARTISTA")
(ADJ,5:4 "GRANDE"))) )>
3. Blame Assignment
S,1
…
NP,1
…
NP,8
…
Grammar
![Page 45: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/45.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Features
Blame Assignment
Rule Refinement
1. Edit: Wi = grande Wi’ = gran
2. Change Word Order: artista gran gran artista
grande [feat1 = −]
gran [feat1 = +]
NP,8
N ADJ ADJ N
![Page 46: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/46.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Features
Blame Assignment
Rule Refinement
1. Edit: Wi = grande Wi’ = gran
2. Change Word Order: artista gran gran artista
grande [feat1 = −]
gran [feat1 = +]
NP,8
N ADJ ADJ N
![Page 47: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/47.jpg)
4. Rule Refinement
NP
DET ADJ N
NP
DET N ADJ
NP,8:
a great artist un artista gran
NP
DET ADJ N
NP
DET ADJ N
NP,8’:
a great artist un gran artista
ADJ feat1 =c +
BIFURCATE
REFINE
![Page 48: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/48.jpg)
4. Rule Refinement
NP
DET ADJ N
NP
DET N ADJ
NP,8:
a great artist un artista gran
NP
DET ADJ N
NP
DET ADJ N
NP,8’:
a great artist un gran artista
ADJ type = pre-nominal
![Page 49: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/49.jpg)
Refinement Steps
Error Correction Elicitation
Finding Triggering Features
Blame Assignment
Rule Refinement
1. Edit: Wi = grande Wi’ = gran
2. Change Word Order: artista gran gran artista
grande [feat1 = −]
gran [feat1 = +]
NP,8
(N ADJ ADJ N)
NP,8 ADJ N N ADJ
NP,8’ ADJ N ADJ N [ADJ feat 1 =c +]
![Page 50: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/50.jpg)
Correct Translation Output
NP,8 ADJ(great-grande) [feat1 = -]
NP,8’ ADJ(great-gran)
[ADJ feat1 =c +] [feat1 = +]
Gaudi era un artista grande
Gaudi era un gran artista
*Gaudi era un grande artista
![Page 51: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/51.jpg)
Done? Not yetNP,8 (R0) ADJ(grande)
[feat1 = -]
NP,8’ (R1) ADJ(gran)
[feat1 =c +] [feat1 = +]
Need to restrict application of general rule (R0) to just post-nominal ADJ
un artista grande
un artista gran
un gran artista
*un grande artista
![Page 52: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/52.jpg)
Add Blocking ConstraintNP,8 (R0) ADJ(grande) [feat1 = -] [feat1 = -]
NP,8’ (R1) ADJ(gran)
[feat1 =c +] [feat1 = +]
Can we also eliminate incorrect translations automatically?
un artista grande
*un artista gran
un gran artista
*un grande artista
![Page 53: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/53.jpg)
Making the grammar tighter
• If Wc = artista Add [feat1= +] to N(artista) Add agreement constraint to NP,8 (R0)
between N and ADJ ((N feat1) = (ADJ feat1))
*un artista grande
*un artista gran
un gran artista
*un grande artista
![Page 54: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/54.jpg)
Generalization Power: abstract feature (feat_i)
Irina is a great friend Irina es una gran amiga (instead of *Irina es una amiga
grande) NP
DET ADJ N
NP
DET ADJ N
NP,8’:
a great friend una gran amiga
NP
DET ADJ N
NP
DET ADJ N
a great person una gran persona
Juan is a great person Juan es una gran persona
(instead of *Juan es una persona grande) ADJ feat1 = +
ADJ feat1 = +
![Page 55: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/55.jpg)
Generalization Power++
When triggering feature already exists in the grammar/lexicon (POS, gender, number, etc.)
I see the red car *veo un auto roja ADJ gender = N gender
Refinements generalize to all lexical entries that have that feature (gender):
The yellow houses are his las casas amarillas son suyas (before: *las casas amarillos son suyas)
We need to go to a dark cave tenemos que ir a una cueva oscura
(before: *cueva oscuro)
gender = fem
gender = masc
gender = fem
gender = masc gender = fem
veo un auto rojoo
![Page 56: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/56.jpg)
Outline• Introduction
• Theoretical Space
• Online Error Elicitation
• Rule Refinement Framework
• MT Evaluation Framework– Relevant (and free) human reference translations– Not punished by BLEU, NIST and METEOR.
• Conclusions
![Page 57: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/57.jpg)
MT Evaluation Framework
Multiple Human Reference
Translations
relevant to specific MT system errors
Bilingual speakers
- MT systems not be punished for picking a different synonym or morpho-syntactic variation by BLEU and METEOR.
- Similar to what [Snover et al. 2006] propose (HTER).
![Page 58: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/58.jpg)
MT Evaluation continued
• Did the refined MT system generate translation as corrected by the user (CTL)? Simple “recall” {0=CTL not in output, 1=CTL in output}
• Did the number of bad translations (implicitly identified by users) generated by the system decrease?Precision at rank k (k=5 in TCTool)
• Did the system successfully managed to reduce ambiguity (number of alternative translations)? Reduction ratio
![Page 59: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/59.jpg)
Outline• Introduction
• Theoretical Space
• Online Error Elicitation
• Rule Refinement Framework
• MT Evaluation Framework
• Conclusions– Impact on MT systems– Work in Progress and Contributions – Future Work
![Page 60: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/60.jpg)
Impact on Transfer-Based MT
Learning
Module
Transfer Rules
Lexical Resources
Transfer System
Translation Candidate
Lattice
Rule Learning and other Resources
Run-Time System
Handcrafted rules
Morpho-logical analyzer
INPUT TEXT
OUTPUT TEXT
![Page 61: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/61.jpg)
Learning
Module
Transfer Rules
Lexical Resources
Transfer System
Translation Candidate
Lattice
Rule Learning and other Resources
Run-Time System
Handcrafted rules
Morpho-logical analyzer
INPUT TEXTOnline
Translation
Correction
Tool
Rule Refinement
Impact on Transfer-Based MT
![Page 62: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/62.jpg)
Learning
Module
Transfer Rules
Lexical Resources
Transfer System
Translation Candidate
Lattice
Rule Learning and other Resources
Run-Time System
Handcrafted rules
Morpho-logical analyzer
INPUT TEXTOnline
Translation
Correction
Tool
Rule Refinement
Impact on Transfer-Based MT
![Page 63: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/63.jpg)
Learning
Module
Transfer Rules
Lexical Resources
Transfer System
Translation Candidate
Lattice
Rule Learning and other Resources
Run-Time System
Handcrafted rules
Morpho-logical analyzer
INPUT TEXTOnline
Translation
Correction
Tool
Rule Refinement
Impact on Transfer-Based MT
![Page 64: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/64.jpg)
Learning
Module
Transfer Rules
Lexical Resources
Transfer System
Translation Candidate
Lattice
Rule Learning and other Resources
Run-Time System
Handcrafted rules
Morpho-logical analyzer
INPUT TEXTOnline
Translation
Correction
Tool
Rule Refinement
OUTPUT TEXT
Impact on Transfer-Based MT
![Page 65: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/65.jpg)
TCTool can help improve
• Rule-based MT (grammar, lexicon, LM)
• EBMT (examples, lexicon, alignments)
• Statistical MT (lexicon, and alignments)+ relevant annotated data
develop smarter training algorithmsPanel this afternoon 3:45-4:45pm
![Page 66: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/66.jpg)
Work in Progress
• Finalizing regression and test sets to perform rigorous evaluation of approach
• Handling incorrect Correction Instances– Have multiple users correct the same set of
sentences
filter out noise (threshold: 90% users agree)
• User study with multiple users
evaluate improvement after refinements
![Page 67: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/67.jpg)
Contributions so far
• An efficient online GUI to display translations and alignments and solicit pinpoint fixes from non-expert bilingual users.
• New Framework to improve MT quality: an expandable set of rule refinement operations
• MT Evaluation Framework, which provides relevant reference translations
![Page 68: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/68.jpg)
Future work
• Explore other ways to make the interaction with users more fun
• Games with a purpose [Von Ahn and Blum, 2004 & 2006]
![Page 69: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/69.jpg)
Future work
• Explore other ways to make the interaction with users more fun
• Games with a purpose [Von Ahn and Blum, 2004 & 2006]
• Second Language Learning
![Page 70: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/70.jpg)
Questions?
![Page 71: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/71.jpg)
Thanks!
MT + MT=
![Page 72: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/72.jpg)
Backup slides• TCTool next step • What if users corrections are different (user n
oise)?• More than one correction per sentence?
• Wc example
• Data set• Where is appropriate to refine vs bifurcate?• Lexical bifurcate• Is the refinement process invariable?
![Page 73: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/73.jpg)
Backup slides (ii)
• TCTool Demo Simulation
• RR operation patterns
• Automatic Evaluation feasibility study
• AMTA paper results
• User studies map
• Precision, recall, F1
• NIST, BLEU, METEOR
![Page 74: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/74.jpg)
Backup slides (iii)
• Done? Not yet… (blocking constraints++)• Minimal pair example• Batch mode implementation• Interactive mode implementation• User studies• Evaluation of refined output• Another Generalization Example• Avenue Architecture• Technical Challenges• Rules Formalism
![Page 75: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/75.jpg)
Player = The Teacher
• Goal: teach the MT system to translate correctly
• Different levels of expertise– Beginner: language learning and improving
(labeled data)– Intermediate-Advanced: provide labeled data
to improve the system and for beginner levels
MT
![Page 76: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/76.jpg)
Two players: cooperation
• In English to Spanish MT:
– Player 1: Spanish native speaker + learning English
– Player 2: English native speaker + learning Spanish
MT
back to mainback to main
![Page 77: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/77.jpg)
Translation rule example
{NP,8}
NP::NP : [DET ADJ N] -> [DET N ADJ]( (X1::Y1) (X2::Y3) (X3::Y2) ((x0 def) = (x1 def)) (x0 = x3) (y2 = x3) ((y1 agr) = (y2 agr)) ((y3 agr) = (y2 agr)))
![Page 78: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/78.jpg)
Translation rule example
{NP,8} SL side (English) TL side (Spanish) ;;X0 Y0 X1 X2 X3 Y1 Y2 Y3NP::NP : [DET ADJ N] -> [DET N ADJ]( (X1::Y1) (X2::Y3) (X3::Y2) ((x0 def) = (x1 def)) (x0 = x3) (y2 = x3) ((y1 agr) = (y2 agr)) ((y3 agr) = (y2 agr)))
Rule ID
;; analysis;; transfer;; transfer;; generation;; generation
![Page 79: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/79.jpg)
Translation rule example
{NP,8};;X0 Y0 X1 X2 X3 Y1 Y2 Y3 NP::NP : [DET ADJ N] -> [DET N ADJ]( (X1::Y1) (X2::Y3) (X3::Y2) ((x0 def) = (x1 def)) ; passing definiteness up (x0 = x3) ; X3 is the head of X0 (y2 = x3) ; pass features of head to
Y2 ((y1 agr) = (y2 agr)) ; det-noun agreement ((y3 agr) = (y2 agr)) ; adj-noun agreement) back to mainback to main
![Page 80: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/80.jpg)
Done? Not yetNP,8 (R0) ADJ(grande)
[feat1 = -]
NP,8’ (R1) ADJ(gran)
[feat1 =c +] [feat1 = +]
Need to restrict application of general rule (R0) to just post-nominal ADJ
Automatic Rule Adaptation
un artista grande
un artista gran
un gran artista
*un grande artista
![Page 81: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/81.jpg)
Add Blocking ConstraintNP,8 (R0) ADJ(grande) [feat1 = -] [feat1 = -]
NP,8’ (R1) ADJ(gran)
[feat1 =c +] [feat1 = +]
Can we also eliminate incorrect translations automatically?
Automatic Rule Adaptation
un artista grande
*un artista gran
un gran artista
*un grande artista
![Page 82: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/82.jpg)
Making the grammar tighter
• If Wc = artista Add [feat1= +] to N(artista) Add agreement constraint to NP,8 (R0)
between N and ADJ ((N feat1) = (ADJ feat1))
Automatic Rule Adaptation
*un artista grande
*un artista gran
un gran artista
*un grande artista
back to mainback to main
![Page 83: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/83.jpg)
Example Requiring Minimal Pair
Automatic Rule Adaptation
1. Run SL sentence through the transfer engine
I see them *veo los Correct TL: los veo
2. Wi = los but no Wi’ nor Wc
Need a minimal pair to determine appropriate refinement:
I see cars veo autos
3. Triggering feature(s): (veo los, veo autos)
(los,autos) = {pos}
PRON(los)[pos=pron] N(autos)[pos=n]
Proposed Work
back to mainback to main
![Page 84: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/84.jpg)
Avenue Architecture
Learning
Module
Learned Transfer
Rules
Lexical Resources
Run Time Transfer System
Decoder
Translation
Correction
Tool
Word-Aligned Parallel Corpus
Elicitation Tool
Elicitation Corpus
Elicitation Rule Learning
Run-Time System
Rule Refinement
Rule
Refinement
Module
Morphology
Morphology Analyzer
Learning Module Handcrafted
rules
INPUT TEXT
OUTPUT TEXT
back to mainback to main
![Page 85: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/85.jpg)
Technical Challenges
Elicit minimal MT information from non-expert users
Automatically Refine and Expand
Translation Rules minimally
Manually written Automatically Learned
Automatic Evaluation of Refinement process
back to mainback to main
![Page 86: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/86.jpg)
Batch Mode Implementation• Given a set of user corrections, apply
refinement module.
• For Refinement Operations of errors that can be refined fully automatically using:
1. Correction information only
2. Correction and error information
Proposed Work Automatic Rule Adaptation
error type, clue word
![Page 87: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/87.jpg)
Rule Refinement Operations
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
![Page 88: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/88.jpg)
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Rule Refinement Operations
1. Correction info only
It is a nice house – *Es una casa bonito Es una casa bonita
Gaudi was a great artist – *Gaudi era un artista grande Gaudi era un gran artista
![Page 89: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/89.jpg)
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Rule Refinement Operations
I am proud of you – *Estoy orgullosa de tu Estoy orgullosa de ti
PP PREP NP
2. Correction and Error info
back to mainback to main
![Page 90: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/90.jpg)
Interactive Mode Implementation
• Extra error information is required to determine triggering context automatically
Need to give other relevant sentences to the user at run-time (minimal pairs)
• For Refinement Operations of errors that can be refined fully automatically but:
3. require a further user interaction
Proposed Work Automatic Rule Adaptation
![Page 91: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/91.jpg)
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Rule Refinement Operations
3. Further user interaction
I see them – *Veo los Los veo
back to mainback to main
![Page 92: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/92.jpg)
Refining and Adding ConstraintsVP,3: VP NP VP NP (veo los, veo autos)
VP,3’: VP NP NP VP + [NP pos =c pron](los veo, *autos veo)
• Percolate triggering features up to the constituent level:
NP: PRON PRON + [NP pos = PRON pos]
• Block application of general rule (VP,3):
VP,3: VP NP VP NP + [NP pos = (*NOT* pron)]
*veo los, veo autos (los veo, *autos veo)
Proposed Work
![Page 93: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/93.jpg)
User Studies• TCTool: new MT classification (Eng2Spa)
• Different language pair Mapudungun or Quechua Spanish
• Batch vs Interactive mode
• Amount of information elicitedjust corrections vs + error information
Proposed Work
back to mainback to main
![Page 94: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/94.jpg)
Evaluation of Refined MT Output
1. Evaluate best translation Automatic evaluation metrics (BLEU, NIST, METEOR)
2. Evaluate translation candidate list size precision (includes parsimony)
![Page 95: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/95.jpg)
1. Evaluate Best translationHypothesis file (translations to be evaluated
automatically)
Raw MT output:– Best sentence (picked by user to be correct or
requiring the least amount of correction) Refined MT output:– Use METEOR score at sentence level to pick best
candidate from the list
Run all automatic metrics on the new hypothesis file using user corrections as reference translations.
![Page 96: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/96.jpg)
2. Evaluate Translation Candidate List
• Precision: tp binary {0,1} (1 =user correction)
tp + fp total number of TLs
SLTL XXTL XXTL XX
SLTLTL XXTL XXTL XXTL XX
SLTLTL XXTL XXTL XXTL
SLTLTL XXTL XX
=
0/30/3 1/51/5 1/51/5 1/31/3
user correction
SLTLTL XX
= 1/21/2back to mainback to main
![Page 97: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/97.jpg)
Generalization PowerWhen triggering feature already exists in
the feature language (pos, gender, number, etc.)
I see them *veo los los veo
- I love him lo amo (before: *amo lo)
- They called me yesterday me llamaron ayer (before: *llamaron me ayer)
- Mary helps her with her homework
Maria le ayuda con sus tareas (before: *Maria ayuda le con sus tareas)
back to mainback to main
![Page 98: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/98.jpg)
Precision, Recall and F1
• Precision: tp
tp + fp (selected, incorrect)
• Recall: tp
tp + fn (correct, not selected)
• F1: 2 PR
(P + R)back to main
![Page 99: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/99.jpg)
Automatic Evaluation Metrics
• BLEU: averages the precision for unigram, bigram and up to 4-grams and applies a length penalty [Papineni, 2001].
• NIST: instead of n-gram precision the information gain from each n-gram is taken into account [NIST 2002].
• METEOR: assigns most of the weight to recall, instead of precision and uses stemming [Lavie, 2004] back to main
![Page 100: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/100.jpg)
Data Set
• Split development set (~400 sentence) into: – Dev set Run User Studies
Develop Refinement Module Validate functionality
– Test set Evaluate effect of Refinement operations
• + Wild test set (from naturally occurring text)
Requirement: need to be fully parsed by grammar
Proposed Work
back to questionsback to questions
![Page 101: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/101.jpg)
Refine vs Bifurcate
• Batch mode bifurcate (no way to tell if the original rule should never apply)
• Interactive mode refine (change original rule) if can get enough evidence that original rule never applies.
• Corrections involving agreement constraints seem to hold for all cases refine (Open research question)
back to questionsback to questions
![Page 102: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/102.jpg)
More than one correction/sentence
A B TL X X
A 1st X
B 2nd X
Tetris approach to Automatic Rule Refinement
Assumption: different corrections to different words different error back to questionsback to questions
![Page 103: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/103.jpg)
Exception: structural divergencesHe danced her out of the room
* La bailó fuera de la habitación
her he-danced out of the room
La sacó de la habitación bailando
her he-take-out of the room dancing
Have no way of knowing that these corrections are related
Do one error at a time, if TQ decreases over the test set, hypothesize that it’s a divergence Feed to the Rule Learner as a new (manually corrected)
training exampleback to questions
![Page 104: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/104.jpg)
Constituent order change
I gave him the tools *di a él las herramientas
I-gave to him the tools
le di las herramientas a él him I-gave the tools to him
desired refinement:VPVP PP(a PRON) NP VPVP NP PP(a PRON)
Can extract constituent information from MT output (parse tree) and treat as one error/correction
![Page 105: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/105.jpg)
More than one correction/error
A+B
TL X
Example: edit and move same word (like: gran, bailó)
Occam’s razor
Assumption: both corrections are part of the same error
back to questions
![Page 106: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/106.jpg)
Wc ExampleI am proud of you
*Estoy orgullosa de tu Wc=de tuti
I-am proud of you-nom
Estoy orgullosa de tiI-am proud of you-oblic
Without Wc information, would need to increase the ambiguity of the grammar significantly!
+[you ti]: I love you *ti quiero (te quiero,…)
you read *ti lees (tu lees,…)
back to questionsback to questions
![Page 107: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/107.jpg)
Lexical bifurcate
• Should the system copy all the features to the new entry?– Good starting point– Might want to copy just a subset of features
(possibly POS-dependent)
Open Research Question
back to questionsback to questions
![Page 108: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/108.jpg)
Automatic Rule Adaptation
![Page 109: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/109.jpg)
Automatic Rule AdaptationSL + best TL picked by user
![Page 110: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/110.jpg)
Automatic Rule AdaptationChanging “grande” into “gran”
![Page 111: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/111.jpg)
Automatic Rule Adaptation
back to main
![Page 112: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/112.jpg)
1
2
3
Automatic Rule Adaptation
![Page 113: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/113.jpg)
Input to RR module
sl: I see them
tl: VEO LOS
tree: <((S,0 (VP,3 (VP,1 (V,1:2 "VEO") )
(NP,0 (PRON,2:3 "LOS") ) ) ) )>
- User correction log file - Transfer engine output (+ parse tree):
Automatic Rule Adaptation
sl: I see cars
tl: VEO AUTOS
tree: <((S,0 (VP,3 (VP,1 (V,1:2 "VEO") )
(NP,2 (N,1:3 “AUTOS") ) ) ) )>back to main
![Page 114: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/114.jpg)
Types of RR Operations• Grammar:
– R0 R0 + R1 [=R0’ + constr] Cov[R0] Cov[R0,R1]– R0 R1[=R0 + constr= -]
R2[=R0’ + constr=c +] Cov[R0] Cov[R1,R2]
– R0 R1 [=R0 + constr] Cov[R0] Cov[R1]
• Lexicon– Lex0 Lex0 + Lex1[=Lex0 + constr] – Lex0 Lex1[=Lex0 + constr]– Lex0 Lex0 + Lex1[Lex0 + TLword] Lex1 (adding lexical item)
bifurcate
refine
Completed Work Automatic Rule Adaptation
back to main
![Page 115: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/115.jpg)
Manual vs Learned Grammars[AMTA 2004]
Automatic Rule Adaptation
NIST BLEU METEOR
Manual grammar 4.3 0.16 0.6
Learned grammar 3.7 0.14 0.55
• Manual inspection:
• Automatic MT Evaluation:
back to main
![Page 116: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/116.jpg)
Human Oracle experiment• As a feasibility experiment, compared
raw output with manually corrected MT:
statistically significant (confidence interval test)
• These is an upper-bound on how much difference we should expect any refinement approach to make.
Automatic Rule Adaptation
Completed Work
back to main
![Page 117: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/117.jpg)
Invariable process?• Rule Refinement process is not invariable
Order of the corrected sentences input to the system matters
• Example:1st: gran artista bifurcate (2 rules)2nd: casa bonito add agr constraint to only 1 rule
(original, general rule)
the specific rule is still incorrect (missing agr constraint)
1st: casa bonito add agr constraint 2nd: gran artista bifurcate
both rules have agreement constraint (optimal order)
back to questionsback to questions
![Page 118: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/118.jpg)
User noise?
Solution:
Have several users evaluate and correct the same test set
threshold, 90% agreement
- correction
- error information (type, clue word)
Only modify the grammar if enough evidence of incorrect rule.
back to questions
![Page 119: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/119.jpg)
User Studies Map
RR moduleManual grammars
Learned grammars
Eng2spa
Only Corrections
Corrections+error info
Active learning
Batch mode
interactive mode
Proposed Work
XX
XX
Mapu2SpaXX
back to main
![Page 120: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/120.jpg)
Recycle corrections of Machine Translation output back into the
system by refining and expanding
existing translation rules
![Page 121: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/121.jpg)
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Rule Refinement Operations
1. Correction info only
It is a nice house – Es una casa bonito Es una casa bonita
John and Mary fell – Juan y Maria cayeron Juan y Maria se cayeron
Gaudi was a great artist – Gaudi era un artista grande Gaudi era un gran artista
![Page 122: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/122.jpg)
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Rule Refinement Operations
Gaudi was a great artist – Gaudi era un artista grande Gaudi era un gran artista
Es una casa bonito Es una casa bonita
J y M cayeron J y M se cayeron
I will help him fix the car – Ayudaré a él a arreglar el auto
Le ayudare a arreglar el auto
1. Correction info only
![Page 123: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/123.jpg)
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Rule Refinement Operations
I will help him fix the car – Ayudaré a él a arreglar el auto
Le ayudare a arreglar el auto
I would like to go – Me gustaria que ir
Me gustaria ir
1. Correction info only
![Page 124: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/124.jpg)
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Rule Refinement Operations
I am proud of you – Estoy orgullosa tu Estoy orgullosa de ti
PP PREP NP
2. Correction and Error info
![Page 125: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/125.jpg)
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Rule Refinement Operations
Focus 3
Wally plays the guitar – Wally juega la guitarra Wally toca la guitarra
I saw the woman – Vi la mujer Vi a la mujer
I see them – Veo los Los veo
![Page 126: Automating Post-Editing To Improve MT Systems Ariadna Font Llitjós and Jaime Carbonell APE Workshop, AMTA – Boston August 12, 2006](https://reader035.vdocuments.site/reader035/viewer/2022062806/5697bfc01a28abf838ca3fa8/html5/thumbnails/126.jpg)
Rule Refinement Operations
Modify Add Delete Change W Order
+Wc –Wc +Wc –Wc +al –al Wi Wc Wi(…) Wc –Wc
= +rule –rule +al –al =Wi WiWi’ RuleLearner
POSi=POSi’ POSiPOSi’
Outside Scope of Thesis
John read the book – A Juan leyó el libro Juan leyó el libro
Where are you from? – Donde eres tu de? De donde eres tu?