who am i?post-editing effort post-editing effort is calculated by comparing the suggestion from the...
TRANSCRIPT
![Page 1: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/1.jpg)
![Page 2: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/2.jpg)
Who am I?
VP Operations at Translated
Project Coordinator at ModernMT
Product Manager at MateCat
![Page 3: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/3.jpg)
Overview
• Phrase-Based Machine Translation vs Neural Machine Translation
• Some key differences (training data, phrases vs sentences, generalisation vs specialisation, learning vs memorisation)
• Analysis (error annotation, quality rating, post-editing effort)
• Conclusions (how to use the data)
![Page 4: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/4.jpg)
PBMT vs NMT
![Page 5: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/5.jpg)
Phrase-Based Machine Translation
![Page 6: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/6.jpg)
Phrase-Based Machine Translation
In a PBMT system, the building blocks (phrases) of
a sentence are deconstructed and their
translations are recombined to form a new
sentence in the target language.
![Page 7: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/7.jpg)
Phrase extraction
Credits: Marcello Federico, Head of HLT-MT Unit at FBK
![Page 8: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/8.jpg)
Translation and Language Model
Phrases from parallel texts are stored in a translation model and retrieved as if they were units from a translation memory.
Phrases are recombined to form the target sentence and reordered based on the samples in the language model.
![Page 9: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/9.jpg)
Translation and reordering
Credits: Marcello Federico, Head of HLT-MT Unit at FBK
![Page 10: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/10.jpg)
Neural Machine Translation
![Page 11: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/11.jpg)
Neural Machine Translation
PBMT memorises translation fragments and uses them as building blocks to compose new translations.
NMT learns and performs translation through an encoding-decoding process which converts source words into a numeric representation, from which it then generates the corresponding target words.
![Page 12: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/12.jpg)
Phrases to Gradients
Credits: Marcello Federico, Head of HLT-MT Unit at FBK
![Page 13: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/13.jpg)
Encoding - Decoding
Credits: Adam Geitgey - https://medium.com/@ageitgey/machine-learning-is-fun-part-5-language-translation-with-deep-learning-and-the-magic-of-sequences-2ace0acca0aa
![Page 14: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/14.jpg)
Feed-Forward Neural Networks
Credits: Marcello Federico, Head of HLT-MT Unit at FBK
![Page 15: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/15.jpg)
Key Differences
![Page 16: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/16.jpg)
Training Data
PBMT:
● Monolingual data: 2-10B words
● Parallel data: 1B
NMT:
● Monolingual data: no need
● Parallel data: 100M (higher quality)
● Impossible to retrieve content from models (privacy)
![Page 17: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/17.jpg)
Phrases vs Sentences
PBMT breaks the original text into phrases and retrieves their translations from the translation model, looking first for longer phrases and then shorter.
NMT takes in entire original sentences and encodes them into a numeric representation and decodes this representation into a target sentence.
![Page 18: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/18.jpg)
Generalisation vs Specialisation
NMT outperforms PBMT in most cases, especially on texts that differ from the training data.
PBMT still is more effective when the content to translate is similar to the training data (e.g. custom engines for a specific product or customer).
![Page 19: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/19.jpg)
Learning vs Memorization
Credits: Marcello Federico, Head of HLT-MT Unit at FBK
![Page 20: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/20.jpg)
Analysis
![Page 21: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/21.jpg)
Data Collection
Data from projects translated by Translated.net in MateCat.
MT system used is Google Translate (API) both for PBMT and NMT.
Data collected before and after April 2017 (when GNMT was introduced).
![Page 22: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/22.jpg)
Language Pairs
● German to English
● English to French
● English to Italian
● English to German
● English to Portuguese
● English to Spanish
![Page 23: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/23.jpg)
Methodology
● Qualitative Analysis
○ Error Annotation
○ Quality Rating
● Quantitative Analysis
○ Post-Editing Effort
![Page 24: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/24.jpg)
Error Annotation
![Page 25: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/25.jpg)
Error Annotation
● Task: Annotate errors found in the raw MT output from PBMT and NMT
● 100 segments per language pair● Randomised data so that the translators didn’t
know which system generated the translation● Four translators per language pair, each
annotated all segments● Seven error categories
![Page 26: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/26.jpg)
Error Annotation - CategoriesGrammar Any issues which affect language quality (e.g. morphology, word order,
concordance etc.).
Mistranslation The translation does not carry the same meaning as the source sentence.
Omission Information from the original sentence is missing in the translation.
Spelling Spelling or typographical errors.
Style Linguistic issues which make the sentence sound awkward in the target language.
Terminology Translation is correct, but the terminology is not adequate for the context.
None No errors detected.
![Page 27: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/27.jpg)
Error AnnotationResults by language pair
![Page 28: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/28.jpg)
![Page 29: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/29.jpg)
Error AnnotationOverall
![Page 30: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/30.jpg)
![Page 31: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/31.jpg)
Quality Rating
![Page 32: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/32.jpg)
Quality Rating
● Task: Select which raw MT output is easier to post-edit to get to a high quality translation.
● 100 segments per language pair● Randomised data so that the translators didn’t
know which system generated the translation● Four translators per language pair, each
evaluated all segments
![Page 33: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/33.jpg)
![Page 34: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/34.jpg)
Post-Editing Effort
![Page 35: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/35.jpg)
Post-Editing Effort - A definition
Post-Editing Effort: the percentage of edits
required to modify the suggestions provided by
the MT system in order to get to a good quality
translation.
![Page 36: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/36.jpg)
Post-Editing Effort
Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation.
The function used to calculate the post-editing effort compares the average number of steps required to edit the suggestion in order to produce a professional translation. These steps could either be changing synonyms, correcting numbers or casing, adjusting punctuation, changing the tag positions, etc.
![Page 37: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/37.jpg)
Post-Editing Effort
Changing synonyms, correcting numbers or casing, adjusting punctuation, changing the tag positions, etc. all have different weights which have been calculated to reflect the effort required to correct each of these issues.
![Page 38: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/38.jpg)
Post-Editing Effort - Examples
Suggestion Translation Post-Editing Effort
hi! Hi! 2%
hi all! Hi all! 2%
tests experiments 100%
Long tests Long experiments 50%
![Page 39: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/39.jpg)
Post-Editing Effort & Productivity
Post-Editing Effort shows a good inverse correlation with translators’ productivity. The better the MT quality, the lower the post-editing effort and the higher the throughput of translators.
![Page 40: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/40.jpg)
Post-Editing Effort - Analysis
We collected data on the Post-Editing Effort for the six language pairs over a period of 18 months and noted the impact due to the introduction of GNMT.
![Page 41: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/41.jpg)
EN-FR (GNMT: April 6, 2017)
EN-ES (GNMT: April 6, 2017)
EN-DE (GNMT: April 6, 2017)
EN-IT (GNMT: April 27, 2017)
EN-PT (GNMT: April 6, 2017)
DE-EN (GNMT: April 6, 2017)
![Page 42: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/42.jpg)
What do I need this for?
![Page 43: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/43.jpg)
Negotiating Rates
Transparency on the actual benefits of MT help set a level playing field with clients and vendors.
Our experience: Clients, account/project managers and translators work on the same platform and have access to the same data on productivity. This makes it easier to negotiate rates.
![Page 44: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/44.jpg)
Turn-Around Times
Post-Editing Effort correlates with the daily throughput of translators.
Our experience: We use post-editing effort together with time to edit to estimate the daily productivity of translators and use that to dynamically assess the required TAT.
![Page 45: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/45.jpg)
Increasing Quality
Machine translation can boost productivity but may lead translators to output sub-optimal translations. Post-editing effort helps to evaluate the final quality.
Our experience: Post-editing effort is one of the quality metrics that we use to identify good translators. Translators with higher post-editing effort rates are preferred.
![Page 46: Who am I?Post-Editing Effort Post-Editing Effort is calculated by comparing the suggestion from the MT system with the final translation. The function used to calculate the post-editing](https://reader033.vdocuments.site/reader033/viewer/2022041504/5e23d14fc4be562c3d37990c/html5/thumbnails/46.jpg)
Thank youAlessandro Cattelan - [email protected]