toin - taus tokyo forum 2015
TRANSCRIPT
-
Criteria for Setting MT Project Unit Prices
TOIN Corporation Takatsugu Kato
Masaru Kawahara
-
Factors that Determine MT Fees
Objective for Using MTLevel of PEMT Quality
-
Decide: Objective for
using translation?
Targetdocument
Assess based on MT evaluation criteria
Can your objective be served by MT
alone?
MT unit price(Level 1)
Assess based on Light PE evaluation criteria
Light or
Full?
Assess based on Full PE evaluation criteria
Light PE unit price
(Level 5)
Full PE unit price
(Level 5)
Yes
No
-
Decide What Your Objective Is I want instant translations and/or to minimize fees MT only I want as close to human translation quality as possible MT+PE
-
MT Translation QualityMT quality is inconsistent, it diers according to various factors. Corpus volume Quality of source les Type of MT engine etc.
Perform sample translation and evaluate MT translation
-
Without evaluating MT... If MT output is good quality without PE, then PE is a useless extra cost Doing PE on low-quality MT translations increases the burden on Post-editors etc.
With a proper evaluation... You get MT+PE at a reasonable unit price A reasonable unit price keeps the client, the LSP, and Post-editors happy! etc.
-
Evaluating Quality when Using MT Alone With MT alone there is no human intervention, so unit price is the same However, to decide whether MT translation alone will serve your purpose requires an evaluation
-
How to Evaluate MT Output
Criterion DescriptionGood - MT translation is perfectly
understandable as-is- Some mistakes (roughly, up to 1 per 10 words), but captures the core meaning of the source
Poor Anything other than Good
MT evaluation criteria
Apply the following criteria to each segment, and divide them into Good and Poor. Then evaluate the percentage of Good.
-
Number of Segments to Evaluate Evaluating the entire target document takes too much time, so use basic statistical techniques to get samples, and calculate the number of segments
n = N/[(/())2*{(N-1)/(1-)}+1]No. of parameters (N)Precision ()Reliability (())Statistical percentage ()
Parameters Samples100 80
200 132
500 217
1000 278
5000 357
10000 370
100000 382
-
Set PE LevelLevel of post-editing Quality criteriaLight PE Accuracy
Full PE Accuracy+Readability+Style
There are two guidelines for PE level
-
PE Guidelines
-
PE evaluation criteriaCriterion DescriptionGood - MT translation can be used as-is,
without editing- Following items need small edits (roughly, up to 1 per 10 words):Word order (not including editing of particles/usage), placement, additions, deletions
Poor Anything other than Good
How to Evaluate MT Output Same as MT alone However, evaluation criteria are dierent
-
Variety of Evaluation CriteriaUse dierent evaluation criteria to suit your ultimate objective for using MT
When using PE
Use PE evaluation criteria Quantify how much eort it will take for PE Sliding unit price depending on evaluation result (Level 5*2)
When using MT alone Use MT evaluation criteria
What level of quality does MT output represent as-is?
-
Why We Use Separate Evaluation Criteria PE evaluation criteria measure how much eort will be required for PE with MT translation MT evaluation criteria measure how accurate and uent MT translation is
Each of these "rulers" is dierent
-
Concrete ExampleSource: You will then be able to use business rule sets and rules to validate that the objects are valid to be processed.
Correct translation:
MT translation PE criterion MT criterion
Just requires changing to armative sentence to match the blue text:Good
The meaning is opposite, so:Poor
-
Table of Unit Prices Set the unit price according to the percentage of Good in the target document (where HT is 100%)
% of Good Light PE Full PE71% or higher 30% 60%
61% to 70% 40% 70%
51% to 60% 50% 80%
31% to 50% 60% 90%
30% or lower 70% 100%
-
TOIN Corporation Takatsugu Kato
Masaru Kawahara