Automatic and Semi-Automatic Evaluation Techniques

Maribel Rodríguez, Language Technology Group Deployment and Development Manager

TAUS Quality Evaluation SummitJune 2016

CONFIDENTIAL

Evaluation Techniques

Semi/Automated EvaluationMachine LearningHuman Evaluation

CONFIDENTIAL

Combining Human & Automated Evaluation• Reduce the scope of Human

Review and focus on what the linguist can do best

• Automated QA Tools and Human Evaluation adjusted to different quality threshold/KPIs

• Keep Data Parameters

CONFIDENTIAL

Combining Content Profiling and Automated Evaluation• Automated selection of a more

relevant sample for human evaluation

• Estimate the scope requiring human evaluation or post-editing

• Help to automate a workflow and assign content based on perplexity to suitable resources

Parting Questions:

• What interesting data can you collect right now for automated quality evaluation purposes?

• How can you take advantage of that data to reduce cost or drive efficiency?