trava – a tool for evaluating machine translation – pedagogical and research possibilities...
TRANSCRIPT
![Page 1: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/1.jpg)
TrAva – a tool for evaluating Machine Translation –
pedagogical and research possibilities
Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro -
Linguateca
![Page 2: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/2.jpg)
Why MT Matters
• Social reasons
• Political reasons
• Commercial importance
• Scientific and philosophical interest
![Page 3: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/3.jpg)
Useful bibliography
• ARNOLD, D, BALKAN, L., LEE HUMPHREYS, R. Lee, MEIJER, S., & SADLER, S. (1994) Machine Translation - An Introductory guide. Manchester & Oxford : NCC Blackwell. ISBN 1-85554-217-X - or at: http://www.essex.ac.uk/linguistics/clmt/MTbook/.
• COLE, Ron (ed) 1996 "Survey of the State of the Art in Human Language Technology" Chapter 8 - Multilinguality, at the Center for Spoken Language Understanding, Oregon. http://cslu.cse.ogi.edu/HLTsurvey/ch8node2.html#Chapter8
• MELBY, Alan K. 1995, The Possibility of Language: A discussion of the Nature of Language, with implications for Human and Machine Translation. Amsterdam: John Benjamins Pub. Co.
![Page 4: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/4.jpg)
Machine Translation (MT) – a few dates
• 1947 Warren Weaver - his ideas led to heavy investment in MT
• 1959 Bar-Hillel - considers FAHQMT - FULLY AUTOMATIC HIGH QUALITY MACHINE TRANSLATION philosophically impossible
• 1964 ALPAC Report - on limitations of MT > withdrawal of funds
• late 1970s the CEC purchase of SYSTRAN and beginning of EUROTRA project.
• Upward trend in the 1970s and 1980s • Today: MT technology - high-end versus low-end
systems • MT and the Internet
![Page 5: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/5.jpg)
From Arnold et al 1995
![Page 6: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/6.jpg)
From Arnold et al 1995
![Page 7: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/7.jpg)
![Page 8: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/8.jpg)
MT architectures – Arnold et al
• Direct architecture - simple grammatical rules + a large lexical and phrasal database
• Transfer architecture - more complex grammar with an underlying approach of transformational-generative theory + considerable research into comparative linguistics in the two languages involved
• Interlingua architecture - L1 > a 'neutral language' (real, artificial, logical, mathematical..) > L2
![Page 9: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/9.jpg)
Major Methods, Techniques and Approaches today
• Statistical vs. Linguistic MT – assimilation tasks: lower quality, broad domains – statistical
techniques predominate– dissemination tasks: higher quality, limited domains – symbolic
techniques predominate– communication tasks: medium quality, medium domain – mixed
techniques predominate
• Rule-based vs. Example-based MT • Transfer vs. Interlingual MT • Multi-Engine MT • Speech-to-Speech Translation
MLIM - Multilingual Information Management: Current levels and Future Abilities - report (1999) Chapter 4 at: http://www-2.cs.cmu.edu/~ref/mlim/chapter4.html
![Page 10: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/10.jpg)
MT – present & future uses
• ‘Gist’ translation• Ephemeral texts with tolerant users• Human aided MT
– Domain specific– Linear sentence structure– Pre-edited text or ‘controlled language’– Post-editing
• Improvement of MT – particularly for restricted domains and registers
![Page 11: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/11.jpg)
MT and the Human translator
• MT is less of a threat to the professional human translator than English
• MT can encourage people’s curiosity for texts in languages they do not understand > and lead to human translation
• MT can be a tool for the human translator
• Professional translators can learn to work with and train MT
![Page 12: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/12.jpg)
PoloCLUP’s experiment
• Background– Master’s seminar in Semantics and Syntax– Wish to raise students’ awareness of the
strengths and weaknesses of MT– Wish to develop their interest in MT as a tool– Need to improve their knowledge of
linguistics. – Availability of free MT online– Automation of process provided by computer
engineer
![Page 13: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/13.jpg)
Phase 1 - METRA
• http://poloclup.linguateca.pt/ferramentas/metra/index.html
• Translation using 7 online MT programmes
• EN > PT
• PT > EN
• At present this tool is getting about 60 hits per day!
![Page 14: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/14.jpg)
![Page 15: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/15.jpg)
![Page 16: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/16.jpg)
![Page 17: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/17.jpg)
BOOMERANG
• http://poloclup.linguateca.pt/ferramentas/boomerang/index.html
• This tool submits a text for translation – and back-translation – and back-translation…. Until it reaches a fixed point
• This shows that the rules programmed for one language direction do not always correspond to the other language direction
![Page 18: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/18.jpg)
![Page 19: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/19.jpg)
![Page 20: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/20.jpg)
EVAL > TrAva
• Informal class experiment led to a useful research tool
• Several versions of EVAL– Different types of classification of input– Different explanations of errors of output
• Production, correction and re-correction of procedure interesting
![Page 21: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/21.jpg)
TrAva - procedure
• Online EN > PT MT using 4 MT systems:– Free Translation– Systran– E T Server– Amikai
• Researcher chooses area for analysis – e.g.– ambiguity – lexical and structural mismatches– Homographs and polysemous lexical items – syntactic complexity – multiword units: idioms and collocations – anaphora resolution
![Page 22: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/22.jpg)
TrAva - procedure
• Selection of ‘genuine’ examples from BNC, Reuter’s corpus, newspapers etc.
• Possible ‘pruning’ of unnecessary text (some systems accept limited text)
• No deliberate attempt to confuse the system
• BUT: avoidance of repetitive ‘test suites’
![Page 23: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/23.jpg)
TrAva - procedure
• Sentence submitted to TrAva• MT results• Researcher:
– Classifies part of sentence being examined in terms of the English lexicon or POS (BNC codes)
– Examines results– Explains errors in terms of Portuguese
grammar
![Page 24: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/24.jpg)
![Page 25: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/25.jpg)
![Page 26: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/26.jpg)
![Page 27: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/27.jpg)
![Page 28: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/28.jpg)
Access to work done
• Researcher may access work done and review it
• Teacher / administrator can access student work and give advice
• FAQ
![Page 29: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/29.jpg)
![Page 30: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/30.jpg)
![Page 31: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/31.jpg)
Present situation
• METRA and BOOMERANG are all free to use online at:
• http://poloclup.linguateca.pt/ferramentas
• TraVa is free to use online at:
• http://www.linguateca.pt/trava/
• The corpus CorTA – over 1000 sentences + 4 MT versions available for consultation at: http://www.linguateca.pt/
![Page 32: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/32.jpg)
![Page 33: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/33.jpg)
![Page 34: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/34.jpg)
Conclusions
• It has been a successful experiment
• It is useful pedagogically– As linguistic analysis– As appreciation of MT
• It has interesting theoretical implications > emphasis on ‘real’ sentences and recognition of interconnection of lexicon + syntax + context
![Page 35: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/35.jpg)
Conclusions
• Further work needs to be done on the classifications– E.g. the analysis of ‘error’ as ‘lexical choice’
needs to be able to combine with other possible reasons for error
• A lone researcher can use it to examine a restricted area
• BUT – a large team is needed to overhaul a system properly
![Page 36: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/36.jpg)
Homographs and Polysemy
• Homographs = words with same spelling but different syntactic use
• Polysemy = words with same spelling, but different meaning according to use or context
• BUT – the difference is not as clear-cut as all that
• However – major problem for MT
![Page 37: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/37.jpg)
![Page 38: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/38.jpg)
![Page 39: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/39.jpg)
![Page 40: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/40.jpg)
![Page 41: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/41.jpg)
![Page 42: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/42.jpg)
![Page 43: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/43.jpg)
![Page 44: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/44.jpg)
![Page 45: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/45.jpg)
![Page 46: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/46.jpg)
Complex Noun Phrases
• DETerminante + ADJectivo + Nome
• DETerminante + ADJectivo Composto + ADJectivo + Nome
• DETerminante + ADVérbio (em –ly) + ADJectivo + ADJectivo + Nome
![Page 47: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/47.jpg)
![Page 48: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/48.jpg)
![Page 49: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/49.jpg)
![Page 50: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/50.jpg)
![Page 51: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/51.jpg)
![Page 52: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/52.jpg)
![Page 53: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/53.jpg)
Lexical Bundles
![Page 54: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/54.jpg)
EXAMPLES
• Now let us look at some examples:– Homographs– Polysemy– Complex noun phrases– Lexical bundles
![Page 55: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/55.jpg)
![Page 56: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/56.jpg)
![Page 57: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/57.jpg)
![Page 58: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/58.jpg)
![Page 59: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/59.jpg)
![Page 60: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/60.jpg)
![Page 61: TrAva – a tool for evaluating Machine Translation – pedagogical and research possibilities Belinda Maia, Diana Santos, Luís Sarmento & Anabela Barreiro](https://reader036.vdocuments.site/reader036/viewer/2022062511/551607d8550346d46f8b5e72/html5/thumbnails/61.jpg)