challenges in neural machine translations · 2020-03-11 · word embeddings • lsa learns word...
TRANSCRIPT
![Page 1: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/1.jpg)
Challenges in Neural Machine Translations
Marc’Aurelio RanzatoFacebook AI Research
ACDL - Pontignano, 19 July 20181
![Page 2: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/2.jpg)
Outline• PART 0 [lecture 1]
• Natural Language Processing & Deep Learning
• Background refresher
• Part 1 [lecture 1]
• Unsupervised Word Translation
• Part 2 [lecture 2]
• Unsupervised Sentence Translation
• Part 3 [lecture 3]
• Uncertainty in Machine Translation
• Sequence-Level Prediction in Machine Translation
2 M. Ranzato
![Page 3: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/3.jpg)
Natural Language Processing
• Language is the most natural and efficient way that people use to communicate.
• A.I. agents must conceivably communicate with humans to perform their tasks efficiently.
• A.I. agents need to understand language (NLU).
• A.I. agents need to generate natural language (NLG).
3 M. Ranzato
![Page 4: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/4.jpg)
Challenges: NLUI saw a man on a hill with a telescope. • There’s a man on a hill, and I’m watching him with my telescope. • There’s a man on a hill, who I’m seeing, and he has a telescope. • There’s a man, and he’s on a hill that also has a telescope on it. • I’m on a hill, and I saw a man using a telescope. • There’s a man on a hill, and I’m sawing him with a telescope.
Prostitutes appeal to Pope. • Prostitutes have asked the Pope for help. • The Pope finds prostitutes appealing.
Language is ambiguous. Its meaning is context dependent, and it may depend on common knowledge of the world.
4 M. Ranzato
![Page 5: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/5.jpg)
Challenges: NLGA: How are you? B: I don’t know. A: Where are you going? B: I don’t know. A: What do you think about Deep Learning B: I don’t know.
Long-range dependencies, grounding, large search space…
How are the startup is a lot of the startup is a lot of the startup is a lot of the startup is a lot of the …
https://cs.stanford.edu/people/karpathy/recurrentjs/
5 M. Ranzato
![Page 6: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/6.jpg)
NLP Today• No model really “understands the meaning”.
• Statistical models leverage vast amounts of data to capture regularities which are sufficient to do well at several non-trivial tasks, such as:
• Search / MT / dialogue systems in restricted domains / Classification of documents…
• Deep Learning: enables learning of features in an end-to-end framework, leveraging big datasets.
6 M. Ranzato
![Page 7: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/7.jpg)
NLP Tasks: Examples
Fixed Length
Variable Length
Variable LengthFixed Lengthoutput
input
7 M. Ranzato
![Page 8: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/8.jpg)
NLP Tasks: Examples
Easy: input and output have fixed length.
Fixed Length
Variable Length
Variable LengthFixed Lengthoutput
input
BoW text classification
8 M. Ranzato
![Page 9: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/9.jpg)
NLP Tasks: Examples
Input is a sequence but output is fixed length.
Fixed Length
Variable Length
Variable LengthFixed Lengthoutput
input
BoW text classification
language modeling
9 M. Ranzato
![Page 10: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/10.jpg)
NLP Tasks: Examples
The model has to generate a variable length sequence at the output.
Fixed Length
Variable Length
Variable LengthFixed Lengthoutput
input
BoW text classification
language modeling
Image Captioning
10 M. Ranzato
![Page 11: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/11.jpg)
NLP Tasks: Examples
The model has to transduce a variable length sequence into another variable length sequence.
Fixed Length
Variable Length
Variable LengthFixed Lengthoutput
input
BoW text classification
language modeling
Image Captioning
MT
11 M. Ranzato
![Page 12: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/12.jpg)
Fixed Length
Variable Length
Variable LengthFixed Lengthoutput
input
BoW text classification
language modeling
Image Captioning
MT
NLP Tasks: Examples
The focus of these lectures will be on Machine Translation: • good use case • important practical applications • metric not too bad…
12 M. Ranzato
![Page 13: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/13.jpg)
NLP & Deep Learning• Language is symbolic, structured and compositional.
• Deep learning is good at learning data dependent representations, and it has a good inductive bias for learning from compositional distributions.
• In order to apply standard deep learning methods to NLP, we need to first map discrete symbols to a continuous space: word embeddings.
13 M. Ranzato
![Page 14: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/14.jpg)
Outline• PART 0 [lecture 1]
• Natural Language Processing & Deep Learning
• Background refresher
• Part 1 [lecture 1]
• Unsupervised Word Translation
• Part 2 [lecture 2]
• Unsupervised Sentence Translation
• Part 3 [lecture 3]
• Uncertainty in Machine Translation
• Sequence-Level Prediction in Machine Translation
14 M. Ranzato
![Page 15: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/15.jpg)
Quick Refresh on the Basics
• Word Embeddings
• Language Modeling
• Machine Translation
15 M. Ranzato
![Page 16: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/16.jpg)
Learning Word Representations
• Learn word representations from raw text (without supervision).
• word2vec review; for more gentle background visit:
• Practical applications:
• Text classification
• Ranking (e.g., Google search, Facebook feeds ranking)
• Machine translation
• Chatbot
http://www.cs.toronto.edu/~ranzato/files/ranzato_deeplearn17_lec2_nlp.pdf
16 M. Ranzato
![Page 17: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/17.jpg)
Latent Semantic Analysis
Deerwester et al. “Indxing by Latent Semantic Analysis” JASIS 1990
term-document matrixxi,j (normalized) number of times word i appears in document j
Example doc1: the cat is furry doc2: dogs are furry
doc1 doc2are 0 1cat 1 0
dogs 0 1furry 1 1
is 1 0the 1 0
17
![Page 18: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/18.jpg)
Latent Semantic Analysis
Deerwester et al. “Indxing by Latent Semantic Analysis” JASIS 1990
term-document matrixxi,j (normalized) number of times word i appears in document j
18
![Page 19: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/19.jpg)
Latent Semantic Analysis
Deerwester et al. “Indxing by Latent Semantic Analysis” JASIS 1990
term-document matrixxi,j (normalized) number of times word i appears in document j
Each column of V , is a representation of a document in the corpus.
is
T
Each column is a D dimensional vector. We can use it to compare & retrieve documents.19
![Page 20: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/20.jpg)
Latent Semantic Analysis
Deerwester et al. “Indxing by Latent Semantic Analysis” JASIS 1990
term-document matrixxi,j (normalized) number of times word i appears in document j
Each row of U, is a representation of a word in the dictionary. Each row of U, is a vectorial representation of a word, a.k.a. embedding.
20
![Page 21: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/21.jpg)
Word Embeddings• Convert words (symbols) into a D dimensional
vector, where D is a hyper-parameter.
• Once embedded, we can:
• Compare words.
• Apply our favorite machine learning method (DL) to represent sequences of words.
• At document retrieval time in LSA, the representation of a new document is a weighted sum of word embeddings (bag-of-words -> bag-of-embeddings): U’ x
21 M. Ranzato
![Page 22: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/22.jpg)
bi-gram• A bi-gram is a model of the probability of a word
given the preceding one:
• The simplest approach consists of building a (normalized) matrix of counts:
p(wk|wk�1)
ci,j number of times word i is preceded by word j
wk 2 V
c(wk|wk�1) =
2
4c1,1 . . . c1,|V |. . . ci,j . . .c|V |,1 . . . c|V |,|V |
3
5
preceding word
curre
nt w
ord
22 M. Ranzato
![Page 23: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/23.jpg)
n-gram• A n-gram is a model of the probability of a word
given the preceding ones:
• The simplest approach consists of building a (normalized) matrix of counts:
ci,j number of times word i is preceded by word in context
wk 2 V
preceding words
curre
nt w
ord
p(wk|wk�1, . . . , wk�n+1)
c(wk|wk�1, . . . , wk�n+1) =
2
4c1,1 . . . c1,M. . . ci,j . . .c|V |,1 . . . c|V |,M
3
5
23 M. Ranzato
![Page 24: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/24.jpg)
Factorized bi-gram
• We can factorize (via SVD, for instance) the bigram to reduce the number of parameters and become more robust to noise (entries with low counts):
c(wk|wk�1) =
2
4c1,1 . . . c1,|V |. . . ci,j . . .c|V |,1 . . . c|V |,|V |
3
5 = UV
U 2 R|V |⇥D
V 2 RD⇥|V |
• Rows of U store “output” word embeddings, and columns of V store “input” word embeddings.
input word
outp
ut w
ord
24 M. Ranzato
![Page 25: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/25.jpg)
Factorized bi-gram• The same can be expressed as a two layer (linear)
neural network:
c(wk|wk�1) =
2
4c1,1 . . . c1,|V |. . . ci,j . . .c|V |,1 . . . c|V |,|V |
3
5 = UV
softmaxV U
2
66666666664
0...010...0
3
77777777775
input word
1-hot representation of the input word
outp
ut w
ord
25 M. Ranzato
![Page 26: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/26.jpg)
Factorized bi-gram
c(wk|wk�1) =
2
4c1,1 . . . c1,|V |. . . ci,j . . .c|V |,1 . . . c|V |,|V |
3
5 = UV
softmaxV U
2
66666666664
0...010...0
3
77777777775
input word
1-hot representation of the input word
outp
ut w
ord
No need to multiply, V is just a look up table!
• The same can be expressed as a two layer (linear) neural network:
26 M. Ranzato
![Page 27: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/27.jpg)
Factorized bi-gram
c(wk|wk�1) =
2
4c1,1 . . . c1,|V |. . . ci,j . . .c|V |,1 . . . c|V |,|V |
3
5 = UV
softmaxV U
2
66666666664
0...010...0
3
77777777775
input word
1-hot representation of the input word
outp
ut w
ord
No need to multiply, V is just a look up table!
NOTE: Since embeddings are free, there is no point adding non-linearities and more layers!Here, depth does not help!
• The same can be expressed as a two layer (linear) neural network:
27 M. Ranzato
![Page 28: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/28.jpg)
Factorized bi-gram
• bi-gram model could be useful for type-ahead applications (in practice, it’s much better to condition upon the past n>2 words).
• Factorized model yields word embeddings as a by-product.
28 M. Ranzato
![Page 29: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/29.jpg)
Word Embeddings• LSA learns word embeddings that take into
account co-occurrences across documents.
• bi-gram instead learns word embeddings that only take into account the next word.
• It seems better to do something in between, using more context but just around the word of interest, yielding a method called word2vec.
Mikolov et al. “Efficient estimation of word representations” rejected by ICLR 2013
![Page 30: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/30.jpg)
skip-gram• Similar to factorized bi-gram model, but
predict N preceding and N following words.
• Words that have the same context will get similar embeddings. E.g.: cat & kitty.
• Input projection is just look-up table. Bulk of computation is the the prediction of words in context.
• Learning by cross-entropy minimization via SGD.
Mikolov et al. “Efficient estimation of word representations” rejected by ICLR 2013
![Page 31: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/31.jpg)
word2vec
• code at: https://code.google.com/archive/p/word2vec/
• see evaluation from Tomas’s NIPS 2013 presentation at:
https://drive.google.com/file/d/0B7XkCwpI5KDYRWRnd1RzWXQ2TWc/edit
Joulin et al. “Bag of tricks for efficient text classification” ACL 2016
![Page 32: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/32.jpg)
from https://drive.google.com/file/d/0B7XkCwpI5KDYRWRnd1RzWXQ2TWc/edit credit T. Mikolov 32
![Page 33: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/33.jpg)
from https://drive.google.com/file/d/0B7XkCwpI5KDYRWRnd1RzWXQ2TWc/edit credit T. Mikolov 33
![Page 34: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/34.jpg)
Recap• Embedding words (from a 1-hot to a distributed
representation) lets you:
• understand similarity between words
• plug them within any parametric ML model
• Several ways to learn word embeddings. word2vec is still one of the most efficient ones.
• Note word2vec leverages large amounts of unlabeled data.
34 M. Ranzato
![Page 35: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/35.jpg)
Quick Refresh on the Basics
• Word Embeddings
• Language Modeling
• Neural Machine Translation
35 M. Ranzato
![Page 36: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/36.jpg)
Language Modeling
• the math…
• with Markov assumption (used by n-grams):
• application: type-ahead.
p✓(w1, w2, . . . , wM ) = p✓(wM |wM�1 . . . , wM�n)p✓(wM�1|wM�2, . . . , wM�n�1) . . . p✓(w2|w1)p✓(w1)
p✓(w1, w2, . . . , wM ) = p✓(wM |wM�1 . . . , w1)p✓(wM�1|wM�2, . . . , w1) . . . p✓(w2|w1)p✓(w1)
36 M. Ranzato
![Page 37: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/37.jpg)
Neural Network LM
Y. Bengio et al. “A neural probabilistic language model” JMLR 2003
![Page 38: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/38.jpg)
Neural Network LM
Y. Bengio et al. “A neural probabilistic language model” JMLR 2003
• Natural extension of the factorized bi-gram model.
• Improved accuracy with more context. A bit better than n-gram (count based methods).
• if we are just interested in word embeddings, much more expensive than word2vec.
• It gives a representation to ordered sequences of n words.
![Page 39: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/39.jpg)
Recurrent Neural Network
• In NN-LM, the hidden state is the concatenation of word embeddings.
• Key idea of RNNs: compute a (non-linear) running average instead, to increase the size of the context.
• Many variants…
39 M. Ranzato
![Page 40: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/40.jpg)
Recurrent Neural Network• Elman RNN:
Elman “Finding structure in time” Cognitive Science 1990
hk = �(Urhk�1 + U i1(wk) + br)
p(wk+1|h) = softmax(Uohk + bo)
LNLL = �nX
i=1
log p(wi|wi�1, . . . , w1)
• Training (cross-entropy / negative log-likelihood loss):
40
![Page 41: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/41.jpg)
RNN: Inference Time• Elman RNN:
Elman “Finding structure in time” Cognitive Science 1990
hk = �(Urhk�1 + U i1(wk) + br)
p(wk+1|h) = softmax(Uohk + bo)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
41
![Page 42: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/42.jpg)
RNN: Inference Time• Elman RNN:
Elman “Finding structure in time” Cognitive Science 1990
hk = �(Urhk�1 + U i1(wk) + br)
p(wk+1|h) = softmax(Uohk + bo)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
42
![Page 43: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/43.jpg)
RNN: Inference Time• Elman RNN:
Elman “Finding structure in time” Cognitive Science 1990
hk = �(Urhk�1 + U i1(wk) + br)
p(wk+1|h) = softmax(Uohk + bo)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
43
![Page 44: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/44.jpg)
RNN: Inference Time• Elman RNN:
Elman “Finding structure in time” Cognitive Science 1990
hk = �(Urhk�1 + U i1(wk) + br)
p(wk+1|h) = softmax(Uohk + bo)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
44
![Page 45: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/45.jpg)
RNN: Inference Time• Elman RNN:
Elman “Finding structure in time” Cognitive Science 1990
hk = �(Urhk�1 + U i1(wk) + br)
p(wk+1|h) = softmax(Uohk + bo)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
45
![Page 46: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/46.jpg)
RNN: Inference Time• Elman RNN:
Elman “Finding structure in time” Cognitive Science 1990
hk = �(Urhk�1 + U i1(wk) + br)
p(wk+1|h) = softmax(Uohk + bo)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
46
![Page 47: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/47.jpg)
RNNs• Inference in an RNN is like a regular forward pass in a deep
neural network, with two differences: • Weights are shared at every layer. • Inputs are provided at every layer.
• Two possible applications: • Scoring: compute the log-likelihood of an input
sequence (sum the log-prob scores at every step). • Generation: sample or take the max from the predicted
distribution over words at each time step, and feed that prediction as input at the next time step.
47 M. Ranzato
![Page 48: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/48.jpg)
RNNs• Inference in an RNN is like a regular forward pass in a deep
neural network, with two differences: • Weights are shared at every layer. • Inputs are provided at every layer.
• Two possible applications: • Scoring: compute the log-likelihood of an input sequence (sum the
log-prob scores at every step). • Generation: sample or take the max from the predicted distribution
over words at each time step, and feed that prediction as input at the next time step.
48 M. Ranzato
![Page 49: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/49.jpg)
RNN: Training Time• Truncated Back-Propagation Through Time:
• Unfold RNN for only N steps and do:
• Forward
• Backward
• Weight update
• Repeat the process on the following sequence of N words, but carry over the value of the last hidden state.
Werbos “Backpropagation through time: what does it do and how to do it” IEEE 199049
![Page 50: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/50.jpg)
RNN: Truncated BPTT
Elman “Finding structure in time” Cognitive Science 1990
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
Forward Pass
50
![Page 51: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/51.jpg)
Elman “Finding structure in time” Cognitive Science 1990
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
RNN: Truncated BPTTForward Pass
51
![Page 52: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/52.jpg)
Elman “Finding structure in time” Cognitive Science 1990
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
RNN: Truncated BPTTForward Pass
52
![Page 53: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/53.jpg)
Elman “Finding structure in time” Cognitive Science 1990
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
RNN: Truncated BPTTBackward Pass
53
![Page 54: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/54.jpg)
Elman “Finding structure in time” Cognitive Science 1990
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
RNN: Truncated BPTTBackward Pass
54
![Page 55: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/55.jpg)
Elman “Finding structure in time” Cognitive Science 1990
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
RNN: Truncated BPTTBackward Pass
55
![Page 56: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/56.jpg)
Elman “Finding structure in time” Cognitive Science 1990
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
RNN: Truncated BPTTParameter Update
56
![Page 57: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/57.jpg)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
o
RNN: Truncated BPTTForward Pass
57 M. Ranzato
![Page 58: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/58.jpg)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
RNN: Truncated BPTTForward Pass
58 M. Ranzato
![Page 59: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/59.jpg)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
RNN: Truncated BPTTForward Pass
59 M. Ranzato
![Page 60: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/60.jpg)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
RNN: Truncated BPTTBackward Pass
60 M. Ranzato
![Page 61: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/61.jpg)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
RNN: Truncated BPTTBackward Pass
61 M. Ranzato
![Page 62: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/62.jpg)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
RNN: Truncated BPTTBackward Pass
62 M. Ranzato
![Page 63: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/63.jpg)
U U U U U U
U U U U U U
r r r r r r
o o o o o o
w1 w2 w3 w4 w5 w6
h0 h1 h2 h3 h4 h5 h6
w2 w3 w4 w5 w6 w7
RNN: Truncated BPTTParameter Update
63 M. Ranzato
![Page 64: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/64.jpg)
Recap• RNNs are more powerful because they capture a
context of potentially “infinite” size.
• The hidden state of a RNN can be interpreted as a way to represent the history of what has been seen so far.
• RNNs can be useful to represent variable length sentences.
• There are lots of RNN variants. The best working ones have gating (units that multiply other units): e.g.: LSTM and GRU.
64 M. Ranzato
![Page 65: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/65.jpg)
Quick Refresh on the Basics
• Word Embeddings
• Language Modeling
• Machine Translation
65 M. Ranzato
![Page 66: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/66.jpg)
Brief History of MT
• Rule-based systems
• Statistical MT
• Neural MT
time
1970
1990
2014
amount of bitexts
~10,000,000
~10,000
0 (dictionary + prior)
66 M. Ranzato
![Page 67: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/67.jpg)
Brief History of MT
• Rule-based systems
• Statistical MT
• Neural MT
time
1970
1990
2014
amount of bitexts
~10,000,000
~10,000
0 (dictionary + prior)
compute
~100Tf
~1Mf
…
67 M. Ranzato
![Page 68: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/68.jpg)
Example: ITA (source) : Il gatto si e’ seduto sul tappetino.
EN (target) : The cat sat on the mat.
Approach: Have one RNN/CNN to encode the source sentence, and another RNN/CNN/MemNN to predict the target sentence. The target RNN learns to (soft) align via attention.
Neural machine translation by jointly learning to align and translate, Bahdanau et al. ICLR 2015
Neural Machine Translation (in 3 slides)
M. Ranzato
68
![Page 69: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/69.jpg)
Y. LeCun’s diagram
the cat sat
cat sat on
69
M. Ranzato
69
![Page 70: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/70.jpg)
the cat sat
cat sat on
70
.* -> softmax
Sum
il gatto si e’ seduto sul tappetino
0.95
Source Target
1) Represent source
M. Ranzato
Source Encoder (RNN/CNN)70
![Page 71: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/71.jpg)
the cat sat
cat sat on
71
Source Encoder (RNN/CNN)
dot product -> softmax
Sum
il gatto si e’ seduto
0.95
Source Target
2) score each source word (attention)
M. Ranzato
sul tappetino
71
![Page 72: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/72.jpg)
the cat sat
cat sat on
72
Sum
il gatto si e’ seduto
0.95
Source Target
3) combine target hidden with source vector
M. Ranzato
sul tappetino
Source Encoder (RNN/CNN)
dot product -> softmax
72
![Page 73: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/73.jpg)
the cat sat
cat sat on
73
Sum
il gatto si e’ seduto
0.95
Source Target
3) combine target hidden with source vector
M. Ranzato
sul tappetino
Source Encoder (RNN/CNN)
dot product -> softmax
Alignment is learnt implicitly.
73
![Page 74: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/74.jpg)
NMT Training & Inference
Training: predict one target token at the time and minimize cross-entropy loss.
Inference: find the most likely target sentence (approximately) using beam search.
Evaluation: BLEU at inference time.
M. Ranzato
LTokNLL = �nX
i=1
log p(ti|t1, . . . , ti�1,x)
74
![Page 75: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/75.jpg)
NMT Training & Inference
Training: predict one target token at the time and minimize cross-entropy loss.
Inference: find the most likely target sentence (approximately) using beam search.
Evaluation: BLEU at inference time.
M. Ranzato
u = argmin� log p(u|x)
75
![Page 76: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/76.jpg)
Beam Searchthe
a
-1
-0.5
-2-4
M. Ranzato76
![Page 77: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/77.jpg)
Beam Searchthe
a
kitty
cat-1
-0.5-2
-3
-1
-0.5
M. Ranzato77
![Page 78: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/78.jpg)
Beam Searchthe
a
kitty
cat
sat
ate
-1
-0.5
-1
-0.5
-1
-0.5
-0.5
-0.25
M. Ranzato78
![Page 79: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/79.jpg)
Beam Searchthe
a
kitty
cat
sat
ate
at
on
-1
-0.5
-1
-0.5-0.5
-0.25
-1
-2
-0.5
-0.25
M. Ranzato79
![Page 80: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/80.jpg)
Beam Searchthe
a
kitty
cat
sat
ate
at
on
-1
-0.5
-1
-0.5-0.5
-0.25
-1
-2
-0.5
-0.25
M. Ranzato80
![Page 81: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/81.jpg)
NMT Training & Inference
Training: predict one target token at the time and minimize cross-entropy loss.
Inference: find the most likely target sentence (approximately) using beam search.
Evaluation: compute BLEU on hypothesis returned by the inference procedure
M. RanzatoBLEU: a method for automatic evaluation of machine translation, Papineni et al. ACL 2002
pn =
Pgenerated sentences
Pngrams Clip(Count(ngram matches))
Pgenerated sentences
Pngrams Count(ngram) BLEU = BP e
PNn=1
1N log pn
![Page 82: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/82.jpg)
Challenges• Most language pairs have little parallel data. How to estimate parameters?
• One-to-many mapping / uncertainty, there does not exist a metric able to account for uncertainty.
• Model is asked to predict a single token at training time, but the whole sequence at test time.
• Exposure bias: training and testing are inconsistent because model has never observed its own predictions at training time.
• At training time, we optimize for a different loss.
• Evaluation criterion is not differentiable.
• Domain shift.
M. RanzatoSix challenges for neural machine translation, Koehn et al. Workshop NMT, ACL 2017
![Page 83: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/83.jpg)
Challenges• Most language pairs have little parallel data. How to estimate parameters?
• One-to-many mapping / uncertainty, there does not exist a metric able to account for uncertainty.
• Model is asked to predict a single token at training time, but the whole sequence at test time.
• Exposure bias: training and testing are inconsistent because model has never observed its own predictions at training time.
• At training time, we optimize for a different loss.
• Evaluation criterion is not differentiable.
• Domain shift.
M. RanzatoSix challenges for neural machine translation, Koehn et al. Workshop NMT, ACL 2017
![Page 84: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/84.jpg)
Outline• PART 0 [lecture 1]
• Natural Language Processing & Deep Learning
• Neural Machine Translation
• Part 1 [lecture 1]
• Unsupervised Word Translation
• Part 2 [lecture 2]
• Unsupervised Sentence Translation
• Part 3 [lecture 3]
• Uncertainty and Sequence-Level Prediction in Machine Translation84 M. Ranzato
![Page 85: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/85.jpg)
Guillaume Lample Ludovic DenoyerAlexis Conneau Herve Jegou
Word Translation Without Parallel Data Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, Herve Jegou ICLR 2018 https://arxiv.org/abs/1710.04087 CODE:https://github.com/facebookresearch/MUSE
85 M. Ranzato
![Page 86: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/86.jpg)
Learning from Low-Resource Language Pairs
• We could leverage:
• Limited amount of parallel data.
• Parallel data from other language pairs.
• Large amount of monolingual data, which is often more easily available.
M. Ranzato86
![Page 87: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/87.jpg)
Goal• Training an NMT system without supervision, using
monolingual data only.
• Admittedly, unrealistic but…
• Baseline for extensions using parallel data (from language pair of interest or others).
• Scientific endeavor, towards our quest for a good unsupervised learning algorithm.
M. Ranzato87
![Page 88: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/88.jpg)
Unsupervised Word Translation
• Motivation: A pre-requisite for unsupervised sentence translation.
• Problem: given two monolingual corpora in two different languages, estimate bilingual lexicon.
• Hint: the context of a word, is often similar across languages since each language refers to the same underlying physical world.
M. Ranzato88
![Page 89: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/89.jpg)
M. Ranzato
Method
En It
1) learn word embeddings (word2vec) separately on each language using lots of monolingual data.
89
![Page 90: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/90.jpg)
catkitty
dog
car
ornitorinc
ornitorinco
auto
gattogattino
cane
M. Ranzato
Method
yx
En It
1) learn word embeddings (word2vec) separately on each language using lots of monolingual data.
90
![Page 91: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/91.jpg)
2) learn a rotation matrix to roughly align the two domains. E.g., via adversarial training: pick a word at random from each language, embed them, project one of the two, and make sure distributions match.
catkitty
dog
car
ornitorinc
ornitorinco
auto
gattogattino
cane
M. Ranzato
xi
yjW
embedding i-th word in En
embedding j-th word in It
orthogonal matrix
LD(✓D|W ) = �Ex [log p(En|Wx; ✓D)]� Ey [log p(It|y; ✓D)]
LW (W✓D) = �Ex [log p(It|Wx; ✓D)]� Ey [log p(En|y; ✓D)]
Method
yx
91
![Page 92: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/92.jpg)
2) learn a rotation matrix to roughly align the two domains. E.g., via adversarial training: pick a word at random from each language, embed them, project one of the two, and make sure distributions match.
cat
dog
car
ornitorinc ornitorinco
auto
gattogattino
cane
M. Ranzato
xi
yjW
embedding i-th word in En
embedding j-th word in It LD(✓D|W ) = �Ex [log p(En|Wx; ✓D)]� Ey [log p(It|y; ✓D)]
LW (W✓D) = �Ex [log p(It|Wx; ✓D)]� Ey [log p(En|y; ✓D)]
Wx y
kitty
Method
orthogonal matrix
92
![Page 93: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/93.jpg)
3) Iterative refinement via orthogonal Procrustes, using the most frequent words. Pick most frequent words, translate them via nearest neighbor, solve least square, and iterate.
cat
kitty
dog
car
ornitorinc ornitorinco
auto
gattogattino
cane
M. Ranzato
xi
yjW
embedding i-th word in En
embedding j-th word in It
Wx y
Wt = argmin ||Wt�1X � Y ||2, s.t. WtWTt = I
Method
orthogonal matrix
93
![Page 94: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/94.jpg)
3) Iterative refinement via orthogonal Procrustes, using the most frequent words. Pick most frequent words, translate them via nearest neighbor, solve least square, and iterate.
catkitty
dog
car
ornitorinc ornitorinco
auto
gattogattino
cane
M. Ranzato
xi
yjW
embedding i-th word in En
embedding j-th word in It
Wx y
Wt = argmin ||Wt�1X � Y ||2, s.t. WtWTt = I
Method
orthogonal matrix
94
![Page 95: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/95.jpg)
4) Build lexicon using metric that compensates for hubness. There are words that have lots of neighbors, while others that are not neighbors of anybody.
catkitty
dog
car
ornitorinc ornitorinco
auto
gattogattino
cane
M. Ranzato
xi
yjW
embedding i-th word in En
embedding j-th word in It
Wx y
CSLS(Wx, y) = 2 cos(Wx, y)� rEn(Wx)� rIt(y)
rEn(Wx) =1
K
X
yt2NEn(Wx)
cos(Wx, yt)
Method
orthogonal matrix
95
![Page 96: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/96.jpg)
catkitty
dog
car
ornitorinc
auto
gattogattino
M. Ranzato
xi
yjW
embedding i-th word in En
embedding j-th word in It
Wx y
CSLS(Wx, y) = 2 cos(Wx, y)� rEn(Wx)� rIt(y)
rEn(Wx) =1
K
X
yt2NEn(Wx)
cos(Wx, yt)
ornitorincocane
orthogonal matrix
4) Build lexicon using metric that compensates for hubness. There are words that have lots of neighbors, while others that are not neighbors of anybody.
96
![Page 97: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/97.jpg)
Results on Word Translation
20
32
44
56
68
en-it it-en
58.7
66.2
56.3
63.7
38.3
45.1
38.5
44.9
31.1
36.8 38
43.1
33.8
39.7
31
36.1
24.6
38.5
24.9
33.8
Mikolovetal.(2013) Dinuetal.(2015) Faruqui&Dyer(2014)2 Artetxeetal.(2017) Smithetal.(2017)Procrustes–NN Procrustes–CSLS Unsupervised–CSLS Procrustes–CSLS(wiki) Unsupervised–CSLS(wiki)
P@1
M. Ranzato
More results on several language pairs, analysis and other tasks in the paper. By using more anchor points and lots of unlabeled data, we even outperform supervised approaches!
supervisedapproaches
supervisedapproaches
unsupervisedapproaches
unsupervisedapproaches
97
![Page 98: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/98.jpg)
MUSE
• 110 ground truth bilingual dictionaries
• code to align embeddings
https://github.com/facebookresearch/MUSE
98 M. Ranzato
![Page 99: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/99.jpg)
Key Idea
• Learn representations of each domain.
• Translate by aligning sets of embeddings.
• How to apply this principle to sentences?
M. Ranzato99
![Page 100: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/100.jpg)
Outline• PART 0 [lecture 1]
• Natural Language Processing & Deep Learning
• Neural Machine Translation
• Part 1 [lecture 1]
• Unsupervised Word Translation
• Part 2 [lecture 2]
• Unsupervised Sentence Translation
• Part 3 [lecture 3]
• Uncertainty and Sequence-Level Prediction in Machine Translation100
![Page 101: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/101.jpg)
Guillaume Lample Ludovic DenoyerAlexis Conneau
Unsupervised Machine Translation Using Monolingual Corpora Only Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato ICLR 2018 https://arxiv.org/abs/1711.00043
Myle Ott
Phrase-Based and Neural Unsupervised Machine Translation Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato https://arxiv.org/abs/1804.07755
101
![Page 102: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/102.jpg)
Naïve Application of MUSE• In general, this may not work on sentences
because:
• Without leveraging compositional structure, space is exponentially large.
• Need good sentence representations.
• Unlikely that a linear mapping is sufficient to align sentence representations of two languages.
M. Ranzato102
![Page 103: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/103.jpg)
Toy Illustration
2D embeddings of valid sentences in the source language.
thank youthank you very much
the cat sat on the matabsolutely despicable
103 M. Ranzato
![Page 104: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/104.jpg)
Toy Illustration
Actually observed source sentences in the monolingual data with underlying manifold.
104 M. Ranzato
![Page 105: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/105.jpg)
Toy Illustration
Similarly for the target sentences (in red).
105 M. Ranzato
![Page 106: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/106.jpg)
Toy Illustration
Empty dots correspond to unobserved translations.
106 M. Ranzato
![Page 107: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/107.jpg)
3 Principles of UnsupMT: #1
Initialization: start by using good token-level (e.g., word-level using MUSE) correspondences.
107 M. Ranzato
![Page 108: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/108.jpg)
3 Principles of UnsupMT: #2
Language Modeling: make sure generations belong to the desired language.
108 M. Ranzato
![Page 109: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/109.jpg)
Back-Translation: reconstruct original sentence from translation (cross markers), in both directions.
Source Language
Target Language
forw
ard
mod
el
3 Principles of UnsupMT: #3
target sentence
noisy source-domain sentenceactual source-domain translation
back
war
d
mod
elSennrich et al. “Improving NMT models with monolingual data” ACL 2016
![Page 110: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/110.jpg)
Source Language
Target Language back
war
d m
odel
noisy target-domain sentence
source sentence
forw
ard
mod
el
Sennrich et al. “Improving NMT models with monolingual data” ACL 2016
3 Principles of UnsupMT: #3
Back-Translation: reconstruct original sentence from translation (cross markers), in both directions.
![Page 111: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/111.jpg)
Generic UnsupMT Algorithm
111
![Page 112: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/112.jpg)
Instantiations
• Phrase-based Machine Translation
• Neural Machine Translation
• Hybrid: PBSMT + NMT
112 M. Ranzato
![Page 113: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/113.jpg)
PBSMT (in 1 slide)• Training consists of:
• alignment of phrases
• construction of phrase tables (count-based)
• training of language model (n-gram)
• This is a good candidate for unsupMT because:
• memorization based, it has less parameters to fit.
• It often beats NMT when labeled data is scarce.
Koehn et al. “Statistical Phrase-Based Translation” NAACL 2003
![Page 114: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/114.jpg)
PBSMT: initialization
• MUSE to align word/phrase embeddings
• Populate unigram (more generally, n-gram) phrase tables by looking at cosine distance of neighbors:
p(tj |si) =e
1T cos(e(tj),We(si))
Pk e
1T cos(e(tk),We(si))
source wordtarget word MUSE rotation114 M. Ranzato
![Page 115: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/115.jpg)
PBSMT: Language Modeling
• Just an n-gram language model.
• Responsible for fixing incorrect entries in phrase table.
115 M. Ranzato
![Page 116: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/116.jpg)
PBSMT: Back-Translation
• Iterative back-translation (5M sentences at the time).
• As we iterate and phrase table gets better, longer spans can be reordered.
116 M. Ranzato
![Page 117: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/117.jpg)
PBSMT: Summary
117
![Page 118: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/118.jpg)
Instantiations
• Phrase-based Machine Translation
• Neural Machine Translation
• Hybrid: PBSMT + NMT
118 M. Ranzato
![Page 119: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/119.jpg)
NMT: Initialization
• For distant languages:
• MUSE unsupervised word alignment
• For languages that share tokens (word roots, etc.)
• Joint learning of embeddings with BPEs.
Sennrich et al. “NMT of rare words with subword units” ACL 2015 M. Ranzato
![Page 120: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/120.jpg)
Joint Learning with BPEsEn
It
- La costituzione e’ stata … - The constitution was … - La luna orbita intorno al.. - Lunar calendar is…
- Merge the monolingual datasets. - Apply BPE tokenization. - Learn token embeddings; as many will be shared
and space is common, there is no need to align.
120 M. Ranzato
![Page 121: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/121.jpg)
NMT: Language Modeling
encoder
y
decodery + n
hen(y + n)
y
NLL
encoder decoder
NLL
x
x
x+ n hit(x+ n)It DAE
En DAE
Since we work with a seq2seq model with attention, we train the decoder LM with a denoising autoencoder task.
121
![Page 122: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/122.jpg)
NMT: Language ModelingSince we work with a seq2seq model with attention, we
train the decoder LM with a denoising autoencoder task.
Arizonawasthefirsttointroducesucharequirement.Arizonawasthefirsttosucharequirement.Arizonawasfirsttointroducesucharequirement.
Ref:
Arizonawasthefirsttointroducesucharequirement.Arizonathefirstwastointroducearequirementsuch.Arizonawasthetointroducefirstsuchrequirementa.
Ref:
Drop
Swap
Even with attention, the model has to learn regularities in the input (not just copy but a good language model).
122 M. Ranzato
![Page 123: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/123.jpg)
NMT: back-translation• given a mini-batch of sentences from the source
monolingual dataset do:
• Use the source-to-target model to translate them.
• Use these translations as input to the target-to-source model and predict original inputs.
• Update parameters of target-to-source model.
• and vice versa, exchanging source with target.
123 M. Ranzato
![Page 124: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/124.jpg)
encoder decoder encoder decoderen enit it
y h(y) x h(x) ˆy
Illustration of the model during back-translation:
An Alternative View
124 M. Ranzato
![Page 125: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/125.jpg)
An Alternative ViewIllustration of the model during back-translation:
innerencoder
inner decoder
innerencoder
inner decoder
en enit it
y h(y) x h(x) ˆy
outer-encoder outer-decoder
M. Ranzato125
![Page 126: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/126.jpg)
An Alternative View
innerencoder
inner decoder
innerencoder
inner decoder
en enit it
y h(y) x h(x) ˆy
outer-encoder outer-decoder
How to constrain the intermediate sentence to be a valid Italian sentence? It has to be a valid sentence and it has to be a translation.
M. Ranzato
Illustration of the model during back-translation:
126
![Page 127: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/127.jpg)
An Alternative View
innerencoder
inner decoder
innerencoder
inner decoder
en enit it
y h(y) x h(x) ˆy
outer-encoder outer-decoder
How to constrain the intermediate sentence to be a valid Italian sentence?
M. Ranzato
- we could add some language modeling constraints directly on , but it would be hard to bprop and would be weak constraint on translation.
- instead, we constraint the latent space.
x
Illustration of the model during back-translation:
127
![Page 128: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/128.jpg)
Adding Language Modeling
innerencoder
inner decoder
innerencoder
inner decoder
it enit en
outer-encoder outer-decoder
M. Ranzato
x+ n y + n
Since inner decoders are shared between the LM and MT task, it should constraint the intermediate sentence to be fluent. But that’s not enough: - translation noise cannot be exactly reproduced (without parallel data). - latent representation produced by the other inner encoder can be
different.128
![Page 129: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/129.jpg)
Adding Language Modeling
innerencoder
inner decoder
innerencoder
inner decoder
enit en
outer-encoder outer-decoder
M. Ranzato
x+ n y + n
Since inner decoders are shared between the LM and MT task, it should constraint the intermediate sentence to be fluent. But that’s not enough: - translation noise cannot be exactly reproduced (without parallel data). - latent representation produced by the “other” inner encoder can be
different. WE NEED TO SHARE LATENT REPRESENTATIONS.
it
latent representation may not be robust to translation noise
129
![Page 130: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/130.jpg)
Adding Language Modeling
innerencoder
inner decoder
innerencoder
inner decoder
enit en
outer-encoder outer-decoder
M. Ranzato
x+ n y + n
Since inner decoders are shared between the LM and MT task, it should constraint the intermediate sentence to be fluent. But that’s not enough: - translation noise cannot be exactly reproduced (without parallel data). - latent representation produced by the “other” inner encoder may be
different.
it
NMT won’t know how to translate.130
![Page 131: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/131.jpg)
Adding Language Modeling
innerencoder
inner decoder
innerencoder
inner decoder
enit en
outer-encoder outer-decoder
M. Ranzato
x+ n y + n
Since inner decoders are shared between the LM and MT task, it should constraint the intermediate sentence to be fluent. But that’s not enough: - translation noise cannot be exactly reproduced (without parallel data). - latent representation produced by the “other” inner encoder may be
different.
it
- WE NEED TO SHARE LATENT REPRESENTATIONS
![Page 132: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/132.jpg)
NMT: Sharing Latent Space
innerencoder
innerdecoder
innerencoder
innerdecoder
enit en
outer-encoder outer-decoder
M. Ranzato
x+ n y + n
it
Sharing achieved via: 1) shared encoder (and also decoder). 2) joint BPE embedding learning.Note: first decoder token specifies the language on the target-side.
132
![Page 133: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/133.jpg)
Instantiations
• Phrase-based Machine Translation
• Neural Machine Translation
• Hybrid: PBSMT + NMT
133 M. Ranzato
![Page 134: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/134.jpg)
PBSMT + NMT
• Train PBSMT
• Use PBSMT to produce data to train NMT in addition to its own back-translated data.
134 M. Ranzato
![Page 135: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/135.jpg)
MethodologyEn Fr
Take monolingual NewsCrawl datasets from 2007 till 2017.
M. Ranzato135
![Page 136: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/136.jpg)
MethodologyEn Fr
Test on original WMT test set (no overlap with training set).
M. Ranzato136
![Page 137: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/137.jpg)
Datasets• WMT’14 En-Fr
• 50M sentences in each language for training
• eval on newstest2014
• WMT’16 En-De
• 50M sentences in each language for training
• eval on newstest2016
M. Ranzato137
![Page 138: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/138.jpg)
Prior work
138 M. Ranzato
![Page 139: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/139.jpg)
139 M. Ranzato
![Page 140: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/140.jpg)
Even after iteration 0, PBSMT is better than prior work! PBSMT works better than NMT, on these language pairs.
140 M. Ranzato
![Page 141: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/141.jpg)
141 M. Ranzato
![Page 142: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/142.jpg)
PBSMT: WMT’14 En-Fr
142 M. Ranzato
![Page 143: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/143.jpg)
NMT BLEU: 12.3 after epoch 1, 17.5 after epoch 3 and 24.2 after epoch 45.PBSMT BLEU: 15.4 after iteration 0, 23.7 after iteration 1 and 24.7 after iteration 4.
143 M. Ranzato
![Page 144: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/144.jpg)
NMT BLEU: 12.3 after epoch 1, 17.5 after epoch 3 and 24.2 after epoch 45.PBSMT BLEU: 15.4 after iteration 0, 23.7 after iteration 1 and 24.7 after iteration 4.
144 M. Ranzato
![Page 145: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/145.jpg)
NMT BLEU: 12.3 after epoch 1, 17.5 after epoch 3 and 24.2 after epoch 45.PBSMT BLEU: 15.4 after iteration 0, 23.7 after iteration 1 and 24.7 after iteration 4.
145 M. Ranzato
![Page 146: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/146.jpg)
WMT’14 En-Fr
146 M. Ranzato
![Page 147: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/147.jpg)
Low-Resource Language Pair: En-Ro
• Similar training set up as in En-Fr and En-De.
• training set: 2.9M monolingual sentences from NewsCrawl + monolingual data from WMT’16.
• test set: newstest 2016.RESULTS
Gu et al. 2018 NMT
PBSMT PBSMT+NMT
NA 21.2 21.3 25.1
22.9 19.4 23.0 23.9
En-Ro Ro-Enthey use: • dictionary • 6K parallel sentences • parallel data in other languages
147 M. Ranzato
![Page 148: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/148.jpg)
Distant Language Pair: En-Ru• Similar training set up as in En-Fr and En-De.
• training set: 50M monolingual sentences from NewsCrawl.
• test set: newstest 2016.
RESULTS
NMT PBSMT
PBSMT+NMT
8.0 13.4 13.8
9.0 16.6 16.6
En-Ru Ru-En
148 M. Ranzato
![Page 149: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/149.jpg)
Distant Language Pair: En-Ru
149 M. Ranzato
![Page 150: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/150.jpg)
Distant & Low-Resource Language Pair: En-Ur
150 M. Ranzato
https://www.bbc.com/urdu/pakistan-44867259
![Page 151: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/151.jpg)
Distant & Low-Resource Language Pair: En-Ur
• Training on 5.5 monolingual sentences (Jawaid et al. 2014) from news sources.
• Test on LDC2010T23 (news related).
RESULTS
PBSMT supervised PBSMT unsupervised
9.8 12.3
Ur-En
they use 800K parallel sentences (out of domain) from Tiedemann (2012).
151 M. Ranzato
![Page 152: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/152.jpg)
PBSMT Ablation: InitializationWMT’14 Fr-En
152 M. Ranzato
![Page 153: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/153.jpg)
PBSMT Ablation: Lang. ModelingWMT’14 Fr-En
153 M. Ranzato
![Page 154: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/154.jpg)
PBSMT Ablation: Back-TranslationWMT’14 Fr-En
154 M. Ranzato
![Page 155: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/155.jpg)
NMT: Ablation
155 M. Ranzato
![Page 156: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/156.jpg)
UnsupMT Summary• 3 principles of unsupMT
• initialization, i.e. token level translation
• language modeling
• back-translation
• PBSMT & NMT version
• Somewhat works also for distant and low resource languages.
M. Ranzato156
![Page 157: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/157.jpg)
UnsupMT Considerations
• General problem: unsupervised learning of the mapping between two domains.
• This is a task where a machine is probably better than humans, as it can easily leverage big data to learn patterns, dependencies and correspondences.
• Trivial extensions to semi-supervised setting.
M. Ranzato157
![Page 158: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/158.jpg)
Questions? Вопросы?
¿Preguntas? Domande?
M. Ranzato158
![Page 159: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/159.jpg)
Outline• PART 0 [lecture 1]
• Natural Language Processing & Deep Learning
• Background refresher
• Part 1 [lecture 1]
• Unsupervised Word Translation
• Part 2 [lecture 2]
• Unsupervised Sentence Translation
• Part 3 [lecture 3]
• Uncertainty
• Sequence-Level Prediction in Machine Translation
159 M. Ranzato
![Page 160: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/160.jpg)
Analyzing Uncertainty in Neural Machine TranslationMyle Ott, Michael Auli, David Grangier, Marc'Aurelio Ranzato ICML 2018 https://arxiv.org/abs/1803.00047
Myle Ott Michael Auli David Grangier
credit to Myle for slides.
![Page 161: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/161.jpg)
Goal: Investigate the effects of uncertainty in NMT model fitting and search
This work
161
M. Ranzato
![Page 162: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/162.jpg)
This work
162
M. Ranzato
![Page 163: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/163.jpg)
This work
Inherent uncertainty in the translation task
163
M. Ranzato
![Page 164: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/164.jpg)
This work
Training data may contain noise
164
M. Ranzato
![Page 165: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/165.jpg)
This work
Model 1 has very little uncertainty
165
M. Ranzato
![Page 166: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/166.jpg)
This work
NMT models can’t assign 0 probability mass to any outputs
166
M. Ranzato
![Page 167: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/167.jpg)
This work
Model 2 has considerable uncertainty
167
M. Ranzato
![Page 168: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/168.jpg)
Goal: Investigate the effects of uncertainty in NMT model fitting and search • Do NMT models capture uncertainty, and how is this
uncertainty represented in the model’s output distribution?
• How does uncertainty affect search? • How closely does the model distribution match the data
distribution? • How do we answer these questions with (typically) only
a single reference translation per source sentence? 168
M. Ranzato
![Page 169: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/169.jpg)
Experimental setup
Convolutional sequence-to-sequence models* (Gehring et al., 2017) Evaluation: compare translations with BLEU (Papineni et al., 2002) • Modified n-gram precision metric, values from 0 (worst)
to 100 (best) Datasets: WMT14 English-French and English-German • Mixture of news, parliamentary and web crawl data
* Results hold for other tested architectures too, e.g., LSTM
169
M. Ranzato
![Page 170: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/170.jpg)
Question: How much uncertainty is there in the model’s output distribution? Experiment: How many independent samples does it take to cover most of the sequence-level probability mass?
Do NMT models capture uncertainty?
170
M. Ranzato
![Page 171: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/171.jpg)
Do NMT models capture uncertainty?
Model’s output distribution is highly uncertain!
Even after 10K samples we cover only 25% of sequence-level probability mass.
What about beam search?
(WMT14 En-Fr)171
M. Ranzato
![Page 172: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/172.jpg)
Beam search is very efficient!
The reference score ( ) is lower than beam hypotheses
What is the quality (BLEU) of these translations?
(WMT14 En-Fr)
Do NMT models capture uncertainty?
172
M. Ranzato
![Page 173: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/173.jpg)
Beam search produces accurate translations
Sampling produces increasingly likely hypotheses, but these get worse BLEU after ~200
(WMT14 En-Fr)
Uncertainty & Search
173
M. Ranzato
![Page 174: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/174.jpg)
Hint: Scatter Plot of Samples
M. Ranzato
![Page 175: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/175.jpg)
Hint: Scatter Plot of Samples
M. Ranzato
Source #2375 (purple):Should this election be decided two months after we stopped voting?
High-BLEU sample:
Low-BLEU sample:
Cette élection devrait-elle ëtre décidé deux mois après que le vote est terminé?
Target #2375 (purple):
Cette élection devrait-elle ëtre décidée deux mois après l'arrêt du scrutin?
Ce choix devrait-il ëtre décidé deux mois après la fin du vote?175
![Page 176: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/176.jpg)
Hint: Scatter Plot of Samples
M. Ranzato
Should this election be decided two months after we stopped voting?
Cette élection devrait-elle ëtre décidé deux mois après que le vote est terminé?
Cette élection devrait-elle ëtre décidée deux mois après l'arrêt du scrutin?
Ce choix devrait-il ëtre décidé deux mois après la fin du vote?
BLEU is just a poor metric.
Source #2375 (purple):
High-BLEU sample:
Low-BLEU sample:
Target #2375 (purple):
176
![Page 177: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/177.jpg)
Hint: Scatter Plot of Samples
M. Ranzato
Source #115 (red):The first nine episodes of Sheriff [unk]'s Wild West will be available from November 24 on the site [unk] or via its application for mobile phones and tablets.
High-logp low BLEU sample:
Les neuf premiers épisodes de [unk] [unk] s Wild West seront disponibles à partir du 24 novembre sur le site [unk] ou via son application pour téléphones et tablettes.
Target #115 (red):
The first nine episodes of Sheriff [unk] s Wild West will be available from November 24 on the site [unk] or via its application for mobile phones and tablets.177
![Page 178: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/178.jpg)
Hint: Scatter Plot of Samples
M. Ranzato
The first nine episodes of Sheriff [unk]'s Wild West will be available from November 24 on the site [unk] or via its application for mobile phones and tablets.
Les neuf premiers épisodes de [unk] [unk] s Wild West seront disponibles à partir du 24 novembre sur le site [unk] ou via son application pour téléphones et tablettes.
The first nine episodes of Sheriff [unk] s Wild West will be available from November 24 on the site [unk] or via its application for mobile phones and tablets.
Model generates copies of source sentence!
Why does beam find this?
Source #115 (red):
High-logp low BLEU sample:
Target #115 (red):
178
![Page 179: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/179.jpg)
Source: The first nine episodes of Sheriff Callie ’s Wild West will be available (…)
Reference: Les neuf premiers épisodes de shérif Callie’ s Wild West seront disponibles (…)
Hypothesis: The first nine episodes of Sheriff Callie ’s Wild West will be available (…)
Uncertainty & Search
179
M. Ranzato
![Page 180: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/180.jpg)
Source: The first nine episodes of Sheriff Callie ’s Wild West will be available (…)
Reference: Les neuf premiers épisodes de shérif Callie’ s Wild West seront disponibles (…)
Hypothesis: The first nine episodes of Sheriff Callie ’s Wild West will be available (…)
Uncertainty & Search
log probs: -4.53 -0.02 -0.28 -0.11 -0.01 -0.001 -0.004 -0.002 …
180
M. Ranzato
![Page 181: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/181.jpg)
Copies* are over-represented in the output of beam search • Copies make up 2.0% of the WMT14 En-Fr training set • Among beam hypotheses, copies account for: Beam=1: 2.6% Beam=5: 2.9% Beam=20: 3.5%
* a copy is a translation that shares >= 50% of its unigrams with the source
Uncertainty & Search
181
M. Ranzato
![Page 182: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/182.jpg)
Copies* are over-represented in the output of beam search • Copies make up 2.0% of the WMT14 En-Fr training set • Among beam hypotheses, copies account for: Beam=1: 2.6% Beam=5: 2.9% Beam=20: 3.5%
* a copy is a translation that shares >= 50% of its unigrams with the source
Uncertainty & Search
A simple idea: filter copies during search
182
M. Ranzato
![Page 183: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/183.jpg)
(WMT17 En-De)
Uncertainty & Search
183
M. Ranzato
![Page 184: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/184.jpg)
(WMT17 En-De)
Uncertainty & Search
184
M. Ranzato
![Page 185: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/185.jpg)
(WMT17 En-De)
Uncertainty & Search
185
M. Ranzato
![Page 186: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/186.jpg)
… and how closely does the model distribution match the data distribution? Challenging because: • We typically observe only a single sample from the data
distribution for each source sentence (i.e., one reference translation)
• The model and data distributions are intractable to enumerate We instead introduce necessary conditions for matching
How is uncertainty represented in the model distribution?
M. Ranzato
![Page 187: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/187.jpg)
Analyzing the model distribution
What are the necessary conditions for the model distribution to match the data distribution: • …at the token level?
• …at the sequence level?
• …when considering multiple reference translations?
187
M. Ranzato
![Page 188: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/188.jpg)
Analyzing the model distribution—Token Level
(WMT14 En-Fr)
Histogram of unigram frequencies
188
M. Ranzato
![Page 189: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/189.jpg)
Histogram of unigram frequencies Beam under-estimates the rarest words, although sampling is not as bad
(WMT14 En-Fr)
Analyzing the model distribution—Token Level
189
M. Ranzato
![Page 190: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/190.jpg)
(WMT14 En-Fr)
Histogram of unigram frequencies Beam under-estimates the rarest words, although sampling is not as bad Beam over-estimates frequent words.We should expect this!
Analyzing the model distribution—Token Level
190
M. Ranzato
![Page 191: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/191.jpg)
(WMT14 En-Fr)
Histogram of unigram frequencies Beam under-estimates the rarest words, although sampling is not as bad Beam over-estimates frequent words.We should expect this! Sampling mostly matches the reference data distribution
Analyzing the model distribution—Token Level
191
M. Ranzato
![Page 192: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/192.jpg)
Analyzing the model distribution—Sequence Level
Synthetic experiment: • Retrain model on news subset of WMT, which does
not contain copies • Artificially introduce copies in the training data with
probability pnoise
• Measure rate of copies among sampled hypotheses
192
M. Ranzato
![Page 193: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/193.jpg)
0.001 0.01 0.1 0.5
0.0001
0.001
0.01
0.1
1.0
rate
perfect matchexact copy
Analyzing the model distribution—Sequence Level
(WMT17 En-De)193
M. Ranzato
![Page 194: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/194.jpg)
0.001 0.01 0.1 0.5
0.0001
0.001
0.01
0.1
1.0
rate
perfect matchexact copy
Model under-estimates copies at a sequence level
Analyzing the model distribution—Sequence Level
(WMT17 En-De)194
M. Ranzato
![Page 195: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/195.jpg)
0.001 0.01 0.1 0.5
0.0001
0.001
0.01
0.1
1.0
rate
perfect matchexact copypartial (incl. exact) copy
Partial copies* do not appear in training, yet…
The model smears probability mass in hypothesis space
(WMT17 En-De)
* A partial copy has a unigram overlap of >= 50% with the source
Analyzing the model distribution—Sequence Level
195
M. Ranzato
![Page 196: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/196.jpg)
Collect 10 additional reference translations from distinct human translators • 500 sentences (En-Fr) and 500 sentences (En-De) • 10K sentences total • Available at: github.com/facebookresearch/analyzing-uncertainty-nmt
Analyzing the model distribution—with Mult. References
196
M. Ranzato
![Page 197: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/197.jpg)
Sent
ence
BLE
U
single reference
38.241.4
44.5
inter-humanbeam (K=5)sampling (K=200)
(WMT14 En-Fr)197
M. Ranzato
![Page 198: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/198.jpg)
Sent
ence
BLE
U
single reference oracle reference
64.1
38.2
70.2
41.4
71
44.5
inter-humanbeam (K=5)sampling (K=200)
(WMT14 En-Fr)
oracle reference: BLEU w.r.t. best matching reference
Source: Thanks a lot! Best hypothesis: Merci!
Ref1: Merci beaucoup! Ref2: Merci beaucoup. Ref3: Merci! Ref4: ….
198
M. Ranzato
![Page 199: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/199.jpg)
Sent
ence
BLE
U
single reference oracle reference
64.1
38.2
70.2
41.4
71
44.5
inter-humanbeam (K=5)sampling (K=200)
(WMT14 En-Fr)
oracle reference: BLEU w.r.t. best matching reference
199
M. Ranzato
![Page 200: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/200.jpg)
Sent
ence
BLE
U
single reference oracle reference
64.1
38.2
70.2
41.4
71
44.5
inter-humanbeam (K=5)sampling (K=200)
(WMT14 En-Fr)
The best beam hypothesis is very close to a reference
oracle reference: BLEU w.r.t. best matching reference
200
M. Ranzato
![Page 201: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/201.jpg)
Sent
ence
BLE
U
single reference oracle reference average oracle
39.1
64.1
38.2
65.770.2
41.4
71
44.5
inter-humanbeam (K=5)sampling (K=200)
(WMT14 En-Fr)
average oracle: average oracle reference BLEU over top-K hypotheses
Source: Thanks a lot! Hypothesis #1: Merci! Hypothesis #2: Merci merci!
Ref1: Merci beaucoup! Ref2: Merci beaucoup. Ref3: Merci! Ref4: …. 201
M. Ranzato
![Page 202: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/202.jpg)
Sent
ence
BLE
U
single reference oracle reference average oracle
39.1
64.1
38.2
65.770.2
41.4
71
44.5
inter-humanbeam (K=5)sampling (K=200)
(WMT14 En-Fr)
average oracle: average oracle reference BLEU over top-K hypotheses
202
M. Ranzato
![Page 203: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/203.jpg)
Most beam hypotheses are close to a reference
Sent
ence
BLE
U
single reference oracle reference average oracle
39.1
64.1
38.2
65.770.2
41.4
71
44.5
inter-humanbeam (K=5)sampling (K=200)
Most sampled hypotheses are far from a reference
(WMT14 En-Fr)203
M. Ranzato
![Page 204: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/204.jpg)
# refs covered7.4
5
1.9
beam K=5beam K=200sampling K=200
# refs covered: number of distinct references (out of 10) matched to at least one hypothesis
Sampling covers more hypotheses (is more diverse) than beam search
M. Ranzato
![Page 205: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/205.jpg)
Conclusion
• NMT models capture uncertainty in their output distributions • Beam search is efficient and effective, but prefers frequent words • Degradation with large beams is mostly due to copying, but this can
be mitigated by filtering the training set • Models are well calibrated at the token level, but smear probability
mass at the sequence level • Smearing may be responsible for lack of diversity in beam search
outputs
Dataset link: github.com/facebookresearch/analyzing-uncertainty-nmt
205
M. Ranzato
![Page 206: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/206.jpg)
Questions? Вопросы?
¿Preguntas? Domande?
M. Ranzato206 M. Ranzato
![Page 207: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/207.jpg)
Outline• PART 0 [lecture 1]
• Natural Language Processing & Deep Learning
• Neural Machine Translation
• Part 1 [lecture 1]
• Unsupervised Word Translation
• Part 2 [lecture 2]
• Unsupervised Sentence Translation
• Part 3 [lecture 3]
• Uncertainty
• Sequence-Level Prediction in Machine Translation
207
![Page 208: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/208.jpg)
Classical Structured Prediction Losses for Sequence to Sequence Learning Sergey Edunov, Myle Ott, Michael Auli, David Grangier, Marc'Aurelio Ranzato NAACL 2018 https://arxiv.org/abs/1711.04956
Myle OttSergey Edunov Michael Auli David Grangier
208
![Page 209: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/209.jpg)
Problems• Model is asked to predict a single token at training time,
but the whole sequence at test time.
• Exposure bias: training and testing are inconsistent because model has never observed its own predictions at training time.
• At training time, we optimize for a different loss.
• Evaluation criterion is not differentiable.
M. RanzatoSequence level training with RNNs, Ranzato et al. ICLR 2016
![Page 210: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/210.jpg)
Selection of Recent Literature• RL-inspired methods
• MIXER
• Actor-Critic
• Using beam search at training time:
• BSO
• Distillation based
Ranzato et al. ICLR 2016
Bahdanau et al. ICLR 2017
Wiseman et al. ACL 2016
Kim et al. EMNLP 2016
M. Ranzato210
![Page 211: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/211.jpg)
QuestionHow do classical structure prediction losses compare against these recent methods?
Classical losses were often applied to log-linear models and/or other problems than MT.
Tsochantaridis et al. “Large margin methods for structured and interdependent output variables” JMLR 2005
Och “Minimum error rate training in statistical machine translation” ACL 2003
Smith and Eisner “Minimum risk annealing for training log-linear models” ACL 2006
Gimpel and Smith “Softmax-margin CRFs: training log-linear models with cost functions” ACL 2010
Taskar et al. “Max-margin Markov networks” NIPS 2003Collins “Discriminative training methods for HMMs” EMNLP 2002
M. Ranzato
Bottou et al. “Global training of document processing systems with graph transformer networks” CVPR 1997
211
![Page 212: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/212.jpg)
QuestionHow do classical structure prediction losses compare against these recent methods?
Classical losses were often applied to log-linear models and/or other problems than MT.
M. Ranzato
Can the Energy-Based Model framework help unifying these different approaches?
LeCun et al. “A tutorial on energy-based learning” MIT Press 2006
![Page 213: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/213.jpg)
Energy-Based Learning
LeCun et al. “A tutorial on energy-based learning” MIT Press 2006 M. Ranzato
space of possible predictions
Energy
target prediction
During training
@L@✓
=@E
@✓
���(x,y+)
� @E
@✓
���(x,y�)
Some losses have explicit negative term, others replace it with constraints in the loss or in the architecture.
![Page 214: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/214.jpg)
Energy-Based Learning
LeCun et al. “A tutorial on energy-based learning” MIT Press 2006 M. Ranzato
space of possible predictionstarget=prediction
After trainingEnergy
![Page 215: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/215.jpg)
Challenges
M. Ranzato
space of possible predictionstarget prediction
Key questions if we want to extend EBMs to MT: • how to search for most likely output? Enumeration & exact search are intractable. • how to deal with uncertainty? • what if target is not reachable?
Energy
215
![Page 216: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/216.jpg)
M. Ranzato
Key questions if we want to extend EBMs to MT: • how to search for most likely output? Enumeration & exact search are intractable. • how to deal with uncertainty? What if we only observe one minimum among many? • what if target is not reachable?
EXAMPLESource: The night before would be practically sleepless .
Target #1: La nuit qui précède pourrait s’avérer quasiment blanche .Target #2: Il ne dormait pratiquement pas la nuit précédente .Target #3: La nuit précédente allait être pratiquement sans sommeil .Target #4: La nuit précédente , on n’a presque pas dormi .Target #5: La veille , presque personne ne connaitra le sommeil .
Challenges
Ott et al. “Analyzing uncertainty in NMT” arXiv:1803.00047 2018
![Page 217: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/217.jpg)
M. Ranzato
Key questions if we want to extend EBMs to MT: • how to search for most likely output? Enumeration & exact search are intractable. • how to deal with uncertainty? What if we only observe one minimum among many? • what if target is not reachable?
EXAMPLESource: nice .
Target #1: chouette .Target #2: belle .Target #3: beau .
Challenges
Ott et al. “Analyzing uncertainty in NMT” arXiv:1803.00047 2018
![Page 218: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/218.jpg)
M. Ranzato
Key questions if we want to extend EBMs to MT: • how to search for most likely output? Enumeration & exact search are intractable. • how to deal with uncertainty? What if we only observe one minimum among many? • what if target is not reachable? E.g.: Not reachable = no hyp. in the beam is close to
the reference.
Challenges
Ott et al. “Analyzing uncertainty in NMT” ICML 2018
![Page 219: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/219.jpg)
Notation
x = (x1, . . . , xm) input sentence
M. Ranzato219
![Page 220: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/220.jpg)
Notation
input sentence
t
x
target sentence
M. Ranzato220
![Page 221: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/221.jpg)
Notation
input sentence
t
x
target sentence
u hypothesis generated by the model
M. Ranzato221
![Page 222: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/222.jpg)
Notation
input sentence
t
x
target sentence
u hypothesis generated by the model
oracle hypothesis
M. Ranzato
u⇤ = arg minu2U(x)
cost(u, t)
222
![Page 223: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/223.jpg)
Notation
input sentence
t
x
target sentence
u hypothesis generated by the model
u⇤ oracle hypothesis
most likely hypothesis
M. Ranzato
u = arg minu2U(x)
� log p(u|x)
223
![Page 224: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/224.jpg)
Baseline: Token Level NLL
LTokNLL = �nX
i=1
log p(ti|t1, . . . , ti�1,x)
for one particular training example and omitting dependence on model parameters.
M. Ranzato224
![Page 225: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/225.jpg)
Sequence Level NLL
LSeqNLL = � log p(u⇤|x) + logX
u2U(x)
p(u|x)
M. Ranzato
The sequence log-probability is simply the sum of the token-level log-probabilities.
}Energy
225
![Page 226: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/226.jpg)
Sequence Level NLL
LSeqNLL = � log p(u⇤|x) + logX
u2U(x)
p(u|x)
M. Ranzato
The sequence log-probability is simply the sum of the token-level log-probabilities.
Homework: compute gradients of loss w.r.t. inputs to token level softmaxes.
Two key differences: choice of target and hypothesis set.
normalize over reachable set
decrease energy of reachable hyp. with lowest cost
226
![Page 227: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/227.jpg)
Sequence Level NLL
LSeqNLL = � log p(u⇤|x) + logX
u2U(x)
p(u|x)
hypothesis space
Energy
t u⇤
U(x)
}gradients
set of hypotheses reachable by the model
M. Ranzato227
![Page 228: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/228.jpg)
ExampleSource:Wir müssen unsere Einwanderungspolitik in Ordnung bringen.
TargetWe have to fix our immigration policy.
Beam:BLEU Model score 75.0 -0.23 We need to fix our immigration policy.36.9 -0.36 We need to fix our policy policy.66.1 -0.42 We have to fix our policy policy.66.1 -0.44 We've got to fix our immigration policy.
M. Ranzato228
![Page 229: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/229.jpg)
ExampleSource:Wir müssen unsere Einwanderungspolitik in Ordnung bringen.
TargetWe have to fix our immigration policy.
Beam:BLEU Model score 75.0 -0.23 We need to fix our immigration policy.36.9 -0.36 We need to fix our policy policy.66.1 -0.42 We have to fix our policy policy.66.1 -0.44 We've got to fix our immigration policy.
M. Ranzato229
![Page 230: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/230.jpg)
Observations• Important to use oracle hypothesis as surrogate target
as opposed to golden target. Otherwise, the model learns to assign very bad scores to its own hypotheses but is not trained to reach the target.
• Evaluation metric only used for oracle selection of target.
• Several ways to generate : beam, sampling, …
• Similar to token level NLL but normalizing over (subset of) hypotheses. Hypothesis score: average token level log-probability.
U(x)
M. Ranzato230
![Page 231: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/231.jpg)
Expected RiskLRisk =
X
u2U(x)
cost(t,u)p(u|x)P
u02U(x) p(u0|x)
• The cost is the evaluation metric; e.g.: 100-BLEU.
• REINFORCE [1] is a special case of this (a single sample Monte Carlo estimate of the expectation over the whole hypothesis space).
M. Ranzato
Homework: compute gradients of loss w.r.t. inputs to token level softmaxes.
[1] Sequence level training with RNNs, Ranzato et al. ICLR 2016
![Page 232: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/232.jpg)
Source:Wir müssen unsere Einwanderungspolitik in Ordnung bringen.
TargetWe have to fix our immigration policy.
Beam:BLEU Model score 75.0 -0.23 We need to fix our immigration policy. 36.9 -0.36 We need to fix our policy policy.66.1 -0.42 We have to fix our policy policy.66.1 -0.44 We've got to fix our immigration policy.
Example
(expected BLEU=42)M. Ranzato232
![Page 233: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/233.jpg)
Example
hypothesis spacet u⇤
U(x)
}gradients
set of hypotheses reachable by the model
M. Ranzato
Energy
233
![Page 234: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/234.jpg)
Max-Margin
• Energy: (negative) un-normalized score (or log-odds).
• Margin:
• The cost is our evaluation metric; e.g.: 100-BLEU.
• Increase score of oracle hypothesis, while decreasing score of most likely hypothesis.
M. Ranzato
Homework: compute gradients of loss w.r.t. inputs to token level softmaxes.
LMaxMargin = max[0,m� (E(u)� E(u⇤))]
m = cost(t, u)� cost(t,u⇤)
234
![Page 235: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/235.jpg)
Source:Wir müssen unsere Einwanderungspolitik in Ordnung bringen.
TargetWe have to fix our immigration policy.
Beam:BLEU Model score 66.1 -0.20 We have to fix our policy policy.75.0 -0.23 We need to fix our immigration policy.36.9 -0.36 We need to fix our policy policy.66.1 -0.44 We've got to fix our immigration policy.
M. Ranzato
Max-Margin
235
![Page 236: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/236.jpg)
hypothesis spacet u⇤
U(x)
}gradients
set of hypotheses reachable by the model
u
M. Ranzato
Max-Margin
Energy
236
![Page 237: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/237.jpg)
Check out the paper for more examples of sequence level training losses!
M. Ranzato237
![Page 238: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/238.jpg)
Practical Tips• Start from a model pre-trained at the token level. Training with
search is excruciatingly slow…
• Even better if pre-trained model had label smoothing.
• Accuracy VS speed trade-off: offline/online generation of hypotheses.
• Cost rescaling.
• Mix token level NLL loss with sequence level loss to improve robustness.
• Need to regularize more.
M. Ranzato238
![Page 239: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/239.jpg)
Results on IWSLT’14 De-EnTEST
TokNLL (Wiseman et al. 2016) 24.0
BSO(Wiseman et al. 2016) 26.4
Actor-Critic(Bahdanau et al. 2016) 28.5
Phrase-based NMT(Huang et al. 2017) 29.2
our TokNLL 31.7
SeqNLL 32.7
Risk 32.9
Perceptron 32.6
M. Ranzato239
![Page 240: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/240.jpg)
Results on IWSLT’14 De-EnTEST
TokNLL (Wiseman et al. 2016) 24.0
BSO(Wiseman et al. 2016) 26.4
Actor-Critic(Bahdanau et al. 2016) 28.5
Phrase-based NMT(Huang et al. 2017) 29.2
our TokNLL 31.7
SeqNLL 32.7
Risk 32.9
Max-Margin 32.6
M. Ranzato240
![Page 241: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/241.jpg)
Observations• Sequence level training does improve evaluation metric
(both on training and) on test set.
• There is not so much difference between the different variants of losses. Risk is just slightly better.
• In our implementation and using the same computational resources, sequence level training is 26x slower per update using online beam generation of 5 hypotheses.
• Hard comparison since each paper has a different baseline!
M. Ranzato241
![Page 242: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/242.jpg)
Observations• Sequence level training does improve evaluation metric
(both on training and) on test set.
• There is not so much difference between the different variants of losses. Risk is just slightly better.
• In our implementation and using the same computational resources, sequence level training is 26x slower per update using online beam generation of 5 hypotheses.
• Hard comparison since each paper has a different baseline!
M. Ranzato242
![Page 243: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/243.jpg)
Fair Comparison to BSOTEST
TokNLL (Wiseman et al. 2016) 24.0
BSO(Wiseman et al. 2016) 26.4
Our re-implementation of their TokNLL 23.9
Risk on top of the above TokNLL 26.7
M. Ranzato243
![Page 244: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/244.jpg)
Fair Comparison to BSOTEST
TokNLL (Wiseman et al. 2016) 24.0
BSO(Wiseman et al. 2016) 26.4
Our re-implementation of their TokNLL 23.9
Risk on top of the above TokNLL 26.7
M. Ranzato
These methods fare comparably once the baseline is the same…244
![Page 245: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/245.jpg)
Diminishing Returns
M. Ranzato
On WMT’14 En-Fr, TokNLL gets 40.6 while Risk gets 41.0The stronger the baseline, the less to be gained.
245
![Page 246: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/246.jpg)
Large Models in MT
Beam search is very effective; only 20% of the tokens with probability < 0.7 (despite exposure bias)!
![Page 247: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/247.jpg)
Large Models in MT
Very large NMT models make almost deterministic transitions. No much to be gained by sequence level training.
![Page 248: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/248.jpg)
Conclusion• Sequence level training does improve, but with
diminishing returns. It’s computationally very expensive.
• If model has little uncertainty (because of the task and because of the model being well (over)fitted), then sequence level training does not help much.
• The particular method to train at the sequence level does not really matter.
• Sequence level training is more prone to overfitting.
M. Ranzato248
![Page 249: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/249.jpg)
EBMs & MT• Nice unifying framework.
• Different losses apply different weights to the “pull-up” and “pull-down” gradients.
• Two key differences two usual EBM learning:
• restrict set of hypotheses to those that are reachable, and
• replace actual target by oracle hypothesis.
M. Ranzato249
![Page 250: Challenges in Neural Machine Translations · 2020-03-11 · Word Embeddings • LSA learns word embeddings that take into account co-occurrences across documents. • bi-gram instead](https://reader035.vdocuments.site/reader035/viewer/2022070811/5f09e96c7e708231d42918f2/html5/thumbnails/250.jpg)
Questions? Вопросы?
¿Preguntas? Domande?
M. Ranzato250