![Page 1: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/1.jpg)
Linguistic Regularities in Sparse and Explicit
Word Representations
Omer Levy Yoav GoldbergBar-Ilan University
Israel
![Page 2: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/2.jpg)
Papers in ACL 2014*
Neural Networks &
Word Embeddings
Other Topics
* Sampling error: +/- 100%
![Page 3: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/3.jpg)
Neural Embeddings
•
![Page 4: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/4.jpg)
Representing words as vectors is not new!
![Page 5: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/5.jpg)
Explicit Representations (Distributional)•
![Page 6: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/6.jpg)
Questions
• Are analogies unique to neural embeddings?Compare neural embeddings with explicit representations
• Why does vector arithmetic reveal analogies?Unravel the mystery behind neural embeddings and their “magic”
![Page 7: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/7.jpg)
Background
![Page 8: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/8.jpg)
Mikolov et al. (2013a,b,c)
• Neural embeddings have interesting geometries
![Page 9: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/9.jpg)
![Page 10: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/10.jpg)
Mikolov et al. (2013a,b,c)
• Neural embeddings have interesting geometries
• These patterns capture “relational similarities”
• Can be used to solve analogies:man is to woman as king is to queen
![Page 11: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/11.jpg)
Mikolov et al. (2013a,b,c)
•
![Page 12: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/12.jpg)
Mikolov et al. (2013a,b,c)
•
![Page 13: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/13.jpg)
Mikolov et al. (2013a,b,c)
![Page 14: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/14.jpg)
Mikolov et al. (2013a,b,c)
![Page 15: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/15.jpg)
Mikolov et al. (2013a,b,c)
![Page 16: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/16.jpg)
Mikolov et al. (2013a,b,c)
![Page 17: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/17.jpg)
Mikolov et al. (2013a,b,c)
![Page 18: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/18.jpg)
Mikolov et al. (2013a,b,c)
![Page 19: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/19.jpg)
Are analogies unique to neural embeddings?
![Page 20: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/20.jpg)
• Experiment: compare embeddings to explicit representations
Are analogies unique to neural embeddings?
![Page 21: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/21.jpg)
Are analogies unique to neural embeddings?• Experiment: compare embeddings to explicit representations
![Page 22: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/22.jpg)
Are analogies unique to neural embeddings?• Experiment: compare embeddings to explicit representations
• Learn different representations from the same corpus:
![Page 23: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/23.jpg)
Are analogies unique to neural embeddings?•
![Page 24: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/24.jpg)
Analogy Datasets
•
![Page 25: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/25.jpg)
Embedding vs Explicit (Round 1)
![Page 26: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/26.jpg)
Embedding vs Explicit (Round 1)
MSR Google0%
10%
20%
30%
40%
50%
60%
70%
Embedding54%
Embedding63%
Explicit29%
Explicit45%
Accu
racy
Many analogies recovered by explicit, but many more by embedding.
![Page 27: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/27.jpg)
Why does vector arithmetic reveal analogies?
![Page 28: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/28.jpg)
Why does vector arithmetic reveal analogies?•
![Page 29: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/29.jpg)
Why does vector arithmetic reveal analogies?•
![Page 30: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/30.jpg)
Why does vector arithmetic reveal analogies?•
![Page 31: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/31.jpg)
Why does vector arithmetic reveal analogies?•
![Page 32: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/32.jpg)
Why does vector arithmetic reveal analogies?•
![Page 33: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/33.jpg)
Why does vector arithmetic reveal analogies?•
![Page 34: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/34.jpg)
Why does vector arithmetic reveal analogies?•
![Page 35: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/35.jpg)
Why does vector arithmetic reveal analogies?•
royal? female?
![Page 36: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/36.jpg)
What does each similarity term mean?• Observe the joint features with explicit representations!
uncrowned Elizabethmajesty Katherinesecond impregnate
… …
![Page 37: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/37.jpg)
Can we do better?
![Page 38: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/38.jpg)
Let’s look at some mistakes…
![Page 39: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/39.jpg)
Let’s look at some mistakes…
![Page 40: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/40.jpg)
Let’s look at some mistakes…
![Page 41: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/41.jpg)
Let’s look at some mistakes…
![Page 42: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/42.jpg)
The Additive Objective
•
![Page 43: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/43.jpg)
The Additive Objective
•
![Page 44: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/44.jpg)
The Additive Objective
•
![Page 45: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/45.jpg)
The Additive Objective
•
![Page 46: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/46.jpg)
The Additive Objective
•
![Page 47: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/47.jpg)
The Additive Objective
•
• Problem: one similarity might dominate the rest• Much more prevalent in explicit representation• Might explain why explicit underperformed
![Page 48: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/48.jpg)
How can we do better?
![Page 49: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/49.jpg)
How can we do better?
• Instead of adding similarities, multiply them!
![Page 50: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/50.jpg)
How can we do better?
•
![Page 51: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/51.jpg)
How can we do better?
•
![Page 52: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/52.jpg)
Embedding vs Explicit (Round 2)
![Page 53: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/53.jpg)
Multiplication > Addition
MSR Google MSR GoogleEmbedding Explicit
0%
10%
20%
30%
40%
50%
60%
70%
80%
Add54%
Add63%
Add29%
Add45%
Mul59%
Mul67% Mul
57%
Mul68%Ac
cura
cy
![Page 54: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/54.jpg)
Explicit is on-par with Embedding
MSR Google0%
10%
20%
30%
40%
50%
60%
70%
80%
Embedding59%
Embedding67%Explicit
57%
Explicit68%Ac
cura
cy
![Page 55: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/55.jpg)
Explicit is on-par with Embedding
• Embeddings are not “magical”
• Embedding-based similarities have a more uniform distribution
• The additive objective performs better on smoother distributions
• The multiplicative objective overcomes this issue
![Page 56: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/56.jpg)
Conclusion
• Are analogies unique to neural embeddings?No! They occur in sparse and explicit representations as well.
• Why does vector arithmetic reveal analogies?Because vector arithmetic is equivalent to similarity arithmetic.
• Can we do better?Yes! The multiplicative objective is significantly better.
![Page 57: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/57.jpg)
More Results and Analyses (in the paper)• Evaluation on closed-vocabulary analogy questions (SemEval 2012)
• Experiments with a third objective function (PairDirection)
• Do different representations reveal the same analogies?
• Error analysis
• A feature-level interpretation of how word similarity reveals analogies
![Page 58: Linguistic Regularities in Sparse and Explicit Word Representations Omer LevyYoav Goldberg Bar-Ilan University Israel](https://reader037.vdocuments.site/reader037/viewer/2022103022/56649d1f5503460f949f3786/html5/thumbnails/58.jpg)