Upload File

word2vec - homepages of uva/fnwi staff€¦ · challenge:)dealing)with)phrases) •...

  • Home
  • Documents
  • word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)
25
word2vec Tom Kenter 20142015.Informa7on Retrieval 1 January 13 2014

Upload: others

Post on 21-Jun-2020

0 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

word2vec  

Tom  Kenter  20142015.Informa7on  Retrieval  1  

January  13  2014  

Page 2: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

word2vec    

seman7cs    

word  embeddings  

Page 3: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Word2vec:  what  is  it  (not)?  It  is:  •  Word  embeddings  =  vectors  representa7on  for  words  

•  Mapping  to  seman7c/syntac7c  space  •  Co-­‐occurrence-­‐based  •  Neural  network  approach  •  Introduced  by  Tomas  Mikolov  (then  at  Google)  

It  is  not:  •  Deep  learning  

Page 4: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Word  embeddings  

Vector  representa7on  of  a  word    

       vector(“nice”)          =  <0.12  0.432  0.2424  …  …  0.65  0.43>            vector(“good”)      =  <0.11  0.322  0.204  …  …  0.53  0.393>          ...          ...          vector(“Paris”)        =  <0.67  0.101  0.74  …  …  0.303  0.112>            vector(“France”)  =  <0.74  0.007  0.568  …  …  0.23  0.102>    

N  dimensions  

V  words  

Page 5: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Vector  similarity  

Cosine  similarity  between  vectors  A  and  B  

Page 6: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Neural  network  language  models  

Figure  from  [6]  

Soc  max  

wI wj

Page 7: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Con7nuous  Bag  Of  Word  architecture  

Figure  from  [6]  

wi-­‐2    wi-­‐1      wi        wi+1    wi+2  

Page 8: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Skip-­‐gram  architecture  

-­‐  Hierarchical  socmax    -­‐  Nega7ve  sampling  

Figure  from  [6]  

wi-­‐2    wi-­‐1      wi        wi+1    wi+2  

Page 9: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

In  short:  two  architectures  

CBOW                            Skip-­‐gram    

•  CBOW  Con7nuous  Bag  Of  Words  Predict  a  missing  word  given  a  window  of  context  words  

•  Skip-­‐gram  Predict  context  words  given  a  word  

Page 10: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Two  architectures  

Figure  from  [4]  

Page 11: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Where  are  the  word  embeddings?  

Figure  from  [6]  

Page 12: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Advantages  •  It  scales  

–  Train  on  billion  word  corpora  –  In  limited  7me  –  Mikolov  men7ons  parallel  training  

•  Word  embeddings  trained  by  one  can  be  used  by  others  –  For  en7rely  different  tasks  –  This  happens  indeed!  

•  Incremental  training  Train  on  one  piece  of  data,  save  results,  con7nue  training  later  on    

•  The  code  is  available  –  Original  source  code  from  Mikolov:  hkps://code.google.com/p/word2vec/  

–  There  is  a  very  nice  Python  module  for  it!  Gensim  word2vec:  hkp://radimrehurek.com/gensim/models/word2vec.html  

Page 13: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

What  can  you  do  with  it?  

A  is  to  B  as  C  is  to  ?    This  is  the  famous  example:  

vector(king)  –  vector(man)  +  vector(woman)  =  

vector(queen)    Actually,  what  the  original  paper  says  is:  if  you  substract  the  vector  for  ‘man’  from  the  one  for  ‘king’  and  add  the  vector  for  ‘woman’,  the  vector  closest  to  the  one  you  end  up  with  turns  out  to  be  the  one  for  ‘queen’.  

Page 14: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

What  can  you  do  with  it?  

A  is  to  B  as  C  is  to  ?    It  also  works  for  syntac7c  rela7ons:    X  =  vector(biggest)  -­‐  vector(big)  +  vector(small).      X  =  vector(smallest)  

Page 15: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

What  can  you  do  with  it?  

A  is  to  B  as  C  is  to  ?    More  interes7ng:  which  country  does  a  par7cular  city  belong  to?    

France  is  to  Paris  as  Germany  is  to  Berlin.    

Page 16: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Why  does  this  happen?  

Figure  from  [5]  

Page 17: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Evalua7on  

•  Mikolov  presents  an  evalua7on  set  in  [4]    

Seman0c  8869  of  these    

Syntac0c  10675  of  these  

Table  from  [4]  

Page 18: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Evalua7on  

Table  from  [4]  

Page 19: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

What  can  you  do  with  it?  

•  Sentence  comple7on  Microsoc  Sentence  Comple7on  Challenge    

•  Selec7ng  out-­‐of-­‐the-­‐list  words    Example:  which  word  does  not  belong  in  [monkey,  lion,  dog,  truck]  

•  Bilingual  Word  Embeddings  for  Phrase-­‐Based  Machine  Transla7on.  EMNLP  –  2013.  

•  Synonym  detec7on    Choose  among  4  candidates  which  one  is  the  best  synonym  for  a  given  word.  

Page 20: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

What  can  you  do  with  it?  

•  Concept  categoriza7on  Group  helicopter  and  motorcycle  together,  and  dog  an  elephant  

•  Selec7onal  preference  Predict  typical  verb  noun  pairs.  Like  people  go  well  with  to  eat  as  a  subject  (but  not  as  an  object).  

Page 21: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Seeing  changes  in  word  meaning/usage  over  7me  

Figure  from  [3]  

Page 22: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Challenge:  dealing  with  phrases  •  Deal  with  this  during  pre-­‐processing  step  

Find  colloca7ons  the  usual  way,  and  feed  them  as  single  ‘words’  to  the  NN    

•  Just  add/average/concatenate  the  vectors  This  is  beker  than  the  previous  approach  if  you  happen  to  be  dealing  with  en77es  like  Bengal  Cger  versus  Sumatran  Tiger    

•  Paragraph2vec  Make  fixed-­‐length  representa7on  of  variable-­‐length  input  

•  Build  a  convolu7onal  NN  on  top  of  word2vec  This  is  done  in  [2]  for  sentence  classifica7on  

Page 23: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Disadvantages/challenges/future  research  

•  Ambiguity  What  if  there  are  two  Amsterdams..???  

•  Evalua7on  is  only  extrinsic  (but  this  goes  for  rankers  as  well)  

•  Parameter  sevng  involves  magic/luck/trial  and  error  There  is  no  such  thing  as  free  lunch  

•  Very  picky  on  literal  words  But  this  affects  all  other  word-­‐based  methods  as  well  

•  Can  not  deal  with  OOV  words  If  a  word  is  met  at  tes7ng  7me  that  was  not  seen  at  training  7me,  it  will  simply  be  ignored.  

Page 24: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

Who  are  the  associated  names  •  Mikolov  (Google)  •  Bengio  (of  Deep  Learning  fame)  He  already  proposed  NN  LMs  and  word  embeddings  a  decade  ago  

 •  There  are  other  algorithms  for  gevng  word  embeddings.  Most  notably:    GloVe:  Global  Vectors  for  Word  Representa7on  Jeffrey  Pennington,  Richard  Socher,  Christopher  D.  Manning  –  2014  All  from  Stanford  

Page 25: word2vec - Homepages of UvA/FNWI staff€¦ · Challenge:)dealing)with)phrases) • Deal)with)this)during)preMprocessing)step) Find)collocaons)the)usual)way,)and)feed)them)as)single)‘words’)to)

(By  no  means  extensive)  list  of  papers  Overview  and  evaluaCon  of  many  different  tasks  [1]    M.  Baroni,  G.  Dinu,  and  G.  Kruszewski.  Dont  count,  predict!  a  systema7c  comparison  of  context-­‐coun7ng  vs.  context-­‐predic7ng  seman7c  vectors.  In  Proceedings  of  the  52nd  Annual  Mee7ng  of  the  Associa7on  for  Computa7onal  Linguis7cs,  volume  1,  2014    Sentence  classificaCon    [2]    Y.  Kim,  Convolu7onal  neural  networks  for  sentence  classifica7on,  2014      This  is  where  the  ‘gay’  and  ‘cell’  example  came  from  [3]    Y.  Kim,  Y.-­‐I.  Chiu,  K.  Hanaki,  D.  Hedge,  and  S.  Petrov.  Temporal  analysis  of  language  through  neural  language  models,  2014      For  more  elaborate  discussion  on  the  architectures  used  [4]    T.  Mikolov,  K.  Chen,  G.  Corrado,  and  J.  Dean.  Efficient  es7ma7on  of  word  representa7ons  in  vector  space.  arXiv  preprint  arXiv:1301.3781,  2013      More  on  Skip-­‐gram,  negaCve  sampling,  hierarchical  soOmax,  and  dealing  with  phrases  [5]    T.  Mikolov,  I.  Sutskever,  K.  Chen,  G.  S.  Corrado,  and  J.  Dean.  Distributed  representa7ons  of  words  and  phrases  and  their  composi7onality.  In  Advances  in  Neural  Informa7on  Processing  Systems,  pages  3111–3119,  2013      Very  nice  detailed  explanaCon  of  how  word2vec  works,  with  derivaCons  of  the  back  propagaCon  equaCons,  etc.  [6]  Xin  Rong,  "word2vec  Parameter  Learning  Explained."  arXiv  preprint  arXiv:1411.2738,  2014    Link  to  a  demo  (scroll  down  a  bit  to  where  it  says  ‘Bonus  App’)  hkp://radimrehurek.com/2014/02/word2vec-­‐tutorial/    


TEMPORAL LOGIC - Homepages of UvA/FNWI staff the next section we discuss some of the most well-known mathemat-ical modellings of time. These are the structures the formal languages

Jan van de Craats 15 november 2005 - Homepages of UvA/FNWI ... · (x,y) van R2 dat behoort bij het punt P heet het coo¨rdinatenpaar, of kortweg de co-ordinaten van P. Daarbij moet

A Crash Course in Arrow Logic - Homepages of UvA/FNWI … ·  · 2014-07-25A Crash Course in Arrow Logic Yde Venema Department of Mathematics and Computer Science Free University

Word2Vec/Glove/FastTextling.snu.ac.kr/class/AI_Agent/lecture/07-2-1-Word2Vec... · 2018-06-15 · Word2Vec/Glove/FastText Hyopil Shin(Seoul National University) Computational Linguistics

Noncommutativity as a colimit - Homepages of UvA/FNWI staff

word2vec - jlu.myweb.cs.uwindsor.ca

ALGEBRA III - Homepages of UvA/FNWI staff

amaha - Homepages

Portfolio Theory - Homepages of UvA/FNWI staff ·  · 2017-10-09Preface These lecture notes have been written for and during the course Portfolio Theory at the Universiteit van Amsterdam

Deep Learning Framework based on Word2Vec and CNN for ... · space [4]. Moreover, with Word2vec features can be obtained without human intervention. Word2Vec can also perform effectively

Science as Art - Homepages of UvA/FNWI staff

Several Complex Variables - Homepages of UvA/FNWI staff · PDF fileSeveral Complex Variables Jaap Korevaar, ... 7.6 Solution of ∂on polydiscs ... 8.7 Some applications of Cncapacities

Syllabus Algebra I - Homepages of UvA/FNWI staff · Deze syllabus is een nieuwe editie anv de syllabus Algebra, Deel A: Grepoen geschreven door de hoog- ... 2

Chapter 1 Representations - Homepages of UvA/FNWI staff · 2015-02-05 · Chapter 1 Representations 1.1 Representations of Groups and Algebras A representation of a group Gon a finite-dimensional

Scarlet : Homepages

BASISBOEK WISKUNDE - Homepages of UvA/FNWI staff Dit is een oefenboek. Elk hoofdstuk begint op de linkerbladzijde met opgaven. Je kunt er direct mee aan de slag, want de eerste opgaven

PDL for ordered trees - Homepages of UvA/FNWI staffPDL for ordered trees Loredana Afanasiev* — Patrick Blackburn** — Ioanna Dimitriou* Bertrand Gaiffe** — Evan Goris* — Maarten

USER’S MANUAL - Homepages of UvA/FNWI staffstaff.science.uva.nl/~aroos/downloads/EBT/EBTmanual.pdfEscalator Boxcar Train User’s Manual 2 the organization and bookkeeping of cohorts)

Beetle Bandit - Homepages of UvA/FNWI staff

word2vec (part 1)

VERVOLGBOEK WISKUNDE - Homepages of UvA/FNWI staff · hoofdstukken 17 en 18 van Basisboek wiskunde en de hoofdstukken 4 en 5 van Vervolgboek wiskunde bekend worden verondersteld

KANSREKENING EN STATISTIEK - Homepages of UvA/FNWI staff · Hoofdstuk 1 Kansen en kansmodellen 1.1 Introductie In het dagelijks leven hebben we het vaak over kansen, bijvoorbeeld

Syllabus Analyse B1 - Homepages of UvA/FNWI staff · 2008. 12. 9. · Syllabus Analyse B1 door T. H. Koornwinder Universiteit van Amsterdam, Faculteit Wiskunde en Informatica, Vakgroep

(pseudo.pdf) - Homepages of UvA/FNWI staff · Volume 728 of the Annals of the New York Academy of Sciences October 31, 1994 Proper Pseudocompact Subgroups of Pseudocompact Abelian

Word2vec and Friends

Modal and temporal logic - Homepages of UvA/FNWI … · 2014-10-28 · Modal and temporal logic ... Proposition 1 Let C 1 and C 2 be classes of frames. If C 1 C ... Proposition 10

Introduction to Word2vec and its application to find ...compling.hss.ntu.edu.sg/courses/hg7017/pdf/word2vec and its application... · Introduction to Word2vec and its application

Dutch Nao Team - Homepages of UvA/FNWI staff · Dutch Nao Team - Team Description for Robocup 2013 page 5/6 4 Activities Besides research in robot soccer, the Dutch Nao Team is also

The Herbrand Topos - Homepages of UvA/FNWI staff · On topos theory: R. Goldblatt { Topoi. The categorial analysis of logic. Second edition. Studies in Logic and the Foundations of

Advanced Statistical Methods - Homepages of UvA/FNWI … · 2015-01-30 · If we want to prove that astrology is true, ... With k bits, one can encode ... where the last step relies

Several Complex Variables - Homepages of UvA/FNWI staff › j.j.o.o.wiegerinck › edu › scv › scvboek.pdf · Several Complex Variables JaapKorevaar,JanWiegerinck [email protected]

word2vec Parameter Learning Explained

Homepages of UvA/FNWI staff

word2vec (часть 2)

What is word2vec?