Upload File

neural turing machine tutorial

67
Neural Turing Machine Mark Chang

Upload: mark-chang

Post on 17-Aug-2015

153 views

Category:

Technology


9 download

Report

TRANSCRIPT

  1. 1. Neural Turing Machine Mark Chang
  2. 2. -> -> (Neural Turing Machine)
  3. 3. http://humanphisiology.wikispaces.com/file/view/neuron.png/216460 814/neuron.png http://upload.wikimedia.org/wikipedia/commons/thumb/4 /4a/Action_potential.svg/1037px-Action_potential.svg.png
  4. 4. http://www.quia.com/files/quia/users/lmcgee/Systems/endocrine-nervous/synapse.gif
  5. 5. nW1 W2 x1 x2 b Wb y nin nout
  6. 6. (0,0) x2 x1 1 0
  7. 7. AND Gate x1 x2 y 0 0 0 0 1 0 1 0 0 1 1 1 (0,0) (0,1) (1,1) (1,0) 0 1 n20 20 b -30 yx1 x2
  8. 8. OR Gate x1 x2 y 0 0 0 0 1 1 1 0 1 1 1 1 (0,0) (0,1) (1,1) (1,0) 0 1 n20 20 b -10 yx1 x2
  9. 9. XOR Gate ? (0,0) (0,1) (1,1) (1,0) 0 0 1 x1 x2 y 0 0 0 0 1 1 1 0 1 1 1 0
  10. 10. XOR Gate n -20 20 b -10 y (0,0) (0,1) (1,1) (1,0) 0 1 (0,0) (0,1) (1,1) (1,0) 1 0 (0,0) (0,1) (1,1) (1,0) 0 0 1 n1 20 20 b -30 x1 x2 n2 20 20 b -10 x1 x2 x1 x2 n1 n2 y 0 0 0 0 0 0 1 0 1 1 1 0 0 1 1 1 1 1 1 0
  11. 11. x y n11 n12 n21 n22W12,y W12,x b W11,y W11,bW12,b b W11,x W21,11 W22,12 W21,12 W22,11 W21,bW22,b z1 z2 Input Layer Hidden Layer Output Layer
  12. 12. http://www.nature.com/neuro/journal/v8/n8/images/nn0805-975-F1.jpg
  13. 13.
  14. 14. http://www.pnas.org/content/102/49/17846/F7.large.jpg
  15. 15. w Forward Propagation Error Function Backward Propagation
  16. 16. Forward Propagation Error Function Backward Propagation
  17. 17. W-NN W x y n11 n12 n21 n22W12,y W12,x b W11,y W11,bW12,b b W11,x W21,11 W22,12 W21,12 W22,11 W21,bW22,b z1 z2
  18. 18. Forward Propagation
  19. 19. Forward Propagation
  20. 20. Error Function n21 n22 z1 z2
  21. 21. w1 w0 Gradient Descent
  22. 22. Backward Propagation
  23. 23. Backward Propagation
  24. 24. Backward Propagation
  25. 25. Backward Propagation
  26. 26. Backward Propagation
  27. 27. Backward Propagation
  28. 28. Backward Propagation
  29. 29. Backward Propagation http://cpmarkchang.logdown.com/posts/277349-neural-network-backward-propagation
  30. 30. Neural Network
  31. 31. ..
  32. 32. n() n() nW1 W2 x1 x2 b Wb y nW1 W2 x1 x2 b Wb y
  33. 33. Recurrent Neural Network n(n(),) n() n(n(n(),),)
  34. 34. Feedforward Neural Network Recurrent Neural Network Long Short Term MemoryNeural Turing Machine
  35. 35. Recurrent Neural Network noutnin
  36. 36. Recurrent Neural Network . x0 y0 y1 x1 x2 y2 yt xt
  37. 37. Recurrent Neural Network x0 x1 xt-1 xt y0 y1 yt-1 yt
  38. 38. Backward Propagation Through Time t = 0 t = 1
  39. 39. Backward Propagation Through Time http://cpmarkchang.logdown.com/posts/278457-neural-network-recurrent-neural-network
  40. 40. Recurrent Neural Network
  41. 41. Vanishing Gradient Problem
  42. 42. Long-Short Term Memory xt m yt Cin c cc k n b nout Memory Cell kout CreadCforgetCwrite mout,t mout,t-1 Coutmin,t
  43. 43. Long-Short Term Memory Cin Cread Cforget Cwrite Cout
  44. 44. Long-Short Term Memory Cwrite
  45. 45. Long-Short Term Memory Cforget
  46. 46. Long-Short Term Memory Cread
  47. 47. Training: Backward Propagation http://www.felixgers.de/papers/phd.pdf
  48. 48. Long-Short Term Memory https://class.coursera.org/neuralnets-2012-001/lecture/95
  49. 49. Neural Turing Machine Input Output Read/Write Head controller Memory
  50. 50. Memory Memory Address Memory Block Block Length 0 1 i n 0 j m
  51. 51. Read Operation 11 2 21 3 42 1 Read Operation: 0 000.9 0.1 0 1 i n Read Vector: Head Location: Memory : 1.1 1.0 2.2
  52. 52. Erase Operation Erase Operation: 0 1 1 11 2 21 3 42 1 0 000.9 0.1 0 1 i n 0 j m 11 2 3 1 0.1 1.8 0.2 3.6 Head Location: Erase Vector: Memory :
  53. 53. Add Operation Add Operation: 1 1 0 0 000.9 0.1 0 1 i n 11 2 3 1 0.1 1.8 0.2 3.6 2 3 10.2 3.6 1.9 1.9 1.1 1.0 Add Vector: Memory : Head Location: 0 j m
  54. 54. Controller controller Input Read Vector: Head Location: Output Add Vector: Erase Vector: Addressing Mechanisms Content Addressing Parameter: Interpolation Parameter: Convolutional Shift Parameter: Sharpening Parameter: Memory Key:
  55. 55. 0 0000 1 .45 .05 .500 0 0 .45 .05 .50 0 0 0 0 0 0 1 0 0 Head Location: 11 2 04 0 21 3 01 1 42 1 15 00 000.9 0.1 Head Location: Memory:Previous State 2 3 1 Memory Key: 00 1 Controller Outputs Content Addressing Interpolation Convolutional Shift Sharpening
  56. 56. Content Addressing 11 2 04 0 21 3 01 1 42 1 15 0 2 3 1 .16 .16 .16 .16 .16 .160 0000 1 .15 .10 .47 .08 .13 .17 Memory Key:Memory : Head Location:
  57. 57. Interpolation 0 000.9 0.1 0 0000 1 0 0000 1 0 000.9 0.1.45 .05 .50 0 0 0
  58. 58. Convolutional Shift .45 .05 .50 0 0 0 .45 .05 .50 0 0 0 .45.05 .50 0 0 0 .45 .05 .500 0 0 .45 .05 .50 0 0 0 .025 .475 .025 .25 0 .225 01 0 00 1 .5 0 .5 -1 0 1-1 0 1 -1 0 1
  59. 59. Sharpening 0 0 0 1 0 0 0 .37 0 .62 0 0 0 .45 .05 .50 0 0 .16 .16 .16 .16 .16 .16
  60. 60. Neural Turing Machine Implementation http://awawfumin.blogspot.tw/2015/03/neural-turing-machines-implementation.html
  61. 61. Experiment: Repeat Copy https://github.com/fumin/ntm
  62. 62. Evolution of Recurrent Neural Network Recurrent Neural Network Long Short Term Memory Neural Turing Machine
  63. 63. Neural Turing Machine
  64. 64. Logistic Regression http://cpmarkchang.logdown.com/posts/189069-logisti-regression-model Overfitting and Regularization http://cpmarkchang.logdown.com/posts/193261-machine-learning-overfitting-and-regularization Model Selection http://cpmarkchang.logdown.com/posts/193914-machine-learning-model-selection Neural Network Backward Propagation http://cpmarkchang.logdown.com/posts/277349-neural-network-backward-propagation Recurrent Neural Network http://cpmarkchang.logdown.com/posts/278457-neural-network-recurrent-neural-network Long Short Term Memory http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf http://www.felixgers.de/papers/phd.pdf Neural Turing Machine http://arxiv.org/pdf/1410.5401.pdf http://awawfumin.blogspot.tw/2015/03/neural-turing-machines-implementation.html
  65. 65. https://www.coursera.org/course/ntumlone https://www.coursera.org/course/ntumltwo https://www.youtube.com/playlist?list=PL6Xpj9I5 qXYEcOhn7TqghAJ6NAPrNmUBH https://www.coursera.org/course/neuralnets
  66. 66. https://github.com/fumin/ntm
  67. 67. Mark Chang facebook https://www.facebook.com/ckmarkoh.chang Githubhttp://github.com/ckmarkoh Bloghttp://cpmarkchang.logdown.com emailckmarkoh at gmail.com Fumin Githubhttps://github.com/fumin Emailawawfumin at gmail.com

About the Tutorial - dcpehvpm.org · 1943 Foundations for neural networks laid. 1945 Isaac Asimov, a Columbia University alumni, coined the term Robotics. 1950 Alan Turing introduced

Machine learning – Neural Network classification tutorial - Scilab… · Machine learning – Neural Network classification tutorial This tutorial is based on the Neural Network

Deep Learning in NLP - University of Ottawadiana/csi5386/Parinaz_Uottawa_presentation_NLP_DL.pdf · •Neural Turing Machines (Graves et al. 2014) A Neural Turing Machine (NTM) architecture

A tutorial on training recurrent neural networks, covering

Tutorial on Neural Network Optimization Problems · Tutorial on Neural Network Optimization Problems Deep Learning Summer School Montreal August 9, 2015 ... - “Qualitatively characterizing

Neural Language Model Tutorial

Neural Turing Machines - Rylan Schaeffer · dub our device a “Neural Turing Machine” (NTM). Unlike a Turing machine, an NTM is a differentiable computer that can be trained by

neural network tutorial 3

Neural Turing Machine - Washington Stateeecs.wsu.edu/~cook/aiseminar/papers/steve.pdf · Neural Turing Machine - Experiments •Goal: to demonstrate NTM is ... Rumelhart, David E.;

Continual and One-Shot Learning through Neural Networks ...sebastianrisi.com/wp-content/uploads/lueders_evoapps17.pdf · one-shot learning at the same time. Keywords: Neural Turing

Neural Signal Processing: Tutorial 1

A tutorial on training recurrent neural networks, covering

EVIC_05 Notes - Forecasting With Neural Networks Tutorial SFCrone

- Tutorial - Neuromorphic Computing with Spintronics...• Deep neural network • Recurrent neural network –Reservoir computing • Hopfield network • Stochastic neural network

Convolution Neural Network CNN A tutorial KH Wong Convolution Neural Network CNN ver. 4.11a1

Tutorial Training Recurrent Neural Networks

Memory Networks, Neural Turing Machines, and Question Answering

Neural Turing Machines - arXiv · Neural Turing Machines Alex Graves [email protected] Greg Wayne [email protected] Ivo Danihelka [email protected] Google …

ICONIP’2008 Tutorial on Computational Resources in Neural ...sima/tutresnn.pdf · ICONIP’2008 Tutorial on Computational Resources in Neural Network Models ... special-purpose

Tutorial: Neural Circuit

Neural Networks Tutorial – A Pathway to Deep Learningdeeplearning.lipingyang.org/wp-content/uploads/2016/12/Neural... · Neural Networks Tutorial – A Pathway to Deep Learning

Neural Networks Tutorial - cs.toronto.edujlucas/teaching/csc411/lectures/tut5... · CSC411 Tutorial #5 Neural Networks Oct, 2017 Shengyang Sun [email protected] *Based on the lectures

HyperNTM: Evolving Scalable Neural Turing Machines through ...sebastianrisi.com/wp-content/uploads/merrild_evoapps18.pdf · HyperNTM: Evolving Scalable Neural Turing Machines through

Neural Turing Machines(Slides)

Multigrid Neural Memorypeople.cs.uchicago.edu/~trihuynh/multigrid_mem/... · to the recent Neural Turing Machines (NTMs) (Graves et al., 2014)). Yet, while useful, none has elevated

Neural Turing Machines - clgiles.ist.psu.educlgiles.ist.psu.edu › IST597 › materials › slides › papers... · IntroducBon’ • Firstapplicaon’of’Machine’Learning’to’logical’

Neural Networks and Lecture 4: Backpropagationvision.stanford.edu/teaching/cs231n/slides/2019/cs231n_2019_lecture04.pdf · Neural Turing Machine Figure reproduced with permission

Artificial Neural Networks: A Tutorial - Computer

Recurrent Neural Network tutorial (2nd)

Neural Structured Turing Machine · duced by Neural Structured Turing Machine with high confidence and very few mistakes. 3.1. Copy To exams whether NSTM can store and recall a long

Practical tutorial on deep neural networks and saliency

Neural Turing Machines

ICONIP’2008 Tutorial on Computational Resources in Neural ...uivty.cs.cas.cz/~sima/tutresnn.pdfICONIP’2008 Tutorial on Computational Resources in Neural Network Models Jiˇr´ı

NLP Programming Tutorial 7 - Neural Networks

Neural Turing Machinev2