ml with tensorflow-new - github pages › ml › lec12.pdf · 2017-10-02 · fei-fei li &...
TRANSCRIPT
![Page 2: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/2.jpg)
Sequence data
• We don’t understand one word only
• We understand based on the previous words + this word. (time series)
• NN/CNN cannot do this
![Page 3: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/3.jpg)
Sequence data
• We don’t understand one word only
• We understand based on the previous words + this word. (time series)
• NN/CNN cannot do this
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
![Page 4: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/4.jpg)
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
![Page 5: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/5.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201613
Recurrent Neural Network
x
RNN
![Page 6: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/6.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201614
Recurrent Neural Network
x
RNN
yusually want to predict a vector at some time steps
![Page 7: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/7.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201615
Recurrent Neural Network
x
RNN
yWe can process a sequence of vectors x by applying a recurrence formula at every time step:
new state old state input vector at some time step
some functionwith parameters W
![Page 8: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/8.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201616
Recurrent Neural Network
x
RNN
yWe can process a sequence of vectors x by applying a recurrence formula at every time step:
Notice: the same function and the same set of parameters are used at every time step.
![Page 9: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/9.jpg)
/HFWXUH����� ��)HE�����)HL�)HL�/L��$QGUHM�.DUSDWK\��-XVWLQ�-RKQVRQ)HL�)HL�/L��$QGUHM�.DUSDWK\��-XVWLQ�-RKQVRQ /HFWXUH����� ��)HE�������
�9DQLOOD��5HFXUUHQW�1HXUDO�1HWZRUN
[
511
\
7KH�VWDWH�FRQVLVWV�RI�D�VLQJOH�³KLGGHQ´�YHFWRU�K�
![Page 10: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/10.jpg)
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
![Page 11: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/11.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201618
Character-levellanguage modelexample
Vocabulary:[h,e,l,o]
Example trainingsequence:“hello”
x
RNN
y
![Page 12: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/12.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201619
Character-levellanguage modelexample
Vocabulary:[h,e,l,o]
Example trainingsequence:“hello”
![Page 13: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/13.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201620
Character-levellanguage modelexample
Vocabulary:[h,e,l,o]
Example trainingsequence:“hello”
![Page 14: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/14.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201620
Character-levellanguage modelexample
Vocabulary:[h,e,l,o]
Example trainingsequence:“hello”
![Page 15: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/15.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201620
Character-levellanguage modelexample
Vocabulary:[h,e,l,o]
Example trainingsequence:“hello”
![Page 16: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/16.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201620
Character-levellanguage modelexample
Vocabulary:[h,e,l,o]
Example trainingsequence:“hello”
![Page 17: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/17.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201621
Character-levellanguage modelexample
Vocabulary:[h,e,l,o]
Example trainingsequence:“hello”
![Page 18: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/18.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201621
Character-levellanguage modelexample
Vocabulary:[h,e,l,o]
Example trainingsequence:“hello”
![Page 19: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/19.jpg)
RNN applications
• Language Modeling
• Speech Recognition
• Machine Translation
• Conversation Modeling/Question Answering
• Image/Video Captioning
• Image/Music/Dance Generation
http://jiwonkim.org/awesome-rnn/
https://github.com/TensorFlowKR/awesome_tensorflow_implementations
![Page 20: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/20.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 20166
Recurrent Networks offer a lot of flexibility:
Vanilla Neural Networks
![Page 21: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/21.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 20167
Recurrent Networks offer a lot of flexibility:
e.g. Image Captioningimage -> sequence of words
![Page 22: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/22.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 20168
Recurrent Networks offer a lot of flexibility:
e.g. Sentiment Classificationsequence of words -> sentiment
![Page 23: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/23.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 20169
Recurrent Networks offer a lot of flexibility:
e.g. Machine Translationseq of words -> seq of words
![Page 24: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/24.jpg)
Lecture 10 - 8 Feb 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 8 Feb 201610
Recurrent Networks offer a lot of flexibility:
e.g. Video classification on frame level
![Page 25: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/25.jpg)
Multi-Layer RNN
![Page 26: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/26.jpg)
Training RNNs is challenging
• Several advanced models- Long Short Term Memory (LSTM)- GRU by Cho et al. 2014
![Page 27: ML with Tensorflow-new - GitHub Pages › ml › lec12.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 -16 8 Feb 2016 Recurrent Neural Network x RNN](https://reader035.vdocuments.site/reader035/viewer/2022081402/5f0e8d717e708231d43fcb5b/html5/thumbnails/27.jpg)
Next
RNN in TensorFlow