elec 576: understanding and visualizing convnets ......elec 576: understanding and visualizing...
TRANSCRIPT
![Page 1: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/1.jpg)
ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks
Ankit B. PatelBaylor College of Medicine (Neuroscience Dept.)
Rice University (ECE Dept.) 10-11-2017
![Page 2: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/2.jpg)
Understand & Visualizing Convnets
![Page 3: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/3.jpg)
Deconvolutional Net
[Zeiler and Fergus]
![Page 4: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/4.jpg)
Feature Visualization
[Zeiler and Fergus]
![Page 5: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/5.jpg)
Activity Maximization
[Simonyan et al.]
![Page 6: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/6.jpg)
Deep Dream Visualization• To produce human viewable images, need to
• Activity maximization (gradient ascent)
• L2 regularization
• Gaussian blur
• Clipping
• Different scales
![Page 7: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/7.jpg)
Image Example
[Günther Noack]
![Page 8: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/8.jpg)
Dumbbell Deep Dream
AlexNet VGGNet GoogleNet
![Page 9: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/9.jpg)
Deep Dream Video
Class: goldfish, Carassius auratus
![Page 10: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/10.jpg)
Texture Synthesis
[Leon Gatys, Alexander Ecker, Matthias Bethge]
![Page 11: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/11.jpg)
Generated Textures
[Leon Gatys, Alexander Ecker, Matthias Bethge]
![Page 12: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/12.jpg)
Deep Style
[Leon Gatys, Alexander Ecker, Matthias Bethge]
![Page 13: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/13.jpg)
Deep Style
[Leon Gatys, Alexander Ecker, Matthias Bethge]
![Page 14: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/14.jpg)
Introduction to Recurrent Neural Networks
![Page 15: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/15.jpg)
What Are Recurrent Neural Networks?
• Recurrent Neural Networks (RNNs) are networks that have feedback
• Output is feed back to the input
• Sequence processing
• Ideal for time-series data or sequential data
![Page 16: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/16.jpg)
History of RNNs
![Page 17: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/17.jpg)
Important RNN Architectures• Hopfield Network
• Jordan and Elman Networks
• Echo State Networks
• Long Short Term Memory (LSTM)
• Bi-Directional RNN
• Gated Recurrent Unit (GRU)
• Neural Turing Machine
![Page 18: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/18.jpg)
Hopfield Network
[Wikipedia]
![Page 19: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/19.jpg)
Elman Networks
[John McCullock]
![Page 20: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/20.jpg)
Echo State Networks
[Herbert Jaeger]
![Page 21: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/21.jpg)
Definition of RNNs
![Page 22: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/22.jpg)
RNN Diagram
[Richard Socher]
![Page 23: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/23.jpg)
RNN Formulation
[Richard Socher]
![Page 24: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/24.jpg)
RNN Example
[Andrej Karpathy]
![Page 25: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/25.jpg)
Different Kinds of Inference Tasks
[Kevin Murphy]
![Page 26: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/26.jpg)
Different Structures for Filtering/Prediction Tasks
[Andrej Karpathy]
Object Recognition
Machine Translation
Action Recognition
Object Tracking
Image Captioning
![Page 27: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/27.jpg)
Universal Expressive Power Results
[John Bullinaria]
![Page 28: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/28.jpg)
Training RNNs
![Page 29: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/29.jpg)
Training an RNN• Use back propagation through time (BPTT)
[Denny Britz]
![Page 30: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/30.jpg)
Back Propagation through Time
[Denny Britz]
![Page 31: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/31.jpg)
RNN Training Issues• Exploding/Vanishing gradients
• Exploding/Vanishing activations
![Page 32: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/32.jpg)
Exploding Gradients
[Richard Socher]
Solution: Gradient Clipping
http://www.jmlr.org/proceedings/papers/v28/pascanu13.pdf
![Page 33: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/33.jpg)
Vanishing Gradient
[Hochreiter, Schmidhuber]
![Page 34: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/34.jpg)
Why Training is Unstable
[Xu, Huang, Li]
Variance of activations/gradients grows multiplicatively
![Page 35: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/35.jpg)
Interesting Question• Are there modifications to an RNN such that it can
combat these gradient problems?
![Page 36: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/36.jpg)
RNNs with Longer Term Memory
![Page 37: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/37.jpg)
Motivation • The need to remember certain events for arbitrarily
long periods of time (Non-Markovian)
• The need to forget certain events
![Page 38: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/38.jpg)
Long Short Term Memory
• 3 gates
• Input
• Forget
• Output
[Zygmunt Z.]
![Page 39: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/39.jpg)
LSTM Formulation
[Alex Graves, Navdeep Jaitly]
![Page 40: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/40.jpg)
Preserving Gradients
[Hochreiter, Schmidhuber]
![Page 41: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/41.jpg)
Gated Recurrent Unit• 2 gates
• Reset
• Combine new input with previous memory
• Update
• How long the previous memory should stay
[Zygmunt Z.]
![Page 42: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/42.jpg)
GRU Formulation
[Danny Britz]
![Page 43: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/43.jpg)
LSTM & GRU Benefits• Remember for longer temporal durations
• RNN has issues for remembering longer durations
• Able to have feedback flow at different strengths depending on inputs
![Page 44: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/44.jpg)
Differences between LSTM & GRU
• GRU has two gates, while LSTM has three gates
• GRU does not have internal memory
• GRU does not use a second nonlinearity for computing the output
![Page 45: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/45.jpg)
Visual Difference of LSTM & GRU
[Chris Olah]
LSTM GRU
![Page 46: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/46.jpg)
LSTM vs GRU Results
[Chung, Gulcehre, Cho, Bengio]
![Page 47: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/47.jpg)
Other Methods for Stabilizing RNN Training
![Page 48: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/48.jpg)
Why Training is Unstable
[Xu, Huang, Li]
Variance of activations/gradients grows multiplicatively
![Page 49: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/49.jpg)
Stabilizing Activations & Gradients
[Xu, Huang, Li]
We want
![Page 50: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/50.jpg)
Taylor Expansions of Different Activation Functions
[Xu, Huang, Li]
![Page 51: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/51.jpg)
Layer Normalization• Similar to batch normalization
• Apply it to RNNs to stabilize the hidden state dynamics
[Ba, Kiros, Hinton]
![Page 52: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/52.jpg)
Layer Normalization Results
[Ba, Kiros, Hinton]
![Page 53: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/53.jpg)
Variants of RNNs
![Page 54: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/54.jpg)
Bidirectional RNNs• The output at time t does not depend on previous
time steps but also the future
• Two RNNs stacked on top of each other
[Danny Britz]
![Page 55: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/55.jpg)
Deep RNNs• Stack them on top of each other
• The output of the previous RNN is the input to the next one
[Danny Britz]
![Page 56: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/56.jpg)
The Power of RNNs: Understanding and
Visualizing
![Page 57: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/57.jpg)
The Effectiveness of an RNN
[Andrej Karpathy]
![Page 58: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/58.jpg)
The Effectiveness of an RNN
[Andrej Karpathy]
![Page 59: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/59.jpg)
The Effectiveness of an RNN
[Andrej Karpathy]
![Page 60: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/60.jpg)
The Effectiveness of an RNN
[Andrej Karpathy]
Trained on War & Peace
Iteration: 100
Iteration: 300
Iteration: 2000
![Page 61: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/61.jpg)
Visualize the Neurons of an RNN
[Andrej Karpathy]
![Page 62: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/62.jpg)
Visualize the Neurons of an RNN
[Andrej Karpathy]
![Page 63: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/63.jpg)
Applications
![Page 64: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/64.jpg)
RNN Applications• Speech Recognition
• Natural Language Processing
• Action Recognition
• Machine Translation
• Many more to come
![Page 65: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/65.jpg)
Speech Recognition• Deep Bidirectional LSTM
[Alex Graves, Navdeep Jaitly, Abdel-rahman Mohamed]
![Page 66: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/66.jpg)
Conversational Speech Recognition
• Achieving human parity
[Xiong et al.]
![Page 67: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/67.jpg)
Natural Language Processing
[Soumith Chantala]
![Page 68: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/68.jpg)
Contextual LSTM for NLP Tasks
[Ghost et al.]
![Page 69: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/69.jpg)
Action Recognition• Long-term Recurrent Convnet
[Donahue et al.]
![Page 70: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/70.jpg)
Google’s Neural Machine Translation System
[Yonghui Wu et al.]
![Page 71: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/71.jpg)
Image Captioning Pt 1
[Vinyals, Toshev, Bengio, Erhan]
![Page 72: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/72.jpg)
Image Captioning Pt 2
[Vinyals, Toshev, Bengio, Erhan]
![Page 73: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/73.jpg)
Object Tracking
[Ning, Zhang, Huang, He, Wang]
![Page 74: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/74.jpg)
Neural Turing Machines
[Chris Olah]
![Page 75: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/75.jpg)
WaveNet
[van den Oord et al.]
![Page 76: ELEC 576: Understanding and Visualizing Convnets ......ELEC 576: Understanding and Visualizing Convnets & Introduction to Recurrent Neural Networks Ankit B. Patel Baylor College of](https://reader036.vdocuments.site/reader036/viewer/2022062916/5ec68cfa579a6a09681250b5/html5/thumbnails/76.jpg)
DoomBot• Doom Competition
• Facebook won 1st place (F1)
• https://www.youtube.com/watch?v=94EPSjQH38Y