convolutional neural networks - university of pennsylvaniacis581/lectures/fall... · 2017-12-05 ·...
TRANSCRIPT
![Page 1: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/1.jpg)
Convolutional Neural Networks
![Page 2: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/2.jpg)
Books» http://www.deeplearningbook.org/
![Page 3: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/3.jpg)
Bookshttp://neuralnetworksanddeeplearning.com/.org/
![Page 4: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/4.jpg)
reviews
» http://www.deeplearningbook.org/contents/linear_algebra.html
» http://www.deeplearningbook.org/contents/prob.html
» http://www.deeplearningbook.org/contents/numerical.html
![Page 5: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/5.jpg)
![Page 6: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/6.jpg)
Input: Output:
Goal: Given an image, we want to identify what class
that image belongs to.
![Page 7: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/7.jpg)
Input
OutputConvolutional Neural Network (CNN)
Pipeline:
A Monitor
![Page 8: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/8.jpg)
Convolutional Neural Nets (CNNs) in a nutshell:
• A typical CNN takes a raw RGB image as an input.
• It then applies a series of non-linear operations on top
of each other.
• These include convolution, sigmoid, matrix
multiplication, and pooling (subsampling) operations.
• The output of a CNN is a highly non-linear function of
the raw RGB image pixels.
![Page 9: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/9.jpg)
How the key operations are encoded in standard CNNs:
• Convolutional Layers: 2D Convolution
• Fully Connected Layers: Matrix Multiplication
• Sigmoid Layers: Sigmoid function
• Pooling Layers: Subsampling
![Page 10: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/10.jpg)
2D convolution:
- convolutional weights of size MxN
- the values in a 2D grid that we want to convolve
A sliding window operation across the entire grid .
h f g
0 0
M N
ij
m n
h f i m j n g m n
( , ) ( , )
![Page 11: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/11.jpg)
0 0 0
0 1 0
0 0 0
0.107 0.113 0.107
0.113 0.119 0.113
0.107 0.113 0.107
-1 0 1
-2 0 2
-1 0 1
Unchanged Image Blurred Image Vertical Edges
1f g 2
f g3
f g
![Page 12: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/12.jpg)
0 0 0
0 1 0
0 0 0
0.107 0.113 0.107
0.113 0.119 0.113
0.107 0.113 0.107
-1 0 1
-2 0 2
-1 0 1
CNNs aim to learn convolutional weights directly from the data
![Page 13: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/13.jpg)
Input: Convolutional Neural Network (CNN)
Early layers learn to detect low level structures such as
oriented edges, colors and corners
![Page 14: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/14.jpg)
Input: Convolutional Neural Network (CNN)
Deep layers learn to detect high-level object structures and their parts.
![Page 15: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/15.jpg)
A Closer Look inside the Convolutional Layer
A Chair Filter
A Person Filter
A Table Filter
A Cupboard Filter
Input Image
![Page 16: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/16.jpg)
Fully Connected Layers:
![Page 17: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/17.jpg)
Fully Connected Layers:
matrix multiplication
![Page 18: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/18.jpg)
Max Pooling Layer:
• Sliding window is applied on a grid of values.
• The maximum is computed using the values in the
current window.
1 2 3
4 5 6
7 8 9
![Page 19: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/19.jpg)
Max Pooling Layer:
5
• Sliding window is applied on a grid of values.
• The maximum is computed using the values in the
current window.
1 2 3
4 5 6
7 8 9
![Page 20: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/20.jpg)
Max Pooling Layer:
5 6
• Sliding window is applied on a grid of values.
• The maximum is computed using the values in the
current window.
1 2 3
4 5 6
7 8 9
![Page 21: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/21.jpg)
Max Pooling Layer:
5 6
8 9
• Sliding window is applied on a grid of values.
• The maximum is computed using the values in the
current window.
1 2 3
4 5 6
7 8 9
![Page 22: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/22.jpg)
Sigmoid Layer:
• Applies a sigmoid function on an input
![Page 23: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/23.jpg)
Let us now consider a CNN with a specific architecture:
• 2 convolutional layers.
• 2 pooling layers.
• 2 fully connected layers.
• 3 sigmoid layers.
Convolutional Networks
![Page 24: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/24.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 25: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/25.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 26: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/26.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 27: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/27.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 28: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/28.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 29: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/29.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 30: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/30.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 31: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/31.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 32: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/32.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 33: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/33.jpg)
Convolutional Networks
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 34: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/34.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 35: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/35.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer
Final Predictions
- sigmoid function - softmax function
![Page 36: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/36.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer
Convolutional layer parameters in layers 1 and 2
- sigmoid function - softmax function
![Page 37: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/37.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer
Fully connected layer parameters in the fully
connected layers 1 and 2
- sigmoid function - softmax function
![Page 38: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/38.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function - softmax function
![Page 39: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/39.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer
1.
- sigmoid function - softmax function
![Page 40: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/40.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer
1.
2.
- sigmoid function - softmax function
![Page 41: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/41.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer
1.
2.
3.
- sigmoid function - softmax function
![Page 42: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/42.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer
1.
2.
3.
4.
- sigmoid function - softmax function
![Page 43: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/43.jpg)
Forward Pass:
- convolutional layer output
Notation:
- fully connected layer output
- pooling layer - sigmoid function
Key Question: How to learn the parameters from the data?
- softmax function
![Page 44: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/44.jpg)
Backpropagation
for
Convolutional Neural Networks
![Page 45: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/45.jpg)
How to learn the parameters of a CNN?
• Assume that we are a given a labeled training dataset
• We want to adjust the parameters of a CNN such that
CNN’s predictions would be as close to true labels as
possible.
• This is difficult to do because the learning objective is
highly non-linear.
Prediction True Label
![Page 46: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/46.jpg)
Gradient descent:
• Iteratively minimizes the objective function.
• The function needs to be differentiable.
![Page 47: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/47.jpg)
Gradient descent:
• Iteratively minimizes the objective function.
• The function needs to be differentiable.
![Page 48: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/48.jpg)
Gradient descent:
• Iteratively minimizes the objective function.
• The function needs to be differentiable.
![Page 49: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/49.jpg)
Gradient descent:
• Iteratively minimizes the objective function.
• The function needs to be differentiable.
![Page 50: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/50.jpg)
1. Compute the gradients of the overall loss
w.r.t. to our predictions and propagate it back:
True Label
![Page 51: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/51.jpg)
2. Compute the gradients of the overall
loss and propagate it back:
True Label
![Page 52: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/52.jpg)
3. Compute the gradients
to adjust the weights:
True Label
![Page 53: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/53.jpg)
4. Backpropagate the
gradients to previous layers:
True Label
![Page 54: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/54.jpg)
5. Compute the gradients
to adjust the weights:
True Label
![Page 55: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/55.jpg)
6. Backpropagate
the gradients to
previous layers:
True Label
![Page 56: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/56.jpg)
7. Compute the gradients
to adjust the weights:
True Label
![Page 57: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/57.jpg)
8. Backpropagate
the gradients to
previous layers:
True Label
![Page 58: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/58.jpg)
9. Compute the gradients
to adjust the weights:
True Label
![Page 59: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/59.jpg)
An Example of
Backpropagation Convolutional Neural Networks
![Page 60: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/60.jpg)
Assume that we have K=5 object classes:
0.5Class 1: Penguin
Class 2: Building
Class 3: Chair
Class 4: Person
Class 5: Bird
0 0.1 0.2 0.1
1 0 0 0 0
True Label
![Page 61: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/61.jpg)
True Label
![Page 62: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/62.jpg)
where
True Label
![Page 63: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/63.jpg)
where
True Label
![Page 64: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/64.jpg)
where
True Label
![Page 65: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/65.jpg)
where
True Label
![Page 66: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/66.jpg)
True Label
![Page 67: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/67.jpg)
True Label
![Page 68: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/68.jpg)
True Label
![Page 69: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/69.jpg)
Assume that we have K=5 object classes:
0.5Class 1: Penguin
Class 2: Building
Class 3: Chair
Class 4: Person
Class 5: Bird
0 0.1 0.2 0.1
1 0 0 0 0
True Label
![Page 70: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/70.jpg)
Assume that we have K=5 object classes:
0.5Class 1: Penguin
Class 2: Building
Class 3: Chair
Class 4: Person
Class 5: Bird
0 0.1 0.2 0.1
1 0 0 0 0
-0.5 0 0.1 0.2 0.1
True Label
![Page 71: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/71.jpg)
Assume that we have K=5 object classes:
0.5Class 1: Penguin
Class 2: Building
Class 3: Chair
Class 4: Person
Class 5: Bird
0 0.1 0.2 0.1
1 0 0 0 0
-0.5 0 0.1 0.2 0.1
Increasing the score corresponding to the true class decreases the loss.
True Label
![Page 72: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/72.jpg)
Assume that we have K=5 object classes:
0.5Class 1: Penguin
Class 2: Building
Class 3: Chair
Class 4: Person
Class 5: Bird
0 0.1 0.2 0.1
1 0 0 0 0
-0.5 0 0.1 0.2 0.1
Decreasing the score of other classes also decreases the loss.
True Label
![Page 73: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/73.jpg)
Adjusting the weights:
True Label
![Page 74: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/74.jpg)
Need to compute the following gradient
Adjusting the weights:
True Label
![Page 75: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/75.jpg)
Adjusting the weights:
Need to compute the following gradient
True Label
![Page 76: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/76.jpg)
was already computed in the previous step
Adjusting the weights:
Need to compute the following gradient
True Label
![Page 77: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/77.jpg)
Adjusting the weights:
Need to compute the following gradient
True Label
![Page 78: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/78.jpg)
Adjusting the weights:
where
Need to compute the following gradient
True Label
![Page 79: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/79.jpg)
Adjusting the weights:
where
Need to compute the following gradient
True Label
![Page 80: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/80.jpg)
Adjusting the weights:
Update rule:
Need to compute the following gradient
True Label
![Page 81: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/81.jpg)
Backpropagating the gradients:
True Label
![Page 82: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/82.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 83: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/83.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 84: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/84.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
was already computed in the previous step
True Label
![Page 85: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/85.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 86: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/86.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
where
True Label
![Page 87: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/87.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
where
True Label
![Page 88: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/88.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 89: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/89.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 90: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/90.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 91: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/91.jpg)
Adjusting the weights:
True Label
![Page 92: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/92.jpg)
Adjusting the weights:
Need to compute the following gradient
True Label
Update rule:
![Page 93: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/93.jpg)
Backpropagating the gradients:
True Label
![Page 94: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/94.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 95: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/95.jpg)
Adjusting the weights:
True Label
![Page 96: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/96.jpg)
Adjusting the weights:
Need to compute the following gradient
True Label
![Page 97: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/97.jpg)
Adjusting the weights:
Need to compute the following gradient
True Label
![Page 98: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/98.jpg)
was already computed in the previous step
Adjusting the weights:
Need to compute the following gradient
True Label
![Page 99: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/99.jpg)
Adjusting the weights:
Need to compute the following gradient
True Label
![Page 100: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/100.jpg)
Adjusting the weights:
where
Need to compute the following gradient
True Label
![Page 101: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/101.jpg)
Adjusting the weights:
where
Need to compute the following gradient
True Label
![Page 102: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/102.jpg)
Adjusting the weights:
Update rule:
Need to compute the following gradient
True Label
![Page 103: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/103.jpg)
Backpropagating the gradients:
True Label
![Page 104: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/104.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 105: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/105.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 106: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/106.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
was already computed in the previous step
True Label
![Page 107: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/107.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 108: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/108.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
where
True Label
![Page 109: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/109.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
where
True Label
![Page 110: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/110.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 111: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/111.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 112: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/112.jpg)
Backpropagating the gradients:
Need to compute the following gradient:
True Label
![Page 113: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/113.jpg)
Adjusting the weights:
True Label
![Page 114: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/114.jpg)
Adjusting the weights:
Need to compute the following gradient
True Label
Update rule:
![Page 115: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/115.jpg)
Visual illustration
BackpropagationConvolutional Neural Networks
![Page 116: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/116.jpg)
Forward:
Fully Connected Layers:
![Page 117: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/117.jpg)
Activation unit of interestForward:
Fully Connected Layers:
![Page 118: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/118.jpg)
Activation unit of interest
The weight that is used in
conjunction with the
activation unit of interest
The output
Forward:
Fully Connected Layers:
![Page 119: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/119.jpg)
Forward:
Fully Connected Layers:
![Page 120: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/120.jpg)
Forward:
Fully Connected Layers:
![Page 121: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/121.jpg)
Forward:
Fully Connected Layers:
![Page 122: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/122.jpg)
Backpropagation
Forward:
Fully Connected Layers:
![Page 123: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/123.jpg)
Forward:
Fully Connected Layers:
![Page 124: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/124.jpg)
Forward:
Fully Connected Layers:
![Page 125: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/125.jpg)
Forward:
Fully Connected Layers:
![Page 126: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/126.jpg)
Backward:Forward:
Fully Connected Layers:
![Page 127: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/127.jpg)
A measure how much an activation
unit contributed to the loss
Backward:Forward:
Fully Connected Layers:
![Page 128: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/128.jpg)
A measure how much an activation
unit contributed to the loss
Backward:Forward:
Fully Connected Layers:
![Page 129: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/129.jpg)
Backward:Forward:
Fully Connected Layers:
![Page 130: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/130.jpg)
Backward:Forward:
Fully Connected Layers:
![Page 131: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/131.jpg)
Backward:Forward:
Fully Connected Layers:
![Page 132: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/132.jpg)
Backward:Forward:
Fully Connected Layers:
![Page 133: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/133.jpg)
Backward:Forward:
Fully Connected Layers:
![Page 134: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/134.jpg)
Backward:Forward:
Fully Connected Layers:
![Page 135: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/135.jpg)
Summary for fully connected layers
BackpropagationConvolutional Neural Networks
![Page 136: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/136.jpg)
1. Let , where n denotes the number of
layers in the network.
Summary:
![Page 137: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/137.jpg)
1. Let , where n denotes the number of
layers in the network.
2. For each fully connected layer :
• For each node in layer set:
Summary:
![Page 138: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/138.jpg)
1. Let , where n denotes the number of
layers in the network.
2. For each fully connected layer :
• For each node in layer set:
• Compute partial derivatives:
Summary:
![Page 139: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/139.jpg)
1. Let , where n denotes the number of
layers in the network.
2. For each fully connected layer :
• For each node in layer set:
• Compute partial derivatives:
• Update the parameters:
Summary:
![Page 140: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/140.jpg)
Visual illustration
BackpropagationConvolutional Neural Networks
![Page 141: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/141.jpg)
Forward:
Convolutional Layers:
1l l la g z ( ) ( ) ( )
![Page 142: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/142.jpg)
Forward:
Convolutional Layers:
1l l la g z ( ) ( ) ( )
![Page 143: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/143.jpg)
Forward:
Convolutional Layers:
1l l la g z ( ) ( ) ( )
![Page 144: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/144.jpg)
1. Let , where c denotes the index of a first fully
connected layer.
2. For each convolutional layer :
• For each node in layer set
Summary:
![Page 145: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/145.jpg)
1. Let , where c denotes the index of a first fully
connected layer.
2. For each convolutional layer :
• For each node in layer set
• Compute partial derivatives:
Summary:
![Page 146: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/146.jpg)
1. Let , where c denotes the index of a first fully
connected layer.
2. For each convolutional layer :
• For each node in layer set
• Compute partial derivatives:
• Update the parameters:
Summary:
![Page 147: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/147.jpg)
Gradient in pooling layers:
• There is no learning done in the pooling layers
• The error that is backpropagated to the pooling layer, is
sent back from to the node where it came from.
True Label
![Page 148: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/148.jpg)
4 5
7 8
Forward Pass
8
Layer Layer
Gradient in pooling layers:
• There is no learning done in the pooling layers
• The error that is backpropagated to the pooling layer, is
sent back from to the node where it came from.
True Label
![Page 149: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/149.jpg)
0 0
0
Backward Pass
Layer Layer
Gradient in pooling layers:
• There is no learning done in the pooling layers
• The error that is backpropagated to the pooling layer, is
sent back from to the node where it came from.
True Label
![Page 150: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/150.jpg)
Recognition as a big table lookup
Y
N
N
Y
Y
N
Y
Y
N
Y
N
Image bits
Recognition
Label
=
m
n
n*m
Image
write down all images
in a table and record
the object label
![Page 151: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/151.jpg)
3
18
Each pixel has 2 = 256 values
3x18 image above has 54 pixels
Total possible 54 pixel images = 256
An tiny image example (can you see what it is?)
54
= 1.1 x 10
8
130
Prof. Kanade’s Theorem: we have not seen anything yet!
“How many images are there?”
![Page 152: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/152.jpg)
3
18
Total possible 54 pixel images = 1.1 x 10130
Total population
10 billion x 1000 x 100 x 356 x 24 x 60 x 60 x 30 = 1024
Compared
generations days
hours
min/sec
frame rate
We have to be clever in writing down this table!
number of images seen by all humans ever:
years
![Page 153: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/153.jpg)
A Closer Look inside the Convolutional Layer
A Chair Filter
A Person Filter
A Table Filter
A Cupboard Filter
Input Image
![Page 154: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/154.jpg)
A Closer Look inside the Convolutional Layer :
![Page 155: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/155.jpg)
A Closer Look inside the Back Propagation Convolutional Layer :
![Page 156: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/156.jpg)
Adjusting the weights:
True Label
![Page 157: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/157.jpg)
( )
( )
( )
( )
( )
- elementwise multiplication
Training Batch
(performed in a sliding window fashion)
Average the Gradient
Across the Batch
Parameter Update
new old
![Page 158: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/158.jpg)
( )
( )
( )
( )
( )
- elementwise multiplication
Training Batch
(performed in a sliding window fashion)
Average the Gradient
Across the Batch
Parameter Update
new old
![Page 159: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/159.jpg)
( )
( )
( )
( )
( )
- elementwise multiplication
Training Batch
(performed in a sliding window fashion)
Average the Gradient
Across the Batch
Parameter Update
new old
![Page 160: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/160.jpg)
Adjusting the weights:
True Label
![Page 161: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/161.jpg)
( )
( )
( )
( )
( )
- elementwise multiplication
(performed in a sliding window fashion)
Average the Gradient
Across the Batch
Parameter Update
new old
Training Batch
![Page 162: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/162.jpg)
( )
( )
( )
( )
( )
- elementwise multiplication
(performed in a sliding window fashion)
Average the Gradient
Across the Batch
Parameter Update
new old
Training Batch
![Page 163: Convolutional Neural Networks - University of Pennsylvaniacis581/Lectures/Fall... · 2017-12-05 · Input: Convolutional Neural Network (CNN) Early layers learn to detect low level](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b3de6b77f8b9a560a8e5603/html5/thumbnails/163.jpg)
( )
( )
( )
( )
( )
- elementwise multiplication
(performed in a sliding window fashion)
Average the Gradient
Across the Batch
Parameter Update
new old
Training Batch