Upload File

introduction to computational vision training neural nets...

  • Home
  • Documents
  • Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline
43
Introduction to Computational Vision Training Neural Nets and CNNs Agastya Kalra 1

Upload: others

Post on 03-Sep-2020

5 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Introduction to Computational VisionTraining Neural Nets and CNNs

Agastya Kalra1

Page 2: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Outline

• Training

• Convolutional Layers

Agastya Kalra2

Page 3: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

SGD Formalized

3

Page 4: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Momentum

• Adds a velocity term•

• Adds a speedup of at most

• Momentum constant usually 0.9, 0.5, 0.99corresponding to a 10x, 2x, 100x increase in max speed

4

Page 5: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Nesterov Momentum

• Applies momentum first then gradient

5 http://cs231n.stanford.edu/slides/2016/winter1516_lecture6.pdf

Page 6: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Setting the learning rate

6 http://cs231n.stanford.edu/slides/2016/winter1516_lecture6.pdf

● Also good to try a setting for 100 iterations and see which is best on validation set

Page 7: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Others

7

• Adagrad• Adaptive Learning Rates

• Adam• Adaptive Learning Rates + Momentum

• RMSProp• Adaptive learning rates with a slightly different decay

Page 8: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

How to choose?

• SGD < SGD+Momentum < SGD+Nesterov Momentum• Adam is a good default• RMSProp is good for RNNs, but also good default• SGD + Nesterov momentum is best if you have

time/resources to optimize learning rate• More of an Art

8

Page 9: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Outline

• Training

• Convolutional Layers

Agastya Kalra9

Page 10: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Fully Connected Layer

10

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 11: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Fully Connected Layer

11

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 12: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

12

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 13: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

13

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 14: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

14

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 15: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

15

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 16: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

16

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 17: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

17

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 18: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

18

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 19: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

19

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 20: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

20

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 21: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

21

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 22: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

22

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 23: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

23

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 24: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

24

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 25: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

25

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 26: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

26

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 27: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

27

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 28: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

28

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 29: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

29

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 30: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

30

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 31: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

31

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 32: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

32

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 33: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

33

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 34: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

34

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 35: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

35

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 36: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

36

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 37: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

37

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 38: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

38

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 39: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

39

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 40: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

40

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 41: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

41

http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Page 42: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

Convolutional Layer: Im2Col

In Practice, this is actually implemented at a matrix multiplication:

https://www.mathworks.com/help/images/ref/im2col.htmlThen Backprop is the same as backprop through a matrix multiply.

Note: If you want to tie the params of 2 weights, initialize them the same and sum their gradients at each timestep.

42

https://www.mathworks.com/help/images/ref/im2col.html
Page 43: Introduction to Computational Vision Training Neural Nets ...yboykov/Courses/cs484_2018/...Introduction to Computational Vision Training Neural Nets and CNNs 1 Agastya Kalra Outline

43


KNN, ID Trees, and Neural Nets Intro to Learning Algorithms · PDF fileKNN-ID and Neural Nets KNN, ID Trees, and Neural Nets Intro to Learning Algorithms KNN, Decision trees, Neural

Deep Learning: Feedforward Neural Nets and Convolutional ... · PDF fileDeep Learning: Feedforward Neural Nets and Convolutional Neural Nets Piyush Rai Machine Learning (CS771A) Nov

Neural Networks Simple Neural Nets For Pattern Classification CHAPTER 2

How Neural Nets Work - Neural Information …papers.nips.cc/paper/59-how-neural-nets-work.pdftherefore provide a good testbed in which to investigate techniques of nonlinear signal

Autobahn: Automorphism-based Graph Neural Nets

Text mining meets neural nets

Stochastic Neural Networks Deep Learning and Neural Nets Spring 2015

Neural Nets and Deep Learning

Machine Learning Deep Learning Neural nets

Basics of Neural Nets and Past-Tense model

Simple Neural Nets for Pattern Classification: McCulloch-Pittsjkalita/work/cs587/2014/02ThresholdLogic.pdf · Simple Neural Nets for Pattern Classification: McCulloch-Pitts ... designers

Brief overview of neural networks Generative …V11: Generative Modeling with Neural Nets Brief overview of neural networks Generative Adversarial Nets Use case: image inpainting With

UI Issues, Neural Nets, RTS

Neural Nets - Mahendran Notes

Deep Convolutional Neural Nets - Duke University · 2020. 10. 13. · Deep Convolutional Neural Nets Carlo Tomasi October 13, 2020 Neural nets are a class of predictors that have

Artificial neural networks - hopfield nets

FTSE Trend Forecasting Using Neural Nets

Neural Net Language Models Deep Learning and Neural Nets Spring 2015

Neural Nets for Optimization

Lecture 13: Recurrent Neural Nets

(Ann) Neural Nets Frohlich

Hebb Nets, Perceptrons and Adaline Nets Based on …jkalita/work/cs587/2014/03SimpleNets.pdf · Simple Neural Nets for Pattern Classification Hebb Nets, Perceptrons and Adaline Nets

SGM-Nets: Semi-global matching with neural networksopenaccess.thecvf.com/content_cvpr_2017/poster/0087_POSTER.pdfSGM-Nets: Semi-global matching with neural networks Akihito Seki1 and

Biologically-Inspired Neural Nets

Learning: Nearest Neighbor, Perceptrons & Neural Nets

Efficient Training of Neural Nets for Nonlinear.pdf

Neural Nets & Cellular Autonama

Intro Ba Neural Nets for Data Mining

Unsupervised Learning With Neural Nets Deep Learning and Neural Nets Spring 2015

Machine Learning Michael M. Richter Neural Networks Email ...pages.cpsc.ucalgary.ca/~mrichter/ML/ML 2010/Neural nets/neural-nets 2(2010).pdfMachine Learning Michael M. Richter Neural

Predictive Coding with Neural Nets: Application to … · PREDICTIVE CODING WITH NEURAL NETS: APPLICATION TO TEXT COMPRESSION J iirgen Schmidhuber Fakultat fiir …

Function Learning and Neural Nets

Neural Nets

Artificial Neural nets

Biologically-Inspired Neural Nets Modeling the Hippocampus