introduction to deep learning - cw.fel.cvut.cz · introduction to deep learning dmytro mishkin...

INTRODUCTION TO DEEP

LEARNING

Dmytro Mishkin

Czech Technical University in Prague

Clear Research Corporation

ducha.aiki@gmail.com

MY BACKGROUND

CTO of Clear Research. Using deep learning

at work since 2014.

PhD student of Czech Technical university in

Prague. Beat Deep Learning approaches at

VPRiCE Challenge 2015 with classical

methods

Now fully work in DL, recent paper “All you

need is a good init” added to Stanford CS231n

course.

Kaggler. 9th out of 1049 teams at National

Data Science Bowl

AGENDA

Why deep learning (DL)? Some applications

and motivations

What is the core idea behind DL?

Basics of convolutional networks (CNN)

Practical recommendation for CNN-based image

classification. State-of-art approaches

Deep Learning libraries overview

How to apply CNNs to different tasks

EC2 hands-on experience on Cats-vs-Dogs

competition. Homework

XKCD. NOT TRUE ANYMORE

DEEP LEARNING APPLICATIONS

Alpha Go :)

Image recognition

Speech Recognition. Cortana, Siri

Translation

Anomaly detection

Fraud detection

Video recognition

Robotics

Recommendation systems

DNA, biology, and more..

ALPHAGO

Mastering the game of Go with deep neural networks and tree search Silver et.al 2016

IMAGE CLASSIFICATION

Select all dogs. Our assignment…almost :)

State-of-art since 2012. Krizhevsky et.al 2012Superhuman level an ImageNet classification since 2015.

He et.al 2015, Szegedy et.al 2015

OBJECT DETECTION

SPEECH RECOGNITION

Cortana

Siri

OK, Google

Figure from Huang et.al. 2015.

ANOMALY DETECTION

VIDEO CAPTIONING

Translating Videos to Natural Language Using Deep Recurrent Neural Networks. Venugopalan et.al. 2015

TEXT TRANSLATION

From [Bahadanau et al., 2015] slides at ICLR 2015.

DEEP LEARNING FRAMEWORKS FOR

REGULATORY GENOMICS AND EPIGENOMICS

https://www.youtube.com/watch?v=2vpKB3j-OY0

ROBOTICS: NAVIGATION

https://www.youtube.com/watch?v=umRdt3zGgpU

FRAUD DETECTION

As simple classificationhttp://www.slideshare.net/0xdata/

paypal-fraud-detection-with-deep-learning-in-h2o-presentationh2oworld2014

AGENDA

Why deep learning (DL)? Some applications and

motivations

What is the core idea behind DL?

Basics of convolutional networks (CNN)

Practical recommendation for CNN-based image

classification. State-of-art approaches

Deep Learning libraries overview

How to apply CNNs to different tasks

EC2 hands-on experience on Cats-vs-Dogs

competition. Homework

DL IS NOT THE BEST CHOICE WHEN

You have little number of heterogenous of

(enumeration) features.

E.g. almost all kaggle competitions:

Given browser, session id, gender, determine if

customer wants revenge :)

Given some anonymized features, predict stock paper

price

Given gender, profession, age, etc. predict insurance

risk

Как нафармить рейтинг на Хабре (sorry :)

WHAT IS COMMON IN DL-FRIENDLY

TASKS?

Extremely hard to explicit write algorithms

Even if features are obvious – how to extract

them?

Lots of structured homogenous data (image,

speech , text).

You can and have to transform input. Could you

transform browser version?

DEEP LEARNING IS HIERARCHICAL

REPRESENTATION LEARNING

Quoc.V.Le et.al.,2011. Building high-level features using large scale unsupervised learning

DEPTH IS ESSENTIAL IN DEEP LEARNING

DEPTH IS ESSENTIAL IN DEEP LEARNING

DEPTH IS ESSENTIAL IN DEEP LEARNING

DEPTH IS ESSENTIAL IN DEEP LEARNING

AGENDA

Why deep learning (DL)? Some applications and

motivations

What is the core idea behind DL?

Basics of convolutional networks (CNN)

Practical recommendation for CNN-based image

classification. State-of-art approaches

Deep Learning libraries overview

How to apply CNNs to different tasks

EC2 hands-on experience on Cats-vs-Dogs

competition. Homework

CONVOLUTIONS? WHY NOT JUST MLP?

http://cs231n.github.io/convolutional-networks/

NN

CNN

WHAT IS CONVOLUTION

https://developer.apple.com/library/ios/documentation/Performance/Conceptual/vImage/ConvolutionOperations/ConvolutionOperations.html

Classical NN

for image

is convolution

with image

size kernel

CONVOLUTIONS? WHY NOT JUST MLP?

Let`s look on filters:

Local

The most values are mean

(non-informative)

Wasted computation

and memory!!!

Also lots of parameters

and low data -> overfitting

http://www.cs.toronto.edu/~ranzato/research/projects.html

CONVOLUTIONS? WHY NOT JUST MLP?

Krizhevsky et al. 2012. conv1 filters

11x11x3. Much less wasted space!

DO WE NEED MATH?

If yes, go to the whiteboard

POOLING

http://cs231n.github.io/convolutional-networks/

MAX POOLING

http://cs231n.github.io/convolutional-networks/

TYPICAL CNN STRUCTURE (LENET-5)

http://eblearn.sourceforge.net/lib/exe/lenet5.png

• (Conv-ReLU-Pool)xN Softmax. Simple• (Conv-ReLU)xN-Pool- (Conv-Relu)x2N-Pool….Softmax. Popular.

• Some Inception arch. Have fun :)

NON-LINEARITIES

NON-LINEARITIES BENCHMARK

https://github.com/ducha-aiki/caffenet-benchmark/blob/master/Activations.md

NON-LINEARITIES

AGENDA

Why deep learning (DL)? Some applications and

motivations

What is the core idea behind DL?

Basics of convolutional networks (CNN)

Practical recommendation for CNN-based

image classification. State-of-art

approaches

Deep Learning libraries overview

How to apply CNNs to different tasks

EC2 hands-on experience on Cats-vs-Dogs

competition. Homework

IMAGE PREPROCESSING

Subtract mean pixel (training set), divide by std.

RGB is the best colorspace for CNN

Do nothing more…

…unless you have specific dataset.

Subtract local mean pixelB.Graham, 2015

Kaggle Diabetic Retinopathy Competition report

TRAINING. SOLVERS AND REGULARIZATION

Use SGD with momentum.

Try learning rates 0.01, 0.005, 0.001

Momentums: 0, 0.5, 0.9, 0.95

Try L2 weight decay 0.0005, 0.0001. Prevents

from overconfidence

Fancy solvers (ADAM, RMSProp, AdaDelta)

sometimes work better, sometimes not.

ARCHITECTURE

Use as small filters as possible

3x3 + ReLU + 3x3 + ReLU > 5x5 + ReLU.

Exception: 1st layer. Too computationally

ineffective to use 3x3 there.

Convolutional Neural Networks at Constrained Time Cost. He and Sun, 2015

WEIGHTS INITIALIZATION

Preserve var=1

of all layers

output.

How?

There are lots of

papers with

variants

Mishkin and Matas. All you need is a good init. ICLR, 2016

WEIGHTS INITIALIZATION

Gaussian noise with some coefficient:

Xavier:

He (0.5 * Xavier for ReLU)

Orthonormal (Saxe et.al. 2013)

Data-dependent: LSUV

Mishkin and Matas. All you need is a good init. ICLR, 2016

BATCH NORMALIZATION

Ioffe et.al 2015

BATCH NORMALIZATION

DROPOUT

DROPOUT

Play with rates. 0.5 is rarely optimal choice (but

often good)

DROPOUT

Dropout_rate * width = constant – doesn`t work!

DATA AUGMENTATION

Common (helps 99% cases):

Random crop: e.g., 227x227 from 256x256 px

(AlexNet)

Horizontal mirror

Dataset dependent:

Random rotation

Affine transform

Random scale

Color augmentation

Noise input

Thin plate deformation

Unleash your imagination

PADDING. VALID AND SAME CONVOLUTION

http://www.johnloomis.org/ece563/notes/filter/conv/convolution.html

Same = padding with zerosby ½ kernel size.

The most common choice

PADDING

Padding:

Preserving spatial size, not “washing out”

information

Dropout-like augmentation by zeros

Caffenet128

with conv padding: 47% top-1 acc

w/o conv padding: 41% top-1 acc.

It is huge difference

RESUME FROM CS231N

AGENDA

Why deep learning (DL)? Some applications and

motivations

What is the core idea behind DL?

Basics of convolutional networks (CNN)

Practical recommendation for CNN-based image

classification. State-of-art approaches

Deep Learning libraries overview. Why

caffe.

How to apply CNNs to different tasks

EC2 hands-on experience on Cats-vs-Dogs

competition. Homework

DEEP LEARNING TOOLBOXES

Caffe

Torch

Theano

TensorFlow

MXNet

…

Nervana

DeepLearning4j

ConvnetJS

CNTK

Veles

H20...sorry, guys :(

MAIN DEEP LEARNING TOOLBOXES

SPEED BENCHMARK ALEXNET

Library Class Time (ms) forward (ms) backward (ms)

CuDNN[R4]-fp16

(Torch)cudnn.SpatialConvolution 71 25 46

Nervana-neon-fp16 ConvLayer 78 25 52

CuDNN[R4]-fp32

(Torch)cudnn.SpatialConvolution 81 27 53

Nervana-neon-fp32 ConvLayer 87 28 58

fbfft (Torch) fbnn.SpatialConvolution 104 31 72

TensorFlow conv2d 151 34 117

Chainer Convolution2D 177 40 136

cudaconvnet2* ConvLayer 177 42 135

CuDNN[R2] * cudnn.SpatialConvolution 231 70 161

Caffe (native) ConvolutionLayer 324 121 203

Torch-7 (native) SpatialConvolutionMM 342 132 210

CL-nn (Torch) SpatialConvolutionMM 963 388 574

Caffe-CLGreenTea ConvolutionLayer 1442 210 1232

https://github.com/soumith/convnet-benchmarks

SPEEDBENCHMARK. GOOGLENET

Library Class Time (ms)forward

(ms)

backward

(ms)

Nervana-

neon-fp16ConvLayer 230 72 157

Nervana-

neon-fp32ConvLayer 270 84 186

CuDNN[R4]-

fp16 (Torch)

cudnn.Spatial

Convolution462 112 349

CuDNN[R4]-

fp32 (Torch)

cudnn.Spatial

Convolution470 130 340

ChainerConvolution2

D687 189 497

TensorFlow conv2d 905 187 718

CaffeConvolutionL

ayer1935 786 1148

CL-nn (Torch)SpatialConvol

utionMM7016 3027 3988

Caffe-

CLGreenTea

ConvolutionL

ayer9462 746 8716

https://github.com/soumith/convnet-benchmarks

CAFFE

CAFFE

AGENDA

Why deep learning (DL)? Some applications and

motivations

What is the core idea behind DL?

Basics of convolutional networks (CNN)

Practical recommendation for CNN-based image

classification. State-of-art approaches

Deep Learning libraries overview

How to apply CNNs to various tasks

EC2 hands-on experience on Cats-vs-Dogs

competition. Homework.

HOW TO DO – LET`S GO TO WHITEBOARD

Image retrieval Babenko et. al (2014)

Person identification Chopra et. al 2006

Ranking Wang et.al 2014

Playing games. Atari (2013) Go (2016)

Text generation https://github.com/karpathy/char-rnn

Image generation Radford et.al 2016

Action recognition Simonyan et.al 2014

Anomaly detection https://www.youtube.com/watch?v=ds73ULGjnpc&feature=youtu.be

Translation Cho et al 2014

Fraud detection at PayPal http://university.h2o.ai/cds-lp/cds02.html

AGENDA

Why deep learning (DL)? Some applications and

motivations

What is the core idea behind DL?

Basics of convolutional networks (CNN)

Practical recommendation for CNN-based image

classification. State-of-art approaches

Deep Learning libraries overview

How to apply CNNs to different tasks

EC2 hands-on experience on Cats-vs-Dogs

competition. Homework.

IMAGE RETRIEVAL

Figure from Babenko et.al.2014

1. Pass image through ImageNet-pretrained CNN.2. Use some layer activations as description

3. L2-normalize (must!)

4. Put in some fast NN search like kd-tree.

EMBEDDINGS WITH SIAMESE NETWORKS

1. Put 2 images through copies of the same networks2. L2 difference < 1 if same person, >1 if different

https://www.cs.nyu.edu/~sumit/research/research.html

WHAT ABOUT 3 COPIES? TRIPLETS

http://arxiv.org/abs/1412.6622

1. Put 2 images through copies of the same networks

2. D(x, x+) < D (x,x-)

Drawback: 1) slow training :(

2) Have to select hard triplets. Random

ones easily satisfy equation above.

GENERATING IMAGES WITH GANS

http://soumith.ch/eyescream/

http://arxiv.org/abs/1511.06434

• Generator tries to generate image undistinguishable from natural.• Discriminatior tries to distinguish.

• Both learn simultaneously

AUTO-ENCODERS

http://deeplearning4j.org/deepautoencoder.html

DE-NOISING AUTO-ENCODERS

Clean Input Corrupted input (what net sees) Reconstructed

If compare input and reconstruction, we can detect anomalies

http://www.cs.toronto.edu/~ranzato/research/projects.html

QUESTIONS?