deep learning for developers - goto conference · deep learning engine micro-sd mini-hdmi usb usb...
TRANSCRIPT
![Page 1: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/1.jpg)
Deep Learning for Developers
Julien SimonPrincipal Technical Evangelist, AI & Machine Learning@julsimon
June 2018
![Page 2: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/2.jpg)
An introduction to Deep Learning
![Page 3: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/3.jpg)
Activation functionsThe neuron
l
∑i=1
xi ∗ wi = u
”Multiply and Accumulate” Source: Wikipedia
![Page 4: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/4.jpg)
x =
x11, x12, …. x1I x21, x22, …. x2I
… … … xm1, xm2, …. xmI
I features
m samples
y =
20…4
m labels,N2 categories
0,0,1,0,0,…,01,0,0,0,0,…,0…0,0,0,0,1,…,0One-hot encoding
Neural networks
![Page 5: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/5.jpg)
x =
x11, x12, …. x1I x21, x22, …. x2I
… … … xm1, xm2, …. xmI
I features
m samples
y =
20…4
m labels,N2 categories
Total number of predictionsAccuracy =
Number of correct predictions
0,0,1,0,0,…,01,0,0,0,0,…,0…0,0,0,0,1,…,0One-hot encoding
Neural networks
![Page 6: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/6.jpg)
Neural networksInitially, the network will not predict correctlyf(X1) = Y’1
A loss function measures the difference between the real label Y1 and the predicted label Y’1error = loss(Y1, Y’1)
For a batch of samples:
The purpose of the training process is to minimize loss by gradually adjusting weights
![Page 7: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/7.jpg)
Train ing
Training data set Training
Trainedneural network
Batch sizeLearning rate
Number of epochsHyper parameters
Backpropagation
![Page 8: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/8.jpg)
Stochastic Gradient Descent (SGD)
Imagine you stand on top of a mountain with skis strapped to your feet. You want to get down to the valley as quickly as possible, but there is fog and you can only see your immediate surroundings. How can you get down the mountain as quickly as possible? You look around and identify the steepest path down, go down that path for a bit, again look around and find the new steepest path, go down that path, and repeat—this is exactly what gradient descent does.
Tim DettmersUniversity of Lugano 2015
https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training/
The « step size » is called the learning rate
z=f(x,y)
![Page 9: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/9.jpg)
Optimizers
https://medium.cim/@julsimon/tumbling-down-the-sgd-rabbit-hole-part-1-740fa402f0d7
![Page 10: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/10.jpg)
Local minima and saddle points
« Do neural networks enter and escape a series of local minima? Do they move at varying speed as they approach and then pass a variety of
saddle points? Answering these questions definitively is difficult, but we present evidence strongly
suggesting that the answer to all of these questions is no. »
« Qualitatively characterizing neural network optimization problems », Goodfellow et al, 2015 https://arxiv.org/abs/1412.6544
![Page 11: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/11.jpg)
Val idat ion
Validation data set Trainedneural network
Validation accuracy
Prediction at the end of
each epoch
![Page 12: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/12.jpg)
Test data set Fully trainedneural network
Test accuracy
Prediction at the end of
experimentation
This data set must have the same distribution as real-life samples, or else test accuracy won’t reflect real-life accuracy.
Test
![Page 13: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/13.jpg)
Early stopping
Training accuracy
Loss function
Accuracy
100%
Epochs
Validation accuracy
Loss
Best epoch
OVERFITTIN
G
« Deep Learning ultimately is about finding a minimum that generalizes well, with bonus points for finding one
fast and reliably », Sebastian Ruder
![Page 14: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/14.jpg)
Demo: fully connected network
![Page 15: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/15.jpg)
Convolutional Neural Networks (CNN)Le Cun, 1998: handwritten digit recognition, 32x32 pixels
https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/
![Page 16: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/16.jpg)
Source: http://timdettmers.com
Extracting features with convolution
Convolution extracts features automatically.Kernel parameters are learned during the training process.
![Page 17: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/17.jpg)
Downsampling images with pooling
Source: Stanford University
Pooling shrinks images while preserving significant information.
![Page 18: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/18.jpg)
Gluon CV: classification, detection, segmentation
https://github.com/dmlc/gluon-cv
[electric_guitar], with probability 0.671
MXNet
![Page 19: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/19.jpg)
Demo: convolutional network
![Page 20: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/20.jpg)
Long Short Term Memory Networks (LSTM)
• A LSTM neuron computes the output based on the input and a previous state• LSTM networks have memory• They’re great at predicting
sequences, e.g. machine translation
![Page 22: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/22.jpg)
GAN: Welcome to the (un)real world, Neo
Generating new ”celebrity” faceshttps://github.com/tkarras/progressive_growing_of_gans
From semantic map to 2048x1024 picture https://tcwang0509.github.io/pix2pixHD/
TF
PyTorch
![Page 23: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/23.jpg)
Scalable training on AWS
Amazon EC2 c5 p3
AWS Deep Learning AMI
Fully managed hosting with auto-scaling
One-click deployment
Pre-built notebooks for
common problems
Built-in, high-performance algorithms
One-click training
Hyperparameter optimization
Build Train
Deploy
Amazon SageMaker
![Page 24: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/24.jpg)
AWS DeepLens
HD video camera
Custom-designedDeep Learning engine
Micro-SD
Mini-HDMI
USB
USB
Reset
Audio out
Power
HD video camera with on-board compute optimized for Deep
Learning
Tutorials, examples, demos, and pre-built
models
From unboxing to prediction in <10
minutes
Integrates with Amazon SageMaker and AWS
Lambda
10 MIN
The world’s first Deep Learning-enabled video camera for developers
![Page 25: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/25.jpg)
Object detection with AWS DeepLens
![Page 26: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/26.jpg)
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://aws.amazon.com/machine-learning | https://aws.amazon.com/blogs/ai
https://mxnet.incubator.apache.org | https://github.com/apache/incubator-mxnethttps://gluon.mxnet.io | https://github.com/gluon-api https://aws.amazon.com/sagemaker https://github.com/awslabs/amazon-sagemaker-examples https://github.com/aws/sagemaker-python-sdk | https://github.com/aws/sagemaker-spark
https://medium.com/@julsimonhttps://youtube.com/juliensimonfrhttps://gitlab.com/juliensimon/dlnotebooks
Gett ing started
![Page 27: Deep Learning for Developers - GOTO Conference · Deep Learning engine Micro-SD Mini-HDMI USB USB Reset Audio out Power HD video camera with on-board compute optimized for Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052611/5f07eb707e708231d41f6b3f/html5/thumbnails/27.jpg)
Thank you!Julien SimonPrincipal Technical Evangelist, AI & Machine Learning@julsimon