인공지능, 기계학습 그리고 딥러닝
TRANSCRIPT
PowerPoint
, Dec 26, 2016 / Seoul National University
? ? ?Convolutional Neural Network CNN(ILSVRC winners)Tensorflow CNN CNN
#
Unsupervised LearningReinforcement LearningRNN
#
Nicolaus Copernicus(1473-1543)
#
?
?
? ?
#
1, 2, 3, 4 Steam EngineElectricity, AutomobilePC, InternetArtificial Intelligence
?1234
#
, , Credit : Nvidia blog
#
(Artificial Intelligence)1956 From Wikipedia:Artificial intelligence(AI) isintelligenceexhibited bymachines. Incomputer science, an ideal "intelligent" machine is a flexiblerational agentthat perceives its environment and takes actions that maximize its chance of success at some goal
AI Translation: (AI) . .
#
?
#
, . .
vsEasyHard
#
(?)
#
?
Credit : MBC
#
(Machine Learning)(Machine Learning)Machine learningis the subfield ofcomputer sciencethat "gives computers the ability to learn without being explicitly programmed " .
ComputerinputsprogramoutputsComputerinputsoutputsprogram programming program !
#
Quiz ?3 x + 2 x = 11 x + 4 x = -35 x + 5 x = 08 x + 3 x = 5 = 1, = -1
(3, 2), (1, 4), (5, 5), (8, 3) input data, 1, -3, 0, 5 label weight weight
#
Supervised Learning()Input labels (classification), (regression)Unsupervised Learning()Input (clustering), (compression)Reinforcement Learning()Label reward Action selection, policy learning
#
Reinforcement Learning
https://youtu.be/Q70ulPJW3Gk
https://youtu.be/SuoouILpjDo
#
#
Perceptron(Artificial Neural Network)
sigmoid activation function
#
Example of ANN(logical AND)
011
weight (-30, 20, 20) !
#
(Regression) ()() (km) 1 ()
#
data ) x 2 + x 0.3 + x (-1) + x 0.1 = ()
#
data ,,, . x + x + x + x = ()? ,,, random ()2 ,,, 1 , ( )2 = 287.524
,,, ()2 0 ,
#
Multi Layer Perceptron(MLP)
? (regression) input (classification) perceptron Linear fitting Non-linear transform
#
(Deep Learning)
(Deep Learning) deep neural network Hidden layer = 2 deep network
weight ??
#
Back Propagation
gradient descent
#
Vanishing gradient problemOverfitting problemGet stuck in local minima
#
Vanishing Gradient ProblemGradient Sigmoid ( : ) layer
#
Overfitting ProblemData data , data(test data)
#
Local Minima local minima
#
?Vanishing gradient problem Sigmoid ReLU Overfitting problem Regularization method ( : dropout)Get stuck in local minima Local minima
#
ReLU : Rectified Linear UnitReLU activation function sparse activationReLU 0 1 vanishing gradient
#
ReLU Piece-wise linear tiling : locally linear mapping
#
Dropout(Regularization Method) , ( : 50%) random hidden layer unit Ensemble model model model
#
Local Minima minimum gradient DNN local minima saddle point local minima global minimum (neural network )
saddle point
#
ConvolutionalNeuralNetwork
Convolutional Neural Network convolution layer, pooling layer, fully-connected layer Parameter(weight) sharingConvolution pooling layer feature fully-connected layer class Picture Credit : The Data Science blog
#
CNN
tile , network tile feature (: )Newtork tile feature ( weight ) feature(: ) network tile network feature
#
Convolution1110001110001110011001100101010101
434243234=convolution1110001110001110011001100101010101
4
=convolutionfilterfeature mapInput or feature mapfilterfeature mapInput or feature map
Convolution : 1x1 + 1x0 + 1x1 + 0x0 + 1x1 + 1x0 + 0x1 + 0x0 + 1x1 = 4Filter
#
Convolution
Credit : Leonardos gitbook
#
Feature Extractor
Credit : Adit Deshpandes blog
#
Feature Extractor
Credit : Adit Deshpandes blog
#
Convolution(Multi Channel, Many Filters)111101111000011011100110001101011000011000111111001110001110001110011001100-10001000-10-10-11-10-10101010101-10-101000-10-10-1100-101010-101011-110-1-1310301-202023
=convolutionInput channel : 3Output channel : 2# of filters : 2
#
Convolution(Multi Channel, Many Filters)111101111000011011100110001101011000011000111111001110001110001110011001100-10001000-10-10-11-10-10101010101301-2020231-110-1-1310-10-1010-10-1010-1-100101010-10101Input channel : 3Output channel : 2# of filters : 2convolutionconvolutionconvolutionconvolutionconvolutionconvolution
#
Visualization of a Convolution Layer
Picture Credit : fundamentals of deep learning by Nikhil
#
ReLU301-2020231-110-1-1310
301002023
101000310ReLUReLU
#
Pooling LayerMax pooling
#
2x2 Max Pooling with Stride=130100202310100031032231131max poolingmax pooling
#
Fully-Connected Layer322311313223113121softmax0.80.2CatDog
#
Convolutional Neural Network layer , , feature , layer feature
#
Large Scale Image ClassificationImageNetOver 15ImageNetOver 15 million labeled high-resolution imagesRoughly 22,000 categoriesCollected from the webLabeled by human labelers using Amazons Mechanical Turk crowd-sourcing toolImageNet Large-Scale Visual Recognition Challenge (ILSVRC)Uses a subset of ImageNet1,000 categories1.2 million training images50,000 validation images150,000 test imagesReport two error rates:Top-1 and top-5
#
ImageNet Classification Results
Krizhevsky et al. 16.4% error(top-5)Next best (non-convnet) 26.2% error
All rankers use deep learning(Convnet)
Revolution of Depth!AlexNet
#
AlexNet(2012 winner)7 hidden layers, 650,000 neurons, 60M parameters2 GPU 1
ImageNet Classification with Deep Convolutional Neural Networks
#
AlexNet Result
ImageNet Classification with Deep Convolutional Neural Networks
#
GoogLeNet(2014 winner)Inception modul 1x1 convolution channel Fully connected layer global average pooling 5M parameters, 1.5B operations/evaluation
Inception moduleGoing Deeper With Convolutions
#
VGG(2nd place in 2014)3x3 filter Why??Convolution filter stack receptive field 2 3x3 filter = 5x5 filter3 3x3 filter = 7x7 filterParameter filter regularization
Very Deep Convolutional Networks for Large-Scale Image Recognition
#
Inception-v3Factorization of filters
Rethinking the Inception Architecture for Computer Vision
#
Residual Net(2015 winner)Revolution of DepthDeep Residual Learning for Image Recognition
#
ResNetLayer ?
56 layer 20 layer training error
Deep Residual Learning for Image Recognition
#
ResNet deep model training error Shallow model parameter deeper model copyExtra layers identity function Deep model optimization (identity )
A shallower model(18 layers)A deeper model(34 layers) Deep Residual Learning for Image Recognition
#
Deep Residual LearningIdentity layer ,
Deep Residual Learning for Image Recognition
#
ResNet ResultImageNet experiments
Deep Residual Learning for Image Recognition
#
Inception-ResNetInception + ResNet
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
#
ILSVRC 2016 Result
#
Tensorflow TensorflowGoogle , 2015 11 open source frameworkPython, computational graph (theano )CPU, GPU, multi-GPU https://tensorflow.org
#
Tensorflow : variable, placeholder a = tf.placeholder(float) , : network y = tf.mul(a, b)Insert coin : session sess = tf.Session() : network sess.run(y, feed_dict={a: 3, b: 3}
#
Global Average PoolingClass Activation MapGlobal average pooling , parameter (CNN parameter fc layer )Class activation map , class , class
Learning Deep Features for Discriminative Localization
#
Fully Connected Layer322311313223113121softmax0.80.2CatDog
#
Global Average Pooling
322311312.51.521softmax0.80.2CatDog
averageaverage
#
PPMI Datasethttp://ai.stanford.edu/~bangpeng/ppmi.html
#
CNN
Detection
SSD: Single Shot MultiBox Detector
#
Segmentation
Learning Deconvolution Network for Semantic Segmentation
#
Super Resolution
Deeply-Recursive Convolutional Network for Image Super-Resolution
#
Texture Synthesis
Texture Synthesis Using Convolutional Neural Networks
#
Artistic Style Transfer
Image Style Transfer Using Convolutional Neural Networks
#
Image Captioning
#
Visual QnA
Q: What is the boy holding?DPPnet:surfboardDPPnet:bat
Q: What is the animal doing?DPPnet:resting (relaxing)DPPnet:swimming (fishing)Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
#
Generative Adversarial Network
StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks
#
Autonomous Driving
#
Summary & Conclusion , () () , network data
#
, ? ? 1 20x20 ? data , ? ,
#
QnAThank You
#