Upload File

deep learning tutorial - university of north carolina at...

  • Home
  • Documents
  • Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics
29
Deep learning Tutorial Tianxiang Gao Aug 31, 2016

Upload: trandan

Post on 11-Jun-2018

218 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Deep learning Tutorial

Tianxiang GaoAug 31, 2016

Page 2: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Content

1. Basics about Deep Learning2. Tutorial in Keras3. Recent Deep Learning techniques

Page 3: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Background machine learning knowledge1. Linear regression (LASSO, Ridge)2. Logistic regression3. Optimization (Gradient Descent, Stochastic mini-batch

gradient descent)

Page 4: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Deep Learning

Deep-learning methods are representation-learning methods with multiple levels of representation, obtained by composing simple but non-linear modules that each transform the representation at one level (starting with the raw input) into a representation at a higher, slightly more abstract level.

LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553 (2015): 436-444.

Page 5: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Activation Unit

Page 6: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

A simple network with 2 layers

X1 Y

X2

w1

w2

Y = f(b + x1w1 + x2w2 )

1b

Input Layer Output Layer

Bias

Bias

Weights

Weights

Page 7: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Feedforward Neural Network

Page 8: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Neural network for classification

Page 9: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Setting up Keras

- Python - Quick Tutorial:

http://deeplearning.net/software/theano/tutorial/python.html

- Numpy - Tutorial:

https://docs.scipy.org/doc/numpy-dev/user/quickstart.html

- Matlab, Numpy, R and Julia:

http://hyperpolyglot.org/numerical-analysis

- Theano (needed for keras, Tensorflow is an alternative) - Tutorial and installation: http://deeplearning.net/software/theano/

- Keras- Tutorial and installation: https://keras.io/

http://deeplearning.net/software/theano/tutorial/python.html
http://deeplearning.net/software/theano/tutorial/python.html
https://docs.scipy.org/doc/numpy-dev/user/quickstart.html
https://docs.scipy.org/doc/numpy-dev/user/quickstart.html
http://hyperpolyglot.org/numerical-analysis
http://hyperpolyglot.org/numerical-analysis
http://deeplearning.net/software/theano/
https://keras.io/
Page 10: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Demo with keras

See the tutorial_class.py:

http://www.cs.unc.edu/~tgao/tutorial_class.py

http://www.cs.unc.edu/~tgao/tutorial_class.py
http://www.cs.unc.edu/~tgao/tutorial_class.py
Page 11: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Convolutional Neural Network

http://inspirehep.net/record/1252539/files/CNN.png

Page 12: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Convolution Layer

Here is a nice animation:

http://cs231n.github.io/convolutional-networks/

http://cs231n.github.io/convolutional-networks/
http://cs231n.github.io/convolutional-networks/
Page 13: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Max pooling layer

Page 14: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Convolutional Neural Network

https://en.wikipedia.org/wiki/File:Typical_cnn.png

Page 15: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

A demo for CNN

How convolutional neural networks see the world

VGG16 (OxfordNet)

https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html

https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html
Page 16: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

State-of-Art result

Classification dataset results

http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html

http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html
http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html
http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html
Page 17: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Recurrent Neural Network

Do not confuse with recursive neural network.

LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553 (2015): 436-444.

Page 18: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Recurrent Neural NetworkA popular structure: LSTM (Long Short-Term Memory)

Example in Keras: Text generation

https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation

.py

Kaparthy’s blog: The Unreasonable Effectiveness of Recurrent Neural

Networks

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Awesome RNN: List of resources, papers about rnn

https://github.com/kjw0612/awesome-rnn

https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py
https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py
https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Page 19: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Combine RNN and CNN

LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553 (2015): 436-444.

Page 20: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Some other techniques

1. Dropout2. AutoEncoder 3. Batch Normalization4. Residual network

Page 21: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Dropout

Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." Journal of Machine Learning Research 15.1 (2014): 1929-1958.

Page 22: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Auto-Encoder (AE)

http://goo.gl/e011Tf

Output X

Code H

Input X

Page 23: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Denoising Auto-Encoder (DAE)

Tan, J., et al. "Unsupervised feature construction and knowledge extraction from genome-wide assays of breast cancer with denoising autoencoders."Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing. Vol. 20. NIH Public Access, 2014.

Output X

Code H

Corrupted input X

Input X

Page 24: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Stacked Auto-Encoder (SAE)

Min, Seonwoo, Byunghan Lee, and Sungroh Yoon. "Deep Learning in Bioinformatics." arXiv preprint arXiv:1603.06430 (2016).

Page 25: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Stacked Auto-Encoder (SAE)

Min, Seonwoo, Byunghan Lee, and Sungroh Yoon. "Deep Learning in Bioinformatics." arXiv preprint arXiv:1603.06430 (2016).http://cs229.stanford.edu/materials/CS229-DeepLearning.pdf

Page 26: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Batch Normalization

Ioffe, Sergey, and Christian Szegedy. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." arXiv preprint arXiv:1502.03167 (2015).

Page 27: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Residual Network

http://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf

http://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
http://icml.cc/2016/tutorials/icml2016_tutorial_deep_residual_networks_kaiminghe.pdf
Page 28: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Questions

Page 29: Deep learning Tutorial - University of North Carolina at …deeplearning.web.unc.edu/.../10/Deep-learning-Tutorial.pdfDeep learning Tutorial Tianxiang Gao Aug 31, 2016 Content 1. Basics

Other resource

Tips on Debugging Theano:

http://udibr.github.io/tips-on-working-with-theano.html

http://udibr.github.io/tips-on-working-with-theano.html
http://udibr.github.io/tips-on-working-with-theano.html

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Deep Learning I - Korea Universitydsba.korea.ac.kr/wp/wp-content/seminar/Deep learning/Deep Learning... · Deep Learning I Deep learning is research on learning models with multilayer

Introducing Deep Learning with MATLAB - MathWorks · Introducing Deep Learning with MATLAB7 How A Deep Neural Network ... relevant features of an image. With deep learning, ... Deep

Deep Learning for Depth Learning CS 229 Course Project ...cs229.stanford.edu/...DepthLearning_FinalReport.pdf · Deep Learning for Depth Learning CS 229 Course Project, ... Deep Learning

Enterprise Application -Lesegruppe WebServices -Zusammenfassung von Tianxiang LU

Binary Deep Learning - TAUweb.eng.tau.ac.il/.../wp-content/uploads/2017/03/Binary-Deep-Learning.pdf · Binary Deep Learning Deep Learning Seminar, School of Electrical Engineering,

Deep Learning with H2O - University of Idahostevel/504/Deep Learning with...Deep Learning with H2O. . 4Deep Learning Overview Unlike the neural networks of the past, modern Deep Learning

Introducing Deep Learning with MATLAB - Systematics · Introducing Deep Learning with MATLAB6 Inside a Deep Neural Network ... Deep learning is a subtype of machine learning. With

Lecture 1: Introduction to Deep Learning - uvadlc.github.io · Lecture 1: Introduction to Deep Learning Efstratios Gavves. UVA DEEP LEARNING COURSE UVA DEEP LEARNING COURSE –EFSTRATIOS

20151216 Computer Vision Meetup Deep Learning Deep Learning - Rene...Deep Learning . René Donner Deep Learning Overview 2 ... coursera ... 20151216 Computer Vision Meetup Deep Learning

Deep learning / representation learning

Introduction to Deep Learning - Computer Graphics€¦ · Introduction to Deep Learning CS468 Spring 2017 Charles Qi. What is Deep Learning? Deep learning allows computational models

Demystifying Deep Learning - MathWorks · Demystifying Deep Learning. 2 What’s Deep Learning and why should I care? A practical approach to Deep Learning (for images) –Transfer

Intro to Deep Learning - bpTT Bright Minds … · 17/08/2020  · Intro to Deep Learning Deep Sequence Modeling Deep Computer Vision Deep Generative Modeling Deep Learning Applications

Deep Learning for Laser Based Odometry Estimation - …juxi.net/workshop/deep-learning-rss-2016/papers/Nicolai - Deep... · Deep Learning for Laser Based Odometry Estimation ... deep

Deep Learning - Oxford Statisticspalamara/teaching/SML19/HT19_lecture15.pdf · SB2b/SM4 - Deep Learning Deep Learning Infrastructure! Computational Infrastructure critical to deep

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

CS 678 – Deep Learning1 Deep Learning Early Work Why Deep Learning Stacked Auto Encoders Deep Belief Networks

Executing Deep Learning Strategies Masterclass Preview - Enterprise Deep Learning

Deep Learning: An Introduction from the NLP Perspective Deep Learning - An...Deep Learning Approach 1 Deep Learning Approach 2 Disclaimer I am not (yet) an expert in Deep Learning

Deep Learning

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

FastVA: Deep Learning Video Analytics Through Edge Processing …mcn.cse.psu.edu/paper/tan-tianxiang/infocom20-tan.pdf · 2020. 1. 8. · the edge server and letting the edge server

Hangzhou Deep Learning Meetup-Deep Reinforcement Learning

Deep learning

Deep Learning in real world @Deep Learning Tokyo

Deep materials informatics: Applications of deep learning in … · deep learning: † Deep learning requires big data: In many cases, the biggest limiting factor for applying deep

THE DEEP LEARNING REVOLUTION - Nvidia...THE DEEP LEARNING REVOLUTION Superhuman Breakthroughs in Modern Artificial Intelligence Powered by GPUs JOIN THE DEEP LEARNING ERA DEEP LEARNING

fast.ai - Learning Deep Learning

Lecture 3: Deep Learning Optimizations - GitHub Pagesuvadlc.github.io/lectures/nov2019/lecture3.pdf · Lecture 3: Deep Learning Optimizations Deep Learning @ UvA. UVA DEEP LEARNING

Deep Learning - South Dakota State UniversityDeep Learning Deep learning (also known as deep structured learning, hierarchical learning or deep machine learning) is a branch of machine

Deep Learning for Reinforcement Learning in Pacman · Deep Learning for Reinforcement Learning in Pacman Deep Learning für Reinforcement Learning in Pacman Vorgelegte Bachelor-Thesis

Deep Learning Early Work Why Deep Learning Stacked Auto Encoders Deep Belief Networks CS 678 – Deep Learning1

Einführung ins Deep Learning · Einführung ins Deep Learning deep space computing AG 6. Dezember 2019 deep-space.ch

Introduction to Deep Learning - Joyce Hojoyceho.github.io/cs534_s17/slide/25-deep-learning.pdf · Deep Learning: Overview ... Review: Backpropogation https: ... • Deep Learning