deep learning – a personal perspectiveintrotodeeplearning.com/2018/materials/2018_6s191... ·...
TRANSCRIPT
![Page 1: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/1.jpg)
Urs Muller
MIT 6.S191 Introduction To Deep Learning
DEEP LEARNING – A PERSONAL PERSPECTIVE
![Page 2: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/2.jpg)
2
NEURAL NETWORK MAGIC
Training data
Random network architecture
Excellent results
Learning algorithm
![Page 3: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/3.jpg)
3
Let’s us solve problems we don’t know how to program
THE REAL MAGIC OF DEEP LEARNING
![Page 4: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/4.jpg)
4
NEURAL NETWORKS RESEARCH AT BELL LABS, HOLMDEL 1985 – 1995
![Page 5: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/5.jpg)
5
BELL LABS BUILDING IN HOLMDELNow called Bell Works
Around 1990:
6,000 employees in Holmdel
~300 in Research
~30 in machine learning, including:
Larry Jackel, Yann LeCun, Leon Bottou, John Denker, Vladimir Vapnik, Yoshua Bengio, Hans Peter Graf, Patrice Simard, Corinna Cortes, and many others
![Page 6: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/6.jpg)
6
ORIGINAL DATABASE ~300 DIGITS
Demonstrated need for large database for benchmarking
→ USPS database
→ MNIST database (60,000 digits)
![Page 7: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/7.jpg)
7 7
USING PRIOR KNOWLEDGE
What class is point “X”?
If representation is north, south, east, west, then choose green
If better representation is elevation above sea level choose red
Using prior knowledge beats requiring tons of training data
Example: use convolutions for OCR
x
![Page 8: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/8.jpg)
8
CONVOLUTIONAL NEURAL NETWORK (CNN)
![Page 9: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/9.jpg)
9 9
LENET 1993
![Page 10: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/10.jpg)
10
HOW WE CAME TO VIEW LEARNING
Choose the right structure – “Structural Risk Minimization”
→ Bring prior knowledge to the learning machine
Capacity control – matching learning machine complexity to available data
→ Examine learning curves
Largely Vladimir Vapnik’s infuence
![Page 11: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/11.jpg)
11
LEARNING CURVES
If test error >> training error: get more training examples or decrease capacity
If test error ~ training error: increase capacity
Smart structure allows low errors with lower capacity
Training set error
Test set error
Number of training examples
Error
Cortes, Jackel, Wan-Ping Chiang: Limits on Learning Machine Accuracy Imposed by Data Qualityhttp://papers.nips.cc/paper/918-limits-on-learning-machine-accuracy-imposed-by-data-quality.pdf
![Page 12: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/12.jpg)
12
PILOTNET LEARNING CURVE
![Page 13: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/13.jpg)
13
THE (FRIENDLY) RECOGNIZER WARWho can read the digits best?
http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=576879
Year 1994
![Page 14: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/14.jpg)
14
Larry Jackel: “In hindsight, the similar test errors are not so surprising since all the researchers were using variants of the same learning principles (everyone talked to Vladimir and studied the learning curves).”
The Optimal Margin (SVM) was remarkable because, unlike the other top performers, it did not include knowledge about the geometry of the problem.
RESULTS ON MNIST DATA SETClassification error on test set Year 1994
![Page 15: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/15.jpg)
15
Inference time in sec on a Sparc 10
Training time in days on a Sparc 10
Classification error on test set
Year 1994
![Page 16: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/16.jpg)
16
LESSONS LEARNED
Look at the data during all processing steps
Solid debugging tools are critical
Validate the training data
The work is experimental in nature
Work with real data
![Page 17: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/17.jpg)
17
EPILOG
1995 Deployed Holmdel neural nets at Wachovia Bank – Eventually our technology was processing 20% of the checks written in the US
1995 AT&T Bell Labs fractured
2002 AT&T Labs – mass layofs
2003 Bell Labs work led to new programs at DARPA –LAGR, Learning Locomotion, Deep Learning, Challenges
2012 Deep Learning becomes popular triggered by availability of data, compute power and ready commercial applications
![Page 18: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/18.jpg)
18
LEARNED AUTONOMY
![Page 19: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/19.jpg)
19 19
DRIVE-AV
DRIVE AV
DRIVEWORKS SDK
DRIVE OS
DRIVE PX – AI CAR COMPUTER
A full self-driving car stack
![Page 20: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/20.jpg)
20
NEW JERSEY TEAM GOAL
Solve the hard and unsolved autonomous vehicles (AV) problems using learning
To provide enhanced safety through algorithmic diversity
To create functionality that may not be achievable otherwise, such as perception-based turns or merges onto a busy highway
Additional AV labs in California, Boulder, Seattle, and Europe
![Page 21: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/21.jpg)
21
NVIDIA OFFICE AT BELL WORKS IN HOLMDEL, NJ
![Page 22: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/22.jpg)
22
Private roads for easy testing
Quick access to diverse public roads
Space to store and work on the cars
An ideal location for autonomous driving R&D
![Page 23: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/23.jpg)
23
New York City
Ferry
Local train
CONVENIENT LOCATION
Close to New York City (45 miles)
Accessible by commuter train, ferry, bus, and car
Family friendly, yet near a major metropolitan area
Close to popular beaches along the Jersey Shore
![Page 24: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/24.jpg)
24
WHERE DO RULES FAIL?
Hand crafted features and rules work well when the problem domain is well understoodSimple rules, such as “follow the lane lines” fail quickly
→ Can you list every feature you look at while driving?
It is extremely hard to write down every single rule required for safe driving – it is relatively easy to collect lots of training data (just drive)
![Page 25: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/25.jpg)
25
Sensordata
CNN predictedpath
Left lane
marking
Right lane
markingRail
Rear axle
Car
LEARNING TO PREDICT THE PATH
Training data
Human driven path
Record data from lots of humansdriving their cars: Sensor data Human driven path
Error (training) signal
CNN
![Page 26: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/26.jpg)
26
EXAMPLE ARCHITECTURESingle frame input with 200x66 pixel resolution
Normalization3@66x200
Convolution24@31x98
Convolution36@14x47
Convolution48@5x22
Convolution64@3x20
Convolution64@1x18
Fully connected
layers
Output: Turn radius
YUV Input 3@66x200
Input is from a single camera with a small patch cropped and scaled from the full image~250,000 neuron weights, not a huge networkThe network grows as we use more sensors and handle harder tasks
![Page 27: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/27.jpg)
27 27
https://www.youtube.com/watch?v=-96BEoXJMs0&t=39s
![Page 28: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/28.jpg)
28 28
DRIVING WITH LEARNED PATH PREDICTION
![Page 29: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/29.jpg)
29 29
LEARNED LANE CHANGES
![Page 30: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/30.jpg)
30
LEARNED TURNShttps://www.youtube.com/watch?v=Sm-NBdSzP6E
![Page 31: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/31.jpg)
31
VISUALIZATIONWhere does the network look?
![Page 32: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/32.jpg)
32 32
VISUALIZATION
![Page 33: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/33.jpg)
33
ATYPICAL VEHICLE CLASS
![Page 34: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/34.jpg)
34
OPEN CHALLENGES
Deal with ambiguous situationsThere is often more than one correct answer
Learn from imperfect behaviorSeveral observations in the same situation, most are correct, some are not correct
![Page 35: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/35.jpg)
35
DRIVING IN THE SNOW
![Page 36: DEEP LEARNING – A PERSONAL PERSPECTIVEintrotodeeplearning.com/2018/materials/2018_6S191... · 2021. 3. 12. · 17 EPILOG 1995Deployed Holmdel neural nets at Wachovia Bank – Eventually](https://reader036.vdocuments.site/reader036/viewer/2022071505/6125eebd0be6782c893af2a2/html5/thumbnails/36.jpg)
Thank You