deep learning for computer vision: saliency prediction (upc 2016)
TRANSCRIPT
![Page 1: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/1.jpg)
Day 3 Lecture 2
Saliency PredictionAcknowledments: Junting Pan, Kevin McGuinness and Xavier Giró-i-Nieto
Elisa Sayrol
[course site]
![Page 2: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/2.jpg)
2
Saliency
![Page 3: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/3.jpg)
3
What have you seen?
Saliency
![Page 4: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/4.jpg)
4
Lighthouse
Saliency
![Page 5: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/5.jpg)
5
HouseLighthouse
Saliency
![Page 6: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/6.jpg)
6
Rocks
HouseLighthouse
Saliency
![Page 7: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/7.jpg)
7
Saliency
![Page 8: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/8.jpg)
Saliency Map
Original Image Ground Truth Saliency Map(Eye-Fixation Map)
The Goal is to obtain the Saliency Map of an Image. Regression problem, not Classification
![Page 9: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/9.jpg)
9
Eye Tracker Mouse Click
Data Bases: Groundtruth generation
![Page 10: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/10.jpg)
10
DataBases
Other databases: http://saliency.mit.edu/datasets.html
TRAIN VALIDATION TEST
SALICONJiang’15
10,000 5,000 5,000
iSun Xu’15 6,000 926 2,000
CAT2000 [Borji’15] 2,000 - 2,000
MIT300 [Judd’12] 300 - -
Pascal-S 850
![Page 11: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/11.jpg)
11
Upsample + filter
2D map
96x96 2340=48x48
Architectures: Junting Net (Shallow Network)
![Page 12: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/12.jpg)
Loss function Mean Square Error (MSE)
Weight initialization Gaussian distribution
Learning rate 0.03 to 0.0001
Mini batch size 128
Training time 7h (SALICON) / 4h (iSUN)
Acceleration SGD+ nesterov momentum (0.9)
Regularisation Maxout norm
GPU NVidia GTX 980
Architectures: Junting Net (Shallow Network)
Shallow and Deep Convolutional Networks for Saliency PredictionJunting Pan, Kevin McGuinness, Elisa Sayrol, Noel O'Connor, Xavier Giro-i-Nieto, CVPR 2016
Winner of the LSUN Challenge 2015!!
![Page 13: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/13.jpg)
Architectures: SalNet (Deep Network)
Loss function Mean Square Error (MSE)
Weight initialization First 3 layers pre-trained with VGG, the rest of the layers random distribution
Learning rate 0,01(halved every 100 iterations)
Mini batch size 2 images for 24.000 iterations
Training time 15h
Acceleration SGD+ nesterov momentum (0.9)
Regularisation L2 weight
GPU NVidia GTX Titan
Shallow and Deep Convolutional Networks for Saliency PredictionJunting Pan, Kevin McGuinness, Elisa Sayrol, Noel O'Connor, Xavier Giro-i-Nieto, CVPR 2016
![Page 14: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/14.jpg)
Quality Results
![Page 15: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/15.jpg)
15
Results from CVPR LSUN Challenge 2015 (iSUN Database)
Architectures: Junting Net (Shallow Network) Winner of the LSUN Challenge 2015!!
![Page 16: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/16.jpg)
Quantitative Results
Metrics: Saliency and Human Fixations: State-of-the-art and Study of Comparison MetricsNicolas Riche, Matthieu Duvinage, Matei Mancas, Bernard Gosselin and Thierry Dutoit, iccv 2013
![Page 17: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/17.jpg)
Similar to VGG_16
Architectures: Saliency Unified ( Very Deep Network)
Saliency Unified: A Deep Architecture for simultaneous Eye Fixation Prediction and Salient Object SegmentationSrinivas S S Kruthiventi, Vennela Gudisa, Jaley H Dholakiya and R. Venkatesh Babu, CVPR 2016
![Page 18: Deep Learning for Computer Vision: Saliency Prediction (UPC 2016)](https://reader030.vdocuments.site/reader030/viewer/2022020203/586f89911a28ab54768b6069/html5/thumbnails/18.jpg)
Quantitative Results
Saliency Unified: A Deep Architecture for simultaneous Eye Fixation Prediction and Salient Object SegmentationSrinivas S S Kruthiventi, Vennela Gudisa, Jaley H Dholakiya and R. Venkatesh Babu, CVPR 2016