cues and constraints for single image 3dseminars/seminars/extra/2015_07_14...cues and constraints...
TRANSCRIPT
![Page 1: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/1.jpg)
Cues and Constraints for Single Image 3D
David Fouhey
Joint work with: Abhinav Gupta, Martial Hebert,
Xiaolong Wang, Wajahat Hussain
![Page 2: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/2.jpg)
2
![Page 3: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/3.jpg)
You See…
3
![Page 4: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/4.jpg)
Unfortunately…
4
![Page 5: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/5.jpg)
Why?
5
![Page 6: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/6.jpg)
Local Cues
6
![Page 7: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/7.jpg)
Constraints
7
![Page 8: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/8.jpg)
Constraints
8
![Page 9: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/9.jpg)
Today
9
Single Image 3D Without a Single 3D Image. Fouhey, Hussain, Gupta, Hebert.
In Submission to ICCV 2015.
Unfolding an Indoor Origami World. Fouhey, Gupta, Hebert. ECCV 2014.
Designing Deep Networks for Surface Normal Estimation. Wang, Fouhey, Gupta. CVPR 2015.
![Page 10: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/10.jpg)
Specific Problem
10
Input Normals
Image Legend
![Page 11: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/11.jpg)
My Other Interests
11
People and Scenes with V. Delaitre, A.A. Efros, I. Laptev, J. Sivic
ECCV ’12, IJCV ‘14
OneWeirdKernelTrick.com Visual Identification of Matrix Rank, Spectral Approaches to Ghost Detection, etc.
with D. Maturana, SIGBOVIK ‘13, ‘15
Visual Prediction with L. Zitnick
CVPR ‘14
![Page 12: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/12.jpg)
Local Cues
12
![Page 13: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/13.jpg)
Local Cues for Surface Normals
13
Hoiem et al. (ICCV ‘05)
Fouhey et al. (ICCV ’13)
Ladicky et al. (ECCV ’14)
Li et al. (CVPR ’15)
Eigen et al. (Arxiv ‘14)
![Page 14: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/14.jpg)
Standard Paradigm
14
Input: RGBD Data Output: Model
![Page 15: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/15.jpg)
Our Paradigm
15
Input: RGBD Data Output: Model
![Page 16: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/16.jpg)
Which One Is Unsupervised?
16
![Page 17: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/17.jpg)
Which One Is Unsupervised?
17
Median Error
23.1 Median Error
19.2
Median Error
17.9 Median Error
21.7
![Page 18: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/18.jpg)
Which One Is Unsupervised?
18
(B) (A)
(D) (C)
![Page 19: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/19.jpg)
Factorization
19
![Page 20: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/20.jpg)
Factorization
20 3D Model from Guo and Hoiem, ICCV13.
3D Structure
![Page 21: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/21.jpg)
Factorization
21
Style 3D Structure
![Page 22: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/22.jpg)
Factorization
22
Style Image
![Page 23: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/23.jpg)
Style Elements
23
Style Image
![Page 24: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/24.jpg)
Detecting with a Style Element
24
Style Element
Input Image
![Page 25: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/25.jpg)
Rectified Images
Detecting with a Style Element
25
Style Element
Input image Detection +
Orientation
Similar to Hedau et al., 2010, Fidler 2012,
![Page 26: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/26.jpg)
Detecting with a Style Element
26
Original Images
Rectified Images
Detection + Orientation
![Page 27: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/27.jpg)
How Do We Discover Elements?
27
…
Automatic Method
Input: RGB Images Output: Style Elements
![Page 28: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/28.jpg)
Modeling Assumptions
28
![Page 29: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/29.jpg)
Overview
29
…
![Page 30: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/30.jpg)
Hypothesize
30
![Page 31: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/31.jpg)
Hypothesize
31
![Page 32: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/32.jpg)
Hypothesize
32 HOG, Dalal and Triggs ’05; ELDA from Hariharan et al. ‘12
![Page 33: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/33.jpg)
What Do We Know?
33
θ
![Page 34: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/34.jpg)
What Do We Know?
34
θ ~Unif([0,360])
θ
H/W ~Unif([1,2])
H
W
![Page 35: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/35.jpg)
Verifying Style Elements
Surface Orientation
X Location 35
![Page 36: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/36.jpg)
Verifying Style Elements
Surface Orientation
Prior = (Rectangular Box)
X Location 36
θ
![Page 37: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/37.jpg)
Verifying Style Elements
Surface Orientation
X Location 37
![Page 38: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/38.jpg)
Verifying Style Elements
Surface Orientation
X Location 38
![Page 39: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/39.jpg)
Verifying Style Elements
X Location 39
𝑃𝑟𝑖𝑜𝑟𝑖 −𝐷𝑎𝑡𝑎𝑖
𝑊
𝑖=1
![Page 40: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/40.jpg)
Verifying Style Elements
Surface Orientation
X Location 40
![Page 41: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/41.jpg)
Verifying Style Elements
Surface Orientation
X Location 41
![Page 42: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/42.jpg)
Discovered Style Elements
Element Detections Element Detections
42
![Page 43: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/43.jpg)
Interpreting
…
…
43
![Page 44: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/44.jpg)
Interpreting
44
![Page 45: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/45.jpg)
Results
45
Input
GT
Output
![Page 46: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/46.jpg)
Results
46
Input
GT
Output
![Page 47: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/47.jpg)
Results
47
Input
GT
Output
![Page 48: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/48.jpg)
Quantitative Results
49
Input Ground Truth Prediction
5⁰
![Page 49: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/49.jpg)
Quantitative Results
50
All Pixels Pixels < 30⁰
Median Error
Proposed 21.7 55.4
Vertical Pixels Pixels < 30⁰
Median Error
19.9 58.8
19.7 59.7
Pixels < 11.25⁰
36.8
3DP 19.2 57.8 39.2
Fouhey et al. ‘14 17.9 58.9
Ladicky et al. ’14 23.5 58.7 40.5 27.7
Eigen et al. ‘14 15.5 71.1
Wang et al. ’15 14.4 68.2 39.2 42.0
![Page 50: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/50.jpg)
Results on Internet Images
51
RGBD Datasets What about?
![Page 51: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/51.jpg)
Results on Internet Images
52
Supermarket
Museum
Laundromat
Locker Room
![Page 52: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/52.jpg)
Results on Internet Images
53
![Page 53: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/53.jpg)
Results on Internet Images
54
![Page 54: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/54.jpg)
Quantitative Results
• Places-205 Dataset (Zhou et al.)
• ~3.7% better than pretrained supervised model (3DP); better in 9/10 categories
55
![Page 55: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/55.jpg)
Observation
56
Element #1
Element #2
Element #3
Element #4
Element #5
Purely Data-Driven Single
Element
Data+Structure
![Page 56: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/56.jpg)
So Far
57
Input Output
What about?
![Page 57: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/57.jpg)
Constraints for Single Image 3D
58
Low Level, Generic
Hoiem et al. 2005, Saxena et al. 2005, 2008, Munoz et al., 2009, etc.
![Page 58: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/58.jpg)
Constraints for Single Image 3D
59
High Level, Physical
Low Level, Generic
Coughlan et al. 2000, Hedau et al. 2009, Del Pero et al., 2011, Wang et al., 2012, Schwing et al. 2012, Lee et al. 2010, Xiao et al. 2012, Zhao et al. 2013, Schwing et al., 2013, etc.
![Page 59: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/59.jpg)
Mid-level in the Past
60 Huffman 71, Clowes 71, Kanade 80, 81 Sugihara 86, Malik 87, etc.
![Page 60: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/60.jpg)
Our Mid-Level Constraints
61
![Page 61: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/61.jpg)
Our Output
62
Input: Single Image
Output: Discrete Scene Parse
![Page 62: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/62.jpg)
Parameterization
63
![Page 63: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/63.jpg)
Parameterization
vp1
vp2
vp3
VP Estimator from Hedau et al., 2009 64
![Page 64: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/64.jpg)
Parameterization
Two VPs give grid cell
65
![Page 65: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/65.jpg)
Encoding Surface Normals
66
![Page 66: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/66.jpg)
Encoding Surface Normals
67
![Page 67: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/67.jpg)
Encoding Surface Normals
68
![Page 68: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/68.jpg)
Encoding Surface Normals
x1,…, x400 x401,…, x800 x801,…, x1200 69
![Page 69: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/69.jpg)
Formulation
70
![Page 70: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/70.jpg)
Constraints
71
![Page 71: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/71.jpg)
Unaries
72
![Page 72: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/72.jpg)
Unaries
Low c Any 3D Evidence
High c
73
![Page 73: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/73.jpg)
Binaries
74
![Page 74: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/74.jpg)
Convex/Concave Constraints
Detected Concave (-) 75
![Page 75: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/75.jpg)
Convex/Concave Constraints
Detected Concave (-) 76
![Page 76: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/76.jpg)
Convex/Concave Constraints
Detected Concave (-) 77
![Page 77: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/77.jpg)
Convex/Concave Constraints
Detected Concave (-) 78
![Page 78: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/78.jpg)
Convex/Concave Constraints
Detected Concave (-) 79
![Page 79: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/79.jpg)
Detecting Convex/Concave
Ground-Truth Discontinuities similar to Gupta, Arbelaez, Malik, 2013 3DP from Fouhey, Gupta, Hebert, 2013
Input 3D Primitive Bank
…
Use 3DP to Transfer Convex/Concave
80
![Page 80: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/80.jpg)
Solving the Model
82
![Page 81: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/81.jpg)
Qualitative Results
83
![Page 82: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/82.jpg)
Qualitative Results
85
![Page 83: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/83.jpg)
Qualitative Results
86
![Page 84: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/84.jpg)
Results – Quantitative
Summary Stats (⁰) (Lower Better)
% Good Pixels (Higher Better)
11.25⁰ 22.5⁰ 30⁰
Proposed 40.5 54.1 58.9
Mean Median
3DP 39.2 52.9 57.8 36.3 19.2
Ladicky ‘14 27.7 49.0 58.7 33.5 23.1
Li ’15* 19.6 40.6 53.7 30.6 27.8
35.2 17.9
88 Fouhey et al. ICCV ’13; Ladicky et al. ECCV ‘14; Li et al. CVPR ‘15
![Page 85: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/85.jpg)
So Far
89
Input Outputs
HOG HOG + Standard Vision Feats + QP
![Page 86: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/86.jpg)
Applying Deep Learning
90
CNN
How do we enforce constraints?
How do we represent the output?
Input Goal Output
![Page 87: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/87.jpg)
Representation
91 Normal Coding from Ladicky, Zeisl, Pollefeys ECCV ‘14
![Page 88: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/88.jpg)
Applying Deep Learning
92
CNN
How do we enforce constraints?
How do we represent the output?
Input Output
![Page 89: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/89.jpg)
What Might We Want?
93
Global Cues Manhattan-World Room Layout
Local Cues Discontinuities
![Page 90: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/90.jpg)
Design
94 Local Edges Global Manhattan Layout
?
![Page 91: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/91.jpg)
Global Network
95
Global
Local Edges Global Manhattan Layout
![Page 92: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/92.jpg)
Local Network
96
Local
Local Edges Global Manhattan Layout
![Page 93: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/93.jpg)
Local Network
97
Local
Local Edges Global Manhattan Layout
![Page 94: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/94.jpg)
Fusion
98
Local
Local Edges Global Manhattan Layout
Global
Fusion
![Page 95: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/95.jpg)
Results
Input Output Input Output Input Output
99
![Page 96: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/96.jpg)
Results
Input Output
100
![Page 97: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/97.jpg)
Results
101
Input Output
![Page 98: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/98.jpg)
Quantitative Results
102
Summary Stats (⁰) % Good Pixels
11.25⁰ 22.5⁰ 30⁰ Mean Median
3DP 39.2 52.9 57.8 36.3 19.2
Ladicky ‘14 27.7 49.0 58.7 33.5 23.1
Li ’15* 19.6 40.6 53.7 30.6 27.8
UNFOLD 40.5 54.1 58.9 35.2 17.9
Proposed 42.0 61.2 68.2 26.9 14.8
Eigen ‘14 39.2 62.0 71.1 23.7 15.5
Eigen et al. Arxiv ’14; Fouhey et al. ICCV ’13, ECCV ‘14; Ladicky et al. ECCV ‘14; Li et al. CVPR ‘15
![Page 99: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/99.jpg)
Summary – Task
103
Input Normals
![Page 100: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/100.jpg)
Summary – Solutions
104
RGB Images Model Surface Normals
![Page 101: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/101.jpg)
Summary – Solutions
105
Edge Constraints Scene Parse
![Page 102: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/102.jpg)
Summary – Solutions
106
Local
Global
Fusion
CNNs + 3D Representations and Constraints
![Page 103: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/103.jpg)
Acknowledgments
107
Collaborators
Xiaolong Wang
Abhinav Gupta Martial Hebert
Sponsors • NSF Graduate Research
Fellowship • NDSEG Fellowship • ONR • NSF • DARPA • Bosch Research &
Technology
Wajahat Hussain
![Page 104: Cues and Constraints for Single Image 3Dseminars/seminars/Extra/2015_07_14...Cues and Constraints for Single Image 3D David Fouhey Joint work with: Abhinav Gupta, Martial Hebert, Xiaolong](https://reader034.vdocuments.site/reader034/viewer/2022050210/5f5d251e736edc55177fcc0a/html5/thumbnails/104.jpg)
Thank You!
108