learning 3d geometry - cs.cornell.edu
TRANSCRIPT
![Page 1: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/1.jpg)
ⓘ
![Page 2: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/2.jpg)
![Page 4: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/4.jpg)
Third Place
![Page 5: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/5.jpg)
![Page 6: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/6.jpg)
Second Place
![Page 7: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/7.jpg)
![Page 8: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/8.jpg)
First Place
![Page 9: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/9.jpg)
![Page 10: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/10.jpg)
![Page 11: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/11.jpg)
![Page 12: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/12.jpg)
![Page 14: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/14.jpg)
![Page 16: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/16.jpg)
•
•
•
![Page 17: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/17.jpg)
![Page 18: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/18.jpg)
![Page 19: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/19.jpg)
●
![Page 20: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/20.jpg)
![Page 21: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/21.jpg)
[Snavely, Seitz, Szeliski. Photo Tourism. SIGGRAPH 2006]
![Page 22: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/22.jpg)
![Page 23: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/23.jpg)
![Page 24: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/24.jpg)
●
●
![Page 25: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/25.jpg)
Input Ground truth Predicted depth
![Page 26: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/26.jpg)
![Page 27: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/27.jpg)
![Page 28: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/28.jpg)
![Page 29: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/29.jpg)
![Page 30: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/30.jpg)
●
●
●
![Page 32: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/32.jpg)
![Page 33: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/33.jpg)
Stereo Photography
![Page 34: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/34.jpg)
Stereo Photography
Viewing Devices
![Page 35: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/35.jpg)
Queen Victoria at World Fair, 1851
Stereo Photography
![Page 36: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/36.jpg)
Stereo Photography
![Page 37: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/37.jpg)
Issue: Narrow Baseline
~1.5 cm~6.5 cm
![Page 38: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/38.jpg)
Left
![Page 39: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/39.jpg)
Right
![Page 40: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/40.jpg)
![Page 41: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/41.jpg)
Problem Statement
View Synthesis
model
…
Output
…
Output Input
![Page 42: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/42.jpg)
Challenges
……InputOutput Output
Extrapolation
Large disocclusion
Non-Lambertian Effects
Reflections, transparencies, etc.
![Page 43: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/43.jpg)
Prior Methods: No Shared Scene Representation
Input views
Neural Net
Output views
[Flynn et al., 2015]
[Kalantari et al. 2016]
![Page 44: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/44.jpg)
Input views
Neural Net
Output views
Prior Methods: No Shared Scene Representation
[Flynn et al., 2015]
[Kalantari et al. 2016]
![Page 45: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/45.jpg)
Input views
Neural Net
Output views
…
Rendered
independently
Prior Methods: No Shared Scene Representation
[Flynn et al., 2015]
[Kalantari et al. 2016]
![Page 46: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/46.jpg)
Input views
Scene
Representation
Ours: Shared Scene Representation
Neural Net
Output views
…
![Page 47: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/47.jpg)
Stereo Magnification: Learning View Synthesis using Multiplane Images
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, Noah Snavely
SIGGRAPH 2018
![Page 48: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/48.jpg)
Multiplane Camera (1937)
Image credits: Disney https://www.youtube.com/watch?v=kN-eCBAOw60 (from 1957)
![Page 49: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/49.jpg)
Multiplane Images (MPIs)
Reference
Viewpoint
Each plane is at a fixed
depth and encoded by
an RGBA image
![Page 50: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/50.jpg)
View Synthesis using Multiplane Images
Reference
Viewpoint
Homography
Target
Viewpoint
Over
![Page 51: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/51.jpg)
View Synthesis using Multiplane Images
Reference
Viewpoint
Homography
Target
Viewpoint
Over
Synthesized image
![Page 52: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/52.jpg)
![Page 53: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/53.jpg)
![Page 54: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/54.jpg)
• Models disocclusion
• Models soft edges and
non-Lambertian effects
• Efficient for view synthesis
• Differentiable rendering
Properties of Multiplane Images
![Page 55: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/55.jpg)
Learning Multiplane Images
Input views
Multiplane Image
Alpha
RGB
Neural net
![Page 56: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/56.jpg)
Learning Multiplane Images
Input views
Rendered views
…
Ground-truthMultiplane Image
Alpha
RGB
Neural net
![Page 57: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/57.jpg)
Training Data
…
Input views Target view
( )
( )
( )
,
,
,
Need massive set of triplets with known
camera poses
![Page 58: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/58.jpg)
![Page 59: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/59.jpg)
![Page 61: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/61.jpg)
Sampling Training Examples
… …
Input TargetInput(Extrapolated)
![Page 62: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/62.jpg)
Sampling Training Examples
… …
InputTargetInput(Interpolated)
![Page 63: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/63.jpg)
Results
![Page 64: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/64.jpg)
Left
![Page 65: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/65.jpg)
Right
![Page 66: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/66.jpg)
Output
![Page 67: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/67.jpg)
Image 1
![Page 68: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/68.jpg)
Image 2
![Page 69: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/69.jpg)
Output
![Page 70: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/70.jpg)
Reference input view
Plane 0
Plane 13
Plane 9
Plane 16
Plane 24 Plane 26
![Page 71: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/71.jpg)
![Page 72: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/72.jpg)
1.4 cm 6.3 cm
Extrapolating Cellphone Footage
![Page 73: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/73.jpg)
![Page 74: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/74.jpg)
![Page 75: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/75.jpg)
●
●
●
○
○
![Page 76: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/76.jpg)
![Page 77: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/77.jpg)
![Page 78: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/78.jpg)
![Page 79: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/79.jpg)
![Page 81: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/81.jpg)
![Page 82: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/82.jpg)
![Page 85: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/85.jpg)
![Page 86: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/86.jpg)
●
●
●
![Page 88: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/88.jpg)
![Page 89: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/89.jpg)
●
●
●
![Page 90: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/90.jpg)
![Page 91: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/91.jpg)
![Page 92: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/92.jpg)
![Page 93: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/93.jpg)
![Page 94: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/94.jpg)
![Page 95: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/95.jpg)
●
●
●
○
![Page 96: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/96.jpg)
●
●
●
○
![Page 97: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/97.jpg)
![Page 98: Learning 3D geometry - cs.cornell.edu](https://reader034.vdocuments.site/reader034/viewer/2022051406/627d7319e7e20839cd6f519d/html5/thumbnails/98.jpg)