multi-view geometry (ch7 new book. ch 10/11 old book)gerig/cs6320-s2013/materials/cs6320-cv... ·...
TRANSCRIPT
![Page 1: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/1.jpg)
Multi-View Geometry (Ch7 New book.
Ch 10/11 old book)
Credits: M. Shah, UCF CAP5415, lecture 23
http://www.cs.ucf.edu/courses/cap6411/cap5415/, Trevor Darrell, Berkeley,
C280, Marc Pollefeys
Guido Gerig
CS 6320 Spring 2013
![Page 2: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/2.jpg)
Material I
• http://vision.middlebury.edu/stereo/
• (online stereo pairs and truth (depth maps)
• Stereo correspondence software: e.g. http://vision.middlebury.edu/stereo/data/scenes2001/data/imagehtml/tsukuba.html
• CVonline compendium: http://homepages.inf.ed.ac.uk/rbf/CVonline/
![Page 3: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/3.jpg)
Material II
• Epipolar Geometry, Rectification: • http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_
COPIES/FUSIELLO2/rectif_cvol.html
• and: http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/OWENS/LECT11/node11.html
• Stereo: • http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_
COPIES/OWENS/LECT11/lect11.html
• 3D Reconstruction:
• http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/OWENS/LECT11/node8.html
![Page 4: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/4.jpg)
• Shading
Visual cues
![Page 5: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/5.jpg)
Visual cues
• Shading
• Texture
The Visual Cliff, by William Vandivert, 1960
![Page 6: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/6.jpg)
Visual cues
From The Art of Photography, Canon
• Shading
• Texture
• Focus
![Page 7: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/7.jpg)
Visual cues
• Shading
• Texture
• Focus
• Motion
![Page 8: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/8.jpg)
Visual cues
• Shading
• Texture
• Focus
• Motion
• Shape From X (X = shading, texture, focus, motion, rotation, ...)
Atmospheric Perspective
Linear Perspective
![Page 9: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/9.jpg)
Visual cues
Cornell CS569 S2008, Lecture 8, slide by Steve Marschner
http://www.cs.cornell.edu/courses/cs569/2008sp/about.stm
Shadows
![Page 10: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/10.jpg)
Visual cues
• Shading
• Texture
• Focus
• Motion
• Shape From X (X = shading, texture, focus, motion, rotation, ...)
• Stereo (disparity, multi-view)
![Page 11: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/11.jpg)
http://www.well.com/~jimg/stereo/stereo_list.html
Grauman
![Page 12: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/12.jpg)
Stereo photography and stereo viewers
Invented by Sir Charles Wheatstone, 1838
Image courtesy of fisher-price.com
Take two pictures of the same subject from two slightly different viewpoints and display so that each eye sees only one of the images.
Grauman
![Page 13: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/13.jpg)
http://www.johnsonshawmuseum.org
Grauman
![Page 14: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/14.jpg)
Human stereopsis: disparity
Disparity occurs when eyes fixate on one object; others appear at different visual angles
![Page 15: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/15.jpg)
Disparity: d = r-l = D-F.
d=0
Human stereopsis: disparity
Adapted from M. Pollefeys
![Page 16: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/16.jpg)
Example: Stereo to Depth Map
![Page 17: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/17.jpg)
Pinhole Camera Model
Z
Xfx
![Page 18: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/18.jpg)
Basic Stereo Derivations
Derive expression for Z as a function of x1, x2, f and B
![Page 19: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/19.jpg)
Basic Stereo Derivations
Z
Xfx 1
Z
Bfx
Z
BXfx
12
21 xx
fBZ
![Page 20: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/20.jpg)
Basic Stereo Derivations
Define the disparity: 21 xxd
d
fBZ
![Page 21: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/21.jpg)
Standard stereo geometry
Disparity d: 𝑑 = |𝑢′ − 𝑢| u’ u
Baseline B
![Page 22: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/22.jpg)
Standard stereo geometry
![Page 23: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/23.jpg)
Standard stereo geometry
![Page 24: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/24.jpg)
J. M. Rehg © 2003
Stereo Correspondence
• Search over disparity to find correspondences • Range of disparities to search over can change
dramatically within a single image pair.
![Page 25: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/25.jpg)
I1 I2 I10
Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on Pattern
Analysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.
Why is disparity important?
Given dense
disparity map,
we can
calculate a
depth/distance/
range map.
![Page 26: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/26.jpg)
27
Goal: 3D from Stereo via Disparity Map
F&P
Chapter 11
image I(x,y) image I´(x´,y´) Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
![Page 27: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/27.jpg)
Example: Stereo to Depth Map
![Page 28: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/28.jpg)
![Page 29: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/29.jpg)
Random dot stereograms
• Julesz 1960: Do we identify local brightness patterns before fusion (monocular process) or after (binocular)?
• To test: pair of synthetic images obtained by randomly spraying black dots on white objects
![Page 30: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/30.jpg)
Random dot stereograms
Forsyth & Ponce
![Page 31: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/31.jpg)
Random dot stereograms
![Page 32: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/32.jpg)
Random dot stereograms
![Page 33: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/33.jpg)
A Cooperative Model (Marr and Poggio, 1976)
Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr.
1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.
![Page 34: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/34.jpg)
Random dot stereograms
From Palmer, “Vision Science”, MIT Press
![Page 35: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/35.jpg)
Random dot stereograms
• When viewed monocularly, they appear random; when viewed stereoscopically, see 3d structure.
• Conclusion: human binocular fusion not directly associated with the physical retinas; must involve the central nervous system
• Imaginary* “cyclopean retina” that combines the left and right image stimuli as a single unit
Grauman
Visual Pathway.jpg wiki.ucl.ac.uk
*This was because it was as though we have a cyclopean
eye inside our brains that can see cyclopean stimuli
hidden to each of our actual eyes.
![Page 36: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/36.jpg)
Autostereograms
Images from magiceye.com
Exploit disparity as depth cue using single image
(Single image random dot stereogram, Single image stereogram)
![Page 37: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/37.jpg)
Images from magiceye.com
Autostereograms
![Page 38: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/38.jpg)
Optical flow Where do pixels move?
![Page 39: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/39.jpg)
Optical flow Where do pixels move?
![Page 40: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/40.jpg)
http://www.johnsonshawmuseum.org
Grauman
![Page 41: Multi-View Geometry (Ch7 New book. Ch 10/11 old book)gerig/CS6320-S2013/Materials/CS6320-CV... · 2013-02-04 · Random dot stereograms • When viewed monocularly, they appear random;](https://reader034.vdocuments.site/reader034/viewer/2022042918/5f5dca74099ff43bfa59b992/html5/thumbnails/41.jpg)
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
Grauman