3-d scene u u’u’ study the mathematical relations between corresponding image points....
TRANSCRIPT
![Page 1: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/1.jpg)
3-D Scene
u
u’
Study the mathematical relations between corresponding image points.
“Corresponding” means originated from the same 3D point.
Objective
![Page 2: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/2.jpg)
Two-views geometryOutline
Background: Camera, Projection models Necessary tools: A taste of projective geometry Two view geometry:
Planar scene (homography ). Non-planar scene (epipolar geometry).
3D reconstruction (stereo).
![Page 3: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/3.jpg)
Perspective Projection
f Xx
Zf Y
yZ
Origin (0,0,0) is the Focal center X,Y (x,y) axis are along the image axis (height / width). Z is depth = distance along the Optical axis f – Focal length
![Page 4: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/4.jpg)
Coordinates in Projective Plane P2
k(0,0,1)
k(x,y,0)
k(1,1,1)
k(1,0,1)
k(0,1,1)
“Ideal point”
Take R3 –{0,0,0} and look at scale equivalence class (rays/lines trough the origin).
z
y
x
z
y
x
![Page 5: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/5.jpg)
2D Projective Geometry: Basics A point:
A line:
we denote a line with a 3-vector
Line coordinates are homogenous
Points and lines are dual: p is on l if
Intersection of two lines/points
2 2( , , ) ( , )T Tx yx y z P
z z
0 ( ) ( ) 0x y
ax by cz a b cz z
0Tl p
1 2 ,l l 1 2p p
( , , )Ta b c
ll
![Page 6: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/6.jpg)
Cross Product in matrix notation [ ]x
0
0
0
xy
xz
yz
x
tt
tt
tt
t1 2 1 2 1 2
1 2 1 2 1 2
1 2 1 2 1 2
x x y z z y
y y z x x z
z z x y y x
0
0
0
x y z z y
y z x z x
z x y y x
t x t z t y t t x
t y t x t z t t y
t z t y t x t t z
Hartley & Zisserman p. 581
ptpt x
![Page 7: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/7.jpg)
2D Projective Transformation
H is defined up to scale
9 parameters 8 degrees of freedom Determined by 4 corresponding points
how does H operate on lines?
0
1: 0 ( )( ) 0T T Tl H l l p l H Hp
Hartley & Zisserman p. 32
HH
![Page 8: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/8.jpg)
Two-views geometryOutline
Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:
Homography Epipolar geometry, the essential matrix Camera calibration, the fundamental matrix
3D reconstruction from two views (Stereo algorithms)
![Page 9: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/9.jpg)
Two View Geometry When a camera changes position and
orientation, the scene moves rigidly relative to the camera
3-D Scene
u
u’
X
Y
Z
d
p
Rotation + translation
![Page 10: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/10.jpg)
Two View Geometry (simple cases) In two cases this results in homography:
1. Camera rotates around its focal point
2. The scene is planar
Then: Point correspondence forms 1:1mapping depth cannot be recovered
![Page 11: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/11.jpg)
Camera Rotation
' , 0
( )
'' ' ( ' ')
' ( ' )'
P RP t
Zp P P p
f
Zp P P p
f
Zp Rp p Rp
Z
(R is 3x3 non-singular)
![Page 12: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/12.jpg)
Planar Scenes
IntuitivelyA sequence of two perspectivities
Algebraically
Need to show:
( )
1'
1, '
' ,'
T
TT
T
n P d aX bY cZ d
n PP RP t RP t R tn P
d d
H R tn P HPd
Zp Hp
Z
Scene
Camera 1
Camera 2
Hpp '
![Page 13: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/13.jpg)
Summary: Two Views Related by HomographyTwo images are related by homography:
One to one mapping from p to p’ H contains 8 degrees of freedom Given correspondences, each point determines
2 equations 4 points are required to recover H Depth cannot be recovered
' ,'
Zp Hp
Z
![Page 14: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/14.jpg)
The General Case: Epipolar Lines
epipolar lineepipolar line
![Page 15: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/15.jpg)
Epipolar Plane
epipolar plane
epipolar lineepipolar lineepipolar lineepipolar line
BaselineBaseline
PP
OO O’O’
![Page 16: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/16.jpg)
Epipole Every plane through the baseline is an epipolar
plane It determines a pair of epipolar lines (one in each image)
Two systems of epipolar lines are obtained Each system intersects in a point, the epipole The epipole is the projection of the center of the
other camera
epipolar planeepipolar linesepipolar linesepipolar linesepipolar lines
BaselineBaselineOO O’O’
![Page 17: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/17.jpg)
Example
![Page 18: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/18.jpg)
Epipolar Lines
epipolar plane
epipolar lineepipolar lineepipolar lineepipolar line
BaselineBaseline
PP
OO O’O’
To define an epipolar plane, we define the plane through the two camera centers O and O’ and some point P. This can be written algebraically (in some worldcoordinates as follows:
' ' 0T
OP OO O P
![Page 19: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/19.jpg)
Essential Matrix (algebraic constraint between corresponding image points) Set world coordinates around the first camera
What to do with O’P? Every rotation changes the observed coordinate in the second image
We need to de-rotate to make the second image plane parallel to the first
Replacing by image points
' ' 0T
OP OO O P
' 0TP t RP
, 'P OP t OO
' 0Tp t Rp Other derivations Hartley & Zisserman p. 241
![Page 20: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/20.jpg)
Essential Matrix (cont.)
Denote this by:
Then
Define
E is called the “essential matrix”
t p t p
' ' 0T Tp t Rp p t Rp
E t R
' 0Tp Ep
' 0Tp t Rp
![Page 21: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/21.jpg)
Properties of the Essential Matrix E is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear E, E can be recovered up to scale using 8 points. Has rank 2.
The constraint detE=0 7 points suffices In fact, there are only 5 degrees of freedom in E,
3 for rotation 2 for translation (up to scale), determined by epipole
0 ': l plpE t
' 0Tp Ep
e) trough lines ( : : 12 all PPEThus
![Page 22: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/22.jpg)
BackgroundThe lens optical axis does not coincide with
the sensor
We model this using a 3x3 matrix the Calibration matrix
Camera Internal Parameters or Calibration matrix
![Page 23: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/23.jpg)
Camera Calibration matrix
The difference between ideal sensor ant the real one is modeled by a 3x3 matrix K:
(cx,cy) camera center, (ax,ay) pixel dimensions, b skew
We end with
0
0 0 1
x x
y y
a b c
K a c
q Kp
![Page 24: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/24.jpg)
Fundamental Matrix
F, is the fundamental matrix.
1 1
1
1
' 0 ( ) ( ') 0
( ) ' 0
T T
T T
T
p Ep K q E K q
q K EK q
F K EK
![Page 25: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/25.jpg)
Properties of the Fundamental Matrix F is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear F, F can be recovered up to scale using 8 points. Has rank 2.
The constraint detF=0 7 points suffices
e) trough lines ( : 12 all PPF
0'Fpp t
![Page 26: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/26.jpg)
Epipolar Plane
l’l’ ll
BaselineBaseline
PP
OO O’O’
Other derivations Hartley & Zisserman p. 223
x X’
ee e’e’
![Page 27: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/27.jpg)
HomographyEpipolar
Form
ShapeOne-to-one mapConcentric epipolar lines
D.o.f.88/5 F/E
Eqs/pnt21
Minimal configuration
45+ (8, linear)
Depth NoYes, up to scale
Scene Planar
(or no translation)
3D scene
Two-views geometry Summary:
0'Fpp tHpp '
![Page 28: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/28.jpg)
Stereo Vision
Objective: 3D reconstruction Input: 2 (or more) images taken with calibrated
cameras Output: 3D structure of scene Steps:
Rectification Matching Depth estimation
![Page 29: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/29.jpg)
Rectification
Image Reprojection reproject image planes onto
common plane parallel to baseline Notice, only focal point of camera
really matters(Seitz)
![Page 30: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/30.jpg)
Rectification
Any stereo pair can be rectified by rotating and scaling the two image planes (=homography)
We will assume images have been rectified so Image planes of cameras are parallel. Focal points are at same height. Focal lengths same.
Then, epipolar lines fall along the horizontal scan lines of the images
![Page 31: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/31.jpg)
Cyclopean Coordinates
Origin at midpoint between camera centers Axes parallel to those of the two (rectified) cameras
( / 2),
( / 2),
( ) ( ),
2 2
l l
r r
l r
l r l r
l r l r l r
f X b fYx y
Z Zf X b fY
x yZ Z
fbx x
Zb x x b y y fb
X Y Zx x x x x x
![Page 32: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/32.jpg)
Disparity
The difference is called “disparity” d is inversely related to Z: greater sensitivity to
nearby points d is directly related to b: sensitivity to small
baseline
l r
fbZ
x x
l rd x x
![Page 33: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/33.jpg)
Main Step: Correspondence Search What to match?
Objects?
More identifiable, but difficult to compute Pixels?
Easier to handle, but maybe ambiguous Edges? Collections of pixels (regions)?
![Page 34: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/34.jpg)
Random Dot Stereogram
Using random dot pairs Julesz showed that recognition is not needed for stereo
![Page 35: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/35.jpg)
Random Dot in Motion
![Page 36: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/36.jpg)
Finding Matches
![Page 37: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/37.jpg)
SSD error
disparity
1D Search More efficient Fewer false matches
![Page 38: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/38.jpg)
Ordering
![Page 39: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/39.jpg)
Ordering
![Page 40: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/40.jpg)
Comparison of Stereo Algorithms
D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,"
International Journal of Computer Vision, 47 (2002), pp. 7-42.
Ground truthScene
![Page 41: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/41.jpg)
Results with window correlation
Window-based matching(best window size)
Ground truth
![Page 42: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/42.jpg)
Scharstein and Szeliski
![Page 43: 3-D Scene u u’u’ Study the mathematical relations between corresponding image points. “Corresponding” means originated from the same 3D point. Objective](https://reader033.vdocuments.site/reader033/viewer/2022051516/56649f445503460f94c65597/html5/thumbnails/43.jpg)
Graph Cuts (next class).
Ground truthGraph cuts