cs 395/495-25: spring 2003
DESCRIPTION
CS 395/495-25: Spring 2003. IBMR: Week 8A Chapter 8... Epipolar Geometry: Two Cameras: Jack Tumblin [email protected]. IBMR-Related Seminars. Light Scattering Models For Rendering Human Hair Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept. Reminders. - PowerPoint PPT PresentationTRANSCRIPT
CS 395/495-25: Spring 2003CS 395/495-25: Spring 2003
IBMR: Week 8A IBMR: Week 8A
Chapter 8... Chapter 8... Epipolar Geometry: Epipolar Geometry:
Two Cameras: Two Cameras:
Jack TumblinJack [email protected]@cs.northwestern.edu
IBMR-Related SeminarsIBMR-Related Seminars
Light Scattering ModelsLight Scattering ModelsFor Rendering Human HairFor Rendering Human Hair
Steve Marschner, Cornell University Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS DeptFriday May 23 3:00pm, Rm 381, CS Dept..
RemindersReminders
• ProjA graded: Good Job! 90,95, 110ProjA graded: Good Job! 90,95, 110• ProjB graded: Good! minor H confusions.ProjB graded: Good! minor H confusions.• MidTerm graded: novel solutions MidTerm graded: novel solutions
encouraged.encouraged.• ProjC due Friday, May 16: many rec’d...ProjC due Friday, May 16: many rec’d...
• ProjD posted, due Friday May 30 ProjD posted, due Friday May 30 • (Last Week’s IAC notes revised...)(Last Week’s IAC notes revised...)• Take-Home Final Exam: Thurs June 5, due Take-Home Final Exam: Thurs June 5, due
June 11June 11
Camera Matrix P links PCamera Matrix P links P33PP22
• Basic cameraBasic camera: : x = Px = P0 0 X X wherewhere PP00 = [K | 0] = = [K | 0] =
• World-space camera:World-space camera: translate X to camera location C, then rotate:translate X to camera location C, then rotate:
x = PX = (Px = PX = (P00·R·T) X·R·T) X
• Rewrite as:Rewrite as:P = K[R | -RC]P = K[R | -RC]
• Redundant notation:Redundant notation:P = [M P = [M | p p44]]
M = KRM = KRpp44 = -K R C = -K R C
Output: Output: xx
2D Camera Image2D Camera Image Input:Input: XX
3D World Space3D World Space
zz
yy
xx
XX (world space)(world space)
zz cc
yy cc
xx cc
CC
xx(c
amer
a (c
amer
a sp
ace)
spac
e)
~~
~~
xxff ss ppxx 00
0 0 yyff ppyy 00
0 0 0 0 11 0 0
~~
Chapter 7: More One-Camera DetailsChapter 7: More One-Camera Details
Full 3x4 camera matrix Full 3x4 camera matrix PP maps maps PP33worldworld to to PP2 2
imageimage
? What does it do to basic 3D world shapes?? What does it do to basic 3D world shapes?
ForwardForward Projection: Projection:
• Line / Ray in Line / Ray in worldworld Line/Ray in Line/Ray in imageimage::– Ray in Ray in PP33 is: is: XX(() = A + ) = A + BB– Camera changes to Camera changes to PP22: : x(x() = PA + ) = PA + PB PB
ff CC
yycc
zzcc
xxcc
pp
PAPA
AA
BB
Chapter 7: More One-Camera DetailsChapter 7: More One-Camera Details
Full 3x4 camera matrix Full 3x4 camera matrix PP maps maps PP33worldworld to to PP2 2
imageimage
? What does it do to basic 3D world shapes?? What does it do to basic 3D world shapes?
BackwardBackward Projection: Projection:
• Line Line LL in in imageimage Plane Plane LL in in worldworld::
– Recall: Line L in Recall: Line L in PP2 2 (a 3-vector): (a 3-vector): L =L = [x[x1 1 xx22 x x33]]TT
– Plane Plane L L in in PP33 (a 4-vector): (a 4-vector):
LL = P = PTT··L = L =
ff CC
yycc
zzcc
xxcc
pp
LL
xx11
xx22
xx33
pp1111 p p2121 p p3131
pp1212 p p2222 p p3232
pp1313 p p2323 p p3333
pp1414 p p2424 p p3434
Cameras as ProtractorsCameras as Protractors
• World-Space Direction:World-Space Direction: D = [xD = [xcc, y, ycc, z, zcc, 0], 0]TT
• Direction from image point Direction from image point xx: : D = (KR)D = (KR)-1-1xx• Point Point C C and points and points xx11,x,x2 2 form angle form angle : : (pg 199)(pg 199)
(K(K-T-TKK-1-1)) = = = = ‘I‘Image ofmage of A Absolutebsolute C Coniconic’’
Line Line LL makes plane makes plane with normal with normal n = Kn = KT T L L (in camera coords)
coscos = = xx11T T (K(K-T-TKK-1-1)) x x22
(x(x11T T (K(K-T-TKK-1-1)) x x11) (x) (x22
TT (K (K-T-TKK-1-1) ) xx22))
CC
xx 22
xx 11
LL
dd 22
dd 11
nn
Cameras as ProtractorsCameras as Protractors
P (P ()) = = (K(K-T-TKK-1-1). ). OK.OK. Now what was Now what was again?again?
• Absolute Conic Absolute Conic =all =all imaginary pointsimaginary points on on
– Satisfies BOTH Satisfies BOTH xx112 2 + x+ x22
2 2 + x+ x332 2 = 0= 0 AND AND xx44
2 2 = 0= 0
– Finds right anglesFinds right angles-- if -- if DD11 DD22, then: , then: DD11TT· · ··DD22 = 0= 0
• Dual of absolute conic is Dual of absolute conic is Dual QuadricDual Quadric QQ**
= all= all imaginaryimaginary planesplanes toto ,, tangent to tangent to
Q*Q*
– Finds right anglesFinds right angles-- if -- if 11 22, then: , then: 11TT· · QQ**··22 = 0= 0
– can writecan write or or Q*Q* as the same matrix: as the same matrix:
1 0 0 01 0 0 00 1 0 00 1 0 00 0 1 00 0 1 00 0 0 00 0 0 0
xx11
xx22
xx33
00
xx1 1 x x22 x x3 3 00
Cameras as ProtractorsCameras as Protractors
Clever vanishing point trick: Clever vanishing point trick: • Perpendicular lines in image?Perpendicular lines in image?• Find their vanishing pts. by construction:Find their vanishing pts. by construction:• Use Use vv11
TT v v22 = 0 = 0, stack, solve for , stack, solve for = = (K(K-T-TKK-1-1))
vv11
vv22
vv33
Cameras as ProtractorsCameras as Protractors
P P = = (K(K-T-TKK-1-1)) = = = = ‘I‘Image ofmage of A Absolutebsolute C Coniconic’’
• Just Just as has a dual as has a dual QQ**, , has dual has dual ** : : * = * = -1-1 = K K = K KTT
• The dual conic The dual conic * * is the image of is the image of QQ** , so, so* = * = P (P (QQ**)) = = P( ) P( )
• Vanishing pointsVanishing points v v11,v,v22 of 2 of 2 world-space lines: world-space lines:vv11
TT v v22 = 0 = 0
• Vanishing linesVanishing lines L L11, L, L22 of 2 of 2 world-space planes: world-space planes:LL11
TT* L* L22 = 0 = 0
1 0 0 0 1 0 0 0 0 1 0 00 1 0 00 0 1 00 0 1 00 0 0 00 0 0 0
Movement Detection?Movement Detection?
• Can we do it from images only?Can we do it from images only?– 2D projective transforms often LOOK like 3-D; 2D projective transforms often LOOK like 3-D; – External cam. calib. affects all elements of External cam. calib. affects all elements of PP
• YES. Camera moved if-&-only-ifYES. Camera moved if-&-only-ifCamera-ray points (Camera-ray points (CCxxXX11,X,X22, etc…, etc…) will) will
map to LINE (not a point) in the other imagemap to LINE (not a point) in the other image
• ‘‘Epipolar LineEpipolar Line’ ==’ == l’ l’ = image of = image of L L
• ‘‘Parallax’ Parallax’ ==== x x11’’xx22’ ’ vectorvectorC’C’
CC
xx LL
XX11
XX22
xx22’’
xx11’’
l’l’
Epipolar Geometry: Chapter 8Epipolar Geometry: Chapter 8
Basic idea:Basic idea:
• Given Given ONLY imagesONLY images from 2 cameras from 2 cameras C, C’C, C’
• Different views of same objects Different views of same objects XX, , butbutwe don’t knowwe don’t know world-space points world-space points XX. .
• If we choose an If we choose an xx, , how can we find how can we find x’x’ ? ?
• How are How are x, x’x, x’ linked? linked?CC
C’C’
x’x’xx
XX
Epipolar Geometry: Chapter 8Epipolar Geometry: Chapter 8
Basic idea:Basic idea:• 2 cameras located at 2 cameras located at C, C’C, C’ in world space. in world space.• Find Find ‘baseline’‘baseline’ through camera centers through camera centers C, C’C, C’• Baseline hits image planes at Baseline hits image planes at ‘epipoles’‘epipoles’
Notice baseline and XNotice baseline and Xform a plane...form a plane...
Many OTHER planesMany OTHER planes thru baseline ...thru baseline ... baselinebaseline
CCC’C’
x’x’xx
XX
Epipolar Geometry: Chapter 8Epipolar Geometry: Chapter 8
Basic idea:Basic idea:• 2 cameras located at C, C’ in world space.2 cameras located at C, C’ in world space.• Find ‘baseline’ through camera centers C, C’Find ‘baseline’ through camera centers C, C’• Baseline hits image planes at ‘epipoles’Baseline hits image planes at ‘epipoles’• Family of planes thru baselineFamily of planes thru baseline
are all the are all the ‘epipolar planes’‘epipolar planes’• Image of planes = lines =Image of planes = lines =
‘epipolar lines’‘epipolar lines’• Lines intersect at Lines intersect at
epipolar pointsepipolar points in both images. in both images.CC
C’C’baselinebaseline
x’x’xx
Epipolar Geometry: Chapter 8Epipolar Geometry: Chapter 8
Summary:Summary:• Connect cameras Connect cameras C, C’C, C’ with a with a baselinebaseline, which, which
hits image planes at hits image planes at epipolesepipoles e, e’. e, e’. • Chose any world pt Chose any world pt X, X, then then everything is coplanar!everything is coplanar!
epipolar planeepipolar plane includes includes image pointsimage points x, x’, and x, x’, and thesethese connect to connect to epipolesepipoles e,e’ by e,e’ by epipolar linesepipolar lines L, L’ L, L’
baselinebaselineCC
C’C’
epipolar line L
epipolar line L epipolar
line L
’
epipolar
line L
’epipolar plane epipolar plane XX
x’x’
e’e’
xx
ee
Epipolar GeometryEpipolar Geometry
Useful properties:Useful properties:• Every image point Every image point xx maps to an epipolar line maps to an epipolar line L’L’
also:also:– Epipoles Epipoles e,e’e,e’ = each camera’s view of the other = each camera’s view of the other – All epipolar linesAll epipolar lines L’ L’ pass through epipole pass through epipole e’e’– Epipolar Line Epipolar Line L’L’ is (image of the is (image of the CCXX ray...) ray...)– Epipolar Line Epipolar Line L’L’ links (image of links (image of CC) to (image of ) to (image of XX))
baselinebaselineCC
C’C’
epipolar line L
epipolar line L epipolar
line L
’
epipolar
line L
’epipolar plane epipolar plane XX
x’x’
e’e’
xx
ee
Fundamental Matrix: Fx = L’Fundamental Matrix: Fx = L’
One Matrix Summarizes ALL of Epipolar GeometryOne Matrix Summarizes ALL of Epipolar Geometry
Fundamental Matrix Fundamental Matrix FF: 3x3, rank 2.: 3x3, rank 2.• Maps image point Maps image point xx to image point to image point x’:x’: x’ x’TT F x = 0 F x = 0
but F is only Rank 2 – given only x, F cannot find x’ for you!! but F is only Rank 2 – given only x, F cannot find x’ for you!!
• Maps image point Maps image point xx to epipolar line to epipolar line L’L’: : F x = L’F x = L’
baselinebaselineCC
C’C’
epipolar line L
epipolar line L epipolar
line L
’
epipolar
line L
’
XX
x’x’
e’e’
xx
ee
epipolar plane epipolar plane
Fundamental Matrix: Fx = L’Fundamental Matrix: Fx = L’
1) How do we find1) How do we find FF??
If we know the camera matrices If we know the camera matrices PP and and P’P’
(we almost never do), book derives: (we almost never do), book derives: (pg 224)(pg 224)
F = [e’]F = [e’] PP’ ’ PP++ ?!?!What?!?! point e’ cross product with a matrix p’p+ ?!?!?!?!What?!?! point e’ cross product with a matrix p’p+ ?!?!
baselinebaselineCC
C’C’
epipolar line L
epipolar line L epipolar
line L
’
epipolar
line L
’
XX
x’x’
e’e’
xx
epipolar plane epipolar plane
X(X(
))
(Recall: P(Recall: P+ = + = PPTT(PP(PPTT))-1-1, the pseudo-inverse), the pseudo-inverse)
Fundamental Matrix: Fx = L’Fundamental Matrix: Fx = L’
F = [e’]F = [e’] PP’ ’ PP++ But what’s this? NEW TRICK:But what’s this? NEW TRICK:
– Cross Product written as matrix multiply Cross Product written as matrix multiply (pg. 554)(pg. 554) a a b =b = = = == [a] = = == [a]··b b
– Note: Note: a a b = -b b = -b a = [a]a = [a]··b = (ab = (aTT··[b][b]))TT
baselinebaselineCC
C’C’
epipolar line L
epipolar line L epipolar
line L
’
epipolar
line L
’
XX
x’x’
e’e’
xx
epipolar plane epipolar plane
X(X(
))
aa11
aa22
aa33
bb11
bb22
bb33
aa22bb33 – a – a33bb22
aa33bb11 – a – a11bb33
aa11bb22 – a – a22bb11
0 -a0 -a33 a a22
a a33 0 -a 0 -a11
-a-a22 a a11 0 0
bb11
bb22
bb33‘‘skew symmetric’ skew symmetric’ matrixmatrix
Fundamental Matrix: Fx = L’Fundamental Matrix: Fx = L’
• 2) What is 2) What is FF if we DON’T know the cameras if we DON’T know the cameras P, P’,P, P’,but we DO know some corresp. point pairs (x, x’)?but we DO know some corresp. point pairs (x, x’)?– FF finds epipolar line finds epipolar line L’L’ from point from point x x : Fx = L’: Fx = L’– (Recall that if (any) point (Recall that if (any) point x’x’ is on line a is on line a L’L’, then , then x’x’TT L’= 0 L’= 0) ) – Substitute Substitute FxFx for L’: for L’: x’x’TTF x = 0F x = 0
AHA! we can find F using DLT-like method! AHA! we can find F using DLT-like method! see Chap. 10see Chap. 10
baselinebaselineCC
C’C’
epipolar line L
epipolar line L epipolar
line L
’
epipolar
line L
’
XX
x’x’
e’e’
xx
epipolar plane epipolar plane
X(X(
))
Fundamental Matrix SummaryFundamental Matrix Summary
FF is 3x3 matrix, maps is 3x3 matrix, maps PP22PP22, rank 2, 7-DOF, rank 2, 7-DOF
• If world space pt If world space pt XX image space pts. image space pts. x x and and x’x’ then then
x’x’TTF x = 0F x = 0
• Every image pt has epipolar line in the other image:Every image pt has epipolar line in the other image:
Fx = L’Fx = L’ FFTTx’ = Lx’ = L
• Baseline pierces image planes at epipoles e, e’ :Baseline pierces image planes at epipoles e, e’ :
Fe = 0Fe = 0 FFTTe’ = 0e’ = 0
(pg. 226)(pg. 226)
Fundamental Matrix SummaryFundamental Matrix Summary
FF is 3x3 matrix, maps is 3x3 matrix, maps PP22PP22, rank 2, 7-DOF, rank 2, 7-DOF
• Given camera matrices Given camera matrices PP, , P’P’, find , find FF matrix by: matrix by:
F = [e’]F = [e’] P’ PP’ P++ (recall: e’ is image of C: e’ = P’C)(recall: e’ is image of C: e’ = P’C)
• FF is unaffected by any proj. transforms done is unaffected by any proj. transforms done on BOTH camerason BOTH cameras
((PHPH, , P’HP’H) has same F matrix as () has same F matrix as (PP, , P’P’) for ) for anyany full-rank full-rank HH
(e.g. F measures camera C vs. Camera C’ only, (e.g. F measures camera C vs. Camera C’ only, no matter where you put them)no matter where you put them)
(pg. 226)(pg. 226)
Fundamental Matrix UsesFundamental Matrix Uses
Special case: camera translate only Special case: camera translate only (no rotations)(no rotations)
• Camera matrices are Camera matrices are P= K[ P= K[ I | 0 I | 0 ]], , P’ = K[P’ = K[I | t I | t ]]– where where KK is internal calib., is internal calib., tt is 3D translation vector is 3D translation vector
• F matrix simplifies to F matrix simplifies to F = [e’]F = [e’]• Epipolar lines are all parallel to direction Epipolar lines are all parallel to direction t t
• x,x’x,x’ displacement depends only on displacement depends only on tt & 3D depth & 3D depth zz::
x’ = x + (Kx’ = x + (Ktt)(1/z)(1/zcc))
ttxx
ttyy
ttzz
Fundamental Matrix UsesFundamental Matrix Uses
Can we find a camera matrix from motion + fundamental Can we find a camera matrix from motion + fundamental FF??
• Let one camera position define the world’s coords:Let one camera position define the world’s coords:
P = PP = P00 = K[ = K[ I | 0 I | 0 ]], , and other is and other is
P’ = [M | m] = K[R P’ = [M | m] = K[R | - -RC] RC] where where KK is internal calib., is internal calib., R R is rotation is rotation CC is position is position
• FF matrix simplifies to matrix simplifies to F = F = [[mm]]M M • If we know how we moved the camera (If we know how we moved the camera (R,C matricesR,C matrices) )
then find then find FF by correspondence and solve for K. by correspondence and solve for K. (pg 237) (pg 237)
• No R, C matrices? Use Essential Matrix No R, C matrices? Use Essential Matrix (pg 238)(pg 238)
~~
~~
Fundamental Matrix PropertiesFundamental Matrix Properties
Why bother with F?Why bother with F?• Can find it from image pt. correspondences onlyCan find it from image pt. correspondences only• Works even for mismatched cameras Works even for mismatched cameras
(example: 100-year time-lapse of Eiffel tower)(example: 100-year time-lapse of Eiffel tower)• Choose your own world-space coordinate system.Choose your own world-space coordinate system.
• SVD lets us recover PSVD lets us recover P00, P’ camera matrices from F, P’ camera matrices from F
(See 8.6 ‘The Essential matrix’(See 8.6 ‘The Essential matrix’pg 240pg 240))
• Complete 2-camera mapping from worldComplete 2-camera mapping from worldimageimage– 2 images + corresponding point pairs (x2 images + corresponding point pairs (xii,x’,x’ii))FF
– Let camera coords == 3D world coords, then Let camera coords == 3D world coords, then (x(xii,x’,x’ii))XXii
(pg. 226)(pg. 226)
Correspondence Problem:Correspondence Problem:
Where Computer Vision, IBMR part ways:Where Computer Vision, IBMR part ways:
Fundamental Matrix + Fundamental Matrix + Corresponding point pairs (x,x’) Corresponding point pairs (x,x’)
How can we How can we ‘blunt’ the correspondence problem?‘blunt’ the correspondence problem?
ENDEND