Download - Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand
![Page 1: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/1.jpg)
Computer Visioncmput 498/613
3D Modeling from images3D Modeling from images
Martin JagersandMartin Jagersand
![Page 2: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/2.jpg)
Pinhole camera• Central projectionCentral projection
• Principal point & aspectPrincipal point & aspect
10100
000
000
~
1
)/,/(),,(
Z
Y
X
f
f
Z
fY
fX
y
x
ZfYZfXZYX TT
1100
10
01
1
1
1
~
1
y
x
cp
cp
cyp
cxp
v
u
yy
xx
yy
xx c
px
py
The projection matrix:
cam
camyy
xx
IK
cp
f
cp
f
X0x
Xx
]|[
0100
00
00
![Page 3: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/3.jpg)
Projective camera
• Camera rotation and translationCamera rotation and translation
• The projection matrixThe projection matrix XtXXtX TT
camcam RRR
Xtx IKRT
PIn general:
•P is a 3x4 matrix with 11 DOF
•Finite: left 3x3 matrix non-singular
•Infinite: left 3x3 matrix singular
Properties: P=[M p4]
•Center:
•Principal ray (projection direction)
0,01
0
41
dd
Cp
C
C
MM
P
3)det( mv M
![Page 4: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/4.jpg)
• Infinite cameras where the last row of Infinite cameras where the last row of PP is is (0,0,0,1)(0,0,0,1)• Points at infinity are mapped to points at infinityPoints at infinity are mapped to points at infinity
Affine cameras
z
z
y
x
Ty
x
td
t
t
t
R
d
t
t
KP
0
0
t
k
j
i
0
j
i
td0
j
k
i{camera}
Z
X
Y{world}
x0
• ErrorError)( 0
0
xxxx
projperspaff d
Good approximation:
• small compared to d0
• point close to principal ray
X
X’
xaff
xpersp
![Page 5: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/5.jpg)
Camera calibration
• 11 DOF => at least 6 points11 DOF => at least 6 points• Linear solutionLinear solution
– Normalization requiredNormalization required– Minimizes algebraic errorMinimizes algebraic error
• Nonlinear solutionNonlinear solution– Minimize geometric error (pixel re-projection)Minimize geometric error (pixel re-projection)
• Radial distortionRadial distortion– Small near the center, increase towards peripherySmall near the center, increase towards periphery
1
0min
p
pA
ii P Xx known?known
...1 221 rKrKr
![Page 6: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/6.jpg)
Application: raysets
Gortler and al.; Microsoft Lumigraph
t
v
s
u (s,t)
(u,v)
H-Y Shum, L-W He; MicrosoftConcentric mosaics
Ck
Cl C0
Li
Lj
vi
vj i j
![Page 7: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/7.jpg)
• Projection equationProjection equation
xxii=P=PiiXX
• Resection:Resection:– xxii,X P,X Pii
Multi-view geometry - resection
Given image points and 3D points calculate camera projection matrix.
![Page 8: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/8.jpg)
• Projection equationProjection equation
xxii=P=PiiXX
• Intersection:Intersection:– xxii,P,Pi i XX
Multi-view geometry - intersection
Given image points and camera projections in at least 2 views calculate the 3D points (structure)
![Page 9: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/9.jpg)
• Projection Projection equationequation
xxii=P=PiiXX
• Structure from Structure from motion (SFM)motion (SFM)– xxii P Pii,, XX
Multi-view geometry - SFM
Given image points in at least 2 views calculate the 3D points (structure) and camera projection matrices (motion)
•Estimate projective structure
•Rectify the reconstruction to metric (autocalibration)
![Page 10: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/10.jpg)
Depth from stereo
•Calibrated aligned camerasCalibrated aligned cameras
Z
X(0,0) (d,0)
Z=f
xl xrrl
rl
xx
dfZ
dXx
fX
x
fZ
)(
Disparity d
Trinocular Vision System
(Point Grey Research)
![Page 11: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/11.jpg)
Application: depth based reprojection
3D warping, 3D warping, McMillanMcMillan
Plenoptic modeling, Plenoptic modeling, McMillan & BishopMcMillan & Bishop
![Page 12: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/12.jpg)
Application: depth based reprojection
Layer depth images,Layer depth images, Shade et al. Shade et al.
Image based objects,Image based objects, Oliveira & Bishop Oliveira & Bishop
![Page 13: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/13.jpg)
Affine camera factorization3D structure from many images
The affine projection equations are The affine projection equations are
1
j
j
j
yi
xi
ij
ij
Z
Y
X
P
Py
x
10001
1
j
j
j
yi
xi
ij
ij
Z
Y
X
P
P
y
x
~
~
4
4
j
j
j
yi
xi
ij
ij
yiij
xiij
Z
Y
X
P
Py
x
Py
Px
![Page 14: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/14.jpg)
Orthographic factorization
The ortographic projection equations are The ortographic projection equations are
where where njmijiij ,...,1,,...,1,Mm P
All equations can be collected for all All equations can be collected for all ii and and jj
wherewhere
n
mmnmm
n
n
M,...,M,M,,
mmm
mmm
mmm
212
1
21
22221
11211
M
P
P
P
Pm
MPm
M ~
~m
j
j
j
jyi
xi
iij
ijij
Z
Y
X
,P
P,
y
xP
Note that P and M are resp. 2mx3 and 3xn matrices and
therefore the rank of m is at most 3
(Tomasi Kanade’92)
![Page 15: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/15.jpg)
Orthographic factorization
Factorize Factorize mm through singular value decomposition through singular value decomposition
An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows
TVUm
TVMUP ~,
~
(Tomasi Kanade’92)
n
mmnmm
n
n
M,...,M,M
mmm
mmmmmm
min 212
1
21
22221
11211
P
P
P
Closest rank-3 approximation yields MLE!
![Page 16: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/16.jpg)
0~~
1~~
1~~
TT1
TT1
TT1
yi
xi
yi
yi
xi
xi
PP
PP
PP
0~~
1~~
1~~
T
T
T
yi
xi
yi
yi
xi
xi
PP
PP
PP
C
C
C
A metric reconstruction is obtained as followsA metric reconstruction is obtained as follows
Where A is computed from Where A is computed from
Orthographic factorization
Factorize Factorize mm through singular value decomposition through singular value decomposition
An affine reconstruction is obtained as followsAn affine reconstruction is obtained as follows
TVUm
TVMUP ~,
~
MQMQPP~
,~ 1
0
1
1
T
T
T
yi
xi
yi
yi
xi
xi
PP
PP
PP 3 linear equations per view on
symmetric matrix C (6DOF)
Q can be obtained from C through Cholesky factorisation
and inversion
(Tomasi Kanade’92)
![Page 17: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/17.jpg)
N-view geometryAffine factorization
[Tomasi &Kanade ’92][Tomasi &Kanade ’92]
•Affine camera Affine camera
•ProjectionProjection
•nn points, points, mm views: measurement matrix views: measurement matrix
]|[ tMP M 2x3 matrix; t 2D vector
t
Z
Y
X
My
x
nmm
nm
n
M
M
W XX
xx
xx
1
1
1
111
~~
~~
W: Rank 3XMVDUW
UDVWT
nm
T
ˆˆˆ33332
Assuming isotropic zero-mean Gaussian noise, factorization achieves ML affine reconstruction.
txx ~
![Page 18: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/18.jpg)
Weak perspective factorization
[D. Weinshall][D. Weinshall]
•Weak perspective cameraWeak perspective camera
•Affine ambiguityAffine ambiguity
•Metric constraintsMetric constraints
j
i
s
sM
)ˆ)(ˆ(ˆˆˆ 11 XQQMXQQMW
0ˆˆ
ˆˆˆˆ 2
ji
jjii
sQQs
ssQQssQQsTT
TTTT
Extract motion parameters Eliminate scale Compute direction of camera axis k = i x j parameterize rotation with Euler angles
![Page 19: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/19.jpg)
Full perspective factorization
The camera equations The camera equations
for a fixed image for a fixed image ii can be written in matrix form can be written in matrix form asas
where where
njmijiijij ,...,1,,...,1,Mmλ P
MPm iii
imiii
mimiii mmm
λ,...,λ,λdiag
M,...,M,M,,...,,
21
2121
Mm
![Page 20: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/20.jpg)
Perspective factorization
All equations can be collected for all All equations can be collected for all ii as as
wherewherePMm
mnn P
P
P
P
m
m
m
m...
,...
2
1
22
11
In these formulas In these formulas mm are known, but are known, but ii,,PP and and MM are unknown are unknown
Observe that Observe that PMPM is a product of a 3 is a product of a 3mmx4 matrix and a 4xx4 matrix and a 4xnn matrix, i.e. it is a rank 4 matrixmatrix, i.e. it is a rank 4 matrix
![Page 21: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/21.jpg)
Perspective factorization algorithm
Assume that i are known, then PM is known.
Use the singular value decomposition PM=U VT
In the noise-free case
S=diag(1,2,3,4,0, … ,0)and a reconstruction can be obtained by setting:
P=the first four columns of U.M=the first four rows of V.
![Page 22: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/22.jpg)
Iterative perspective factorization
When i are unknown the following algorithm can be used:
1. Set ij=1 (affine approximation).
2. Factorize PM and obtain an estimate of P and M. If 5 is sufficiently small then STOP.
3. Use m, P and M to estimate i from the camera equations (linearly) mi i=PiM
4. Goto 2.
In general the algorithm minimizes the proximity measure P(,P,M)=5Note that structure and motion recovered
up to an arbitrary projective transformation
![Page 23: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/23.jpg)
Projective factorizationsummary
[Sturm & Triggs’96][ Heyden ‘97 ][Sturm & Triggs’96][ Heyden ‘97 ]
•Measurement matrixMeasurement matrix
•Known projective depthKnown projective depth
– Projective ambiguity Projective ambiguity
• Iterative algorithmIterative algorithm– Reconstruct withReconstruct with– Reestimate depth and iterate Reestimate depth and iterate
nmm
nmn
mm
nn
P
P
W XX
xx
xx
1
1
11
1111
11
3mxn matrix
Rank 4
ij
XPVDUW
UDVWT
nm
T
ˆˆˆ44442
1ij
ij
![Page 24: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/24.jpg)
Further Factorization work
Factorization with uncertaintyFactorization with uncertainty
Factorization for dynamic scenesFactorization for dynamic scenes
(Irani & Anandan, IJCV’02)
(Costeira and Kanade ‘94)
(Bregler et al. 2000, Brand 2001)
![Page 25: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/25.jpg)
Sequential 3D structure from motionusing 2 and 3 view geom
• Initialize structure and motion from two viewsInitialize structure and motion from two views• For each additional viewFor each additional view
– Determine poseDetermine pose– Refine and extend structureRefine and extend structure
• Determine correspondences robustly by jointly Determine correspondences robustly by jointly estimating matches and epipolar geometry estimating matches and epipolar geometry
Images
![Page 26: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/26.jpg)
2 view geometry (Epipolar geometry)
![Page 27: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/27.jpg)
Fundamental matrix
• Algebraic representation of epipolar geometryAlgebraic representation of epipolar geometry
Step 1: X on a plane
Step 2: epipolar line l’xxe
xexel
xx
FH
H
]'[
']'['''
'
0' xx FT
F
•3x3, Rank 2, det(F)=0
•Linear sol. – 8 corr. Points (unique)
•Nonlinear sol. – 7 corr. points (3sol.)
•Very sensitive to noise & outliers
[Faugeras ’92, Hartley ’92 ]
Epipolar lines:
Epipoles:
Projection matrices:
'|']'['
]|[
0'0
''
evee
0
ee
xlxl
T
T
T
FP
IP
FF
FF
![Page 28: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/28.jpg)
3,4,N view geometry
• Trifocal tensor (3 view geometry)Trifocal tensor (3 view geometry)
],,[: 321 TTTT 3x3x3 tensor;
27 params. (18 indep.)
0]"[)(]'[
"]3,2,1['
xxx
lll
ii
i
T
T
TTT lines
points
• Quadrifocal tensor (4 view geometry) Quadrifocal tensor (4 view geometry) [Triggs ’95]
•Multiview tensors Multiview tensors [Hartley’95][ Hayden ‘98]
There is no additional constraint between more than 4 images. All the constraints There is no additional constraint between more than 4 images. All the constraints can be expressed using F,triliear tensor or quadrifocal tensor. can be expressed using F,triliear tensor or quadrifocal tensor.
[Hartley ’97][Torr & Zisserman ’97][ Faugeras ’97]
![Page 29: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/29.jpg)
Initial structure and motion
eeaFeP
0IPT
x2
1
Epipolar geometry Projective calibration
012 FmmT
compatible with F
Yields correct projective camera setup(Faugeras´92,Hartley´92)
Obtain structure through triangulation
Use reprojection error for minimizationAvoid measurements in projective space
![Page 30: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/30.jpg)
Compute Pi+1 using robust approachFind additional matches using predicted projectionExtend, correct and refine reconstruction
2D-2D
2D-3D 2D-3D
mimi+1
M
new view
Determine pose towards existing structure
![Page 31: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/31.jpg)
Non-sequential image collections
4.8im/pt64 images
3792
po
ints
Problem:Features are lost and reinitialized as new features
Solution:Match with other close views
![Page 32: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/32.jpg)
For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure
Relating to more views
Problem: find close views in projective frame
For every view For every view iiExtract featuresExtract featuresCompute two view geometry Compute two view geometry ii-1/-1/ii and matches and matches Compute pose using robust algorithmCompute pose using robust algorithmFor all For all closeclose views views kk
Compute two view geometry Compute two view geometry kk//ii and matches and matchesInfer new 2D-3D matches and add to listInfer new 2D-3D matches and add to list
Refine pose using all 2D-3D matchesRefine pose using all 2D-3D matchesRefine existing structureRefine existing structureInitialize new structureInitialize new structure
![Page 33: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/33.jpg)
Determining close views
• If viewpoints are If viewpoints are closeclose then most image changes then most image changes can be modelled through a can be modelled through a planar homographyplanar homography
• Qualitative distance measureQualitative distance measure is obtained by is obtained by looking at the looking at the residual errorresidual error on the on the best possible best possible planar homographyplanar homography
Distance = m´,mmedian min HD
![Page 34: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/34.jpg)
9.8im/pt
4.8im/pt
64 images
64 images
3792
po
ints
2170
po
ints
Non-sequential image collections (2)
![Page 35: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/35.jpg)
Refining structure and motion
• Minimize reprojection errorMinimize reprojection error
– Maximum Likelyhood Estimation Maximum Likelyhood Estimation (if (if error zero-mean Gaussian noise)error zero-mean Gaussian noise)
– Huge problem but can be solved efficientlyHuge problem but can be solved efficiently(Bundle adjustment)(Bundle adjustment)
m
k
n
iikD
ik 1 1
2
kiM̂,P̂
M̂P̂,mmin
![Page 36: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/36.jpg)
Refining a captured model:Bundle adjustment
• Refine structure XRefine structure Xjj and motion P and motion Pii
• Minimize geometric errorMinimize geometric error• ML solution, assuming noise is GaussianML solution, assuming noise is Gaussian• Tolerant to missing dataTolerant to missing data
ji
ijj
iPd,
2),ˆˆ(min xX
![Page 37: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/37.jpg)
Projective ambiguity andself-calibration
Given an uncalibrated image sequence with corresponding point it is possible to reconstruct the object up to an unknown projective transformation
• Autocalibration (self-calibration): Determine a projective transformation T that upgrades the projective reconstruction to a metric one.
T
´M´M))((Mm 1 PTPTP
![Page 38: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/38.jpg)
Remember:Stratification of geometry
15 DOF 12 DOFplane at infinity
parallelism
More general
More structure
Projective Affine Metric
7 DOFabsolute conicangles, rel.dist.
![Page 39: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/39.jpg)
Constraints ?
• Scene constraintsScene constraints– Parallellism, vanishing points, horizon, ...Parallellism, vanishing points, horizon, ...– Distances, positions, angles, ...Distances, positions, angles, ...
Unknown scene no constraints
• Camera extrinsics constraints–Pose, orientation, ...
Unknown camera motion no constraints • Camera intrinsics constraints
–Focal length, principal point, aspect ratio & skew
Perspective camera model too general some constraints
![Page 40: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/40.jpg)
Euclidean projection matrix
tRRKP TT
1yy
xx
uf
usf
K
Factorization of Euclidean projection matrix
Intrinsics:
Extrinsics: t,R
Note: every projection matrix can be factorized,
but only meaningful for euclidean projection matrices
(camera geometry)
(camera motion)
![Page 41: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/41.jpg)
Constraints on intrinsic parameters
Constant Constant e.g. fixed camera:e.g. fixed camera:
KnownKnowne.g. rectangular pixels:e.g. rectangular pixels:
square pixels:square pixels: principal point known:principal point known:
21 KK
0s
1yy
xx
uf
usf
K
0, sff yx
2,
2,
hwuu yx
![Page 42: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/42.jpg)
Self-calibration
Upgrade from Upgrade from projectiveprojective structure to structure to metricmetric structure using structure using constraintsconstraints onon intrinsicintrinsic camera parameters camera parameters– Constant intrinsicsConstant intrinsics
– Some known intrinsics, others varyingSome known intrinsics, others varying
– Constraints on intrincs and restricted motionConstraints on intrincs and restricted motion(e.g. pure translation, pure rotation, planar motion)(e.g. pure translation, pure rotation, planar motion)
(Faugeras et al. ECCV´92, Hartley´93,
Triggs´97, Pollefeys et al. PAMI´98, ...)
(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)
(Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)
![Page 43: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/43.jpg)
A counting argument
• To go from projective (15DOF) to metric (7DOF) at least 8 To go from projective (15DOF) to metric (7DOF) at least 8 constraints are neededconstraints are needed
• Minimal sequence length should satisfyMinimal sequence length should satisfy
• Independent of algorithmIndependent of algorithm• Assumes general motion (i.e. not critical)Assumes general motion (i.e. not critical)
8#1# fixednknownn
![Page 44: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/44.jpg)
Conics•Conic: Conic:
– Euclidean geometry: hyperbola, ellipse, parabola & degenerateEuclidean geometry: hyperbola, ellipse, parabola & degenerate– Projective geometry: equivalent under projective transformProjective geometry: equivalent under projective transform– Defined by 5 pointsDefined by 5 points
•TangentTangent
•Dual conic C*Dual conic C*
0
022
xx C
feydxcybxyaxT
fed
ecb
dba
C
2/2/
2/2/
2/2/
xl C
0* ll CT
![Page 45: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/45.jpg)
Quadrics
Quadrics: Quadrics: QQ 4x4 symmetric matrix 4x4 symmetric matrix
9 DOF (defined by 9 points in general pose)9 DOF (defined by 9 points in general pose)
•Dual: Dual: Q*Q*Planes tangent to the quadricPlanes tangent to the quadric
0XX QT
0* ππ QT
![Page 46: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/46.jpg)
Summary:Conics & Quadrics
0mmT C 0ll *T C
1* CC
conics
0MMT Q 0*T Q
1* QQ
quadrics
1T´~ CHHCC
T*** ´~ HHCCC T´ TTQ~QQ ***
1T´~ QTTQQ
transformations
T** ~ PPQC
projection
![Page 47: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/47.jpg)
The absolute conic
• Absolute conic is a imaginary circle onAbsolute conic is a imaginary circle on• The absolute dual quadric (rim quadric)The absolute dual quadric (rim quadric)
• In a metric frameIn a metric frame
π
0),,(),,(
0
)1,0,0,0(
321321
4
23
22
21
TxxxIxxx
x
xxx
π
*
0
0*
*
T
T
I
0
0
Note: is the nullspace of
*
On : π
*
Fixed under similarity transf.
![Page 48: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/48.jpg)
Self-calibration
•Theoretically formulated by Theoretically formulated by [Faugeras ’92][Faugeras ’92]
•2 basic approaches2 basic approaches– Stratified: recover Stratified: recover – Direct: recover Direct: recover [Triggs’97][Triggs’97]
•Constraints:Constraints:– Camera internal constraintsCamera internal constraints
– Constant parameters Constant parameters [Hartley’94][ Mohr’93][Hartley’94][ Mohr’93]
– Known skew and aspect ratioKnown skew and aspect ratio [Hayden&[Hayden&ÅÅstrströöm’98][Pollefeys’98]m’98][Pollefeys’98]
– Scene constraints (angles, ratios of length)Scene constraints (angles, ratios of length)
•Choice of H:Choice of H: Knowing camera Knowing camera KK and and
π
*
TTT K
KH )1,(,
1p
p
0
π
![Page 49: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/49.jpg)
Absolute Dual Quadric and Self-calibration
Eliminate extrinsics from equationEliminate extrinsics from equation
Equivalent to projection of dual quadric
))(Ω)((Ω *1* TTTTT PTTTPTPPKK
Abs.Dual Quadric also exists in projective world
T´Ω´´ * PP Transforming world so thatreduces ambiguity to metric
** ΩΩ´
![Page 50: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/50.jpg)
*
*
projection
constraints
Absolute conic = calibration object which is always present but can only be observed through constraints on the intrinsics
Tii
Tiii Ωω KKPP
Absolute Dual Quadric and Self-calibration
Projection equation:Projection equation:
Translate constraints on Translate constraints on K K through projection equationthrough projection equation to to constraints on constraints on **
![Page 51: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/51.jpg)
KRd0d]C
~|KR[IPXx
mapping between ∞ to an image is given by the planar homogaphy x=Hd, with H=KR
image of the absolute conic (IAC) = I
1-T-1T KKKKω 1TCHHC
(i) IAC depends only on intrinsics(ii) angle between two rays(iii) DIAC=*=KKT
(iv) K (cholesky factorisation)(v) image of circular points
2T
21T
1
2T
1
ωxxωxx
ωxxcos
Image of the absolute conic
HZ 7.5.1:
![Page 52: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/52.jpg)
Constraints on *
1
ω 22
222
*
yx
yyyyxy
xyxyxx
cc
ccfccsf
cccsfcsf
Zero skewZero skew quadraticquadratic mm
Principal pointPrincipal point linearlinear 22mm
Zero skew (& p.p.)Zero skew (& p.p.) linearlinear mm
Fixed aspect ratio (& Fixed aspect ratio (& p.p.& Skew)p.p.& Skew)
quadraticquadratic m-1m-1
Known aspect ratio Known aspect ratio (& p.p.& Skew)(& p.p.& Skew)
linearlinear mm
Focal length Focal length
(& p.p. & Skew)(& p.p. & Skew)
linearlinear mm
*23
*13
*33
*12 ωωωω
0ωω *23
*13
0ω*12
*11
*22
*22
*11 ω'ωω'ω
*22
*11 ωω
*11
*33 ωω
condition constraint type #constraints
![Page 53: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/53.jpg)
Summary: Self calibration based on the IADC
•Calibrated cameraCalibrated camera– Dual absolute quadric (DAC)Dual absolute quadric (DAC)– Dual image of the absolute conic (DIAC)Dual image of the absolute conic (DIAC)
•Projective cameraProjective camera– DACDAC– DIACDIAC
•AutocalibrationAutocalibration– Determine based on constraints onDetermine based on constraints on– Decompose Decompose
TKK
diagI
*
)0,1,1,1(~
Tii
iTii
T
KKPQP
HIHQ
**
* ~
* i*THIHQ
~*
i
j
Ci
Cj
![Page 54: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/54.jpg)
Illustration of self-calibration
Projective Affine Metric
![Page 55: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/55.jpg)
Degenerate configurations
• Pure translation: Pure translation: affine transformation (5 DOF)affine transformation (5 DOF)
• Pure rotation: Pure rotation: arbitrary pose for (3 DOF) arbitrary pose for (3 DOF)
• Planar motion: Planar motion: scaling axis perpendicular to plane scaling axis perpendicular to plane (1DOF)(1DOF)
• Orbital motion: Orbital motion: projective distortion along rotation axis projective distortion along rotation axis (2DOF) (2DOF)
π
Not unique solution !Not unique solution !
![Page 56: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/56.jpg)
A complete modeling systemprojective
Sequence of frames scene structureSequence of frames scene structure
1.1. Get corresponding points (tracking).Get corresponding points (tracking).
2.2. 2,3 view geometry: 2,3 view geometry: compute F,T between consecutive frames compute F,T between consecutive frames (recompute correspondences).(recompute correspondences).
3.3. Initial reconstruction: Initial reconstruction: get an initial structure from a get an initial structure from a subsequence with big baseline (trilinear tensor, factorization …) subsequence with big baseline (trilinear tensor, factorization …) and bind more frames/points using resection/intersection.and bind more frames/points using resection/intersection.
4.4. Self-calibration.Self-calibration.
5.5. Bundle adjustment.Bundle adjustment.
![Page 57: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/57.jpg)
A complete modeling systemaffine
Sequence of frames scene structureSequence of frames scene structure
1.1. Get corresponding points (tracking).Get corresponding points (tracking).
2.2. Affine factorization. (This already computes ML Affine factorization. (This already computes ML estimate over all frames so no need for bundle estimate over all frames so no need for bundle adjustment for simple scenes.adjustment for simple scenes.
3.3. Self-calibration.Self-calibration.
4.4. If several model segments: Merge, bundle adjust.If several model segments: Merge, bundle adjust.
![Page 58: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/58.jpg)
Examples – modeling with dynamic texture
Cobzas,Yerex,JagersandCobzas,Yerex,Jagersand
![Page 59: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/59.jpg)
Examples: geometric modeling
Debevec and Taylor:Debevec and Taylor: Façade Façade
![Page 60: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/60.jpg)
Examples: geometric modeling
Pollefeys: Pollefeys: Arenberg CastleArenberg Castle
![Page 61: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/61.jpg)
Examples: geometric modeling
INRIA –INRIA –VISIRE projectVISIRE project
![Page 62: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/62.jpg)
Examples: geometric modeling
CIP Prague –CIP Prague –Projective Reconstruction Based on Cake ConfigurationProjective Reconstruction Based on Cake Configuration
![Page 63: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/63.jpg)
Reading: FP Chapter 11.
• The Stereopsis Problem: Fusion and Reconstruction• Human Stereopsis and Random Dot Stereograms• Cooperative Algorithms• Correlation-Based Fusion• Multi-Scale Edge Matching• Dynamic Programming• Using Three or More Cameras
Dense stereo
•Go back to original images, do dense matching.Go back to original images, do dense matching.
•Try to get dense depth mapsTry to get dense depth maps
![Page 64: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/64.jpg)
Stereo image rectification
![Page 65: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/65.jpg)
Stereo image rectification
• reproject image planes onto a commonreproject image planes onto a common• plane parallel to the line between optical centersplane parallel to the line between optical centers• pixel motion is horizontal after this transformationpixel motion is horizontal after this transformation• two homographies (3x3 transform), one for each input two homographies (3x3 transform), one for each input
image reprojectionimage reprojectionC. Loop and Z. Zhang. C. Loop and Z. Zhang. Computing RectifyingComputing Rectifying Homographies Homographies
for Stereo Vision for Stereo Vision. IEEE Conf. Computer Vision and Pattern . IEEE Conf. Computer Vision and Pattern Recognition, 1999Recognition, 1999..
![Page 66: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/66.jpg)
All epipolar lines are parallel in the rectified image plane.
Rectification
![Page 67: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/67.jpg)
simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
He
0
0
1
Image rectification throughhomography warp
![Page 68: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/68.jpg)
ExampleUnrectified
Rectified
![Page 69: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/69.jpg)
Depth from disparity
f
x x’
baseline
z
C C’
X
f
![Page 70: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/70.jpg)
Stereo matching algorithms•Match Pixels in Conjugate Epipolar LinesMatch Pixels in Conjugate Epipolar Lines
– Assume brightness constancyAssume brightness constancy– This is a tough problemThis is a tough problem– Numerous approachesNumerous approaches
– A good survey and evaluation: A good survey and evaluation: http://www.http://www.middleburymiddlebury..eduedu/stereo//stereo/
![Page 71: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/71.jpg)
Your basic stereo algorithm
For each epipolar line
For each pixel in the left image• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
Improvement: match windows• This should look familar...
![Page 72: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/72.jpg)
Stereo as energy minimization
• Find disparities d that minimize an energy function
• Simple pixel / window matching
SSD distance between windows I(x, y) and J(x, y + d(x,y))=
![Page 73: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/73.jpg)
Stereo as energy minimization
I(x, y) J(x, y)
y = 141
C(x, y, d); the disparity space image (DSI)x
d
![Page 74: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/74.jpg)
Stereo as energy minimization
y = 141
x
d
Simple pixel / window matching: choose the minimum of each column in the DSI independently:
![Page 75: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/75.jpg)
Matching windowsSimilarity MeasureSimilarity Measure FormulaFormula
Sum of Absolute Differences Sum of Absolute Differences (SAD)(SAD)
Sum of Squared Differences (SSD)Sum of Squared Differences (SSD)
Zero-mean SADZero-mean SAD
Locally scaled SADLocally scaled SAD
Normalized Cross Correlation Normalized Cross Correlation (NCC)(NCC)
http://siddhantahuja.wordpress.com/category/stereo-vision/ SAD SSD NCC Ground truth
![Page 76: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/76.jpg)
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
Trade-off
• Matching cost (data)
• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 77: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/77.jpg)
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
![Page 78: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/78.jpg)
Hierarchical stereo matchingD
ow
nsam
plin
g
(Gau
ssia
n p
yra
mid
)
Dis
pari
ty p
rop
ag
ati
on
Allows faster computation
Deals with large disparity ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 79: Computer Vision cmput 498/613 3D Modeling from images 3D Modeling from images Martin Jagersand](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649f3e5503460f94c5f2b0/html5/thumbnails/79.jpg)
Example: reconstruct image from neighboring images