a dvanced c omputer v ision feature-based alignment
DESCRIPTION
A dvanced C omputer V ision Feature-based Alignment. Lecturer: Lu Yi & Prof. Fuh l ynn. [email protected] CSIE NTU. Content. Pre-requisites: mainly about camera model Alignment Algorithm RANdom Sample Consensus (RANSAC ) Pose Estimation. Sources. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/1.jpg)
Advanced Computer Vision
Feature-based AlignmentLecturer: Lu Yi & Prof. Fuh
![Page 2: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/2.jpg)
Content• Pre-requisites: mainly about camera model• Alignment Algorithm• RANdom Sample Consensus (RANSAC)• Pose Estimation
![Page 3: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/3.jpg)
Sources• EGGN 512 – Computer Vision course by Prof. Willliam Hoff• http://inside.mines.edu/~whoff/courses/EENG512/• Playlist@Youtube:• http://www.youtube.com/playlist?list=PL4B3F8D4A5CAD8DA3
• Digital Visual Effects course by Prof. Yung-Yu Chuang• http://www.csie.ntu.edu.tw/~cyy/courses/vfx/14spring/news/
• Both are highly recommended
![Page 4: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/4.jpg)
Taught Way• Cover the main parts of textbook chapter 6• Lecture mostly depended on EGGN 512 course videos
![Page 5: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/5.jpg)
Frames of Reference• Image frames are 2D; other are 3D
• The “pose” (position and orientation) of a 3D rigid body has 6 degrees of freedom
![Page 6: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/6.jpg)
Pre-requisites• Image Formation• Geometric Transformations (2D-2D, 3D-3D, 3D-2D)• Camera Model (Camera Matrix = Intrinsic + Extrinsic camera matrix)
• All related contents are in textbook chapter 2: Image formation
![Page 7: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/7.jpg)
Pre-requisites• Review1. Projective Transformation2. Camera Matrix• EGGN 512 – Lecture 5-1 3D-2D Transforms• http://
www.youtube.com/watch?v=DFNjOUMuecU&index=11&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 5-2 3D-2D Transforms• http://
www.youtube.com/watch?v=5gesrLgNuQo&list=PL4B3F8D4A5CAD8DA3
![Page 8: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/8.jpg)
Alignment Algorithm• EGGN 512 – Lecture 14-1 Alignment• http://www.youtube.com/watch?v=UcU4814hvR8&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 14-2 Alignment• http://
www.youtube.com/watch?v=XxEKMecNZk0&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 14-3 Alignment• http://www.youtube.com/watch?v=lSXfv4baMwk&list=PL4B3F8D4A5CAD8DA3• Slides • http://inside.mines.edu/~
whoff/courses/EENG512/lectures/15-AlignmentNonlinear.pdf
![Page 9: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/9.jpg)
Alignment Algorithm• Feature-based methods: only use feature points to estimate
parameters• Features: SIFT, SURF, Harris .etc.
![Page 10: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/10.jpg)
Determine pairwise alignment• p’=Mp, where M is a transformation matrix, p and p’ are feature
matches• It is possible to use more complicated models such as affine or
perspective• For example, assume M is a 2x2 matrix
• Find M with the least square error
yx
mmmm
yx
2221
1211
''
n
i
pMp1
2'
![Page 11: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/11.jpg)
Determine pairwise alignment
• Overdetermined system
yx
mmmm
yx
2221
1211
''
'1221211
'1121111
ymymx
xmymx
'
'
'2
'1
'1
22
21
12
11
22
11
11
0000
0000
00
n
n
nn
nn
yx
xyx
mmmm
yxyx
yxyx
yx
![Page 12: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/12.jpg)
Normal Equation• Given an overdetermined system
• the normal equation is that which minimizes the sum of the square differences between left and right sides
• Why?
bAx
bAAxA TT
![Page 13: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/13.jpg)
Normal Equation 2)( bAxx E
n
m
nmn
m
b
b
x
x
aa
aa
:::
:
...::::::
... 1
1
1
111
nxm, n equations, m variables
![Page 14: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/14.jpg)
Normal Equation
n
m
jjnj
i
m
jjij
m
jjj
n
i
m
jjnj
m
jjij
m
jjj
bxa
bxa
bxa
b
b
b
xa
xa
xa
1
1
11
11
1
1
11
:
:
:
:
:
:
bAx
n
ii
m
jjij bxaE
1
2
1
2)( bAxx
![Page 15: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/15.jpg)
Normal Equation
n
ii
m
jjij bxaE
1
2
1
2)( bAxx
1
0xE
n
iii
n
ij
m
jiji baxaa
11
1 11 22
11 1
2 i
n
ii
m
jjij abxa
)(20 bAAxAx
TT
E bAAxA TT →
![Page 16: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/16.jpg)
Normal Equation 2bAx
![Page 17: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/17.jpg)
Normal Equation
bbxbAxbAAxAx
bbbAxAxbAxAxbAxbAxbAxbAx
bAxbAx
bAx
TTTTTTT
TTTTTT
TTT
TT
T
2
bAAxAx
TT 22 E
![Page 18: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/18.jpg)
Determine pairwise alignment• p’=Mp, where M is a transformation matrix, p and p’ are feature
matches• For translation model, it is easier.
• What if the match is false? Avoid impact of outliers. (Ans: RANSAC)
n
iiiii yymxxmE
1
2'2
2'1
1
0mE
![Page 19: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/19.jpg)
Application: Image Stitching• Stitching = alignment + blending
geometricalregistration
photometricregistration
![Page 20: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/20.jpg)
Application: Image Stitching• Panorama – the whole picture
• Compact Camera FOV = 50 x 35°
![Page 21: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/21.jpg)
Application: Image Stitching• Panorama – the whole picture
• Compact Camera FOV = 50 x 35°• Human FOV = 200 x 135°
![Page 22: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/22.jpg)
Application: Image Stitching• Panorama – the whole picture
• Compact Camera FOV = 50 x 35°• Human FOV = 200 x 135°• Panoramic Mosaic = 360 x 180°
![Page 23: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/23.jpg)
Application: Image Stitching• Example:• http://graphics.stanford.edu/courses/cs178/applets/projection.html
![Page 24: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/24.jpg)
Application: Face Alignment• Face alignment is the key preprocessing step for face recognition
![Page 25: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/25.jpg)
Application: Face Alignment• Examples from LFW dataset:
![Page 26: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/26.jpg)
RANSAC Algorithm• RANSAC = RANdom Sample Consensus• An iterative method to estimate parameters of a mathematical model
from a set of observed data which contains outliers [Wiki]• Compare to robust statistics• Given N data points xi, assume that majority of them are generated from a
model with parameters , try to recover .
• Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381-395.
![Page 27: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/27.jpg)
RANSAC AlgorithmRun k times: (1) draw n samples randomly (2) fit parameters with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, cOutput with the largest c
How many times?How big? Smaller is better
How to define?Depends on the problem.
![Page 28: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/28.jpg)
RANSAC Algorithm (How to determine K)
p: probability of real inliersP: probability of success after k trials
knpP )1(1 n samples are all inliers
a failurefailure after k trials
)1log()1log(
npPk
n p k
3 0.5 356 0.6 976 0.5 293
for P=0.99
![Page 29: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/29.jpg)
RANSAC Algorithm (Example: line fitting)
![Page 30: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/30.jpg)
RANSAC Algorithm (Example: line fitting)
n=2
![Page 31: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/31.jpg)
RANSAC Algorithm (Example: line fitting)
![Page 32: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/32.jpg)
RANSAC Algorithm (Example: line fitting)
![Page 33: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/33.jpg)
RANSAC Algorithm (Example: line fitting)
c=3
![Page 34: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/34.jpg)
RANSAC Algorithm (Example: line fitting)
c=3
![Page 35: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/35.jpg)
RANSAC Algorithm (Example: line fitting)
c=15
![Page 36: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/36.jpg)
RANSAC Algorithm• EGGN 512 – Lecture 27-1 RANSAC• http://www.youtube.com/watch?v=NKxXGsZdDp8&list=PL4B3F8D4A5
CAD8DA3&index=54
![Page 37: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/37.jpg)
RANSAC Algorithm (Example: Homography)
![Page 38: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/38.jpg)
RANSAC Algorithm (Example: Homography)
![Page 39: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/39.jpg)
RANSAC Algorithm (Example: Homography)
![Page 40: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/40.jpg)
Pose Estimation• EGGN 512 – Lecture 16-1 Pose Estimation• http://
www.youtube.com/watch?v=kq3c6QpcAGc&list=PL4B3F8D4A5CAD8DA3
• EGGN 512 – Lecture 17-1 Pose from Lines• http://
www.youtube.com/watch?v=D_4eUoqgWdc&list=PL4B3F8D4A5CAD8DA3
![Page 41: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/41.jpg)
Pose Estimation• EGGN 512 – Lecture 18-1 SVD• http://
www.youtube.com/watch?v=C852P1JrHXI&list=PL4B3F8D4A5CAD8DA3
• EGGN 512 – Lecture 18-2 SVD• http://
www.youtube.com/watch?v=aIkzK4CdYes&list=PL4B3F8D4A5CAD8DA3
![Page 42: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/42.jpg)
Pose Estimation• EGGN 512 – Lecture 19-1 Linear Pose Estimation• http://
www.youtube.com/watch?v=HojSSrxsB4Q&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 19-2 Linear Pose Estimation• http://
www.youtube.com/watch?v=ik8dFybnyPY&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 19-3 Linear Pose Estimation• http://
www.youtube.com/watch?v=OS5b-3Xfn1M&list=PL4B3F8D4A5CAD8DA3
![Page 43: A dvanced C omputer V ision Feature-based Alignment](https://reader036.vdocuments.site/reader036/viewer/2022062410/56816178550346895dd105a7/html5/thumbnails/43.jpg)
END