lucas-kanade optical flo16385/s15/lectures/lecture21.pdf · to obtain the least squares solution...
TRANSCRIPT
![Page 1: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/1.jpg)
Lucas-Kanade Optical FlowComputer Vision 16-385
Carnegie Mellon University (Kris Kitani)
![Page 2: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/2.jpg)
spatial derivative optical flow temporal derivative
Ix
u+ Iy
v + It
= 0
I
x
=@I
@x
Iy =@I
@y u =dx
dt
v =dy
dtIt =
@I
@t
How can we use the brightness constancy equation to estimate the optical flow?
![Page 3: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/3.jpg)
Ix
u+ Iy
v + It
= 0
known
unknown
We need at least ____ equations to solve for 2 unknowns.
![Page 4: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/4.jpg)
Ix
u+ Iy
v + It
= 0
known
unknown
Where do we get more equations (constraints)?
![Page 5: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/5.jpg)
Ix
u+ Iy
v + It
= 0
Where do we get more equations (constraints)?
Assume that the surrounding patch (say 5x5) has constant flow
![Page 6: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/6.jpg)
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations
Assumptions:
![Page 7: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/7.jpg)
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations
Assumptions:
Ix
(p1)u+ Iy
(p1)v = �It
(p1)
Ix
(p2)u+ Iy
(p2)v = �It
(p2)
...
Ix
(p25)u+ Iy
(p25)v = �It
(p25)
![Page 8: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/8.jpg)
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations2
6664
Ix
(p1) Iy
(p1)Ix
(p2) Iy
(p2)...
...Ix
(p25) Iy
(p25)
3
7775
uv
�= �
2
6664
It
(p1)It
(p2)...
It
(p25)
3
7775
Assumptions:
Matrix form
![Page 9: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/9.jpg)
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations2
6664
Ix
(p1) Iy
(p1)Ix
(p2) Iy
(p2)...
...Ix
(p25) Iy
(p25)
3
7775
uv
�= �
2
6664
It
(p1)It
(p2)...
It
(p25)
3
7775
Ax
b25⇥ 2 2⇥ 1 25⇥ 1
How many equations? How many unknowns? How do we solve this?
Assumptions:
![Page 10: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/10.jpg)
Least squares approximation
x = argminx
||Ax� b||2A
>Ax = A
>b
is equivalent to solving
![Page 11: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/11.jpg)
To obtain the least squares solution solve:
Least squares approximation
x = argminx
||Ax� b||2A
>Ax = A
>b
is equivalent to solving
2
4
Pp2P
Ix
Ix
Pp2P
Ix
Iy
Pp2P
Iy
Ix
Pp2P
Iy
Iy
3
5
uv
�= �
2
4
Pp2P
Ix
It
Pp2P
Iy
It
3
5
where the summation is over each pixel p in patch P
A>A A>bx
x = (A>A)�1
A
>b
![Page 12: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/12.jpg)
To obtain the least squares solution solve:
Least squares approximation
x = argminx
||Ax� b||2A
>Ax = A
>b
is equivalent to solving
2
4
Pp2P
Ix
Ix
Pp2P
Ix
Iy
Pp2P
Iy
Ix
Pp2P
Iy
Iy
3
5
uv
�= �
2
4
Pp2P
Ix
It
Pp2P
Iy
It
3
5
where the summation is over each pixel p in patch P
A>A A>bx
Sometimes called ‘Lucas-Kanade Optical Flow’(special case of the LK method with a translational warp model)
![Page 13: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/13.jpg)
A
>Ax = A
>b
When is this solvable?
ATA should be invertible
ATA should not be too small
ATA should be well conditioned
λ1 and λ2 should not be too small
λ1/λ2 should not be too large (λ1=larger eigenvalue)
![Page 14: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/14.jpg)
2
4
Pp2P
Ix
Ix
Pp2P
Ix
Iy
Pp2P
Iy
Ix
Pp2P
Iy
Iy
3
5
Where have you seen this before?
A>A =
![Page 15: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/15.jpg)
2
4
Pp2P
Ix
Ix
Pp2P
Ix
Iy
Pp2P
Iy
Ix
Pp2P
Iy
Iy
3
5
Where have you seen this before?
Harris Corner Detector!
![Page 16: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/16.jpg)
Implications• Corners are when λ1, λ2 are big; this is also when
Lucas-Kanade optical flow works best
• Corners are regions with two different directions of gradient (at least)
• Corners are good places to compute flow!
What happens when you have no ‘corners’?
![Page 17: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/17.jpg)
You want to compute optical flow. What happens if the image patch contains only a line?
![Page 18: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/18.jpg)
������������
In which direction is the line moving?
small visible image patch
![Page 19: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/19.jpg)
������������
In which direction is the line moving?
small visible image patch
![Page 20: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/20.jpg)
������������
![Page 21: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/21.jpg)
������������
![Page 22: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/22.jpg)
������������
![Page 23: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/23.jpg)
������������
![Page 24: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/24.jpg)
Want patches with different gradients to the avoid aperture problem
![Page 25: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/25.jpg)
Want patches with different gradients to the avoid aperture problem
![Page 26: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/26.jpg)
1 1 1 12 2 2 23 3 3 34 4 4 45 5 5 5
- - - -- 1 1 1- 2 2 2- 3 3 3- 4 4 4
y
x
y
x
optical flow: (1,1)
H(x,y) = y I(x,y)
It(3, 3) = I(3, 3)�H(3, 3) = �1
Ix
u+ Iy
v + It
= 0
Iy(3, 3) = 1
Ix
(3, 3) = 0v = 1
We recover the v of the optical flow but not the u. This is the aperture problem.
Compute gradients Solution:
![Page 27: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/27.jpg)
Applied to optical flow: Assume that the flow over a small image patch is constant
Note: The Lucas-Kanade method is a general
framework for image matching (not only optical flow)
![Page 28: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/28.jpg)
Horn-Shunck!Optical Flow
(1981)
Lucas-Kanade!Optical Flow!
(1981)
‘constant’ flow‘smooth’ flow
Constraint: Flow of neighboring pixels
should be smooth
Contraint: Flow over a small patch
should be the same
![Page 29: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/29.jpg)
Horn-Schunck16-385 Computer Vision
Carnegie Mellon University (Kris Kitani)
![Page 30: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/30.jpg)
Horn-Schunck !Optical Flow (1981)
Lucas-Kanade!Optical Flow (1981)
‘constant’ flow!(flow is constant for all pixels)
‘smooth’ flow!(flow can vary from pixel to pixel)
brightness constancymethod of differences
global method (dense)
local method (sparse)
small motion
![Page 31: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/31.jpg)
Optical Flow (Problem definition)
Estimate the motion (flow) between these
two consecutive images
I(x, y, t)I(x, y, t0)
How is this different from estimating a 2D transform?
![Page 32: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/32.jpg)
Key Assumptions (unique to optical flow)
Color Constancy!(Brightness constancy for intensity images)
Small Motion!(pixels only move a little bit)
Implication: allows for pixel to pixel comparison (not image features)
Implication: linearization of the brightness constancy constraint
![Page 33: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/33.jpg)
Approach
I(x, y, t)I(x, y, t0)
Look for nearby pixels with the same color(small motion) (color constancy)
![Page 34: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/34.jpg)
Brightness constancy
I(x, y, t)
(x, y)
(x+ u�t, y + v�t)
(x, y)
(u, v) (�x, �y) = (u�t, v�t)Optical flow (velocities): Displacement:
I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)
I(x, y, t+ �t)
For a really small time step…
![Page 35: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/35.jpg)
Optical Flow Constraint equationI(x+ u�t, y + v�t, t+ �t) = I(x, y, t)
I(x, y, t) +@I
@x
�x+@I
@y
�y +@I
@t
�t = I(x, y, t)
Ix
u+ Iy
v + It
= 0
@I
@x
dx
dt
+@I
@y
dy
dt
+@I
@t
= 0
![Page 36: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/36.jpg)
Ix
u+ Iy
v + It
= 0
u
v
Solution lies on a straight line
The solution cannot be determined uniquely with a single constraint (a single pixel)
![Page 37: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/37.jpg)
Where can we get an additional constraint?
![Page 38: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/38.jpg)
Horn-Schunck !Optical Flow (1981)
Lucas-Kanade!Optical Flow (1981)
‘constant’ flow!(flow is constant for all pixels)
‘smooth’ flow!(flow can vary from pixel to pixel)
brightness constancymethod of differences
global method local method
small motion
![Page 39: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/39.jpg)
Smoothness
most objects in the world are rigid or deform elastically !
moving together coherently
we expect optical flow fields to be smooth
![Page 40: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/40.jpg)
Key idea (of Horn-Schunck optical flow)
Enforce brightness constancy
Enforce smooth flow field
to compute optical flow
![Page 41: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/41.jpg)
Enforce brightness constancy
Ix
u+ Iy
v + It
= 0
minu,v
Ix
uij
+ Iy
vij
+ It
�2For every pixel,
![Page 42: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/42.jpg)
Enforce brightness constancy
Ix
u+ Iy
v + It
= 0
minu,v
Ix
uij
+ Iy
vij
+ It
�2For every pixel,
lazy notation for Ix
(i, j)
![Page 43: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/43.jpg)
Ix
Iy
changes these
displacement of object
u v changes these
It
+ =
Find the optical flow such that it satisfies:
![Page 44: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/44.jpg)
Enforce smooth flow field
uij ui+1,jui�1,j
ui,j+1
ui,j�1
minu
(ui,j � ui+1,j)2
u-component of flow
![Page 45: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/45.jpg)
Which flow field is more likely?
X
ij
(uij � ui+1,j)2
X
ij
(uij � ui+1,j)2?
![Page 46: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/46.jpg)
Which flow field is more likely?
more likelyless likely
![Page 47: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/47.jpg)
optical flow objective function
Optical flow constraint
Smoothness constraint
(uij � ui+1,j)
ij
i, j + 1
i+ 1, j ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
i� 1, j
i, j � 1
(uij � ui,j+1)
ij
i, j + 1
i+ 1, j ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
i� 1, j
i, j � 1
(vij � vi+1,j) (vij � vi,j+1)
Ed
(i, j) =
Ix
uij
+ Iy
vij
+ It
�2
Es(i, j) =1
4
(uij � ui+1,j)
2 + (uij � ui,j+1)2 + (vij � vi+1,j)
2 + (vij � vi,j+1)2
�
![Page 48: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/48.jpg)
Horn-Schunck optical flow
minu, v
X
ij
⇢Ed(i, j) + �Es(i, j)
�
Ed
(i, j) =
Ix
uij
+ Iy
vij
+ It
�2
where
and
Es(i, j) =1
4
(uij � ui+1,j)
2 + (uij � ui,j+1)2 + (vij � vi+1,j)
2 + (vij � vi,j+1)2
�
![Page 49: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/49.jpg)
How do we solve this minimization problem?
minu, v
X
ij
⇢Ed(i, j) + �Es(i, j)
�
![Page 50: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/50.jpg)
How do we solve this minimization problem?
minu, v
X
ij
⇢Ed(i, j) + �Es(i, j)
�
Compute partial derivative, derive update equations (gradient decent)
![Page 51: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/51.jpg)
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
![Page 52: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/52.jpg)
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
it’s not so bad…
how many terms depend on k and l?
![Page 53: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/53.jpg)
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
it’s not so bad…
how many terms depend on k and l?
![Page 54: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/54.jpg)
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
(uij � ui+1,j)
ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
uij =1
4
⇢ui+1,j + ui�1,j + ui,j+1 + ui,j�1
�local average
(u2ij � 2uijui+1,j + u2
i+1,j) (u2ij � 2uijui,j+1 + u2
i,j+1)
(variable will appear four times in sum)
![Page 55: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/55.jpg)
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
![Page 56: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/56.jpg)
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
![Page 57: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/57.jpg)
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
Ax = b
how do you solve this?
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
this is a linear system
![Page 58: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/58.jpg)
ok, take a step back, why are we doing all this math?
![Page 59: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/59.jpg)
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
We are solving for the optical flow (u,v) given two constraints
smoothness brightness constancy
We need the math to minimize this (back to the math)
![Page 60: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/60.jpg)
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
Ax = b
how do you solve this?
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
Partial derivatives of Horn-Schunck objective function E:
![Page 61: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/61.jpg)
x = A�1b =
adjA
detAb
Recall
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
![Page 62: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/62.jpg)
x = A�1b =
adjA
detAb
Recall
(determinant of A)
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
(determinant of A)
Same as the linear system:{1 + �(I2
x
+ I2y
)}ukl
= (1 + �I2x
)ukl
� �Ix
Iy
vkl
� �Ix
It
{1 + �(I2x
+ I2y
)}vkl
= (1 + �I2y
)vkl
� �Ix
Iy
ukl
� �Iy
It
![Page 63: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/63.jpg)
(determinant of A)
(determinant of A)
Rearrange to get update equations:
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
1 + �(I2x
+ I2y
)Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
1 + �(I2x
+ I2y
)Iy
new value
old average
{1 + �(I2x
+ I2y
)}ukl
= (1 + �I2x
)ukl
� �Ix
Iy
vkl
� �Ix
It
{1 + �(I2x
+ I2y
)}vkl
= (1 + �I2y
)vkl
� �Ix
Iy
ukl
� �Iy
It
check for missing lambda
![Page 64: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/64.jpg)
ok, take a step back, why did we do all this math?
![Page 65: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/65.jpg)
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
We are solving for the optical flow (u,v) given two constraints
smoothness brightness constancy
We needed the math to minimize this (now to the algorithm)
![Page 66: Lucas-Kanade Optical Flo16385/s15/lectures/Lecture21.pdf · To obtain the least squares solution solve: Least squares approximation xˆ = argmin x ||Ax b||2 is equivalent to solving](https://reader036.vdocuments.site/reader036/viewer/2022071220/605a0a647568cc68c2174e4b/html5/thumbnails/66.jpg)
Horn-Schunck Optical Flow Algorithm
1. Precompute image gradients!
2. Precompute temporal gradients!
3. Initialize flow field!
4. While not converged Compute flow field updates for each pixel:
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
1 + �(I2x
+ I2y
)Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
1 + �(I2x
+ I2y
)Iy
Ix
Iy
It
u = 0
v = 0