lecture-4 - cs.ucf.edu · lecture-4 computing optical flow hamburg taxi seq. horn&schunck...
TRANSCRIPT
![Page 1: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/1.jpg)
Lecture-4Computing Optical Flow
Hamburg Taxi seq
![Page 2: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/2.jpg)
![Page 3: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/3.jpg)
Horn&Schunck Optical Flow
df (x, y, t)dt
=∂f∂x
dxdt
+∂f∂y
dydt
+∂f∂t
= 0
Sequence Image ),,( tyxf
![Page 4: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/4.jpg)
Horn&Schunck Optical Flow
Taylor Series
f x y t f x y tfx
dxfy
dyft
dt( , , ) ( , , )= + + +∂∂
∂∂
∂∂
),,(),,( dttdyydxxftyxf +++=
0=++ dtfdyfdxf tyx
Interpretation of optical flow eq
d=normal flowp=parallel flow
Equation of st.line
0=++ tyx fvfuf
y
t
y
x
ffu
ffv −−=
df
f ft
x y
=+2 2
![Page 5: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/5.jpg)
Horn&Schunck (contd)
variational calculus
discrete version
min
{( ) ( )}∫∫ + + + + + +f u f v f u u v v dxdyx y t x y x y2 2 2 2 2λ
( ) ( )
( ) (( )
f u f v f f u
f u f v f f vx y t x
x y t y
+ + + =
+ + + =
λ
λ
∆
∆
2
2
0
0
( ) ( )
( ) (( )
f u f v f f u u
f u f v f f v vx y t x av
x y t y av
+ + + − =
+ + + − =
λ
λ
0
0
u u fPD
v v fPD
av x
av y
= −
= −
P f u f v f
D f fx av y av t
x y
= + +
= + +λ 2 2
yyxx uuu +=∆2
Algorithm-1
k=0Initialize
• Repeat until some error measure is satisfied(converges)
u vK K
22yx
tavyavx
ffD
fvfufP
++=
++=
λu u f
PD
v v fPD
av x
av y
= −
= −
![Page 6: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/6.jpg)
Derivatives
Derivative: Rate of change of some quantity Speed is a rate of change of a distanceAcceleration is a rate of change of speed
Derivative
speed
acceleration
xx fxfx
xxfxfdxdf
=′=∆
∆−−= →∆ )()()(lim 0
dtdva
dtds
v
=
=
![Page 7: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/7.jpg)
Examples
x
x
exdxdy
exy
−
−
−+=
+=
)1(cos
sin3
42
42 xxdxdy
xxy
+=
+=
Second Derivative
xxx fxf
dxdf
=′′= )(
22
2
3
42
122
42
xdx
yd
xxdxdy
xxy
+=
+=
+=
![Page 8: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/8.jpg)
Discrete Derivative
(Finite Difference)
)()()(lim 0 xfx
xxfxfdxdf
x ′=∆
∆−−= →∆
)(1
)1()( xfxfxfdxdf ′=
−−=
)()1()( xfxfxfdxdf ′=−−=
Discrete Derivative
Left difference
Right difference
Center difference
)()1()( xfxfxfdxdf ′=−−=
)()1()( xfxfxfdxdf ′=+−=
)()1()1( xfxfxfdxdf ′=−−+=
![Page 9: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/9.jpg)
Example
F(x)=10 10 10 10 20 20 20F’(x)=0 0 0 0 10 0 0F’’(x)=0 0 0 0 10 -10 0
-1 1 left difference1 -1 right difference-1 0 1 center difference
Left difference
Derivatives in Two Dimensions
(partial Derivatives)
yyyxfyxff
yf
xyxxfyxff
xf
yxf
yy
xx
∆∆−−
==∂∂
∆∆−−
==∂∂
→∆
→∆
),(),(lim
),(),(lim
),(
0
0
x
y
yx
yx
ff
ff
ff
1
22
tandirection
)(magnitude
VectorGradient ),(
−==
+=
θ
Laplacian2 =+=∆ yyxx fff
![Page 10: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/10.jpg)
Derivatives of an Image
Derivative & average
Prewit
xf 101101101
−
−−
yf 111000111 −−−
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
20202010102020201010202020101020202010102020201010
),( yxI
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
0000000303000030300003030000000
xI
Derivatives of an Image
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
0000000000000000000000000
yI
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
20202010102020201010202020101020202010102020201010
),( yxI
![Page 11: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/11.jpg)
Laplacian
0 −14
0
−14
1 −14
0 −14
0
fxx + fyy
Convolution
![Page 12: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/12.jpg)
Convolution (contd)
),(*),(),(
),(),(),(1
1
1
1
yxgyxfyxh
jigjyixfyxhi j
=
++= ∑ ∑−= −=
h(x, y) = f (x −1,y −1)g(−1,−1) + f (x, y −1)g(0,−1) + f (x +1,y −1)g(1,−1) + f (x −1, y)g(−1,0) + f (x, y)g(0,0) + f (x +1,y)g(1,0) + f (x −1, y +1)g(−1,1) + f (x, y +1)g(0,1) + f (x +1, y +1)g(1,1)
Derivative Masks
Apply first mask to 1st imageSecond mask to 2nd imageAdd the responses to get f_x, f_y, f_t
xf
image second1111
imagefirst 1111
⎥⎦
⎤⎢⎣
⎡−−
⎥⎦
⎤⎢⎣
⎡−−
yf
image second1111
imagefirst 1111
⎥⎦
⎤⎢⎣
⎡ −−
⎥⎦
⎤⎢⎣
⎡ −−
tf
image second1111
imagefirst 1111
⎥⎦
⎤⎢⎣
⎡
⎥⎦
⎤⎢⎣
⎡−−−−
![Page 13: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/13.jpg)
Synthetic Images
Horn & Schunck Results
One iteration 10 iterations
4=λ
![Page 14: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/14.jpg)
Lucas & Kanade (Least Squares)
Optical flow eq
• Consider 3 by 3 windowtyx fvfuf −=+
999
111
tyx
tyx
fvfuf
fvfuf
−=+
−=+
M
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
−
−
=⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
9
1
9
1
9
1
t
t
y
y
x
x
f
f
vu
f
f
f
fMMM
tfAu =
Lucas & Kanade
tTT
tTT
t
fAAAu
fAAuA
fAu
1)( −=
=
=
2)(min tyixi fvfuf ++∑
![Page 15: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/15.jpg)
Lucas & Kanade2)(min tyixi fvfuf ++∑
0)(
0)(
=++
=++
∑
∑
yitiyixi
xitiyixi
ffvfuf
ffvfuf
fxi2∑ u + fxi∑ fyiv = − fxi∑ f ti
fxi fyi∑ u + fyi2∑ v = − fyi∑ f ti
fxi2∑ fxi fyi∑
fxi fyi∑ fyi2∑
⎡
⎣
⎢ ⎢ ⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥ ⎥ ⎥
uv
⎡
⎣ ⎢
⎤
⎦ ⎥ =
− fxi f ti∑
− fyi f ti∑
⎡
⎣
⎢ ⎢ ⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥ ⎥ ⎥
( fxi∑ u + fyiv + f ti) f xi = 0
( fxi∑ u + fyiv + f ti) f yi = 0
![Page 16: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/16.jpg)
Lucas & Kanade
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−
−
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=⎥⎦
⎤⎢⎣
⎡
∑
∑
∑∑
∑∑−
tiyi
tixi
yiyixi
yixixi
ff
ff
fff
fff
vu
1
2
2
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−
−
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−
−
−=⎥
⎦
⎤⎢⎣
⎡
∑
∑
∑∑
∑∑
∑ ∑∑ tiyi
tixi
xiyixi
yix iyi
yixiyixi ff
ff
fff
fff
ffffvu
2
2
222 )(
1
∑ ∑∑
∑ ∑ ∑∑
∑ ∑∑
∑ ∑ ∑∑
−
−=
−
+−=
222
2
222
2
)(
)(
yixiyixi
tiy ix iyixitix i
y ix iyixi
tiyiyixitix iyi
ffff
fffffffv
ffff
fffffffu
Lucas-Kanadewithout pyramids
Fails in areas of large motion
![Page 17: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/17.jpg)
Lucas-Kanade with Pyramids
Comments
Horn-Schunck and Lucas-Kanade optical method works only for small motion.If object moves faster, the brightness changes rapidly, 2x2 or 3x3 masks fail to estimate spatiotemporal derivatives.Pyramids can be used to compute large optical flow vectors.
![Page 18: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/18.jpg)
![Page 19: Lecture-4 - cs.ucf.edu · Lecture-4 Computing Optical Flow Hamburg Taxi seq. Horn&Schunck Optical Flow df (x,y,t) dt = ... Algorithm-1 k=0 Initialize](https://reader033.vdocuments.site/reader033/viewer/2022050410/5f87b0e0e23a5a6fb1177f5d/html5/thumbnails/19.jpg)