optical flow estimation using variational techniques darya frolova
Post on 22-Dec-2015
234 views
TRANSCRIPT
![Page 1: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/1.jpg)
Optical Flow Estimation Optical Flow Estimation using using
Variational Techniques Variational Techniques
Darya FrolovaDarya Frolova
![Page 2: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/2.jpg)
Part IPart I
Mathematical Preliminaries
![Page 3: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/3.jpg)
AgendaAgenda
Minimization problem: ? )( min
xFAx
Investigate the existence of a solution with respect to reasonable assumptions on functional F and space A
- conditions for A and F,- main theorem: existence and uniqueness of minimum
Optical Flow - what is this? - different approaches - formulation of minimization problem for Optical Flow - existence of solution (using main theorem)
![Page 4: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/4.jpg)
minimization problem: ? )( min
xFAx
TheoremA is a compact set, function F is continuous on A.
Ax 0 Then such that )( )( 0 min xFxFAx
What happens when A is not a compact?
What happens when functional F is not continuous?
Does the minimum exist?
a bx0
F (x)
![Page 5: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/5.jpg)
New TheoremNew Theorem
Theorem
A is a compact set
Ax 0 Then such that )( )( 0 min xFxFx
function F is continuous on A
A = BV (Ω) – space of functions of bounded variation
functional F is coercive and lower semicontinuous on A
![Page 6: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/6.jpg)
Bounded Variation -1D caseBounded Variation -1D case
Rbaxf ],[:)(
f has bounded variation over ],[ ba if exists const M such that
|)()(| ... |)()(||)()(| 1121 nxfbfxfxfafxf
for all bxxxa n 121 ...
Consider
a x1 x2 xn -1 b……
)(af )( 1nxf )(bf
)( 2xf)( 1xf
M | )( )(| 1
1
n
iii xfxf
Definition
![Page 7: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/7.jpg)
Bounded VariationBounded Variation
Definition
The space of functions of bounded variation on Ω is denoted BV (Ω)
Total variation of function f can be represented as follows:
dx
dffD
sup|)(|φ
where - function φ is compactly supported (is zero outside of a compact set),
] 1 ,1[ -
![Page 8: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/8.jpg)
If f is differentiable, then dx
df
|f
dx
df
Supremum for
and
1
-1
f(x)
sign(df/dx)
dx
dfsign
dx
df
dx
d
Then variation of function f =
max loc min loc
)( )(x x
xfxf
Function f does not need to be differentiable
![Page 9: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/9.jpg)
Bounded Variation – ND caseBounded Variation – ND case
sup |)(| dxdivffD
)( x
N
1i i
i xdiv
1 || ,)( ), ..... ,( )(
101
L
NN C
bounded open subset, function NR )( 1 Lf
Variation of over f
where
φ
![Page 10: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/10.jpg)
Bounded Variation – exampleBounded Variation – example
xxxf 1sin)( 21(
xxxxf 1cos1sin2)(
1
0
1
0
|1cos1sin2||)(| dxxxxdxxf
functions under integral are bounded on [0,1]
xxxf 1sin)( 2(
xxxxf 11cos1sin)(
1
0
1
0
|11cos1sin||)(| dxxxxdxxf - problem at x = 0, 2nd term is not bounded
![Page 11: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/11.jpg)
DefinitionsDefinitions
Definition Banach space – complete, normed linear space
Definition
Space X is said to be complete, when any Cauchy sequence xn from X converges
DefinitionConsider sequence xn from X.
If N 0 natural, such that for any m, n > N
|| || mn xx , then xn - Cauchy sequence
Space of functions of bounded variation BV(Ω) is a Banach space
![Page 12: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/12.jpg)
Definition
Functional f is coercive if
)( lim||
xfx
DefinitionConsider sequence xn. Denote . inf n
knk xX
Then lower limit of sequence xn is a limit of sequence Xk:
nknk
kk
nn
xXx inflimlimlim
+ +
x
f (x)
![Page 13: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/13.jpg)
Dual SpaceDual Space
Let X denote a real Banach space. Dual space of X : X * – space of linear bounded operators
X * = f : X → R
Definition
Linear operator:
f (αx + βy) = α f (x) + β f (y)
Bounded operator:
bounded set converts to bounded set
|||| ||)(||
such that 0
xcxfx
c
![Page 14: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/14.jpg)
Dual spaceDual space
Definition
X is called reflexive if ( X *) *
Reflexive Not reflexive
e.g. finite-dimensional (normed) spaces,
Hilbert spaces
e.g. the space of sequences ℓ∞
||max nxℓ∞ = xn :
( if bidual space of X is equal to X )
![Page 15: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/15.jpg)
Topologies on XTopologies on X
Definition Sequence xn from X strongly converges to point x if
)(n 0 | | Xn xx
If sequence strongly converges, then it weakly converges
Definition Sequence xn from X weakly converges to point x if
)(n )( )( xfxf n
for every functional f from X*
( if converges sequence of real numbers f (xn) for any linear bounded f )
![Page 16: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/16.jpg)
Topologies on XTopologies on X**Definition
Sequence fn from X* strongly converges to f if
)(n 0 | | * Xn ff
Definition Sequence fn from X weakly converges to f if
)(n )( )( fgfg n
for every g from (X*)* (bidual space of X)
Definition Sequence fn from X weakly* converges to f if
)(n ffn
for every x from X
![Page 17: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/17.jpg)
Direct MethodDirect Method
Problemf : X → R , where X is a Banach space
inf f (x)xX
Does the solution exist?
The proof consists of three steps , which is called
Direct Method of Variational Calculus
![Page 18: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/18.jpg)
TheoremTheorem
If
functional f is coercive and lower semicontinuous on X,
X is Banach and reflexive space
Thenfunctional f has its minimum on X :
)( )( such that min00 xfxfXxXx
![Page 19: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/19.jpg)
Direct MethodDirect Method
Step A Construct minimizing sequence xn for functional f :
)( )( inflim xfxfXx
nn
Step Ba) If f is coercive, then minimizing sequence is bounded
Step C If f is lower semicontinuous at point x0 then x0 is a point of minimum: )( )( min0 xfxf
Xx
b) If X is reflexive, then there exists weakly convergent subsequence of minimizing sequence:
Xxxjn 0
weakly exists
![Page 20: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/20.jpg)
Step A
There exists an infimum of the set f (x), x X R
and exists a sequence R , which converges to this infimum
f (x), x X R
![Page 21: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/21.jpg)
Step B (a)If f is coercive, then minimizing sequence is bounded :
CxC n || such that 0const
Proof )( )( inflim xfxfXx
nn
Consider minimizing sequence xn:
Assume that it is not bounded: CxnC n || such that 0 (equivalently, for any ball with radius C = 1,2… there
exists an element outside this ball)cnx
cnxIf we form a subsequence of these , it will satisfy: ) ( || cn nxc
f is coercive
)( lim||
xfx
)( lim||
c
cn
nx
xf
But is subsequence of f (xn), that is why )(cnxf
)( lim||
nx
xfn
So, unbounded xn cannot be minimizing sequence.
To prove
![Page 22: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/22.jpg)
Step B (b)
we can conclude that exists x0 X and exists subsequence
such that:
jnx
0weakly xx
jn
We proved that minimizing sequence is bounded.
If X is reflexive ( ( X* )* = X ) then using theorem about weak sequential compactness, which states that:
for any bounded sequence in reflexive Banach space there exists weakly convergent subsequence
If X is reflexive, then there exists weakly convergent subsequence of minimizing sequence
To prove
Proof
![Page 23: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/23.jpg)
Step C
If f is lower semicontinuous at x0 x0 is a minimum point of f
0weakly xx
jn We proved the existence of such that : jnx
)()( )( inflimlim xfxfxfXx
nn
nn
j
j
If there exists limit of f (xn) then there exists a limit of its subsequence
and these limits are equal:
)( jnxf
On the other hand, from the lower semicontinuity it follows that )( )( 0lim xfxfjn
Hence )( )( inf0 xfxfXx
)( )( min0 xfxfXx
This means that
To prove
Proof
![Page 24: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/24.jpg)
Part IIPart II
Optical Flow
![Page 25: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/25.jpg)
Image SequenceImage Sequence
Sequence of images contains information about the scene,We want to estimate motion (using variational formulation)
![Page 26: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/26.jpg)
2D motion field2D motion field
Optical center
2D motion field
Projection on the image plane of the 3D velocity of the scene
3D motion field
Image intensity
I1
I2
Motion vector - ?
![Page 27: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/27.jpg)
Optical FlowOptical FlowWhat we are able to perceive is just an apparent motion, called
Optical flow
(motion, observable only through intensity variations)
Intensity remains constant – no motion is perceived
No object motion, moving light source produces intensity variations
![Page 28: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/28.jpg)
Optical flow-methodsOptical flow-methods
Correlation-based techniques - compare parts of the first image with parts of the second in terms of the similarity in brightness patterns in order to determine the motion vectors
Feature-based methods - compute and analyze Optical Flow at small number of well-defined image features
Gradient-based methods - use spatiotemporal partial derivatives to estimate flow at each point
![Page 29: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/29.jpg)
Brightness constancyBrightness constancy
Intensity of a point keeps constant along its trajectory
(reasonable for small displacements)
),( xtI intensity of the pixel ),( 21 xxx at time t
Start from point x0 at time t0. Trajectory → ( t, x (t) )
txtItxtI allfor ),( ))( ,( 00) ,( ) )( ,( 0000 xttxt
0at 0 ttt
II
dt
dx
Differentiating →
)( )( 00 tdt
dxtv We will search the Optical Flow as the velocity field:
![Page 30: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/30.jpg)
Brightness constancyBrightness constancy
Given sequence I (t, x) and time t0
Find the velocity v (x) such that:
We need to find the velocity field - 2 componentsWe have 1 scalar equation
Can find only “normal flow”
Component in the direction of gradient I
0 ),( ),( )( 000 xtIxtIxv t
Optical Flow constraint (OFC)
![Page 31: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/31.jpg)
Aperture problemAperture problem
![Page 32: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/32.jpg)
Solving the aperture problemSolving the aperture problem
Second order derivative constraint : conservation of along trajectories
I
0 ) ,(
xtdt
Id
Rigid deformations are not considered (object moves locally in one direction)
0
0
2
1
2221
1211
2
1
tx
tx
xxxx
xxxx
I
I
v
v
II
II
Sensitive to noise
![Page 33: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/33.jpg)
Weighted least squaresWeighted least squares
Velocities are constant in small window (spatial neighborhood)
dxIIvxwrx
tv
)( 2
),(Ball
2
0
inf
w (x) is a window function ( gives more influence to the constraint at the center of the neighborhood than at the periphery)
Too local, no global regularity
![Page 34: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/34.jpg)
Regularizing the velocity fieldRegularizing the velocity field
)( )( inf vSvAv
smoothing term
dxvdxIIvj
tv
j ||
22
1
2
inf
A (v) S (v)
Velocity should change slowly in spatial domain (in image plain)
Optical Flow constraint
Horn and Schunck
![Page 35: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/35.jpg)
DiscontinuitiesDiscontinuities
But smoothing term does not allow to save discontinuities
Discontinuities near edges are
lost
Synthetic example(method of Horn and Schunck)
![Page 36: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/36.jpg)
Discontinuity-preserving approachDiscontinuity-preserving approach
2
inf
dxIIv tv
dxvj
j ||
22
1
dxvj
j) || (
2
1
Where function η permits noise removal and edge conservation
2
inf
dxIIv tv
dxvj
j ||
22
1
dxv div ) )( (
[Black et.al, Cohen, Kumar, Balas, Tannenbaum, Blanc-Feraud]
[Suter, Gupta and Prince, Guichard and Rudin]
![Page 37: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/37.jpg)
Discontinuity-preserving approach Discontinuity-preserving approach summarysummary
Given: sequence I (t, x)
Find: velocity field v that minimizes the energy functional E
22
1
||)( )( | | )( vDIcDvIDIvvE h
j
st j
S (v)
Smoothing term, we need to find
conditions on η for saving discontinuities
H (v)
Is related to homogeneous
regions
A (v)
L1 norm of the Optical Flow
constraint
![Page 38: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/38.jpg)
Smoothing termSmoothing term
)( )(2
1
j
jDvvS
Function η : R+ → R+ is strictly convex, nondecreasing
η (0) = 0 (without loss of generality)
)( lim ss
bassbas )( η (s)
bas
bas
![Page 39: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/39.jpg)
Homogeneous termHomogeneous term
2||)( )( vDIcvH
Idea: if there is no texture (there is no gradient), then there is no possibility to correctly estimate the flow field
So, we may force it to be zero
0 )( and 1 )( limlim0
scscss
)( Cc
]1 ; [ )( such that const exists cc mxcm
Without loss of generality:
![Page 40: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/40.jpg)
Existence of solutionExistence of solution
Theorem Under the following hypotheses:
]1 ; [ )( such that const exists cc mxcm
)( Cc
bassbas )(
)( lim ss
η : R+ → R+ is strictly convex, nondecreasing, η (0) = 0
22
1
||)( )( | | )( min vDIcDvIDIvvE h
j
st j
The minimization problem
has a unique solution in BV(Ω)
![Page 41: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/41.jpg)
Existence of a solutionExistence of a solution
ProofUsing direct method of Variational Calculus:
Step A Construct minimizing sequence vn for functional E :
)( )( inflim)(
vEvEBVv
nn
Step Ba) If E is coercive, then minimizing sequence is bounded
b) If BV (Ω) is reflexive, then there exists weakly convergent subsequence of minimizing sequence:
Step C If E is lower semicontinuous at point x0 then x0 is a point of minimum: )( )( min
)(0 vEvE
BVv
![Page 42: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/42.jpg)
Existence of a solutionExistence of a solution
22
1
||)( )( | | )( vDIcDvIDIvvE h
j
st j
)( lim ss
v v v
+ +
c is bounded
+
So, functional E is coercive
![Page 43: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/43.jpg)
Existence of a solutionExistence of a solution
ProofUsing direct method of Variational Calculus:
Step A Construct minimizing sequence vn for functional E :
)( )( inflim)(
vEvEBVv
nn
Step Ba) If E is coercive, then minimizing sequence is bounded
b) If BV (Ω) is reflexive, then there exists weakly convergent subsequence of minimizing sequence:
Step C If E is lower semicontinuous at point x0 then x0 is a point of minimum: )( )( min
)(0 vEvE
BVv
![Page 44: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/44.jpg)
Existence of a solutionExistence of a solution
Space of functions of bounded variation is not reflexive: ( BV *)* ≠ BV
But it has such a property that
every bounded sequence Ij from BV (Ω) has a subsequence
that weakly* converges to some element I from BV (Ω)
![Page 45: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/45.jpg)
Existence of a solutionExistence of a solution
ProofUsing direct method of Variational Calculus:
Step A Construct minimizing sequence vn for functional E :
)( )( inflim)(
vEvEBVv
nn
Step Ba) If E is coercive, then minimizing sequence is bounded
b) If BV (Ω) is reflexive, then there exists weakly convergent subsequence of minimizing sequence:
Step C If E is lower semicontinuous at point x0 then x0 is a point of minimum: )( )( min
)(0 vEvE
BVv
![Page 46: Optical Flow Estimation using Variational Techniques Darya Frolova](https://reader036.vdocuments.site/reader036/viewer/2022062516/56649d815503460f94a66d08/html5/thumbnails/46.jpg)
The End