Optical Flow

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

Optical Flow:Where do pixels move to?

Motion is a basic cueMotion can be the only cue for segmentation

Motion is a basic cue

Even impoverished motion data can elicit a strong percept

Applications tracking structure from motion motion segmentation stabilization compression mosaicing …

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

Definition of optical flow

OPTICAL FLOW = apparent motion of brightness patterns

Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image

Caution required !

Two examples :

1. Uniform, rotating sphere

O.F. = 0

2. No motion, but changing lighting

O.F. 0

Mathematical formulation

I (x,y,t) = brightness at (x,y) at time t

Optical flow constraint equation :

0

tI

dtdy

yI

dtdx

xI

dtdI

),,(),,( tyxItttdtdyyt

dtdxxI

Brightness constancy assumption:

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

The aperture problem

0 tyx IvIuI

1 equation in 2 unknowns

dtdxu

dtdyv

yII x

yII y t

II t

Aperture Problem: Animation

The Aperture Problem

0

What is Optical Flow, anyway? Estimate of observed projected motion field Not always well defined! Compare:

Motion Field (or Scene Flow)projection of 3-D motion field

Normal Flowobserved tangent motion

Optical Flowapparent motion of the brightness pattern(hopefully equal to motion field)

Consider Barber pole illusion

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

Horn & Schunck algorithm

Additional smoothness constraint :

,))()(( 2222 dxdyvvuue yxyxs besides OF constraint equation term

,)( 2 dxdyIvIuIe tyxc

minimize es+ec

Horn & Schunck

The Euler-Lagrange equations :

0

0

yx

yx

vvv

uuu

Fy

Fx

F

Fy

Fx

F

In our case ,

,)()()( 22222tyxyxyx IvIuIvvuuF

so the Euler-Lagrange equations are

,)(

,)(

ytyx

xtyx

IIvIuIv

IIvIuIu

2

2

2

2

yx

is the Laplacian operator

Horn & Schunck

Remarks :

1. Coupled PDEs solved using iterative methods and finite differences

2. More than two frames allow a better estimation of It

3. Information spreads from corner-type patterns

,)(

,)(

ytyx

xtyx

IIvIuIvtv

IIvIuIutu

Horn & Schunck, remarks

1. Errors at boundaries

2. Example of regularization (selection principle for the solution of ill-posed problems)

Results of an enhanced system

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

02),(

02),(

tyxy

tyxx

IvIuIIdv

vudE

IvIuIIdu

vudE

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow

Textured Motion http://

www.cs.ucla.edu/~wangyz/Research/wyzphdresearch.htm

Natural scenes contain rich stochastic motion patterns which are characterized by the movement of a large amount of particle andwave elements, such as falling snow, water waves, dancing grass, etc. We call these motion patterns "textured motion".