Optical Flow Methods

2007/8/9

Outline

Introduction to 2-D Motion The Optical Flow Equation The Solution of Optical Flow

Equation Comparison of different methods Reference

The 2-D Motion

The projection of 3-D motion into the image plane.

X

Z

Y

The 2-D Motion(2)

A 2-D displacement field is a collection of 2-D displacement vectors.

Definition of Optical Flow

Optical flow is a vector field of pixel velocities based on the observable variations form the time-varying image intensity patter.

Difference between Optical flow and 2-D displacement(1)

There must be sufficient gray-level variation for the actual motion to be observable.

Difference between Optical flow and 2-D displacement(2)

An observable optical flow may not always correspond to actual motion. For example: changes in external illumination.

Outline

Introduction to 2-D Motion The Optical Flow Equation The Solution of Optical Flow

Equation Comparison of different methods Reference

The Optical Flow Equation(1)

Let the image brightness at the point (x, y) in the image plane at time t be denoted by

The brightness of a particular point in the pattern is constant, so that

Using the chain rule for differentiation we see that, 0

t

E

dt

dy

y

E

dt

dx

x

E

0dt

dE

),,( tyxE

The Optical Flow Equation(2)

If we let and , for the partial derivatives, we have a single linear equation in two unknowns: u and v.

Writing the equation in the two unknowns u and v,

dt

dxu

dt

dyv ),( yx EE

0 tyx EvEuE

The Optical Flow Equation(3) Writing the equation in another form,

The component of the movement in the direction of the brightness gradient equals

tyx EvuEE ,,

22),(

yx

t

EE

Evu

The Optical Flow Equation(4)

The velocity has to lie along a line perpendicular to the brightness gradient vector.

x

y

),( yx EE

Constraint Line

tyx EvuEE ,,

22),(

yx

t

EE

Evu

(u,v)

Outline

Introduction to 2-D Motion The Optical Flow Equation The Solution of Optical Flow

Equation Comparison of different methods Reference

Second-Order Differential Methods(1)

Based on the conservation of the spatial image gradient.

The flow field is given by

0)),,((

dt

tyxEd

yt

E

xt

E

y

E

yx

E

xy

E

x

E

v

v

y

x

2

21

2

22

2

2

2

,

,

Second-Order Differential Methods(2)

The deficiencies: The constraint does not allow for

some motion such as rotation and zooming.

Second-order partials cannot always be estimated with sufficient accuracy.

Block Motion Model (1) (Lucas and Kanade Method) Based on the assumption that the mo

tion vector remains unchanged over a particular block of pixels.

for x,y inside block B

)(),,( tvtyxv

2

,)()(

t

Etv

y

Etv

x

EE yx

Byx

Block Motion Model (2)

Computing the partials of error with respect to and , then setting them equal to zero, we have

xv yv

0)()(,

x

E

t

Etv

y

Etv

x

Eyx

Byx

0)()(,

y

E

t

Etv

y

Etv

x

Eyx

Byx

Block Motion Model (3)

Solving the equations, we have

t

E

y

Et

E

x

E

y

E

y

E

y

E

x

E

y

E

x

E

x

E

x

E

tv

tv

Byx

Byx

ByxByx

ByxByx

y

x

,

,

1

,,

,,

,

,

)(

)(

Block Motion Model (4)

It is possible to increase the influence of the constraints towards the center of the block by weighted summations.

The accuracy of estimation depends on the accuracy of the estimated spatial and temporal partial derivatives.

Horn and Schunck Method(1) The additional constraint is to mini

mize the sum of the squares of the Laplacians of the optical flow velocity:

and2

2

2

22

y

u

x

uu

2

2

2

22

y

v

x

vv

Horn and Schunck Method(2) The minimization of the sum of the

errors in the equation for the rate of changes of image brightness.

and the measure of smoothness in the velocity flow.

2

2

2

2

2

2

2

2

y

v

x

v

y

u

x

uc

tyxb EvEuE

Horn and Schunck Method(3) Let the total error to be minimized be

The minimization is to be accomplished by finding suitable values for optical flow velocity (u ,v).

The solution can be found iteratively.

dxdybc 2222

Horn and Schunck Method: Directional-Smoothness constraint

The directional smoothness constraint:

W is a weight matrix depending on the spatial changes in gray level content of the video.

The directional-smoothness method minimizes the criterion function:

yTyx

Txds vWvvWv 2

dxdybds 2222

Gradient Estimation Using Finite Differences(1)

To obtain the estimates of the partials, we can compute the average of the forward and backward finite differences.

Gradient Estimation Using Finite Differences(2) The three partial derivatives of images

brightness at the center of the cube are estimated form the average of differences along four parallel edges of the cube.

Gradient Estimation by Local Polynomial Fitting(1) An approach to approximate E(x,y,t) local

ly by a linear combination of some low-order polynomials in x, y, and t; that is,

Set N equal to 9 and choose the following basis functions

1

0

),,(),,(N

iii tyxatyxE

ytxtxyyxtyxtyxi ,,,,,,,,1),,( 22

Gradient Estimation by Local Polynomial Fitting(2) The coefficients are estimated by using

the least squares method.

The components of the gradient can be found by differentiation,

x y t

N

iii tyxatyxEe 2

1

0

2 )),,(),,((

yaxaat

tyxE

taxayaay

tyxE

tayaxaax

tyxE

873

8652

7641

),,(

2),,(

2),,(

Estimating the Laplacian of the Flow Velocities(1) The approximation takes the following

form

and

The local averages u and v are defined as:

)( ,,,,2

kjikji vvv )( ,,,,2

kjikji uuu

Estimating the Laplacian of the Flow Velocities(2) The Laplacian is estimated by subtrac

ting the value at a point form a weighted average of the values at neighboring points.

Outline

Introduction to 2-D Motion The Optical Flow Equation The Solution of Optical Flow

Equation Comparison of different methods Reference

Comparison of different methods(1)

Three different method to be compared: Lucas-Kanade method based on block moti

on model. (11x11 blocks with no weighting) Horn-Schunck method imposing a global s

moothness constraint.( , allowed for 20 to 150 iterations)

The directional-smoothness method of Nagel( with 20 iterations)

6252

5,252

Comparison of different methods(2) These methods have been applied to the

7th and 8th frames of a video sequence, known as the “Mobile and Calendar.”

The gradients have been approximated by average finite differences and polynomial fitting.

The images are spatially pre-smoothed by a 5x5 Gaussian kernel with the variance 2.5 pixels.

Comparison of different methods(3)

Comparison of the differential methods.

278

10)),(),,((),(

255255log10

yxdyyxdxEyxEPSNR

yx

Method PSNR(dB) Entropy(bits)Polynomial Difference Polynomia

lDifference

Frame-DifferenceLucas-KanadeHorn-SchunckNagel

23.4530.8928.1429.08

-32.0930.7131.84

-6.444.225.83

-6.825.045.95

Outline

Introduction to 2-D Motion The Optical Flow Equation The Solution of Optical Flow

Equation Comparison of different methods Reference

Reference A. M. Tekalp, Digital Video Processing. Engle

wood Cliffs, NJ: Prentice-Hall, 1995. Horn, B.K.P. and Schunck, B.G. Determining

optical flow:A retrospective, Artificial Intelligence, vol. 17, 1981, pp.185-203.

J.L. Barron, D.J. Fleet, and S.S. Beauchemin, “Performance of Optical Flow Techniques,” in International Journal of Computer Vision, February 1994, vol. 12(1), pp. 43-77.