Effective Optical Flow Estimation

Jan Kamenický 21.10.2011

Motivation

Usage

• Motion detection• Object segmentation• Video encoding (compression)• Stereo disparity measurement

Optical flow equation

• Color constancy

• Taylor series

• Optical flow equation

Estimating optical flow

• Basic equation

• Local methods– Lucas & Kanade• flow field is locally constant (or affine)• least squares minimization• cannot handle interior parts of objects

Estimating optical flow

• Basic equation

• Global methods– Horn & Schunk

• more sensitive to noisedata term smoothing term

• Data term– non-homogenous surface (shading, reflections)– non-flat scene / non-uniform lighting– spatial discontinuities

• Smoothness term– discontinuities (moving objects boundaries)

Problems

Estimating partial derivatives

• Discrete approximation by differences– forward, backward – not exact– use 2x2x2 cube in (x,y,t) space• compute the difference as an average of 4 adjacent first

order differences

– use larger support• e.g. [1, -8, 0, 8, -1]/12

i i+1

j

j+1

kk+1

Data term

• L2 norm

• L1 norm

• Many modifications– generalized Charbonnier

Regularization term

• Enforces smooth flow field• Similar norms can be used– L2, L1 (total variation), …

• Other possibilities– Laplacian instead of gradient

Dealing with larger displacements

• Smoothing (blurring)– usually Gaussian kernel– decreases flow field accuracy

• Pyramidal approach– compute flow on down-sampled images– up-sample the flow to next level– compute the warping (using the optical flow)– repeat

More optimizations

• Graduated non-convexity– iteratively move from convex energy function to

the more robust non-convex form

• Median filtering (5x5)– weighted modification

• More warping steps on one pyramid level

OF methods comparison

• Optical flow estimation benchmark– http://vision.middlebury.edu/flow/

• Average end-point error

References

• Main described method– D. Sun, S. Roth, M. J. Black: Secrets of Optical Flow

Estimation and Their Principles, CVPR 2010– http://www.cs.brown.edu/~black/code.html