Motion Detail Preserving Optical Flow Estimation

Li Xu1, Jiaya Jia1, Yasuyuki Matsushita2

1 The Chinese University of Hong Kong 2 Microsoft Research Asia

Conventional Optical Flow

• Middlebury Benchmark [Baker et al. 07]• Dominant Scheme: Coarse-to-Fine Warping

Large Displacement Optical Flow

• Region Matching [Brox et al. 09, 10]• Discrete Local Search [Steinbrucker et al. 09]

Both Large and Small Motion Exist

• Capture large motion• Preserve sub-pixel accuracy

Our Work

• Framework– Extended coarse-to-fine motion estimation for both

large and small displacement optical flow• Model– A new data term to selectively combine constraints

• Solver– Efficient numerical solver for discrete-continuous

optimization

Outline

• Framework– Extended coarse-to-fine motion estimation for both

large and small displacement optical flow• Model– A new data term to selectively combine constraints

• Solver– Efficient numerical solver for discrete-continuous

optimization

The Multi-scale Problem

The Multi-scale Problem

Ground truthGround truthGround truth

The Multi-scale Problem

Ground truthGround truthGround truth

Ground truth

…Estimate EstimateEstimate

Ground truthGround truth

The Multi-scale Problem

• Large discrepancy between initial values and optimal motion vectors

• Our solution – Improve flow initialization to reduce the reliance

on the initialization from coarser levels

Extended Flow Initialization

• Sparse feature matching for each level

Extended Flow Initialization

• Identify missing motion vectors

Extended Flow Initialization

• Identify missing motion vectors

Extended Flow Initialization

…

…

Extended Flow Initialization

Fuse

Outline

• Framework: extended initialization for coarse-to-fine motion estimation

• Model: selective data term • Efficient numerical solver

Data Constraints

• AverageI

xI

1 1(u, x) (u, x) (u, x)2 2DE D D

I 2 1(u, x) (x u) (x)I ID

I 2 1(u, x) (x u) (I I x)D • Gradient constancy

• Color constancy

I 2 1(u, x) (x u) (I I x)D

• Pixels moving out of shadow

Problems

pI 1 1p(u , ) 6.63D

• Color constancy is violated

I Ip1 1 p1 11 (u , ) (u , ) = 3.482

p pD D

• Average:

p1u : ground truth motion of p1

• Gradient constancy holdsp 1I 1 p(u , ) 0.32D

• Pixels undergoing rotational motion

Problems

• Color constancy holds

• Gradient constancy is violatedp2u : ground truth motion of p2

p 2I 2 p(u , ) 4.20D

• Average:

I Ip2 2 p2 21 (u , ) (u , ) = 2.242

p pD D

pI 2 2p(u , ) 0.29D

Our Proposal

• Selectively combine the constraints

where

I Ix

(u, ) (x) (u,x) (1 (x)) (u,x)DE D D 2(x) : {0,1}

I Ix

(u, ) (u,x(x) 1) ( ) (u,x)(x)DE D D

Comparisons

RubberWhale Urban22

2.5

3

3.5

4

4.5

5

colorgradientaverageours

AAE

Outline

• Framework: extended initialization for coarse to fine motion estimation

• Model: selective data term

• Efficient numerical solver

I Ix

(x) (u, x) (1 (x)) (u, x)D D

Energy Functions and Solver

• Total energy

• Probability of a particular state of the system

(u, )1(u, ) EP eZ

I Ix

(u, ) (x) (u, x) (1 (x)) (u, x) ( u, x)E D D S

(u, )1(u, ) EP eZ

(u, )1(u, ) EP eZ

Ix

I(x) (u, x) (1 (x)) (u, x)(u, ) ( u, x)E SD D

Mean Field Approximation

• Partition function

• Sum over all possible values of α

(u, )

{u} { 0,1}

EZ e

I Ix

( u,x) (u, x) ((x) (x)

{ 0,

1 ) (u, x)

{u 1}}

x

S D D

e e

(u, x)(u, x) II

x

1{ ( u,x) ln( )}

{u}

DDS e e

e

. . .

The effective potential Eeff (u) [Geiger & Girosi, 1989]

• Optimal condition (Euler-Lagrange equations)

• It decomposes to

II (u, x)(u, x)

x

1(u) ( u,x) ln( )DDeffE S e e

I I

I II I

(u,x) (u,x)

u I u I(u,x) (u,x)(u,x) (u,x)

u

(u, x) (u, x)

div( ( u,x)) 0

D D

D DD D

e eD De e e e

S

I I( (u,x) (u,x))

1(x)1 D De

u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S

I I

I II I

(u,x) (u,x)

(u,x) (u,u I u I

u

x)(u,x) (u,x)(u, x) (u, x)

div( ( u,x)) 0

D D

D DD D

e ee e e e

D D

S

( )x 1 ( )x

{

I I( (u,x) (u,x))

1(x)1 D De

u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S {

Algorithm Skeleton

• For each level

• Extended Flow Initialization (QPBO)• Continuous Minimization (Iterative reweight)– Update– Compute flow field (Variable Splitting)

I I( (u,x) (u,x))

1(x)1 D De

u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S {

Results

Averaging constraints Ours

Difference

Middlebury Dataset

EPE=0.74

Results from Different Steps

Coarse-to-fine

Extended coarse-to-fine

EPE=0.15 rank =1

EPE=0.24 rank =1

Large Displacement

Overlaid Input

Large Displacement

• Motion Estimates

Coarse-to-fine Our Result Warping Result

Comparison

• Motion Magnitude Maps

LDOP [Brox et al. 09 ] [Steinbrucker et al. 09] Ours

More Results

Overlaid Input

Conventional Coarse-to-fine Our Result

More Results

Overlaid Input

Coarse-to-fine Our Result

Conclusion

• Extended initialization (Framework)• Selective data term (Model)• Efficient numerical scheme (Solver)

• Limitations– Featureless motion details – Large occlusions

Thank you!

More Results

Overlaid Input

Coarse-to-fine Our Results