Transcript
Page 1: Optical flow and Tracking

Optical flow and Tracking

CISC 649/849Spring 2009

University of Delaware

Page 2: Optical flow and Tracking

Outline

• Fusionflow• Joint Lucas Kanade Tracking• Some practical issues in tracking

Page 3: Optical flow and Tracking

What smoothing to choose?

Page 4: Optical flow and Tracking

Stereo Matching results…

Page 5: Optical flow and Tracking

Difficulties in optical flow

• Cannot directly apply belief propagation or graph cut– Number of labels too high

• Brightness variation higher than stereo matching

Page 6: Optical flow and Tracking

Can we combine different flows?

???

Page 7: Optical flow and Tracking

Formulation as a labeling problem

• Given flows x0 and x1, find a labeling y• Combine the flows to get a new flow xf

Page 8: Optical flow and Tracking

Graph Cut formulation

Page 9: Optical flow and Tracking

Graph cut

Page 10: Optical flow and Tracking

Proposal Solutions

• Horn and Shunck with different smoothing

• Lucas Kanade with different window sizes

• Shifted versions of above

Page 11: Optical flow and Tracking

Discrete Optimization

• Choose one of the proposals randomly as initial flow field

• Visit other proposals in random order and update labeling

• Combine the proposals according to the labeling to give fused estimate

Page 12: Optical flow and Tracking

Continuous Optimization

• Some areas may have same solution in all proposals

• Use conjugate gradient method on the energy function to decrease the energy further

• Use bicubic interpolation to calculate gradient

Page 13: Optical flow and Tracking

Results

Page 14: Optical flow and Tracking

Recap…Lucas Kanade

(sparse feature tracking)Horn Schunck

(dense optic flow)

• assumes unknown displacement u of a pixel is constant within some neighborhood• i.e., finds displacement of a small window centered around a pixel by minimizing:

• regularizes the unconstrained optic flow equation by imposing a global smoothness term• computes global displacement functions u(x, y) v(x, y) by minimizing:

• λ: regularization parameter, Ω: image domain• minimum of the functional is found by solving the corresponding Euler-Lagrange equations, leading to:

• denotes convolution with an integration window of size ρ • differentiating with respect to u and v, setting the derivatives to zero leads to a linear system:

Page 15: Optical flow and Tracking

Limitations of Lucas-Kanade Tracking

• Tracks only those features whose minimum eigenvalue is greater than a fixed threshold

• Do edges satisfy this condition?• Are edges bad for tracking?• How can this be corrected?

Page 16: Optical flow and Tracking

Ambiguity on edges

?

Page 17: Optical flow and Tracking

Joint Lucas Kanade Tracking

Page 18: Optical flow and Tracking

Matrix Formulation

Page 19: Optical flow and Tracking

Iterative Solution

Page 20: Optical flow and Tracking

Joint Lucas Kanade TrackingFor each feature i,1. Initialize ui ← (0, 0)T

2. Initialize iFor pyramid level n − 1 to 0 step −1,1. For each feature i, compute Zi

2. Repeat until convergence: (a) For each feature i, i. Determine ii. Compute the difference It between the first image and the shifted second image: It (x, y) = I1(x, y) − I2(x + ui , y + vi) iii. Compute ei

iv. Solve Zi u′i = ei for incremental motion u’i

v. Add incremental motion to overall estimate: ui ← ui + u′i

3. Expand to the next level: ui ← ui, where is the pyramid scale factor

Page 21: Optical flow and Tracking

How to find mean flow?

• Average of neighboring features?– Too much variation in the flow vectors even if the

motion is rigid

• Calculate an affine motion model with neighboring features weighted according to their distance from tracked feature

Page 22: Optical flow and Tracking

What features to track?

Given the Eigen values of a window are emax and emin

• Standard Lucas Kanade chooses windows with emin > Threshold

• This restricts the features to corners• Joint Lucas Kanade chooses windows with

max(emin,K emax ) > Threshold where K<1.

Page 23: Optical flow and Tracking

Results

LK

JLK

Page 24: Optical flow and Tracking

Observations

• JLK performs better on edges and untextured regions

• Aperture problem is overcome on edges

• Future improvements– Does not handle occlusions– Does not account for motion discontinuities

Page 25: Optical flow and Tracking

Some issues in tracking

• Appearance change• Sub pixel accuracy• Lost Features/Occlusion

Page 26: Optical flow and Tracking

Further reading

• Joint Tracking of Features and Edges. Stanley T. Birchfield and Shrinivas J. Pundlik. CVPR 2008

• FusionFlow: Discrete-Continuous Optimization for Optical Flow Estimation. V. Lempitsky, S. Roth, C. Rother. CVPR 2008

• The template update problem, Matthews, L.; Ishikawa, T.; Baker, S. PAMI 2004


Top Related