Optical flow and Tracking

Download Optical flow and Tracking

Post on 31-Dec-2015




5 download

Embed Size (px)


Optical flow and Tracking. CISC 649/849 Spring 2009 University of Delaware. Outline. Fusionflow Joint Lucas Kanade Tracking Some practical issues in tracking. What smoothing to choose?. Stereo Matching results. Difficulties in optical flow. - PowerPoint PPT Presentation


  • Optical flow and TrackingCISC 649/849Spring 2009University of Delaware

  • OutlineFusionflowJoint Lucas Kanade TrackingSome practical issues in tracking

  • What smoothing to choose?

  • Stereo Matching results

  • Difficulties in optical flowCannot directly apply belief propagation or graph cutNumber of labels too highBrightness variation higher than stereo matching

  • Can we combine different flows????

  • Formulation as a labeling problemGiven flows x0 and x1, find a labeling yCombine the flows to get a new flow xf

  • Graph Cut formulation

  • Graph cut

  • Proposal SolutionsHorn and Shunck with different smoothing

    Lucas Kanade with different window sizes

    Shifted versions of above

  • Discrete OptimizationChoose one of the proposals randomly as initial flow field

    Visit other proposals in random order and update labeling

    Combine the proposals according to the labeling to give fused estimate

  • Continuous OptimizationSome areas may have same solution in all proposalsUse conjugate gradient method on the energy function to decrease the energy furtherUse bicubic interpolation to calculate gradient

  • Results

  • RecapLucas Kanade(sparse feature tracking)Horn Schunck(dense optic flow) assumes unknown displacement u of a pixel is constant within some neighborhood i.e., finds displacement of a small window centered around a pixel by minimizing: regularizes the unconstrained optic flow equation by imposing a global smoothness term computes global displacement functions u(x, y) v(x, y) by minimizing: : regularization parameter, : image domain minimum of the functional is found by solving the corresponding Euler-Lagrange equations, leading to:

  • Limitations of Lucas-Kanade TrackingTracks only those features whose minimum eigenvalue is greater than a fixed thresholdDo edges satisfy this condition?Are edges bad for tracking?How can this be corrected?

  • Ambiguity on edges?

  • Joint Lucas Kanade Tracking

  • Matrix Formulation

  • Iterative Solution

  • Joint Lucas Kanade TrackingFor each feature i,1. Initialize ui (0, 0)T2. Initialize iFor pyramid level n 1 to 0 step 1,1. For each feature i, compute Zi2. Repeat until convergence: (a) For each feature i, i. Determine ii. Compute the difference It between the first image and the shifted second image: It (x, y) = I1(x, y) I2(x + ui , y + vi) iii. Compute ei iv. Solve Zi ui = ei for incremental motion ui v. Add incremental motion to overall estimate: ui ui + ui3. Expand to the next level: ui aui, where a is the pyramid scale factor

  • How to find mean flow?Average of neighboring features?Too much variation in the flow vectors even if the motion is rigidCalculate an affine motion model with neighboring features weighted according to their distance from tracked feature

  • What features to track?Given the Eigen values of a window are emax and eminStandard Lucas Kanade chooses windows with emin > ThresholdThis restricts the features to cornersJoint Lucas Kanade chooses windows with max(emin,K emax ) > Threshold where K
  • ResultsLKJLK

  • ObservationsJLK performs better on edges and untextured regionsAperture problem is overcome on edges

    Future improvementsDoes not handle occlusionsDoes not account for motion discontinuities

  • Some issues in trackingAppearance changeSub pixel accuracyLost Features/Occlusion

  • Further readingJoint Tracking of Features and Edges. Stanley T. Birchfield and Shrinivas J. Pundlik. CVPR 2008 FusionFlow: Discrete-Continuous Optimization for Optical Flow Estimation. V. Lempitsky, S. Roth, C. Rother. CVPR 2008The template update problem, Matthews, L.; Ishikawa, T.; Baker, S. PAMI 2004



View more >