motion estimation i
TRANSCRIPT
![Page 2: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/2.jpg)
Last time
β’ Motion perception
β’ Motion representation
β’ Parametric motion: Lucas-Kanade
β’ Dense optical flow: Horn-Schunck
ππ’ππ£= βππ₯πππ₯ ππ₯
πππ¦
ππ₯πππ¦ ππ¦
πππ¦
β1ππ₯πππ‘ππ¦πππ₯
ππ₯2 + πΌπ ππ₯ππ¦
ππ₯ππ¦ ππ¦2 + πΌπ
ππ= βππ₯ππππ¦ππ
![Page 3: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/3.jpg)
Who are they?
Berthold K. P. Horn Takeo Kanade
![Page 4: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/4.jpg)
Content
β’ Robust optical flow estimation
β’ Applications
β’ Feature matching
β’ Discrete optical flow
β’ Layer motion analysis
β’ Other representations
![Page 5: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/5.jpg)
Content
β’ Robust optical flow estimation
β’ Applications
β’ Feature matching
β’ Discrete optical flow
β’ Layer motion analysis
β’ Other representations
![Page 6: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/6.jpg)
Spatial regularity
β’ Horn-Schunck is a Gaussian Markov random field (GMRF)
πΌπ₯π’ + πΌπ¦π£ + πΌπ‘2+ πΌ π»π’ 2 + π»π£ 2 ππ₯ππ¦
β’ Spatial over-smoothness is caused by the quadratic smoothness term
β’ Nevertheless, real optical flow fields are sparse!
π’ π’π₯ π’π¦
π£ π£π₯ π£π¦
![Page 7: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/7.jpg)
Data term
β’ Horn-Schunck is a Gaussian Markov random field (GMRF)
πΌπ₯π’ + πΌπ¦π£ + πΌπ‘2+ πΌ π»π’ 2 + π»π£ 2 ππ₯ππ¦
β’ Quadratic data term implies Gaussian white noise
β’ Nevertheless, the difference between two corresponded pixels is caused byβ Noise (majority)
β Occlusion
β Compression error
β Lighting change
β β¦
β’ The error function needs to account for these factors
![Page 8: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/8.jpg)
Noise model
β’ Explicitly model the noise π
πΌ2 π₯ + π’, π¦ + π£ = πΌ1 π₯, π¦ + π
β’ It can be a mixture of two Gaussians, inlier and outlier
π ~ ππ 0, ππ2 + 1 β π π(0, ππ
2)
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
5
10
15
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
5
10
15
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
5
10
15
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1-4
-2
0
2
4
6
8
10
12
14
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1-4
-2
0
2
4
6
8
10
12
14
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1-4
-2
0
2
4
6
8
10
12
14
Inlier Outlier Mixture (π = 0.9)
Potential
![Page 9: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/9.jpg)
More components in the mixture
β’ Consider a Gaussian mixture model
π~1
π
π=1
πΎ
πππ(0, ππ 2)
β’ Varying the decaying rate π, we obtain a variety of potential functions
-4 -3 -2 -1 0 1 2 3 4-5
0
5
10
15
20
25
30
-4 -3 -2 -1 0 1 2 3 4-2
0
2
4
6
8
10
12
π = 0.1 π = 0.9
![Page 10: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/10.jpg)
Typical error functions
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
3
L2 normπ π§ = π§2
L1 normπ π§ = |π§|
Truncated L1 normπ π§ = min( π§ , π)
Lorentzianπ π§ = log(1 + πΎπ§2)
![Page 11: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/11.jpg)
Robust statistics
β’ Traditional L2 norm: only noise, no outlier
β’ Example: estimate the average of 0.95, 1.04, 0.91, 1.02, 1.10, 20.01
β’ Estimate with minimum error
π§β = argminπ§ π π π§ β π§π
β L2 norm: π§β = 4.172
β L1 norm: π§β = 1.038
β Truncated L1: π§β = 1.0296
β Lorentzian: π§β = 1.0147
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
3
L2 normπ π§ = π§2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
3
L1 normπ π§ = |π§|
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
3
Truncated L1 normπ π§ = min( π§ , π)
-2 -1.5 -1 -0.5 0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
3
Lorentzianπ π§ = log(1 + πΎπ§2)
![Page 12: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/12.jpg)
The family of robust power functions
β’ Can we directly use L1 norm π π§ = π§ ?β Derivative is not continuous
β’ Alternative forms
β L1 norm: π π§2 = π§2 + π2
β Sub L1: π π§2; π = π§2 + π2 π, π < 0.5
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
1.2
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
1.2
|π§|
π§2 + π2
π = 0.5π = 0.4π = 0.3π = 0.2
![Page 13: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/13.jpg)
Modification to Horn-Schunck
β’ Let x = (π₯, π¦, π‘), and w x = π’ x , π£ x , 1 be the flow vector
β’ Horn-Schunck (recall)
πΌπ₯π’ + πΌπ¦π£ + πΌπ‘2+ πΌ π»π’ 2 + π»π£ 2 ππ₯ππ¦
β’ Robust estimation
π πΌ x + w β πΌ x 2 + πΌπ π»π’ 2 + π»π£ 2 ππ₯ππ¦
β’ Robust estimation with Lucas-Kanade
π β π πΌ x + w β πΌ x 2 + πΌπ π»π’ 2 + π»π£ 2 ππ₯ππ¦
![Page 14: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/14.jpg)
A unifying framework
β’ The robust object function
π β π πΌ x + w β πΌ x 2 + πΌπ π»π’ 2 + π»π£ 2 ππ₯ππ¦
β Lucas-Kanade: πΌ = 0, π π§2 = π§2
β Robust Lucas-Kanade: πΌ = 0, π π§2 = π§2 + π2
β Horn-Schunck: π = 1, π π§2 = π§2, π π§2 = π§2
β’ One can also learn the filters (other than gradients), and robust function π β , π(β ) [Roth & Black 2005]
![Page 15: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/15.jpg)
Derivation strategies
β’ Euler-Lagrange β Derive in continuous domain, discretize in the end
β Nonlinear PDEβs
β Outer and inner fixed point iterations
β Limited to derivative filters; cannot generalize to arbitrary filters
β’ Energy minimizationβ Discretize first and derive in matrix form
β Easy to understand and derive
β Iteratively reweighted least square (IRLS)
β’ Variational optimization
β’ Euler-Lagrange = Variational optimization = IRLS
![Page 16: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/16.jpg)
Iteratively reweighted least square (IRLS)
β’ Let π π§ = π§2 + π2 π be a robust function
β’ We want to minimize the objective function
Ξ¦ ππ₯ + π =
π=1
π
π ππππ₯ + ππ
2
where π₯ β βπ , π΄ = π1 π2β―πππ β βπΓπ , π β βπ
β’ By setting πΞ¦
ππ₯= 0, we can derive
πΞ¦
ππ₯β
π=1
π
πβ² ππππ₯ + ππ
2ππππ₯ + ππ ππ
= π=1
π
π€ππππππ₯ππ +π€ππππππ
= π=1
π
ππππ€πππ₯ππ + πππ€ππππ
= πππππ₯ + ππππ
π€ππ = πβ² ππ
ππ₯ + ππ2
π = diag Ξ¦β²(ππ₯ + π)
![Page 17: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/17.jpg)
Iteratively reweighted least square (IRLS)
β’ Derivative: πΞ¦
ππ₯= πππππ₯ + ππππ = 0
β’ Iterate between reweighting and least square
β’ Convergence is guaranteed (local minima)
1. Initialize π₯ = π₯0
2. Compute weight matrix π = diag Ξ¦β²(ππ₯ + π)
3. Solve the linear system πππππ₯ = βππππ
4. If π₯ converges, return; otherwise, go to 2
![Page 18: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/18.jpg)
IRLS for robust optical flow
β’ Objective function
π β π πΌ x + w β πΌ x 2 + πΌπ π»π’ 2 + π»π£ 2 ππ₯ππ¦
β’ Discretize, linearize and increment
π₯,π¦
π β π πΌπ‘ + πΌπ₯ππ’ + πΌπ¦ππ£2+ πΌπ π» π’ + ππ’ 2 + π» π£ + ππ£ 2
β’ IRLS (initialize ππ’ = ππ£ = 0)β Reweight:
β Least square:
πΏπ₯π₯β² = diag π β πβ²ππ₯ππ₯ , πΏπ₯π¦
β² = diag π β πβ²ππ₯ππ¦ ,
πΏπ¦π¦β² = diag π β πβ²ππ¦ππ¦ , πΏπ₯π‘
β² = diag π β πβ²ππ₯ππ‘ ,
πΏπ¦π‘β² = diag π β πβ²ππ¦ππ‘ , π = ππ₯
ππ½β²ππ₯ + ππ¦ππ½β²ππ¦
πΏπ₯π₯β² + πΌπ πΏπ₯π¦
β²
πΏπ₯π¦β² πΏπ¦π¦
β² + πΌπππππ= βπΏπ₯π‘β² + πΌππ
πΏπ¦π‘β² + πΌππ
![Page 19: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/19.jpg)
Whatβs changed?
β’ Optical flow with robust function
β’ Horn-Schunck
πΏπ₯π₯β² = diag π β πβ²ππ₯ππ₯ , πΏπ₯π¦
β² = diag π β πβ²ππ₯ππ¦ ,
πΏπ¦π¦β² = diag π β πβ²ππ¦ππ¦ , πΏπ₯π‘
β² = diag π β πβ²ππ₯ππ‘ ,
πΏπ¦π‘β² = diag π β πβ²ππ¦ππ‘ , π = ππ₯
ππ½β²ππ₯ + ππ¦ππ½β²ππ¦
πΏπ₯π₯β² + πΌπ πΏπ₯π¦
β²
πΏπ₯π¦β² πΏπ¦π¦
β² + πΌπππππ= βπΏπ₯π‘β² + πΌππ
πΏπ¦π‘β² + πΌππ
ππ₯2 + πΌπ ππ₯ππ¦
ππ₯ππ¦ ππ¦2 + πΌπ
ππππ= βππ₯ππ‘ + πΌππππ¦ππ‘ + πΌππ
![Page 20: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/20.jpg)
Example
Flow visualization
Coarse-to-fine LK with median filtering
Input two frames
Horn-Schunck
Robust optical flow
![Page 21: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/21.jpg)
Content
β’ Robust optical flow estimation
β’ Applications
β’ Feature matching
β’ Discrete optical flow
β’ Layer motion analysis
β’ Other representations
![Page 22: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/22.jpg)
Video stabilization
![Page 23: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/23.jpg)
Video denoising
![Page 24: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/24.jpg)
Video super resolution
![Page 25: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/25.jpg)
Content
β’ Robust optical flow estimation
β’ Applications
β’ Feature matching
β’ Discrete optical flow
β’ Layer motion analysis
β’ Contour motion analysis
β’ Obtaining motion ground truth
![Page 26: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/26.jpg)
Block matching
β’ Both Horn-Schunck and Lucas-Kanade are sub-pixel accuracy algorithms
β’ But in practice we may not need sub-pixel accuracy
β’ MPEG: 16 Γ 16 block matching using MMSE
β’ H264: variable block size and quarter-pixel precision
![Page 27: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/27.jpg)
Tracking reliable features
β’ Idea: no need to work on ambiguous region pixels (flat regions & line structures)
β’ Instead, we can track features and then propagate the tracking to ambiguous pixels
β’ Good features to track [Shi & Tomashi 94]
β’ Block matching + Lucas-Kanade refinement
ππ’ππ£= βππ₯πππ₯ ππ₯
πππ¦
ππ₯πππ¦ ππ¦
πππ¦
β1ππ₯πππ‘ππ¦πππ‘
![Page 28: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/28.jpg)
Feature detection & tracking
![Page 29: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/29.jpg)
From sparse to dense
β’ Interpolation: given values {ππ} at { π₯π , π¦π }, reconstruct a smooth plane π(π₯, π¦)
β’ Membrane model (first order smoothness)
π
π€π π π₯π , π¦π β ππ2 + πΌ ππ₯
2 + ππ¦2 ππ₯ππ¦
β’ Thin plate model (second order smoothness)
π
π€π π π₯π , π¦π β ππ2 + πΌ ππ₯π₯
2 + ππ₯π¦2 + ππ¦π¦
2 ππ₯ππ¦
![Page 30: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/30.jpg)
Membrane vs. thin plate
![Page 31: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/31.jpg)
Dense flow field from sparse tracking
![Page 32: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/32.jpg)
Pros and Cons of Feature Matching
β’ Prosβ Efficient (a few feature points vs. all pixels)
β Reliable (with advanced feature descriptors)
β’ Consβ Independent tracking (tracking can be unreliable)
β Not all information is used (may not capture weak features)
β’ How to improveβ Track every pixel with uncertainty
β Integrate spatial regularity (neighboring pixels go together)
![Page 33: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/33.jpg)
Content
β’ Robust optical flow estimation
β’ Applications
β’ Feature matching
β’ Discrete optical flow
β’ Layer motion analysis
β’ Other representations
![Page 34: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/34.jpg)
Discrete optical flow
β’ The objective function is similar to that of continuous flow
β’ x = (π₯, π¦) is pixel coordinate, w = (π’, π£) is flow vector
πΈ w =
x
min( πΌ1 x β πΌ2 x + w x , π‘) +
x
π( π’ x + π£ x ) +
x1,x2 βπ
min πΌ π’ x1 β π’ x2 , π + min πΌ π£ x1 β π£ x2 , π
β’ Truncated L1 norms:β Account for outliers in the data term
β Encourage piecewise smoothness in the smoothness term
Data term
Small displacement
Spatial regularity
![Page 35: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/35.jpg)
Decoupled smoothness
Smoothness O(L4)
Smoothness: O(L2)
Smoothness: O(L2)
Data term: O(L2) Data term: O(L2)
Coupled smoothness
Decoupled smoothnessSmoothness: O(L)
Smoothness: O(L)
![Page 36: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/36.jpg)
Combinatorial optimization on graph
πΈ w =
x
min( πΌ1 x β πΌ2 x + w x , π‘) +
x
π( π’ x + π£ x ) +
x1,x2 βπ
min πΌ π’ x1 β π’ x2 , π + min πΌ π£ x1 β π£ x2 , π
β’ Optimization strategiesβ Belief propagation
β Graph cuts
β MCMC (simulated annealing)
![Page 37: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/37.jpg)
Dual-layer belief propagation
[Shekhovtsov et al. CVPR 07]
Horizontal flow u
Vertical flow v
u
v
w = (π’, π£)
Data term
min( πΌ1 x β πΌ2 x + w , π‘)
Smoothness term on u
min πΌ π’ x1 β π’ x2 , π
Smoothness term on v
min πΌ π£ x1 β π£ x2 , π
Regularization term on u π|π’ x |
Regularization term on v π|π£ x |
![Page 38: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/38.jpg)
Dual-layer belief propagation
u
v
Message πππ: given
all the information at node π, predict the distribution at node π
Update within π’ plane
π π
![Page 39: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/39.jpg)
Dual-layer belief propagation
u
v
Update within π£ plane
![Page 40: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/40.jpg)
Dual-layer belief propagation
u
v
Update from π’ plane to π£ plane
π π
![Page 41: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/41.jpg)
Dual-layer belief propagation
u
v
Update from π£ plane to π’ plane
![Page 42: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/42.jpg)
Example
Flow visualization
Coarse-to-fine LK with median filtering
Input two frames
Robust optical flow
Discrete optical flow
![Page 43: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/43.jpg)
Content
β’ Robust optical flow estimation
β’ Applications
β’ Feature matching
β’ Discrete optical flow
β’ Layer motion analysis
β’ Other representations
![Page 44: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/44.jpg)
Layer representation
β’ Optical flow field is able to model complicated motion
β’ Different angle: a video sequence can be a composite of several moving layers
β’ Layers have been widely used
β Adobe Photoshop
β Adobe After Effect
β’ Compositing is straightforward, but inference is hard
Wang & Adelson, 1994
![Page 45: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/45.jpg)
Wang & Adelson, 1994
β’ Strategyβ Obtaining dense optical flow field
β Divide a frame into non-overlapping regions and fit affine motion for each region
β Cluster affine motions by k-means clustering
β Region assignment by hypothesis testing
β Region splitter: disconnected regions are separated
![Page 46: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/46.jpg)
Results
Optical flow field Clustering to affine regions Clustering with error metric
Three layers with affine motion superimposed
Reconstructed background layer
Flower garden
![Page 47: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/47.jpg)
Weiss & Adelson, 1996
β’ Chicken & egg problemβ Good motion β good segmentation
β Good segmentation β good motion
β’ We donβt have either of them, so iterate!
β’ Perceptually organized expectation & maximization (POEM)β E-step: estimate the motion parameter of each layer
β M-step: estimate the likelihood that a pixel belongs to each of the layers (segmentation)
![Page 48: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/48.jpg)
Liu et. al. 2005
β’ Reliable layer segmentation for motion magnification
β’ Layer segmentation pipeline
Feature point
tracking
Trajectory
clustering
Dense optical
flow interpolation
Layer
segmentation
![Page 49: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/49.jpg)
Normalized Complex Correlation
β’ The similarity metric should be independent of phase and magnitude
β’ Normalized complex correlation
tt
t
tCtCtCtC
tCtCCCS
)()()()(
|)()(|),(
2211
2
21
21
Feature point
tracking
Trajectory
clustering
Dense optical
flow interpolation
Layer
segmentation
![Page 50: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/50.jpg)
Spectral Clustering
Affinity matrix Clustering Reordering of affinity matrix
Two clustersTrajectory
Tra
jecto
ry
Feature point
tracking
Trajectory
clustering
Dense optical
flow interpolation
Layer
segmentation
![Page 51: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/51.jpg)
Clustering Results
Feature point
tracking
Trajectory
clustering
Dense optical
flow interpolation
Layer
segmentation
![Page 52: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/52.jpg)
Flow vectors of
clustered sparse
feature points
Dense optical flow
field of cluster 1
(leaves)
Dense optical flow
field of cluster 2
(swing)
From Sparse Feature Points to Dense Optical Flow Field
Cluster 1: leaves
Cluster 2: swing
β’ Interpolate dense optical flow field using locally weighted linear regression
Feature point
tracking
Trajectory
clustering
Dense optical
flow interpolation
Layer
segmentation
![Page 53: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/53.jpg)
Motion Layer Assignment
β’ Assign each pixel to a motion cluster layer, using four cues:
β Motion likelihoodβconsistency of pixelβs intensity if it moves with
the motion of a given layer (dense optical flow field)
β Color likelihoodβconsistency of the color in a layer
β Spatial connectivityβadjacent pixels favored to belong the same
group
β Temporal coherenceβlabel assignment stays constant over time
β’ Energy minimization using graph cuts
Feature point
tracking
Trajectory
clustering
Dense optical
flow interpolation
Layer
segmentation
![Page 54: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/54.jpg)
![Page 55: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/55.jpg)
How good is optical flow?
β’ The AAE (average angular error) race on the Yosemitesequence for over 15 years
#I. Austvoll. Lecture Notes in Computer Science, 2005*Brox et al. ECCV, 2004.
Yosemite sequence State-of-the-art optical flow*
Improvement#
![Page 56: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/56.jpg)
Middlebury flow database
Baker et. al. A Database and Evaluation Methodology for Optical Flow. ICCV 2007
![Page 57: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/57.jpg)
Middlebury flow database
![Page 58: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/58.jpg)
Human-assisted motion annotation
β’ Ground truth is essential to progress the field
β’ A first step in computer vision to obtain ground-truth motion for arbitrary scenes (a different approach from Baker et al. 2007)
β’ An interactive system to combine human perception and the state-of-the-art computer vision algorithms to annotate motion
Liu et al. Human-assisted motion annotation. CVPR 2008
![Page 59: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/59.jpg)
Demo: interactive layer segmentation
![Page 60: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/60.jpg)
Demo: interactive motion labeling
![Page 61: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/61.jpg)
Motion database of natural scenes
Color map
Bruhn et al. Lucas/Kanade meets Horn/Schunck: combining local and global optical flow methods. IJCV, 2005
![Page 62: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/62.jpg)
Content
β’ Robust optical flow estimation
β’ Applications
β’ Feature matching
β’ Discrete optical flow
β’ Layer motion analysis
β’ Other representations
![Page 63: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/63.jpg)
Particle video
P. Sand and S. Teller. Particle Video: Long-Range Motion Estimation using Point Trajectories. CVPR 2006
![Page 64: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/64.jpg)
Particle video
![Page 65: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/65.jpg)
Seemingly Simple Examples
Kanizsa square
From real video
![Page 66: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/66.jpg)
Output from the State-of-the-Art Optical Flow Algorithm
T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004
Optical flow fieldKanizsa square
![Page 67: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/67.jpg)
Output from the State-of-the-Art Optical Flow Algorithm
T. Brox et al. High accuracy optical flow estimation based on a theory for warping. ECCV 2004
Optical flow field
Dancer
![Page 68: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/68.jpg)
Optical flow representation: aperture problem
Corners Lines Flat regions
Spurious junctions Boundary ownership Illusory boundaries
![Page 69: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/69.jpg)
Optical flow representation: aperture problem
Corners Lines Flat regions
Spurious junctions Boundary ownership Illusory boundaries
We need motion representation beyond pixel level!
![Page 70: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/70.jpg)
Challenge: Textureless Objects under Occlusion
β’ Corners are not always trustworthy (junctions)
β’ Flat regions do not always move smoothly (discontinuous at illusory boundaries)
β’ How about boundaries?
β Easy to detect and track for textureless objects
β Able to handle junctions with illusory boundaries
![Page 71: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/71.jpg)
Frame 1
![Page 72: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/72.jpg)
Frame 2
![Page 73: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/73.jpg)
Extracted boundary fragments
![Page 74: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/74.jpg)
Optical flow from Lucas-Kanade algorithm
![Page 75: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/75.jpg)
Estimated motion by our system, after grouping
![Page 76: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/76.jpg)
Boundary grouping and illusory boundaries (frame 1)
![Page 77: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/77.jpg)
Boundary grouping and illusory boundaries (frame 2)
![Page 78: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/78.jpg)
Rotating Chair
![Page 79: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/79.jpg)
Frame 1
![Page 80: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/80.jpg)
Frame 2
![Page 81: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/81.jpg)
Extracted boundary fragments
![Page 82: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/82.jpg)
Estimated flow field from Brox et al.
![Page 83: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/83.jpg)
Estimated motion by our system, after grouping
![Page 84: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/84.jpg)
Boundary grouping and illusory boundaries (frame 1)
![Page 85: Motion Estimation I](https://reader030.vdocuments.site/reader030/viewer/2022012603/6199485a17e354378118b26c/html5/thumbnails/85.jpg)
Boundary grouping and illusory boundaries (frame 2)