motion detail preserving optical flow estimation
DESCRIPTION
Motion Detail Preserving Optical Flow Estimation. Tzu ming Su Advisor : S.J.Wang. L. Xu , J. Jia , and Y. Matsushita. Motion detail preserving optical flow estimation. In CVPR, 2010. Outline. Previous Work Optical flow Conventional optical flow estimation. CCD. 3D motion vector. - PowerPoint PPT PresentationTRANSCRIPT
1
Tzu ming Su
Advisor : S.J.Wang
MOTION DETAIL PRESERVING OPTICAL FLOW ESTIMATION
2013/1/28
L. Xu, J. Jia, and Y. Matsushita. Motion detail preserving
optical flow estimation. In CVPR, 2010.
2
OUTLINE
2013/1/28
• Previous Work
• Optical flow
• Conventional optical flow estimation
3
MOTION FIELD
2013/1/28
• Definition : an ideal representation of 3D motion as it is projected onto a camera image.
3D motion vector
2D optical flow vector
CCD
4
MOTION FIELD
2013/1/28
• Applications :• Video enhancement : stabilization, denoising, super resolution
• 3D reconstruction : structure from motion (SFM)
• Video segmentation
• Tracking/recognition
• Advanced video editing (label propagation)
5
MOTION FIELD ESTIMATION
2013/1/28
• Optical flowRecover image motion at each pixel from spatio-temporal
image brightness variations
• Feature-trackingExtract visual features (corners, textured areas) and “track”
them over multiple frames
6
OPTICAL FLOW
2013/1/28
• Definition : the apparent motion of brightness patterns in the images
• Map flow vector to color
• Magnitude: saturation
• Orientation: hue
7
OPTICAL FLOW
2013/1/28
• Key assumptions
• Brightness constancy
• Small motion
• Spatial coherence
Remark : Brightness constancy is often violatedÞ Use gradient constancy for addition , both of them are called data constraint
8
BRIGHTNESS CONSISTENCY
2013/1/28
• 1-D case
Ix v It
9
BRIGHTNESS CONSISTENCY
x
)1,( txII ( x , t )
p?v
• 1-D case
2013/1/28
10
BRIGHTNESS CONSISTENCY
x
)1,( txII ( x , t )
p
xI
Spatial derivative
Temporal derivative v
• 1-D case
2013/1/28
11
BRIGHTNESS CONSISTENCY• 1-D case
• 2-D case
One equation, two velocity (u,v) unknowns…
u v
2013/1/28
12
APERTURE PROBLEM
2013/1/28
13
APERTURE PROBLEM
2013/1/28
14
APERTURE PROBLEM
2013/1/28
Time t
?Time t+dt
• We know the movement parallel to the direction of gradient , but not the movement orthogonal to the gradient
• We need additional constraints
15
CONVENTIONAL ESTIMATION
2013/1/28
• Use data consistency & additional constraint to estimate optical flow
• Horn-Schunck
• Minimize energy function with smoothness term
• Lucas-Kanade
• Minimize least square error function with local region coherence
16
HORN-SCHUNCK ESTIMATION
2013/1/28
• Imposing spatial smoothness to the flow field
• Adjacent pixels should move together as much as possible
• Horn & Schunck equation
17
HORN-SCHUNCK ESTIMATION
2013/1/28
• Use 2D Euler Lagrange
• Can be iteratively solved
18
COARSE TO FINE ESTIMATION
2013/1/28
• Optical flow is assumed to be small motions , but in fact most motions are not
• Solved by coarse to fine resolution
19
image Iimage It-1
COARSE TO FINE ESTIMATION
run iteratively
run iteratively...
2013/1/28
20
OUTLINE
2013/1/28
• Previous Work
• Contributions
• Extended Flow Initialization
• Selective data term
• Efficient optimization solver
• Experimental result
• Conclusion
21
OUTLINE
2013/1/28
• Previous Work
• Contributions
• Extended Flow Initialization
• Selective data term
• Efficient optimization solver
• Experimental result
• Conclusion
22
MUTI-SCALE PROBLEM
2013/1/28
• Conventional coarse to fine estimation can’t deal with large displacement.
• With different motion scales between foreground & background , even small motions can be miss detected.
23
MUTI-SCALE PROBLEM
2013/1/28
24
Ground truth
… Estimate EstimateEstimate
Ground truthGround truth
2013/1/28
25
MUTI-SCALE PROBLEM
2013/1/28
• Large discrepancy between initial values and optimal motion vectors
• Solution : Improve flow initialization to reduce the reliance on the initialization from coarser levels
262013/1/28
Sparse feature matching
Fusion
Dense nearest-neighbor patch matching
Selection
EXTENDED FLOW INITIALIZATION• Sparse feature matching for each level
28
EXTENDED FLOW INITIALIZATION• Identify missing motion vectors
2013/1/28
29
EXTENDED FLOW INITIALIZATION• Identify missing motion vectors
2013/1/28
30
EXTENDED FLOW INITIALIZATION
…
2013/1/28
31
…
EXTENDED FLOW INITIALIZATION
Fuse
2013/1/28
322013/1/28
Sparse feature matching
Fusion
Dense nearest-neighbor patch matching
Selection
33
OUTLINE
2013/1/28
• Previous Work
• Contributions
• Extended Flow Initialization
• Selective data term
• Efficient optimization solver
• Experimental result
• Conclusion
34
CONSTRAINTS
2013/1/28
• Brightness consistency
• Gradient consistency
• Average
I 2 1(u, x) (x u) (x)I ID
I 2 1(u, x) (x u) (I I x)D
Ix
I1 1(u, x) (u, x) (u, x)2 2DE D D
35
• Pixels moving out of shadow
CONSTRAINTS
pI 1 1p(u , ) 6.63D
• Color constancy is violated
I Ip1 1 p1 11 (u , ) (u , ) = 3.482
p pD D
• Average:
p1u : ground truth motion of p1
• Gradient constancy holdsp 1I 1 p(u , ) 0.32D
2013/1/28
36
• Pixels undergoing rotational motion
CONSTRAINTS
• Color constancy holds
• Gradient constancy is violatedp2u : ground truth motion of p2
p 2I 2 p(u , ) 4.20D • Average:
I Ip2 2 p2 21 (u , ) (u , ) = 2.242
p pD D
pI 2 2p(u , ) 0.29D
2013/1/28
37
SELECTIVE DATA TERM• Selectively combine the constraints
where 2(x) : {0,1}
I Ix
(u, ) (u,x(x) 1) ( ) (u,x)(x)DE D D
2013/1/28
38
SELECTIVE DATA TERM
RubberWhale Urban22
2.5
3
3.5
4
4.5
5
colorgradientaverageours
AAE
selective
2013/1/28
39
OUTLINE
2013/1/28
• Previous Work
• Contributions
• Extended Flow Initialization
• Selective data term
• Efficient optimization solver
• Experimental result
• Conclusion
40
DISCRETE-OPTIMIZATION
2013/1/28
• Minimizing energy including discrete α & continuous u :
• Try to separate α & u
• For α• Probability of a particular state of MRF system
Ix
I(x) (u, x) (1 (x)) (u, x)(u, ) ( u, x)E SD D
(u, )1(u, ) EP eZ
41
DISCRETE-OPTIMIZATION
2013/1/28
• Partition function
• Sum over all possible values of α
(u, )
{u} { 0,1}
EZ e
I Ix
( u,x) (u, x) ((x) (x)
{ 0,
1 ) (u, x)
{u 1}}
x
S D D
e e
(u, x)(u, x) II
x
1{ ( u,x) ln( )}
{u}
DDS e e
e
• Optimal condition (Euler-Lagrange equations)
• It decomposes to
II (u, x)(u, x)
x
1(u) ( u,x) ln( )DDeffE S e e
I I( (u,x) (u,x))
1(x)1 D De
u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S
I I
I II I
(u,x) (u,x)
(u,x) (u,u I u I
u
x)(u,x) (u,x)(u, x) (u, x)
div( ( u,x)) 0
D D
D DD D
e ee e e e
D D
S
( )x 1 ( )x
DISCRETE-OPTIMIZATION
• Minimization – Update α– Compute flow field
43
CONTINUOUS-OPTIMIZATION
2013/1/28
• Energy function
• Variable splitting
44
CONTINUOUS-OPTIMIZATION
2013/1/28
• Fix u , estimate w,p
• Fix w,p , estimate u
• The Euler-Lagrange equation Is linear.
45
OUTLINE
2013/1/28
• Previous Work
• Contributions
• Extended Flow Initialization
• Selective data term
• Efficient optimization solver
• Experimental result
• Conclusion
46
SELECTIVE DATA TERM
Averaging Selective
Difference
2013/1/28
47
EXPERIMENTAL RESULTS
2013/1/28
48
RESULTS FROM DIFFERENT STEPS
Coarse-to-fine
Extended coarse-to-fine2013/1/28
492013/1/28
50
LARGE DISPLACEMENT
Overlaid Input 2013/1/28
51
LARGE DISPLACEMENT • Motion Estimates
Coarse-to-fine Result Warping Result2013/1/28
52
LARGE DISPLACEMENT • Motion Magnitude Maps
LDOP [Brox et al. 09 ] [Steinbrucker et al. 09]] Result2013/1/28
53
OVERLAID INPUT
2013/1/28
54Conventional Coarse-to-fine Result2013/1/28
55
EXPERIMENTAL RESULTS
Overlaid Input2013/1/28
56Coarse-to-fine Result2013/1/28
57
OUTLINE
2013/1/28
• Previous Work
• Contributions
• Extended Flow Initialization
• Selective data term
• Efficient optimization solver
• Experimental result
• Conclusion
58
CONCLUSION
2013/1/28
• To solve the coarse-to-fine problem , it seems more easier to make a correctness in every level.
• Using optical flow for small motion & other tracking skill for large displacement seems reasonable.
• It takes 40s ~ 3mins to compute an optical flow field respect to the amount of missing parts. Tradeoff problem.
592013/1/28
Thank you for your listening
60
FEATURE MATCHING
2013/1/28
• Feature :” interesting “ , ” unique” part of image
• Two components of feature :
Test image Detector: where are the local features?
Descriptor: how to describe them?
61
FEATURE MATCHING
2013/1/28
• Local measure of feature uniqueness Shifting the window in any direction causes a big change
“flat” :no change in all directions
“edge”: no change along the edge direction
“corner”:significant change in all directions
62
SIFT FEATURE MATCHING
2013/1/28
• SIFT : Scale Invariant Feature Transform
• Problem: non-invariant between image scales
All points will be classified as edges
Corner
63
SIFT FEATURE MATCHING
2013/1/28
• Find scale that gives local maxima of some function f in both position and scale
64
SIFT FEATURE MATCHING
2013/1/28
• Function f : Laplacian-of-Gaussian
65
SIFT FEATURE MATCHING
2013/1/28
• We define the characteristic scale as the scale that produces peak of Laplacian response
66
ALGORITHM• Scale-space extrema detection
• Keypoint localization
• Orientation assignment
• Keypoint descriptor
( )local descriptor
detector
descriptor
67
ALGORITHM• Scale-space extrema detection
• Keypoint localization
• Orientation assignment
• Keypoint descriptor
( )local descriptor
detector
descriptor
68
DETECTOR
2013/1/28
69
SCALE-SPACE EXTREMA DETECTION
2013/1/28
• Use Difference of Gaussian instead of LOG
• More efficient
DOG & LOG
70
KEYPOINT LOCALIZATION
2013/1/28
• X is selected if it is larger or smaller than all 26 neighbors
• Eliminating edge responses
71
ALGORITHM• Scale-space extrema detection
• Keypoint localization
• Orientation assignment
• Keypoint descriptor
( )local descriptor
detector
descriptor
72
ORIENTATION ASSIGNMENT
2013/1/28
• Use orientation histogram in the window to vote for total orientation
• Rotation-Invariant
73
KEYPOINT DESCRIPTOR
2013/1/28
• Describe the orientation histogram in 8x8 window near the pixel
• Illumination-robust
Back
74
PATCH MATCHING
2013/1/28
• SIFT still lose information about objects lacking features
• Using “patch” as a unit , minimizing
• Without smoothness term , it can detect large replacement , but also produce errors . Errors can be eliminate by fusion step.
75
PATCH MATCHING
2013/1/28
• Randomized Correspondence Algorithm
• Idea : Coherent matches with neighbors
• Algorithm
• Initialization
• Propagation
• Search
76
PATCH MATCHING
2013/1/28Back
77
GRAPHIC CUT
2013/1/28
• Regard every pixel of image as a random variable , then the image is a “ random field .”
• Every pixel is only related to its neighbors , the filed is a “ Markov random field. ”(MRF)
• MRF can be viewed as a graph.
78
GRAPHIC CUT
2013/1/28
• Regard the optical field as a MRF.
• The value of a pixel is chosen within optical flow frames produced previously , it’s a “ labeling problem. ”
• The edges between pixels are smoothness relation.
• Cut the graph with minimum energy.
79
GRAPHIC CUT
2013/1/28
• Minimize the energy function
• Multi-labeling problem
• expansion move algorithm : expanse the label which can decrease the energy
V. Lempitsky, S. Roth, and C. Rother, “Fusionflow: Discrete-Continuous Optimization for Optical Flow Estimation,”Proc. IEEE Conf. Computer Vision and Pattern Recognition,2008Back
80
FEATURE MATCHING
2013/1/28
• Find corners
Change of intensity for the shift [u,v]:
IntensityShifted intensity
Window function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
81
FEATURE MATCHING
2013/1/28
• For small shifts [u,v] we have a bilinear approximation:
where M is a 22 matrix computed from image derivatives:
82
FEATURE MATCHING
2013/1/28
2
“Corner”1 and 2 are large, 1 ~ 2;E increases in all directions
1 and 2 are small;E is almost constant in all directions
“Edge” 1 >> 2
“Edge” 2 >> 1
“Flat” region
Classification of image points using eigenvalues of M: