consistent optical flow forconsistent optical flow for stereo video · consistent optical flow for...
TRANSCRIPT
Consistent Optical Flow forConsistent Optical Flow forConsistent Optical Flow for Consistent Optical Flow for Stereo VideoStereo Video
ICIP 2010
A it S ll t Ch i ti Li M MAnita Sellent, Christian Linz, Marcus MagnorTU Braunschweig, Germany
Monocular Video EditingMonocular Video EditingPropagating keyframe information based on
Monocular Video EditingMonocular Video Editing
Optical Flow
2Consistent Optical Flow for Stereo Video Anita Sellent
Monocular Video EditingMonocular Video EditingPropagating keyframe information based on
Monocular Video EditingMonocular Video Editing
Optical Flow
[Eisemann et al. 2009]
Inaccurate flow estimation requires manual
3Consistent Optical Flow for Stereo Video Anita Sellent
Inaccurate flow estimation requires manual interaction
Stereoscopic Video EditingStereoscopic Video EditingStereoscopic Video EditingStereoscopic Video EditingCorrespondence between temporally and spatially neighboring images
left right
4Consistent Optical Flow for Stereo Video Anita Sellent
Uncalibrated and non-synchronized cameras
Consistent Optical FlowConsistent Optical FlowConsistent Optical FlowConsistent Optical FlowOptical flow estimation between image pairs
Small errors accumulate quickly
time
5Consistent Optical Flow for Stereo Video Anita Sellent
left right
Key IdeaKey IdeaKey IdeaKey IdeaExploit redundant data in stereo videoCalculate flow on three images simultaneously
6Consistent Optical Flow for Stereo Video Anita Sellent
OutlineOutlineOutlineOutlineThe basic two image flowThe symmetry constraintThe loop constraintIterative flow estimationPropagation over a sequencep g qResults
7Consistent Optical Flow for Stereo Video Anita Sellent
BasicBasic TwoTwo Image FlowImage FlowBasic Basic TwoTwo Image FlowImage FlowBrightness constancy assumption
Smoothness term: total variation, Huber norm
0)()( 2,121 ≈+− wxIxISmoothness term: total variation, Huber norm
, G
2,1w∇ 2,15.0 wD ∇
Global energy
∫∫Ω
∇++− dxwwxIIw 2,12,121 )(min
2,1
λ
8Consistent Optical Flow for Stereo Video Anita Sellent
Splitting the Energy FunctionSplitting the Energy FunctionSplitting the Energy FunctionSplitting the Energy Function[Zach et al. 2007][Werlberger et al 2009]
Auxiliary variable
∫2
[Werlberger et al. 2009]
2,1z
∫Ω
∇+−++− dxzzwwxIIzw 2,12,12,12,121,
2)(min2,12,1 θλ
Pointwise problemPointwise problem
2,12,12,1211)(min
21
zwwxIIw
−++−θ
λ
Image denoising problem2,1w θ
∫ ∇ d1i
9Consistent Optical Flow for Stereo Video Anita Sellent
∫Ω
∇+− dxzzwzw 2,12,12,1,
min2,12,1 θ
The Symmetry ConstraintThe Symmetry ConstraintThe Symmetry ConstraintThe Symmetry Constraint[Alvarez et al. 2007]
Symmetrie error:
0≈++= )()( 2,11,22,1 wxwxwSρ
2I :2,1w flow from to1I 2I
:1,2w flow from to2I 1I
1I
10Consistent Optical Flow for Stereo Video Anita Sellent
The Loop ConstraintThe Loop ConstraintThe Loop ConstraintThe Loop ConstraintLoop error:
0≈+++++= )()()( 3,22,11,32,13,22,1 wwxwwxwxwLρ
2I
I1I
3I
11Consistent Optical Flow for Stereo Video Anita Sellent
UpdateUpdate schemeschemeUpdate Update schemeschemeDetermine incremental update jidw ,
from brightness constancyEvaluate satisfaction of symmetry and loop
fconstraint for this estimate22LS ba eep ρρ −−=
jik
jik
ji dwpww ,,1
, +=+Update current flow fieldUse current flow estimation to determine
jiz ,
hjw ,
TV-optimized versionProceed with next flow field
12Consistent Optical Flow for Stereo Video Anita Sellent
Consistent UpdateConsistent UpdateConsistent UpdateConsistent UpdateUpdate flow field only when symmetry and loop constraint are satisfiedRely on smooth fill-in otherwiseBrightness constancy for occluded points is suppressed
13Consistent Optical Flow for Stereo Video Anita Sellent
Propagation in aPropagation in a SequenceSequencePropagation in a Propagation in a SequenceSequenceUse known flow fields to constrain the new flow fields
14Consistent Optical Flow for Stereo Video Anita Sellent
Propagation IIPropagation IIPropagation IIPropagation II… …
15Consistent Optical Flow for Stereo Video Anita Sellent
Propagation IIPropagation IIPropagation IIPropagation II… …
16Consistent Optical Flow for Stereo Video Anita Sellent
Propagation IIPropagation IIPropagation IIPropagation II… …
17Consistent Optical Flow for Stereo Video Anita Sellent
Propagation IIPropagation IIPropagation IIPropagation II… …
18Consistent Optical Flow for Stereo Video Anita Sellent
Propagation IIPropagation IIPropagation IIPropagation II… …
19Consistent Optical Flow for Stereo Video Anita Sellent
Propagation IIPropagation IIPropagation IIPropagation II… …
20Consistent Optical Flow for Stereo Video Anita Sellent
ResultsResultsResultsResultsTest sequences with ground-truth
[V d t l 2008][Vaudrey et al. 2008]
21Consistent Optical Flow for Stereo Video Anita Sellent
[Scharstein et al. 2007]
Results AEEResults AEEResults AEEResults AEE
AEE Wave enpeda Art Books DollsAEE Wave .enpeda Art Books Dolls
TV-L2 1.03 0.71 10.62 14.60 5.81
Symmetry 1.01 0.61 10.02 10.73 2.93
Loop 0.97 0.59 9.34 6.43 2.85
22Consistent Optical Flow for Stereo Video Anita Sellent
Results ConsistencyResults ConsistencyResults ConsistencyResults Consistency
Consistency Wave .enpeda Art Books DollspTV-L2 1.24 4.01 3.00 17.45 11.37
Symmetry 0.16 1.29 2.69 6.30 7.60
23Consistent Optical Flow for Stereo Video Anita Sellent
Loop 0.13 0.87 1.71 4.27 2.84
ConclusionConclusionConclusionConclusionEstimate flow fields between three images simultaneously Prevent error accumulationDisable brightness constancy for occluded points
Consistent and more accurate flow fields
24Consistent Optical Flow for Stereo Video Anita Sellent
Thank You for Your Attention!Thank You for Your Attention!
htt // hi t b d
25Consistent Optical Flow for Stereo Video Anita Sellent
https://graphics.tu-bs.de