tracking deformable surfaces with optical flow in the ...€¦ · nordia’08, 27 june 2008 optical...

Image Processing DepartmentComputer Vision & Graphics Group

Berlin, Germany

• Anna Hilsmann

Tracking Deformable Surfaces with Optical Flow in the Presence of Self-Occlusion in Monocular

Image Sequences

NORDIA’08, 27-28 June 2008

Anna HilsmannNORDIA’08, 27 June 2008

Optical Flow Based Deformable Surface Tracking



Outline

• Optical Flow Based Tracking and Formulation of the Motion Model

• Handling Optical Flow Constraints• Occlusion Handling• Experimental Results



Optical Flow Constraint Equation

At each pixel in the image:

Minimize the error functional



Optical Flow Constraint Equation



BUT: one equation in two unknowns for each pixel !



Optical Flow Constraint provides one equation for two unknownsRegularization by motion model

triangulated, regular 2D mesh with vertices



Back to the Optical Flow Constraint Equation


Use the parameterized motion model to regularize optical flow field !

with



Back to the Optical Flow Constraint Equation


Use the parameterized motion model to regularize optical flow field !

with

Overdetermined equation system that can efficiently be solved in a linear least squares sense



Incorporating spatial smoothing constraints

with

?

‘Laplacian vertex displacement deviation‘defined locally on the mesh topology



Incorporating spatial smoothing constraints

with

?

Overdetermined equation system that can efficiently be solved in a linear least squares sense

‘Laplacian vertex displacement deviation‘defined locally on the mesh topology



Handling Optical Flow Constraints





However...

... only valid for small displacments between two successive frames hierarchical framework





However...


... only valid for constant and uniform illumination

bandpass filtered images & model-to frame estimation



Handling Optical Flow Constraints... only valid for constant and uniform illumination


Illumination is not constant ...

bandpass filter

use a model frame ......and estimate transformationto next frame

warp it with current estimate...





However...


... only valid for constant and uniform illumination

bandpass filtered images & model-to frame estimation

… aperture problem motion model


Self-Occlusion Handling

Normally, a 2D mesh folds under self-occlusions...




bad state in 2D monocular tracking




Idea: force the mesh to shrink instead of fold to stabelize the trackingGay-Bellile et al. ICCV 2007

bad state in 2D monocular tracking



We defined the mesh smoothness locally on vertex topology

with



We defined the mesh smoothness locally on vertex topology:

Idea: weight the smoothing constraints locally according to the occlusion of a region and force the mesh to shrink instead of fold

with



Occlusion maps from local shrinking regions in the mesh:


Results

T-Shirt Logo Sequence, 500 frames, 25fps


Results

T-Shirt Folding Sequence, 500 frames, 25fps Occlusion Map


Results

Paper Sequence, 100 frames, 25fps Occlusion Map


Results

without occlusion handling the mesh folds...

with occlusion handling the mesh shrinks at the boundary.Paper Sequence


Results

Knut Sequence, 100 frames, 25fps Occlusion Map


Results

Picasso Sequence, 500 frames, 25fps Augmentation


Results


Conclusion and applications

• straight-forward optical flow based algorithm for single view tracking

• mesh based motion model

• closed-form solution

• suitable for real-time applications:

e.g. Virtual Mirror, Virtual Clothing, Virtual Dressing Rooms


Thank You

The work presented in this paper has been developed with the support of the European Network of Excellence VISNET II (Contract IST-1-038398).

tracking deformable surfaces with optical flow in the ...€¦ · nordia’08, 27 june 2008 optical...

Documents