optical flow – part 2 #motion estimation and occlusion detection #blurred video with layers liron...

OPTICAL FLOW – PART 2#MOTION ESTIMATION AND

OCCLUSION DETECTION#BLURRED VIDEO WITH LAYERS

Liron Gruber

OUTLINELocal Layering for Joint Motion Estimation and Occlusion Detection (Sun1 Liu2 Pfister1):• The problem• Some layers models history• Local layers model • Probabilistic model• Results

Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black):• The problem• The two layered model• Initialization and optimization• Results

Joint Motion Estimation and Occlusion Detection

• The problem:Most motion estimation algorithms (optical flow,

layered models) cannot handle large amount of occlusion• Their solution:Local layering model where motion and occlusion relationships are inferred jointly

What we have today

• Optical flow methods: -> X and T junctions problem

• Layered models : 1. Number of layers 2. The layer ownership 3. Depth ordering 4. The motion for each layer

Layers ‘HISTORY’

3. ( estimate the motion of superpixels and the occlusion boundaries between superpixels for epipolar constrained optical flow estimation )

Previous work on layers relies on either motion or color cues to initialize layer segmentation

For example:

2. optical flow algorithms -> different features ‘random forest’ classifiers (Humayun)

1. optical flow algorithm -> motion vectors clustering -> layer primitive (Limor’s Sun)

[1] [2]

Their method

The Problem:

Mid - summary

Global layers : • limited in capturing mutual or self occlusions• often only contain a few number of layers

(complexity explodes)

• all layers methods separate motion estimation (Do not use the detected occlusion to improve optical flow)

(exp 3 – does that , but not vise versa...)

So – Local Layers…

local layering model

Jointly infer motion and occlusion:• superpixel representations (from over segmentation)• Each superpixel’s occlusion-relationships with its

neighbors • In the inference - keeping the uncertainties of both motion and occlusion relationships

motion is inferred by considering all the possibilities of local occlusion relationships and vice versa

In practice – given I1, I2

Unknowns: (after segmentation of I1 to local layers)

• m - movement of each layer• o - occlusion map for I1 • R=1,0,-1 - occlusion relationship between spatially close local layers

R

From R to Pseudo Depth (d)

A: A(i ,i) =| Ni |A(i, j) = -1 if j in Ni 0 otherwise

b:b(i) = sum of Rij (over all j in Ni )

Probabilistic Model

• Data term

• Motion prior

• Motion and occlusion

Probabilistic Model – Data term

(intuition only…)

Probabilistic Model - Motion prior

(intuition only…)

• Similar to optical flow:Motion is smooth and slow

Occasionally abrupt near object boundaries

Probabilistic Model - Motion and occlusion

Only Make sure o is consistent with R,m :

There are pixels in the set

overlap No pixels in the set

No overlap

Inference process

Results(Average EPE)

Classical /baseline optical flow methods

(motion)

State-of-the-art learning based

approach (occlusion)

Summary

• Local layering model can handle motion and occlusion well for both challenging synthetic and real sequences

(“two bars” sequence and the MPI Sintel dataset)

• This method improves the baseline algorithms that provide the motion estimations and performs comparably with one learning-based occlusion detection algorithm

OUTLINELocal Layering for Joint Motion Estimation and Occlusion Detection (Sun1 Liu2 Pfister1):• The problem• Some layers models history• Local layers model • Probabilistic model• Results

Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black):• The problem• The two layered model• Initialization and optimization• Results

Modeling Blurred Video with Layers

The problem: Videos contain complex spatially-varying motion

blur due to finite shutter speeds. Existing methods (to estimate optical flow, deblur the images, and segment the scene) fail in such cases and fail specifically at object boundaries.

Their solution: A novel 2 layered model of scenes in motion.

Jointly estimate the layer segmentation and each layer's appearance and motion.

Example

Notation

observed color image

unblurred color “appearance" of layer l segmentation mask for l * assumed to be constant across the sequence * only consider opaque layers (is binary)

the transformation (motion) parameters for layer l at frame t

a blur matrix (s is the shutter speed)

• Two Layers with Foreground Motion and Blur

• ……. ->

A walk through

• A Single Layer with Motion Blur

• Two Layers without Motion Blur

The Two-Layer Model

observed image

blur matrix segmentation mask

unblurred “appearance"

transformation matrices (according to )

They minimized: + Regularization term

(Regularization term)

Spatial smoothness:

Background preference:

The generative model - example

Initialization• Good initialization is important - (the choice of initial dense optical flow algorithm is not critical, they use MDP-Flow) [b] • 2 dominant motions are robustly estimated [c]

Pyramids levels(large motions)

Optimization

Iterative, alternating optimization method: 1. optimize one variable at a time (using gradient descent)2. Terminate the optimization after 3 iterations (to avoid reaching local optima) and switch to the next variable

• Relax the binary-valued • To deal with large motions - a Gaussian pyramid

“…the shape of the person and rims of the bicycle being evident”

Results

layered optical flow model (no blur)

motion blur in an algorithm for optical flow (no layers)

accurate optical flow method (no layers, no blur)

• They developed a principled formulation of motion blur in layers.

• They jointly estimated parametric motion, deblurred appearance, and scene segmentation.

• The layered model captures the blur at boundaries and by modeling the blur process one achieves better motion estimation, layer segmentation, and layer deblurring.

Summary

The end

Thank you

EXTRA SLIDES (1)

Extra equations +explanations 1

Extra equations +explanations 2

Extra equations +explanations 3

Inference

EXTRA SLIDES (2)

Evaluation 2