optical flow – part 2 #motion estimation and occlusion detection #blurred video with layers liron...

40
OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Upload: beverly-stafford

Post on 02-Jan-2016

224 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

OPTICAL FLOW – PART 2#MOTION ESTIMATION AND

OCCLUSION DETECTION#BLURRED VIDEO WITH LAYERS

Liron Gruber

Page 2: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

OUTLINELocal Layering for Joint Motion Estimation and Occlusion Detection (Sun1 Liu2 Pfister1):• The problem• Some layers models history• Local layers model • Probabilistic model• Results

Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black):• The problem• The two layered model• Initialization and optimization• Results

Page 3: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Joint Motion Estimation and Occlusion Detection

• The problem:Most motion estimation algorithms (optical flow,

layered models) cannot handle large amount of occlusion• Their solution:Local layering model where motion and occlusion relationships are inferred jointly

Page 4: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

What we have today

• Optical flow methods: -> X and T junctions problem

• Layered models : 1. Number of layers 2. The layer ownership 3. Depth ordering 4. The motion for each layer

Page 5: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Layers ‘HISTORY’

3. ( estimate the motion of superpixels and the occlusion boundaries between superpixels for epipolar constrained optical flow estimation )

Previous work on layers relies on either motion or color cues to initialize layer segmentation

For example:

2. optical flow algorithms -> different features ‘random forest’ classifiers (Humayun)

1. optical flow algorithm -> motion vectors clustering -> layer primitive (Limor’s Sun)

Page 6: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

[1] [2]

Their method

The Problem:

Page 7: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Mid - summary

Global layers : • limited in capturing mutual or self occlusions• often only contain a few number of layers

(complexity explodes)

• all layers methods separate motion estimation (Do not use the detected occlusion to improve optical flow)

(exp 3 – does that , but not vise versa...)

So – Local Layers…

Page 8: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

local layering model

Jointly infer motion and occlusion:• superpixel representations (from over segmentation)• Each superpixel’s occlusion-relationships with its

neighbors • In the inference - keeping the uncertainties of both motion and occlusion relationships

motion is inferred by considering all the possibilities of local occlusion relationships and vice versa

Page 9: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

In practice – given I1, I2

Unknowns: (after segmentation of I1 to local layers)

• m - movement of each layer• o - occlusion map for I1 • R=1,0,-1 - occlusion relationship between spatially close local layers

Page 10: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

R

Page 11: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

From R to Pseudo Depth (d)

A: A(i ,i) =| Ni |A(i, j) = -1 if j in Ni 0 otherwise

b:b(i) = sum of Rij (over all j in Ni )

Page 12: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Probabilistic Model

• Data term

• Motion prior

• Motion and occlusion

Page 13: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Probabilistic Model – Data term

(intuition only…)

Page 14: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Probabilistic Model - Motion prior

(intuition only…)

• Similar to optical flow:Motion is smooth and slow

Occasionally abrupt near object boundaries

Page 15: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Probabilistic Model - Motion and occlusion

Only Make sure o is consistent with R,m :

There are pixels in the set

overlap No pixels in the set

No overlap

Page 16: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Inference process

Page 17: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Results(Average EPE)

Classical /baseline optical flow methods

(motion)

State-of-the-art learning based

approach (occlusion)

Page 18: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Summary

• Local layering model can handle motion and occlusion well for both challenging synthetic and real sequences

(“two bars” sequence and the MPI Sintel dataset)

• This method improves the baseline algorithms that provide the motion estimations and performs comparably with one learning-based occlusion detection algorithm

Page 19: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

OUTLINELocal Layering for Joint Motion Estimation and Occlusion Detection (Sun1 Liu2 Pfister1):• The problem• Some layers models history• Local layers model • Probabilistic model• Results

Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black):• The problem• The two layered model• Initialization and optimization• Results

Page 20: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Modeling Blurred Video with Layers

The problem: Videos contain complex spatially-varying motion

blur due to finite shutter speeds. Existing methods (to estimate optical flow, deblur the images, and segment the scene) fail in such cases and fail specifically at object boundaries.

Their solution: A novel 2 layered model of scenes in motion.

Jointly estimate the layer segmentation and each layer's appearance and motion.

Page 21: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Example

Page 22: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Notation

observed color image

unblurred color “appearance" of layer l segmentation mask for l * assumed to be constant across the sequence * only consider opaque layers (is binary)

the transformation (motion) parameters for layer l at frame t

a blur matrix (s is the shutter speed)

Page 23: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

• Two Layers with Foreground Motion and Blur

• ……. ->

A walk through

• A Single Layer with Motion Blur

• Two Layers without Motion Blur

Page 24: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

The Two-Layer Model

observed image

blur matrix segmentation mask

unblurred “appearance"

transformation matrices (according to )

They minimized: + Regularization term

Page 25: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

(Regularization term)

Spatial smoothness:

Background preference:

Page 26: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

The generative model - example

Page 27: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Initialization• Good initialization is important - (the choice of initial dense optical flow algorithm is not critical, they use MDP-Flow) [b] • 2 dominant motions are robustly estimated [c]

Pyramids levels(large motions)

Page 28: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Optimization

Iterative, alternating optimization method: 1. optimize one variable at a time (using gradient descent)2. Terminate the optimization after 3 iterations (to avoid reaching local optima) and switch to the next variable

• Relax the binary-valued • To deal with large motions - a Gaussian pyramid

Page 29: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

“…the shape of the person and rims of the bicycle being evident”

Page 30: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Results

layered optical flow model (no blur)

motion blur in an algorithm for optical flow (no layers)

accurate optical flow method (no layers, no blur)

Page 31: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

• They developed a principled formulation of motion blur in layers.

• They jointly estimated parametric motion, deblurred appearance, and scene segmentation.

• The layered model captures the blur at boundaries and by modeling the blur process one achieves better motion estimation, layer segmentation, and layer deblurring.

Summary

Page 32: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

The end

Thank you

Page 33: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber
Page 34: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

EXTRA SLIDES (1)

Page 35: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Extra equations +explanations 1

Page 36: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Extra equations +explanations 2

Page 37: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Extra equations +explanations 3

Page 38: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Inference

Page 39: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

EXTRA SLIDES (2)

Page 40: OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

Evaluation 2