o bj c ut

50
OBJ CUT M. Pawan Kumar Philip Torr Andrew Zisserman UNIVERSITY OF OXFORD

Upload: corina

Post on 22-Jan-2016

34 views

Category:

Documents


0 download

DESCRIPTION

UNIVERSITY OF OXFORD. O BJ C UT. M. Pawan Kumar Philip Torr Andrew Zisserman. Aim. Given an image, to segment the object. Object Category Model. Segmentation. Cow Image. Segmented Cow. Segmentation should (ideally) be shaped like the object e.g. cow-like - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: O BJ  C UT

OBJ CUT

M. Pawan Kumar

Philip Torr

Andrew Zisserman

UNIVERSITYOF

OXFORD

Page 2: O BJ  C UT

Aim• Given an image, to segment the object

Segmentation should (ideally) be• shaped like the object e.g. cow-like• obtained efficiently in an unsupervised manner• able to handle self-occlusion

Segmentation

ObjectCategory

Model

Cow Image Segmented Cow

Page 3: O BJ  C UT

Challenges

Self Occlusion

Intra-Class Shape Variability

Intra-Class Appearance Variability

Page 4: O BJ  C UT

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Cow Image

Object Seed Pixels

Page 5: O BJ  C UT

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Cow Image

Object Seed Pixels

Background Seed Pixels

Page 6: O BJ  C UT

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Segmented Image

Page 7: O BJ  C UT

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Cow Image

Object Seed Pixels

Background Seed Pixels

Page 8: O BJ  C UT

MotivationMagic Wand

Current methods require user intervention• Object and background seed pixels (Boykov and Jolly, ICCV 01)• Bounding Box of object (Rother et al. SIGGRAPH 04)

Segmented Image

Page 9: O BJ  C UT

Problem • Manually intensive

• Segmentation is not guaranteed to be ‘object-like’

Non Object-like Segmentation

Motivation

Page 10: O BJ  C UT

Our Method• Combine object detection with segmentation

– Borenstein and Ullman, ECCV ’02– Leibe and Schiele, BMVC ’03

• Incorporate global shape priors in MRF

• Detection provides– Object Localization– Global shape priors

• Automatically segments the object– Note our method completely generic– Applicable to any object category model

Page 11: O BJ  C UT

Outline

• Problem Formulation

• Form of Shape Prior

• Optimization

• Results

Page 12: O BJ  C UT

Problem• Labelling m over the set of pixels D• Shape prior provided by parameter Θ

• Energy E (m,Θ) = ∑Φx(D|mx)+Φx(mx|Θ) + ∑ Ψxy(mx,my)+ Φ(D|mx,my)

• Unary terms– Likelihood based on colour– Unary potential based on distance from Θ

• Pairwise terms– Prior– Contrast term

• Find best labelling m* = arg min ∑ wi E (m,Θi)– wi is the weight for sample Θi

Unary terms Pairwise terms

Page 13: O BJ  C UT

MRF

Probability for a labelling consists of• Likelihood

• Unary potential based on colour of pixel• Prior which favours same labels for neighbours (pairwise potentials)

Prior Ψxy(mx,my)

Unary Potential Φx(D|mx)

D (pixels)

m (labels)

Image Plane

x

y

mx

my

Page 14: O BJ  C UT

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior

x …

y …

x …

y …

Φx(D|obj)

Φx(D|bkg)Ψxy(mx,my)

Likelihood Ratio (Colour)

Page 15: O BJ  C UT

Example

Cow Image Object SeedPixels

Background SeedPixels

PriorLikelihood Ratio (Colour)

Page 16: O BJ  C UT

Contrast-Dependent MRF

Probability of labelling in addition has• Contrast term which favours boundaries to lie on image edges

D (pixels)

m (labels)

Image Plane

Contrast Term Φ(D|mx,my)

x

y

mx

my

Page 17: O BJ  C UT

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + Contrast

x …

y …

x …

y …

Likelihood Ratio (Colour)

Ψxy(mx,my)+Φ(D|mx,my)

Φx(D|obj)

Φx(D|bkg)

Page 18: O BJ  C UT

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + ContrastLikelihood Ratio (Colour)

Page 19: O BJ  C UT

Our Model

Probability of labelling in addition has• Unary potential which depend on distance from Θ (shape parameter)

D (pixels)

m (labels)

Θ (shape parameter)

Image Plane

Object CategorySpecific MRFx

y

mx

my

Unary PotentialΦx(mx|Θ)

Page 20: O BJ  C UT

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + ContrastDistance from Θ

Shape Prior Θ

Page 21: O BJ  C UT

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + ContrastLikelihood + Distance from Θ

Shape Prior Θ

Page 22: O BJ  C UT

Example

Cow Image Object SeedPixels

Background SeedPixels

Prior + ContrastLikelihood + Distance from Θ

Shape Prior Θ

Page 23: O BJ  C UT

Outline

• Problem Formulation– E (m,Θ) = ∑Φx(D|mx)+Φx(mx|Θ) + ∑ Ψxy(mx,my)+ Φ(D|mx,my)

• Form of Shape Prior

• Optimization

• Results

Page 24: O BJ  C UT

Layered Pictorial Structures (LPS)• Generative model

• Composition of parts + spatial layout

Layer 2

Layer 1

Parts in Layer 2 can occlude parts in Layer 1

Spatial Layout(Pairwise Configuration)

Page 25: O BJ  C UT

Layer 2

Layer 1

Transformations

Θ1

P(Θ1) = 0.9

Cow Instance

Layered Pictorial Structures (LPS)

Page 26: O BJ  C UT

Layer 2

Layer 1

Transformations

Θ2

P(Θ2) = 0.8

Cow Instance

Layered Pictorial Structures (LPS)

Page 27: O BJ  C UT

Layer 2

Layer 1

Transformations

Θ3

P(Θ3) = 0.01

Unlikely Instance

Layered Pictorial Structures (LPS)

Page 28: O BJ  C UT

LPS for Detection• Learning

– Learnt automatically using a set of examples

• Detection– Matches LPS to image using Loopy Belief Propagation– Localizes object parts

Page 29: O BJ  C UT

Outline

• Problem Formulation

• Form of Shape Prior

• Optimization

• Results

Page 30: O BJ  C UT

Optimization

• Given image D, find best labelling as m* = arg max p(m|D)

• Treat LPS parameter Θ as a latent (hidden) variable

• EM framework– E : sample the distribution over Θ

– M : obtain the labelling m

Page 31: O BJ  C UT

E-Step

• Given initial labelling m’, determine p(Θ|m’,D)

• Problem Efficiently sampling from p(Θ|m’,D)

• Solution• We develop efficient sum-product Loopy Belief

Propagation (LBP) for matching LPS.

• Similar to efficient max-product LBP for MAP estimate– Felzenszwalb and Huttenlocher, CVPR ‘04

Page 32: O BJ  C UT

Results

• Different samples localize different parts well.• We cannot use only the MAP estimate of the LPS.

Page 33: O BJ  C UT

M-Step

• Given samples from p(Θ|m’,D), get new labelling mnew

• Sample Θi provides– Object localization to learn RGB distributions of object and background– Shape prior for segmentation

• Problem– Maximize expected log likelihood using all samples– To efficiently obtain the new labelling

Page 34: O BJ  C UT

M-Step

Cow Image Shape Θ1

w1 = P(Θ1|m’,D)

RGB Histogram for Object RGB Histogram for Background

Page 35: O BJ  C UT

Cow Image Shape Θ1

M-Step

w1 = P(Θ1|m’,D)

Θ1

Image PlaneD (pixels)

m (labels)

• Best labelling found efficiently using a Single Graph Cut

Page 36: O BJ  C UT

Segmentation using Graph Cuts

x …

y … … …

z … …

Obj

Bkg

CutΦx(D|bkg) + Φx(bkg|Θ)

m

Φz(D|obj) + Φz(obj|Θ)

Ψxy(mx,my)+

Φ(D|mx,my)

Page 37: O BJ  C UT

Segmentation using Graph Cuts

x …

y … … …

z … …

Obj

Bkg

m

Page 38: O BJ  C UT

M-Step

Cow Image Shape Θ2

w2 = P(Θ2|m’,D)

RGB Histogram for BackgroundRGB Histogram for Object

Page 39: O BJ  C UT

M-Step

Cow Image Shape Θ2

w2 = P(Θ2|m’,D)

Θ2

Image PlaneD (pixels)

m (labels)

• Best labelling found efficiently using a Single Graph Cut

Page 40: O BJ  C UT

M-Step

Θ2

Image Plane

Θ1

Image Plane

w1 + w2 + ….

• Best labelling found efficiently using a Single Graph Cut

m* = arg min ∑ wi E (m,Θi)

Page 41: O BJ  C UT

Outline

• Problem Formulation

• Form of Shape Prior

• Optimization

• Results

Page 42: O BJ  C UT

SegmentationImage

ResultsUsing LPS Model for Cow

Page 43: O BJ  C UT

In the absence of a clear boundary between object and background

SegmentationImage

ResultsUsing LPS Model for Cow

Page 44: O BJ  C UT

SegmentationImage

ResultsUsing LPS Model for Cow

Page 45: O BJ  C UT

SegmentationImage

ResultsUsing LPS Model for Cow

Page 46: O BJ  C UT

SegmentationImage

ResultsUsing LPS Model for Horse

Page 47: O BJ  C UT

SegmentationImage

ResultsUsing LPS Model for Horse

Page 48: O BJ  C UT

Our Method Leibe and SchieleImage

Results

Page 49: O BJ  C UT

AppearanceShape Shape+Appearance

Results

Without Φx(D|mx) Without Φx(mx|Θ)

Page 50: O BJ  C UT

• Conclusions

– New model for introducing global shape prior in MRF– Method of combining detection and segmentation– Efficient LBP for detecting articulated objects

• Future Work

– Other shape parameters need to be explored– Method needs to be extended to handle multiple

visual aspects