photomontage

GrabCut GrabCut Interactive ImageInteractive Image

((andand Stereo) Stereo) SSegmentation egmentation

Carsten RotherCarsten RotherVladimir Kolmogorov Vladimir Kolmogorov

Andrew BlakeAndrew BlakeAntonio CriminisiAntonio Criminisi

Geoffrey CrossGeoffrey Cross [based on Siggraph 2004 and ECCV 2004][based on Siggraph 2004 and ECCV 2004]

PhotomontagePhotomontage

GrabCut – Interactive Foreground Extraction 1

Talk OutlineTalk Outline

Hard Image Segmentation: Fore- vs. BackgroundHard Image Segmentation: Fore- vs. Background

Soft Segmentation: Alpha Matting

Stereo Segmentation: Exploit Depth

GrabCut – Interactive Foreground Extraction 2

Problem Problem

GrabCut – Interactive Foreground Extraction 3

Fast & Accurate ?

What GrabCut doesWhat GrabCut does

User Input

Result

Magic Wand (198?)

Intelligent ScissorsMortensen and Barrett (1995)

GrabCut

Regions Boundary Regions & Boundary

GrabCut – Interactive Foreground Extraction 4

FrameworkFramework

Input: Image

Output: Segmentation

Parameters: Colour ,Coherence

Energy:

Optimization:

GrabCut – Interactive Foreground Extraction 5

Maximum a posteriori estimator (MAP):

• Gibbs Distribution of the MRF

same as

Energy – Probabilistic Energy – Probabilistic ViewView

GrabCut – Interactive Foreground Extraction 6

- log

Graph CutsGraph Cuts - - Boykov and Jolly Boykov and Jolly (2001)(2001)

GrabCut – Interactive Foreground Extraction 7

ImageImage Min CutMin Cut

Cut: separating source and sink; Energy: collection of edges

Min Cut: Global minimal enegry in polynomial time

Foreground Foreground (source)(source)

BackgroundBackground(sink)(sink)

Iterated Graph CutIterated Graph Cut

User Initialisation

K-means for learning

colour distributions

Graph cuts to infer the

segmentation

?

GrabCut – Interactive Foreground Extraction 8

1 2 3 4

Iterated Graph CutsIterated Graph Cuts

GrabCut – Interactive Foreground Extraction 9

Energy after each IterationResult

Guaranteed to

converge

Colour ModelColour Model

Gaussian Mixture Model Gaussian Mixture Model (typically 5-8 components)(typically 5-8 components)

Foreground &Background

Background

Foreground

BackgroundG

GrabCut – Interactive Foreground Extraction 10

R

G

RIterated graph cut

Coherence ModelCoherence ModelAn object is a coherent set of pixels:

Error (%) over training set:

25

How do we choose ?

25

Gaussian MRF:

Linear regression gives in closed-form

Pseudo-Likelihood:

Parameter Learning Parameter Learning (Blake (Blake 2004)2004)

GrabCut – Interactive Foreground Extraction 12

approximation

=

A Gaussian MRF is not a realistic texture model

Gaussian? Gaussian!Real Image syntheticGMRF

Parameter Learning - Parameter Learning - ProblemsProblems

GrabCut – Interactive Foreground Extraction 13

Moderately simple Moderately simple examplesexamples

… … GrabCut completes automaticallyGrabCut completes automatically GrabCut – Interactive Foreground Extraction 14

Difficult ExamplesDifficult Examples

Camouflage & Camouflage & Low ContrastLow Contrast No telepathyNo telepathyFine structureFine structure

Initial Rectangle

InitialResult

GrabCut – Interactive Foreground Extraction 15

Evaluation – Labelled Evaluation – Labelled DatabaseDatabase

Available online: http://research.microsoft.com/vision/cambridge/segmentation/

GrabCut – Interactive Foreground Extraction 16

Comparison Comparison GrabCutBoykov and Jolly (2001)

Error Rate: 0.72%Error Rate: 1.87%Error Rate: 1.81%Error Rate: 1.32%Error Rate: 1.25%Error Rate: 0.72%

GrabCut – Interactive Foreground Extraction 17

User Input

Result

ComparisonComparison

Trimap Boykov and Jolly

Error Rate: 1.36%

Input Image Ground Truth BimapGrabCut

Error Rate: 2.13%

GrabCut – Interactive Foreground Extraction 18

Error rate - modestly increase

User Interactions - considerable reduced

Results Parameter Results Parameter LearningLearning

GrabCut – Interactive Foreground Extraction 19

ComparisonComparison

Magic Wand (198?)

Intelligent Scissors Mortensen and Barrett (1995)

GrabCutRother et al. (2004)

Graph Cuts Boykov and Jolly (2001)

LazySnappingLi et al. (2004)

GrabCut – Interactive Foreground Extraction 20

““Mixed pixels”: Combination of fore- and Mixed pixels”: Combination of fore- and background background

Alpha Mask: Proportion of fore- and background Alpha Mask: Proportion of fore- and background Natural Matting Problem: Determine alpha,F,B Natural Matting Problem: Determine alpha,F,B

from C from C

Under-determined System: 3 Equations and 7 unknowns

Digital MattingDigital Matting

GrabCut – Interactive Foreground Extraction 21

1. Simple Alpha & Simple Colour 2. Difficult Alpha & Simple Colour

3. Simple Alpha & Difficult Colour

4. Difficult Alpha & Difficult Colour

Existing Methods

Human ?

GrabCut

GrabCut – Interactive Foreground Extraction 22

Transparency - Transparency - TaxonomieTaxonomie

Border Matting Border Matting

Hard Segmentation Automatic Trimap Soft Segmentation

GrabCut – Interactive Foreground Extraction 23

to

Input Bayes MattingChuang et. al. (2001)

Knockout 2Photoshop Plug-In

ComparisonComparison

GrabCut – Interactive Foreground Extraction 24

With no regularisation over alpha

Shum et. al. (2004): Coherence matting in “Pop-up light fields”

Solve

Mean Colour Foreground

Mean ColourBackground

GrabCut – Interactive Foreground Extraction 25

Natural Image MattingNatural Image Matting

Ruzon and Tomasi (2000): Alpha estimation in natural images

Noisy alpha-profile

Border MattingBorder Matting

GrabCut – Interactive Foreground Extraction 26

1

0

Foreground

Mix

Back-ground

Foreground Mix Background

Fit a smooth alpha-profile with parameters

Result using DP Border Matting

DP

t

Dynamic ProgrammingDynamic Programming

GrabCut – Interactive Foreground Extraction 27

Noisy alpha-profile Regularisation

t+1

GrabCut BorderGrabCut Border Matting -Matting - ColourColour

Compute MAP of p(F|C,alpha) (marginalize over Compute MAP of p(F|C,alpha) (marginalize over B)B)

To avoid colour bleeding use colour stealing To avoid colour bleeding use colour stealing (“exemplar based inpainting” – Patches do not work)(“exemplar based inpainting” – Patches do not work)

[Chuang et al. ‘01] Grabcut Border Matting

GrabCut – Interactive Foreground Extraction 28

ResultsResults

GrabCut – Interactive Foreground Extraction 29

Stereo Video + Stereo Video + SegmentationSegmentation

Criminisi et. al. (2003): 4-Plane DP to handle occlusions properly

Left Sequence Right Sequence

Disparity Sequence

GrabCut – Interactive Foreground Extraction 30

Occusion, left and rightOccusion, left and right

GrabCut – Interactive Foreground Extraction 31

Background SubstitutionBackground Substitution

Criminisi et. al. (2004): Remove boundary artefacts (SPS algorithm)

GrabCut – Interactive Foreground Extraction 32

Object InsertionObject Insertion

GrabCut – Interactive Foreground Extraction 33

Focus on ForegroundFocus on Foreground

GrabCut – Interactive Foreground Extraction 34

Conclusions & FutConclusions & Future ure WorkWork

GrabCut – powerful interactive extraction tool

Iterated Graph Cut based on colour and contrast

Regularized alpha matting by Dynamic Programming

Stereo and Segmentation give supportive information

How to solve the difficult hair problem ?

GrabCut – Interactive Foreground Extraction 35

[Argawall et.al.2004]