photomontage

GrabCut GrabCut Interactive ImageInteractive Image

((andand Stereo) Stereo) SSegmentation egmentation

Carsten RotherCarsten RotherVladimir Kolmogorov Vladimir Kolmogorov

Andrew BlakeAndrew BlakeAntonio CriminisiAntonio Criminisi

Geoffrey CrossGeoffrey Cross [based on Siggraph 2004 and ECCV 2004][based on Siggraph 2004 and ECCV 2004]

PhotomontagePhotomontage

GrabCut – Interactive Foreground Extraction 1

Talk OutlineTalk Outline

Hard Image Segmentation: Fore- vs. BackgroundHard Image Segmentation: Fore- vs. Background

Soft Segmentation: Alpha Matting

Stereo Segmentation: Exploit Depth


Problem Problem


Fast & Accurate ?

What GrabCut doesWhat GrabCut does

User Input

Result

Magic Wand (198?)

Intelligent ScissorsMortensen and Barrett (1995)

GrabCut

Regions Boundary Regions & Boundary


FrameworkFramework

Input: Image

Output: Segmentation

Parameters: Colour ,Coherence

Energy:

Optimization:


Maximum a posteriori estimator (MAP):

• Gibbs Distribution of the MRF

same as

Energy – Probabilistic Energy – Probabilistic ViewView


- log

Graph CutsGraph Cuts - - Boykov and Jolly Boykov and Jolly (2001)(2001)


ImageImage Min CutMin Cut

Cut: separating source and sink; Energy: collection of edges

Min Cut: Global minimal enegry in polynomial time

Foreground Foreground (source)(source)

BackgroundBackground(sink)(sink)

Iterated Graph CutIterated Graph Cut

User Initialisation

K-means for learning

colour distributions

Graph cuts to infer the

segmentation

?


1 2 3 4

Iterated Graph CutsIterated Graph Cuts


Energy after each IterationResult

Guaranteed to

converge

Colour ModelColour Model

Gaussian Mixture Model Gaussian Mixture Model (typically 5-8 components)(typically 5-8 components)

Foreground &Background

Background

Foreground

BackgroundG


R

G

RIterated graph cut

Coherence ModelCoherence ModelAn object is a coherent set of pixels:

Error (%) over training set:

25

How do we choose ?

25

Gaussian MRF:

Linear regression gives in closed-form

Pseudo-Likelihood:

Parameter Learning Parameter Learning (Blake (Blake 2004)2004)


approximation

=

A Gaussian MRF is not a realistic texture model

Gaussian? Gaussian!Real Image syntheticGMRF

Parameter Learning - Parameter Learning - ProblemsProblems


Moderately simple Moderately simple examplesexamples

… … GrabCut completes automaticallyGrabCut completes automatically GrabCut – Interactive Foreground Extraction 14

Difficult ExamplesDifficult Examples

Camouflage & Camouflage & Low ContrastLow Contrast No telepathyNo telepathyFine structureFine structure

Initial Rectangle

InitialResult


Evaluation – Labelled Evaluation – Labelled DatabaseDatabase

Available online: http://research.microsoft.com/vision/cambridge/segmentation/


Comparison Comparison GrabCutBoykov and Jolly (2001)

Error Rate: 0.72%Error Rate: 1.87%Error Rate: 1.81%Error Rate: 1.32%Error Rate: 1.25%Error Rate: 0.72%


User Input

Result

ComparisonComparison

Trimap Boykov and Jolly

Error Rate: 1.36%

Input Image Ground Truth BimapGrabCut

Error Rate: 2.13%


Error rate - modestly increase

User Interactions - considerable reduced

Results Parameter Results Parameter LearningLearning



Magic Wand (198?)

Intelligent Scissors Mortensen and Barrett (1995)

GrabCutRother et al. (2004)

Graph Cuts Boykov and Jolly (2001)

LazySnappingLi et al. (2004)


““Mixed pixels”: Combination of fore- and Mixed pixels”: Combination of fore- and background background

Alpha Mask: Proportion of fore- and background Alpha Mask: Proportion of fore- and background Natural Matting Problem: Determine alpha,F,B Natural Matting Problem: Determine alpha,F,B

from C from C

Under-determined System: 3 Equations and 7 unknowns

Digital MattingDigital Matting


1. Simple Alpha & Simple Colour 2. Difficult Alpha & Simple Colour

3. Simple Alpha & Difficult Colour

4. Difficult Alpha & Difficult Colour

Existing Methods

Human ?

GrabCut


Transparency - Transparency - TaxonomieTaxonomie

Border Matting Border Matting

Hard Segmentation Automatic Trimap Soft Segmentation


to

Input Bayes MattingChuang et. al. (2001)

Knockout 2Photoshop Plug-In



With no regularisation over alpha

Shum et. al. (2004): Coherence matting in “Pop-up light fields”

Solve

Mean Colour Foreground

Mean ColourBackground


Natural Image MattingNatural Image Matting

Ruzon and Tomasi (2000): Alpha estimation in natural images

Noisy alpha-profile

Border MattingBorder Matting


1

0

Foreground

Mix

Back-ground

Foreground Mix Background

Fit a smooth alpha-profile with parameters

Result using DP Border Matting

DP

t

Dynamic ProgrammingDynamic Programming


Noisy alpha-profile Regularisation

t+1

GrabCut BorderGrabCut Border Matting -Matting - ColourColour

Compute MAP of p(F|C,alpha) (marginalize over Compute MAP of p(F|C,alpha) (marginalize over B)B)

To avoid colour bleeding use colour stealing To avoid colour bleeding use colour stealing (“exemplar based inpainting” – Patches do not work)(“exemplar based inpainting” – Patches do not work)

[Chuang et al. ‘01] Grabcut Border Matting


ResultsResults


Stereo Video + Stereo Video + SegmentationSegmentation

Criminisi et. al. (2003): 4-Plane DP to handle occlusions properly

Left Sequence Right Sequence

Disparity Sequence


Occusion, left and rightOccusion, left and right


Background SubstitutionBackground Substitution

Criminisi et. al. (2004): Remove boundary artefacts (SPS algorithm)


Object InsertionObject Insertion


Focus on ForegroundFocus on Foreground


Conclusions & FutConclusions & Future ure WorkWork

GrabCut – powerful interactive extraction tool

Iterated Graph Cut based on colour and contrast

Regularized alpha matting by Dynamic Programming

Stereo and Segmentation give supportive information

How to solve the difficult hair problem ?


[Argawall et.al.2004]

photomontage

Documents

difficult alpha simple

simple alpha difficult

simple alpha simple

alpha estimation

background alpha mask

alpha shum et

smooth alphaprofile

coherence matting