graph cut & energy minimization yevgeny doctor ip seminar 2008, idc

Graph Cut &

Energy Minimization

Yevgeny Doctor

IP Seminar 2008, IDC

Yevgeny Doctor, March 2008, IDC 2

Outline

Motivation Graph Cuts Energy Minimization in Computer Vision Solving Energy Minimization with Graph Cuts Results


Graph Cut

G(V,E) is a finite directed graph and every edge (u,v) has a capacity c(u,v) (a non-negative real number).

Assume two vertices, the source s and the sink t, have been distinguished.

A cut is a split of the nodes into two sets S and T, such that s is in S and t is in T.

ts


Min-Cut

The capacity of a cut (S,T ) is defined as

Min-Cut – finding the cut with the minimal capacity

Sx Ty

yxcTSc ,,


Max Flow

Find the maximum flow from s to t


Min-Cut Max-Flow

Min-Cut and Max-Flow are equivalent problems

Fast polynomial algorithms for max flow exist For example:

Ford-Fulkerson algorithm – O(E*maxflow)Edmonds-Karp algorithm – O(VE2)


Energy Minimization Problems

In general terms, given some problem, we: Formulate the known constraints Build an “energy function” (aka “cost function”) Look for a solution that minimizes it

If we have no further knowledge: The problem can be NP-Hard (requires exponential

solution time) Use slow, generic approximation algorithms for

optimization problems (such as simulated annealing)


EM in Computer Vision

Consider a broad class of problems called Pixel Labeling Given some images we want to “say something about the pixels” For each pixel p, give it a label fp from a finite set of labels L,

such that we minimize some energy function.

Many applications Image Segmentation Image Restoration Stereo and Motion Medical Imaging Multicamera Scene Reconstruction


Pixel Labeling Example

"Graph Cut Matching In Computer Vision", Toby Collins, February 2004http://homepages.inf.ed.ac.uk/rbf/CVonline/


EM in Computer Vision

Consider a specific family of Energy Functions Powerful enough to formulate many useful problems Can be reduced to solving a graph min-cut problem

Problems defined with these functions: Can be solved quickly (using max-flow algorithms) In many cases – optimal solution or within a known

factor of the optimum


Nqp

qpqpp

pp ffVfDfE,

, ,

Input: set of pixels P, set of labels L, is a neighbourhood system on pixels.

Goal: find a labeling that minimizes . . is a function derived from the observed data that

measures the cost of assigning label to pixel p. . measures the cost of assigning the labels

to adjacent pixels p, q. Used to impose spatial smoothness.

Energy Function Definition

PPN

LPf : fE pp fD

qpqp ffV ,, qp ff ,pf

Esmooth(f)Edata(f)


The Edata(f) component: Look at each pixel independently Given it’s current value, what would it cost to label it with

each of the labels? Examples:

Cost based on a-priori known pixel intensity or color distribution is the observed intensity of pixel p

What if we used only this component in E(f)? Label each pixel independently with the most likely

(cheap) label

Energy Function - Edata Component

ppp iif ,2


What would be the problem? For example (object segmentation):

Images from “Nearest Neighbor” lecture – here for illustration purposes only

We need to add a “smoothness” cost

Optimizing Edata Only – Illustration


Energy Function - Esmooth Component

Look at all pairs of neighbor pixels Penalize adjacent pixels with different labels What smoothness cost function to use?

Noised diamond imageFast Approximate Energy Minimization via Graph Cuts

Yuri Boykov, Olga Veksler, Ramin Zabih


Potts Interaction Penalty: T – indicator function, K – constant The solution will be piecewise constant, with

discontinuities at the boundaries

Smoothness Cost Functions

Nqp

qpsmooth ffTKfE,

Fast Approximate Energy Minimization via Graph Cuts



L2 distance:

What is the problem? High penalties at object boundaries We want smooth objects, but allow different labels at

object boundary – a discontinuity-preserving function.


Nqp

qpsmooth fffE,


Truncated L2 distance:

K – constant The solution will be piecewise smooth, with

discontinuities at the boundaries




Nqp

qpsmooth ffKfE,

,min


Normalize for neighbor distance, image contrast:

This function penalizes a lot for discontinuities between pixels of similar intensities when |Ip − Iq| < .

However, if pixels are very different, |Ip − Iq| > , then the penalty is small.


Nqp

II

qpdist

ffT

smoothqpqpfE

,2, 2

2

exp


Solving Energy Minimization with Graph Cuts

Many classes of Energy Minimization problems in Computer Vision can be reduced to Graph Cuts

Solve multiple-labels problems with binary decisions


“Fast Approximate Energy Minimization via Graph Cuts.” Yuri Boykov, Olga Veksler, Ramin Zabih, 1999

For two classes of interaction potentials V (Esmooth): V is semi-metric on a label space L if for every :

V is metric on L if in addition, triangle inequality holds:

For example, truncated L2 distance and Potts Interaction Penalty are both metric.

Approximate Energy Minimization

L ,

0,, VV 0,V

LVVV ,,,,,


Swap-Move algorithm: 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each pair of labels

3.1. Find f* = argmin E(f') among f' within one swap of f 3.2. If E(f*) < E(f), set f := f* and success := 1

4. If success = 1 goto 2 5. Return f

swap: In the new labeling f’, some pixels that were labeled

in f are now labeled , and vice versa.

Solution for Semi-metric Class

L ,


Solve swap step with Graph Cut

Graph:


Yuri Boykov, Olga Veksler, Ramin Zabih, 1999


Solve swap step with Graph Cut

Cut and Labeling:

Weights:




Expansion Move algorithm: 1. Start with an arbitrary labeling f 2. Set success := 0 3. For each label

3.1. Find f* = argmin E(f') among f' within one expansion of f 3.2. If E(f*) < E(f), set f := f* and success := 1

4. If success = 1 goto 2 5. Return f

expansion: In the new labeling some pixels change their label to .

Solution for Metric Class

L


Example:

Guaranteed approximation ratio by the algorithm: Produces a labeling f such that ,

where f* is the global minimum and .

Expansion Move Algorithm

"What Energy Functions Can Be Minimized via Graph Cuts?", Vladimir Kolmogorov and Ramin Zabih, February 2004IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 26, NO. 2.

*2* fEkfEfE

:,min

:,max

V

Vk


Solve expansion step with Graph Cut

Graph:




Solve expansion step with Graph Cut

Cut and Labeling:

Weights:




Three energy functions:Truncated quadratic – semi-metric (swap-move):

Potts model – metric (-expansion):

Truncated L2 distance – metric (-expansion):

Compared with:Best simulated annealing algorithm

Experimental Results

Nqp

qpPp

pp ffKifE,

221 ,min

Nqp

qpPp

pp ffTKifE,

22

Nqp

qpPp

pp ffKifE,

23 ,min


Convergence Rate




Image Restoration




Motion and Stereo




What Energy Functions Can Be Minimized via Graph Cuts?

A precise characterization of the class of functions in F3 that can be minimized using graph cuts

A general purpose graph construction for minimizing any function in this class

A necessary condition for any function of binary variables to be minimized via graph cuts.



Define general Energy Function classes:Given F2:

F3:

F2 is contained in F3




Regular F2 functions:




Graph construction for F2:




Regular F3 functions:

A function E of more than two variables is called regular if all projections of E of two variables are regular

And Generally: Let E be a function of binary variables. If E is

not regular, then E is not graph-representable.



GrabCut

"Interactive Foreground Extraction using Iterated Graph Cuts"

Carsten Rother, Vladimir Kolmogorov, Andrew Blake, 2004Microsoft Research Cambridge, UK


GrabCut


GrabCut

Transparency:


GrabCut


Video Cutout

"Video Object Cut and Paste"Yin Li, Jian Sun, Heung-Yeung Shum, 2005

Microsoft Research Asia

Figure 1: Given a video (a) containing an opaque video object on a complicated background, our system can generate clear alpha mattes and foreground colors (b) for the video object, which can be pasted onto another background.


Video Cutout

3D segmentation of spatio-temporal video volume Partition into foreground and background, while

preserving temporal coherence Local refinement of initial segmentation Provide brush tools for the user to control the object

boundary precisely wherever needed Apply coherent matting to extract the alpha mattes and

foreground colors of the object based on the accurate binary segmentation result


Video Cutout


Video Cutout

Figure 7: Clip #2, frame 27. (a) 3D graph cut result is shown by the overlaid dashed line. The flag is a rapidly deforming object, but 3D graph cut can apture the shape very well. (b) Dashed lines indicate the boundaries after both the local refinement and overriding operations. The white pixels record the actions of foreground brushing and the black pixels for background brushing. (c) coherence matting result pasted on a blue screen.

graph cut & energy minimization yevgeny doctor ip seminar 2008, idc

Documents

idcyevgeny doctor

tyevgeny doctor

esmoothfedatafyevgeny

optimumyevgeny doctor

ukrbfcvonline yevgeny

minimal capacityyevgeny

pixel intensity

pixel labelinggiven