measuring uncertainty in graph cut solutions

Measuring Uncertainty in Graph Cut Solutions

Pushmeet Kohli Philip H.S. Torr

Department of Computing Oxford Brookes University

Objective

Labelling problem

t

s

Graph Cut

st-mincut

Belief or Confidence

Most ProbableSolution

No uncertainty measureassociated with the solution

Outline

Inference in Graphical Models Inference using Graph Cuts Computing Min-marginals using Graph

Cuts Flow Potentials and Min-marginals Results

Inference in Graphical Models

x* = arg max Pr(x|D) x

E(x|D) = -log Pr(x|D) + constant

x* = arg min E(x|D) x

x: Set of latent variablesD: Observed Data

x*: Most Probable (MAP) SolutionPr: Joint Posterior Probability


.2 .1 .05

.35 .05 .2

.01 .03 .01

A

B 0 1 2

0

1

2

Joint Distribution

Pr(A,B)

MAP Solution

arg max Pr(A,B) {A=1, B=0}


Marginal- Sum the joint probability over all other variables.

.2 .1 .05

.35 .05 .2

.01 .03 .01

A

B 0 1 2

0

1

2

P(A=1) = 0.6


.2 .1 .05

.35 .05 .2

.01 .03 .01

A

B 0 1 2

0

1

2

Max- Marginals (µ)- Maximum joint probability over all other variables.

µA,1 = 0.35


.2 .1 .05

.35 .05 .2

.01 .03 .01

A

B 0 1 2

0

1

2

µA,1 = 0.35

µA,0 = 0.2

µA,2 = 0.03

Confidence or Belief (σ)- Normalized max-marginals σA,1 = µA,1 / Σx µA,x

= 0.35/ 0.58= 0.603


Min-Marginals Energies(ψ)- Minimize joint energy over all other variables.

- Related to max-marginals as:

µj = (1/z)*exp(-ψj)




- Can be used to compute confidence as:

σj = µj / Σa µa = exp(-ψi) / Σa exp(-ψa)





- Can be used to compute confidence as:

How to compute min-marginal energies using Graph Cuts?

σj = µj / Σa µa = exp(-ψi) / Σa exp(-ψa)



Graphical Model

Topology

Tree Graph with cycles

Belief Propagation and variants

Exact solution

True Marginals/ min-marginals

Approximate solution

Approximate Marginals/ min-marginals

Graph Cuts

No Marginals/

Min-Marginals

Class 1: Single Max-flow Computation, Exact Solution

Class 2: Expansions/ Swap Moves, Approximate Solution

Outline



• Given energy functions E(x|D),

Compute: arg min E(x|D)

Inference using Graph cuts

x


• Certain E(x|D) can be minimized using graph cuts exactly.


Compute: arg min E(x|D)x


• Class of energy function and graph contruction

- Binary random variables

- Submodular functions (Kolmogorov & Zabih, ECCV 2002)

- Multi-valued variables - Convex Pair-wise Terms (Ishikawa, PAMI 2003)

• Certain E(x|D) can be minimized using graph cuts exactly.


Compute: arg min E(x|D)x

EMRF(a1,a2)

Sink (0)

Source (1)

a1

a2

Graph Construction for Binary Random Variables


a1

a2

EMRF(a1,a2) = 2a1

2t-edges

(unary terms)


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1

a1

a2

2

5


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2

a1

a2

2

5

9

4


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2

a1

a2

2

5

9

4

2

n-edges(pair-wise term)


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1

a1 = 1 a2 = 1

EMRF(1,1) = 11

Cost of st-cut = 11


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1

a1 = 1 a2 = 0

EMRF(1,0) = 8

Cost of st-cut = 8


Sink (0)

Source (1)

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1

a1 = 1 a2 = 0

EMRF(1,0) = 8

Cost of st-cut = 8


MAP Solutiona1,map = 1 a2,map = 0 Sink (0)

Source (1)

Outline



Computing Min-marginals using Graph Cuts


Instead of minimizing E(.), minimize a projection of E(.) where the value of latent

variable xv is fixed to label j.

All projections of a sub-modular function aresub-modular [Kolmogorov and Zabih, ECCV 2002]


Instead of minimizing E(.), minimize a projection of E(.) where the value of latent

variable xv is fixed to label j.

Problem Solved? Not Really!


• Computing a min-marginal requires computation of a st-cut.

• Typical Segmentation problem- 640x480 image, 2 labels- Variables = 640x480 = 307200- Number of Min-marginals = 307200x2 = 614400 - Time taken for 1 graph cut = .3 seconds- Total computation time = 614400x0.3 = 184320 sec

= 51.2 hours!!!

Energy Projections and Graph Construction

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)


EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a2= 1

EMRF(a1,1) = 2a1 + 5ā1+ 9 + ā1

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)


EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

a1

a2

2

6

9

Sink (0)

Source (1) a2= 1

EMRF(a1,1) = 2a1 + 5ā1+ 9 + ā1

EMRF(a1,1) = 2a1 + 6ā1+ 9


EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 + Kā2

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)

∞

A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.

Alternative Construction

K


a1

a2

2

5

9

4

2

1

Sink (0)

Source (1) A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.

• The minimum value of the energy projection can be calculated by using the same graph.



∞K


a1

a2

2

5

9

4

2

1

Sink (0)

Source (1) A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.



K> Sum of Outgoing/Incoming edges


∞K





• Results in a small change in the graph.

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)

∞K




a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)

∞


K


• Results in a small change in the graph.

Solve using Dynamic Graph Cuts


Dynamic Graph Cuts

- Kohli and Torr [ICCV 2005]

Graph 1

Graph 2

Energy function

Projection of Energy function

Dynamic Graph Cuts


Graph 1

Graph 2

Energy function

Projection of Energy function

Graph 1 and 2 are similar

Dynamic Graph Cuts


Graph 1

Graph 2

Dynamic Graph Cuts


Graph 1 Residual Graph 1 st-cut 1


Computationally Expensive Procedure

Compute Max-flow

Compute Max-flow

Dynamic Graph Cuts




Re-parameterized Graph 2

Compute Max-flow

Dynamic Graph Cuts




Re-parameterized Graph 2same

solution

Dynamic Graph Cuts




Re-parameterized Graph 2similar edge

weights

Dynamic Graph Cuts





Extremely Fast Operation

Compute Max-flow

Compute Max-flow

Dynamic Graph Cuts





Extremely Fast Operation

300 msec

0.002msec

Summary of the Algorithm

• Construct graph G for minimizing energy E

• For computing min-marginals do:

- Obtain graph G* ≡ energy function projection E*(By adding constraining edges)

- Find the maximum flow in G* using dynamic graph cut algorithm [Kohli and Torr, ICCV 2005]

Extension to Multiple Labels

Graph construction for multi-label random variables (Ishikawa PAMI 2003)

Labels: l1 …. ln

Latent variables: x1 …. xn

x1 = l3 x2 = l2

MAP Labels

x3 = l2 x4 = l1

Extension to Multiple Labels

Graph construction for multi-label random variables (Ishikawa PAMI 2003)

Labels: l1 …. ln

Latent variables: x1 …. xn

x4 = l3

Graph for Projection

Outline



Min-Marginals and Flow Potentials

• Flow Potentials- maximum amount of flow that can be passed from the node to a particular terminal.


1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph



1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph


Source (s) flow potential of node 4


1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph

Source (s) flow potential of node 4

1 2 3

4 5

s

t

1 1

1 1



1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph


Sink (t) flow potential of node 4


1 2 3

4 5

s

t

3

29

7

1 1 4

3 1

2 11

Graph

Sink (t) flow potential of node 4

1 2 3

4 5

s

t

2

29



Relationship between Min-marginal of binary latent variables and flow potential.

MAP Solution energy

Flow potential in residual graph

+Min-marginal =

For details: See theorem 1.

Implications: Any algorithm for computing flow potentials can be used to compute min-marginals.

Outline



• Typical Segmentation problem- 640x480 image, 2 labels

• Computation Times- Naïve Approach = 51.2 hours!!!- Our Algorithm (Dynamic Graph Cuts) = 1.2 seconds

Experimental Evaluation

Experimental Evaluation

Computation Times (in seconds) for binary variables

Problem Size

(No. of Variables/

Neighbourhood)

MAP solution

(single maxflow computation)

Computing all min-marginals

1x105, 4-neighbourhood 0.18 0.70

2x105, 4-neighbourhood 0.46 1.34

4x105 , 4-neighbourhood 0.92 3.15






Min-Marginals in Image segmentation

Unary likelihood Contrast Term

Uniform Prior(Potts Model)

Image Segmentation Energy Boykov and Jolly [ICCV 2001], Blake et al. [ECCV 2004]

xi = binary variable representing label (‘fg’ or ‘bg’) of pixel i


ImageMAP Solution Belief - Foreground

Lowsmoothness

Highsmoothness

Moderatesmoothness

Colour Scale

1

0

0.5

[MSR]


encourage smoothness








How smoothness effects solutions?

encourage smoothness


Effect of increasing pair-wise terms of the energy function

Image


Effect of increasing pair-wise terms of the energy function

MAP Segmentation Foreground Confidence Map

Concluding Remarks

• Efficient method for computing exact min-marginals for certain labelling problems• Relationship between Flow-potentials and min-marginals

Applications• Computing M Most Probable Solutions• Min-marginals for parameter learning• Hierarchical Segmentation

Future Work• Efficient min-marginal computation for general problems

Thank You

measuring uncertainty in graph cut solutions

Documents

graph cutscomputing

graphical modelsinference

graphical modelsa b

graphical modelsx

joint energy

graphical modelsmarginal

graph cutsflow potentials

energy functions exd