graph cut based optimization for computer vision

7/31/2019 Graph Cut based Optimization for Computer Vision

http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 1/240



Overview

• Motivation

• Min-Cut / Max-Flow (Graph Cut) Algorithm

• Markov and Conditional Random Fields

• Random Field Optimisation using Graph Cuts

• Submodular vs. Non-Submodular Problems

• Pairwise vs. Higher Order Problems

• 2-Label vs. Multi-Label Problems

• Recent Advances in Random Field Optimisation

• Conclusions



Overview

• Motivation








• Conclusions



Image Labelling Problems

Image Denoising Geometry Estimation Object Segmentation

Assign a label to each image pixel

Building

Sky

TreeGrass

Depth Estimation




• Labellings highly structured

Possible labelling Unprobable labelling Impossible labelling




• Labellings highly structured

• Labels highly correlated with very complex dependencies

• Neighbouring pixels tend to take the same label

• Low number of connected components

• Classes present may be seen in one image

• Geometric / Location consistency

• Planarity in depth estimation

• … many others (task dependent)




• Labelling highly structured


• Independent label estimation too hard







• Whole labelling should be formulated as one optimisationproblem








• Number of pixels up to millions

• Hard to train complex dependencies• Optimisation problem is hard to infer








• Number of pixels up to millions

• Hard to train complex dependencies• Optimisation problem is hard to infer

Vision is hard !




• You can

either

• Change the subject from the ComputerVision to the History of Renaissance Art

Vision is hard !




• You can

either

• Change the subject from the ComputerVision to the History of Renaissance Art

or

• Learn everything about Random Fields and

Graph Cuts

Vision is hard !



Foreground / Background Estimation

Rother et al. SIGGRAPH04




Data term Smoothness term

Data term

Estimated using FG / BG

colour models

Smoothness term

where

Intensity dependent smoothness





How to solve this optimisation problem?





How to solve this optimisation problem?

• Transform into min-cut / max-flow problem

• Solve it using min-cut / max-flow algorithm



Overview

• Motivation







• Applications

• Conclusions



Max-Flow Problem

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

Task :

Maximize the flow from the sink to the source such that

1) The flow it conserved for each node

2) The flow for each pipe does not exceed the capacity



Max-Flow Problem

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3



Max-Flow Problem

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

flow from thesourcecapacity

flow from node ito node j

conservation of flow

set of edges

set of nodes



Max-Flow Problem

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

Ford & Fulkerson algorithm (1956)

Find the path from source to sink

While (path exists)

flow += maximum capacity in the path

Build the residual graph (“subtract” the flow)

Find the path in the residual graph

End



Max-Flow Problem

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3



While (path exists)




End



Max-Flow Problem

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3



While (path exists)




End

flow = 3



Max-Flow Problem

source

sink

9

5

6-3

8-3

42+3

2

2

5-3

3-3

5

5

311

6

2

3



While (path exists)




End

flow = 3

+3



Max-Flow Problem

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

3



While (path exists)




End

flow = 3

3

l bl



Max-Flow Problem

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

3



While (path exists)




End

flow = 6

3

M Fl P bl



Max-Flow Problem

source

sink

9-3

5

3

5-3

45

2

2

2

5

5

3-311

6

2

3-3



While (path exists)




End

flow = 6

3

+3

+3

M Fl P bl



Max-Flow Problem

source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2



While (path exists)




End

flow = 6

3

3

3

M Fl P bl



Max-Flow Problem

source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2



While (path exists)




End

flow = 11

3

3

3

M Fl P bl



Max-Flow Problem

source

sink

6-5

5-5

3

2

45

2

2

2

5-5

5-5

1+51

6

2+5



While (path exists)




End

flow = 11

3

3

3

M Fl P bl



Max-Flow Problem

source

sink

1

3

2

45

2

2

2

61

6

7



While (path exists)




End

flow = 11

3

3

3

M Fl P bl



Max-Flow Problem

source

sink

1

3

2

45

2

2

2

61

6

7



While (path exists)




End

flow = 13

3

3

3

Max Flow Problem



Max-Flow Problem

source

sink

1

3

2-2

45

2-2

2-2

2-2

61

6

7



While (path exists)




End

flow = 13

3

3

3

+2

+2

Max Flow Problem



Max-Flow Problem

source

sink

1

3

45

61

6

7



While (path exists)




End

flow = 13

3

3

3

2

2

Max Flow Problem



Max-Flow Problem

source

sink

1

3

45

61

6

7



While (path exists)




End

flow = 15

3

3

3

2

2

Max Flow Problem



Max-Flow Problem

source

sink

1

3-2

4-25

61

6-2

7



While (path exists)




End

flow = 15

3

3

3+2

2

2-2

+2

Max Flow Problem



Max-Flow Problem

source

sink

1

1

5

61

4

7



While (path exists)




End

flow = 15

3

3

5

2

2

2

Max Flow Problem



Max-Flow Problem

source

sink

1

1

5

61

4

7



While (path exists)




End

flow = 15

3

3

5

2

2

2

Max-Flow Problem



Max-Flow Problem

source

sink

1

1

5

61

4

7


Why is the solution globally optimal ?

flow = 15

3

3

5

2

2

2

Max-Flow Problem



Max-Flow Problem

source

sink

1

1

5

61

4

7



1. Let S be the set of reachable nodes in the

residual graph

flow = 15

3

3

5

2

2

2

S

Max-Flow Problem



Max-Flow Problem



1. Let S be the set of reachable nodes in the

residual graph

2. The flow from S to V - S equals to the sum

of capacities from S to V – S

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S

Max-Flow Problem



Max-Flow Problem



1. Let S be the set of reachable nodes in theresidual graph

2. The flow from S to V - S equals to the sum of

capacities from S to V – S3. The flow from any A to V - A is upper bounded

by the sum of capacities from A to V – A

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

A

Max-Flow Problem



Max-Flow Problem

flow = 15







4. The solution is globally optimal

source

sink

8/9

5/5

5/6

8/8

2/4

0/2

2/2

0/2

5/5

3/3

5/5

5/5

3/30/10/1

2/6

0/2

3/3

Individual flows obtained by summing up all paths

Max-Flow Problem



Max Flow Problem

flow = 15







4. The solution is globally optimal

source

sink

8/9

5/5

5/6

8/8

2/4

0/2

2/2

0/2

5/5

3/3

5/5

5/5

3/30/10/1

2/6

0/2

3/3

Max-Flow Problem



Max Flow Problem

source

sink

1000

1000


Order does matter

1000

1000

1

Max-Flow Problem



Max Flow Problem

source

sink

999

999


Order does matter

1000

1000

1

Max-Flow Problem



Max Flow Problem

source

sink

999

999


Order does matter

999

999

1

Max-Flow Problem



Max Flow Problem

source

sink

999

999


Order does matter

• Standard algorithm not polynomial

• Breath first leads to O(VE2)

• Path found in O(E)

• At least one edge gets saturated

• The saturated edge distance to thesource has to increase and is atmost V leading to O(VE)

(Edmonds & Karp, Dinic)

999

999

1

Max-Flow Problem



Max Flow Problem

source

sink

999

999


Order does matter

• Standard algorithm not polynomial

• Breath first leads to O(VE2)

(Edmonds & Karp)

• Various methods use different algorithm

to find the path and vary in complexity

999

999

1

Min-Cut Problem



Min Cut Problem

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

Task :

Minimize the cost of the cut

1) Each node is either assigned to the source S or sink T

2) The cost of the edge (i, j) is taken if (iS) and (jT)

Min-Cut Problem



Cu ob e

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

source set sink set

edge costs

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

source set sink set

edge costsS

T

cost = 18

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

source set sink set

edge costsS

T

cost = 25

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

source set sink set

edge costsS

T

cost = 23

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1x2

x3

x4

x5

x6

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1x2

x3

x4

x5

x6

transformable into Linear program (LP)

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1x2

x3

x4

x5

x6

Dual to max-flow problem

transformable into Linear program (LP)

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1x2

x3

x4

x5

x6

After the substitution of the constraints :

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x i = 0 x i S x i = 1 x i T

x1x2

x3

x4

x5

x6

source edges sink edges

Edges betweenvariables

Min-Cut Problem



source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

C( x ) = 5x 1

+ 9x 2

+ 4x 3

+ 3x 3 (1-x

1 ) + 2x

1(1-x

3 )

+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )

+ 1x 5 (1-x 1 ) + 6x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )

+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )

+ 8(1-x 5 ) + 5(1-x 6 )

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 5x 1

+ 9x 2

+ 4x 3

+ 3x 3 (1-x

1 ) + 2x

1(1-x

3 )

+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )

+ 1x 5 (1-x 1 ) + 6x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )

+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )

+ 8(1-x 5 ) + 5(1-x 6 )

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 2x 1

+ 9x 2

+ 4x 3

+ 2x 1(1-x

3 )

+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )

+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )

+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )

+ 5(1-x 5 ) + 5(1-x 6 )

+ 3x 1 + 3x 3 (1-x 1 ) + 3x 5 (1-x 3 ) + 3(1-x 5 )

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 2x 1

+ 9x 2

+ 4x 3

+ 2x 1

(1-x 3

)

+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )

+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )

+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )

+ 5(1-x 5 ) + 5(1-x 6 )

+ 3x 1 + 3x 3 (1-x 1 ) + 3x 5 (1-x 3 ) + 3(1-x 5 )

3x 1

+ 3x 3 (1-x

1 ) + 3x

5 (1-x

3 ) + 3(1-x

5 )

=

3 + 3x 1(1-x 3 ) + 3x 3 (1-x 5 )

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 3 + 2x 1

+ 9x 2

+ 4x 3

+ 5x 1

(1-x 3

)

+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )

+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )

+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )

+ 5(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 )

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

33

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 3 + 2x 1

+ 6x 2

+ 4x 3

+ 5x 1

(1-x 3

)

+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )

+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )

+ 2x 5 (1-x 4 ) + 6(1-x 4 )

+ 2(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 )

+ 3x 2 + 3x 6 (1-x 2 ) + 3x 5 (1-x 6 ) + 3(1-x 5 )

3x 2

+ 3x 6 (1-x

2 ) + 3x

5 (1-x

6 ) + 3(1-x

5 )

=

3 + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

33

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 6 + 2x 1

+ 6x 2

+ 4x 3

+ 5x 1

(1-x 3

)

+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )

+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )

+ 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )

+ 2(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 )

source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2

3

3

3

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 11 + 2x 1

+ 1x 2

+ 4x 3

+ 5x 1

(1-x 3

)

+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 4 (1-x 1 )

+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 )

+ 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )

+ 2(1-x 5 ) + 3x 3 (1-x 5 )

source

sink

1

3

2

45

2

2

2

61

6

7

3

3

3

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 13 + 1x 2

+ 4x 3

+ 5x 1

(1-x 3

)

+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )

+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 )

+ 2x 4 (1-x 5 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )

+ 3x 3 (1-x 5 )

source

sink

1

3

45

61

6

7

3

3

3

2

2

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 13 + 1x 2

+ 2x 3

+ 5x 1

(1-x 3

)

+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )

+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 )

+ 2x 4 (1-x 5 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )

+ 3x 3 (1-x 5 )

source

sink

1

3

45

61

6

7

3

3

3

2

2

x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 15 + 1x 2 + 4x 3 + 5x 1(1-x 3 )

+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )

+ 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 )

+ 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )

source

sink

1

1

5

61

4

7

3

3

5

2

2

2x1x2

x3

x4

x5

x6

Min-Cut Problem



C( x ) = 15 + 1x 2 + 4x 3 + 5x 1(1-x 3 )

+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )

+ 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 )

+ 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )

source

sink

1

1

5

61

4

7

3

3

5

2

2

2x1x2

x3

x4

x5

x6 cost = 0

min cut

S

T

cost = 0

• All coefficients positive• Must be global minimum

S – set of reachable nodes from s



Overview



• Motivation



• Random Field Optimisation using Graph Cuts• Submodular vs. Non-Submodular Problems


• 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation

• Conclusions

Markov and Conditional RF



• Markov / Conditional Random fields modelprobabilistic dependencies of the set of randomvariables




• Markov / Conditional Random fields model conditionaldependencies between random variables

• Each variable is conditionally independent of all othervariables given its neighbours






• Posterior probability of the labelling x given data D is :

partition function potential functionscliques






• Posterior probability of the labelling x given data D is :

• Energy of the labelling is defined as :

partition function potential functionscliques




• The most probable (Max a Posteriori (MAP)) labellingis defined as:





• The only distinction (MRF vs. CRF) is that in the CRFthe conditional dependencies between variables

depend also on the data





• The only distinction (MRF vs. CRF) is that in the CRFthe conditional dependencies between variables

depend also on the data• Typically we define an energy first and then pretend

there is an underlying probabilistic distribution there,but there isn’t really (Pssssst, don’t tell anyone)

Overview



• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm


• Random Field Optimisation using Graph Cuts• Submodular vs. Non-Submodular Problems


• 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation

• Conclusions

Pairwise CRF models



Standard CRF Energy


Graph Cut based Inference



The energy / potential is called submodular (only) if for everypair of variables :




For 2-label problems x {0, 1} :





Pairwise potential can be transformed into :

where






where




After summing up :




After summing up :

Let :




After summing up :

Let : Then :




After summing up :

Let : Then :

Equivalent st-mincut problem is :





Data term

Estimated using FG / BGcolour models

Smoothness term

where

Intensity dependent smoothness




source

sink

Rother et al. SIGGRAPH04




Extendable to (some) Multi-label CRFs

• each state of multi-label variable encoded using

multiple binary variables

• the cost of every possible cut must be the same as

the associated energy• the solution obtained by inverting the encoding

Obtain

solutionGraph Cut

Build

Graph

Encoding

multi-label energy solution

Dense Stereo Estimation



Left Camera Image Right Camera Image Dense Stereo Result

• For each pixel assigns a disparity label : z• Disparities from the discrete set {0 , 1, .. D }




Data term

Left image

feature

Shifted right

image feature

Left Camera Image Right Camera Image




Data term

Left image

feature

Shifted right

image feature

Smoothness term




Ishikawa PAMI03

Encoding sour

ce

xi0

xi D

x k0

x k D

. .

.

x k1 x k

2

source

sink

.




Ishikawa PAMI03

Unary Potential

sour

ce

xi0

xi D

x k0

x k D

. .

.

x k1 x k

2 ∞

∞

∞

∞

∞

∞

∞ ∞

source

sink

.

cost = +

The cost of every cut should beequal to the corresponding

energy under the encoding



Ishikawa PAMI03

Unary Potential

sour

ce

xi0

xi D

x k0

x k D

. .

.

x k1 x k

2 ∞

∞

∞

∞

∞

∞

∞ ∞

source

sink

.

cost = ∞






Ishikawa PAMI03

Pairwise Potential

sour

ce

xi0

xi D

x k0

x k D

. .

.

x k1 x k

2 ∞

∞

∞

∞

∞

∞

∞ ∞

K

K

K

K

source

sink

.

cost = + + K





See [Ishikawa PAMI03] for more details

sour

ce

xi0

xi D

x k0

x k D

. .

.

x k1 x k

2 ∞

∞

∞

∞

∞

∞

∞ ∞

source

sink

.

Extendable to any convex cost

Convex function




Move making algorithms

• Original problem decomposed into a series of subproblemssolvable with graph cut

• In each subproblem we find the optimal move from the

current solution in a restricted search space

Update

solutionGraph Cut

Build

Graph

Encoding

Propose

move

Initial solutionsolution


Example : [B k t l 01]



Example : [Boykov et al. 01]

α-expansion

– each variable either keeps its old label or changes to α

– all α-moves are iteratively performed till convergenceTransformation function :

α-swap

– each variable taking label α or can change its label to α or

– all α-moves are iteratively performed till convergence


Sufficient conditions for move making algorithms :




(all possible moves are submodular)

α-swap : semi-metricity

Proof:






(all possible moves are submodular)

α-expansion : metricity

Proof:

Object-class Segmentation



Data term Smoothness termData term

Smoothness term

where

Discriminatively trained classifier




Original Image Initial solution

grass




Original Image Initial solution Building expansion

grass

grass

building





Sky expansion

grass

grass

grass

building

building

sky





Sky expansion Tree expansion

grass

grass

grass grass

building

building building

sky sky

tree





Sky expansion Tree expansion Final Solution

grass

grass

grass grass grass

building

building buildingbuilding

sky sky sky

tree treeaeroplane




Range-expansion [Kumar, Veksler & Torr 1]

– Expansion version of range swap moves

– Each varibale can change its label to any label in a convexrange or keep its old label

Range-(swap) moves [Veksler 07] – Each variable in the convex range can change its label to any

other label in the convex range

– all range-moves are iteratively performed till convergence

Dense Stereo Reconstruction



Data term

Smoothness term

Same as before





Data term

Smoothness term

Same as before


Convex range Truncation




Original Image Initial Solution




Original Image Initial Solution After 1st expansion





After 2nd expansion

After 1st expansion





After 2nd expansion

After 1st expansion

After 3rd expansion





After 2nd expansion

After 1st expansion

After 3rd expansion Final solution


E t dibl t t i l f bi hi h d i



• Extendible to certain classes of binary higher order energies

• Higher order terms have to be transformable to the pairwise submodularenergy with binary auxiliary variables z

C

Higher order term Pairwise term

Update

solutionGraph Cut

Transf. to

pairwise subm.

Encoding

Propose

move



E t dibl t t i l f bi hi h d i



• Extendible to certain classes of binary higher order energies

• Higher order terms have to be transformable to the pairwise submodularenergy with binary auxiliary variables z

C

Higher order term Pairwise term

Example :

Kolmogorov ECCV06, Ramalingam et al. DAM12




Enforces label consistency of the clique as a weak constraint

Input image Pairwise CRF Higher order CRF

Kohli et al. CVPR07




Update

solutionGraph Cut

Transf. to

pairwise subm.

Encoding

Propose

move

Initial solution solution

If the energy is not submodular / graph-representable – Overestimation by a submodular energy

– Quadratic Pseudo-Boolean Optimisation (QPBO)


Overestimation by a submodular energy (convex / concave)



Update

solutionGraph Cut

Overestim. by

pairwise subm.

Encoding

Propose

move


y gy

– We find an energy E’(t) s.t.

– it is tight in the current solution E’(t0) = E(t0)

– Overestimates E(t) for all t E’(t) ≥ E(t)

– We replace E(t) by E’(t)

–

The moves are not optimal, but quaranteed to converge – The tighter over-estimation, the better

Example :


Quadratic Pseudo-Boolean Optimisation




–

Each binary variable is encoded using two binaryvariables x i and x i s.t. x i = 1 - x i _ _






source

sink

xix j

xi x j

_ _ – Energy submodular in x i and x i

– Solved by dropping the constraint x i = 1 - x i

– If x i = 1 - x i the solution for xi is optimal

_

_ _

Overview



• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm







• Conclusions

Motivation



• To propose formulations enforcing certain structureproperties of the output useful in computer vision

– Label Consistency in segments as a weak constraint

– Label agreement over different scales

– Label-set consistency

– Multiple domains consistency

• To propose feasible Graph-Cut based inference method

Structures in CRF



• Taskar et al. 02 – associative potentials

• Woodford et al. 08 – planarity constraint

• Vicente et al. 08 – connectivity constraint

• Nowozin & Lampert 09 – connectivity constraint

• Roth & Black 09 – field of experts

• Woodford et al. 09 – marginal probability

• Delong et al. 10 – label occurrence costs

Object-class Segmentation Problem

• Aims to assign a class label for each pixel of an image



• Aims to assign a class label for each pixel of an image

• Classifier trained on the training set

• Evaluated on never seen test images

Pairwise CRF models



Standard CRF Energy for Object Segmentation

Cannot encode global consistency of labels!!

Local context

Ladický, Russell, Kohli, Torr ECCV10

Encoding Co-occurrence

C i f l[Heitz et al. '08]



• Thing – Thing

• Stuff - Stuff

• Stuff - Thing

[ Images from Rabinovich et al. 07 ]

Co-occurrence is a powerful cue[ ]

[Rabinovich et al. ‘07]


Encoding Co-occurrence

Co occurrence is a powerful cue[Heitz et al. '08]



• Thing – Thing

• Stuff - Stuff

• Stuff - Thing

Co-occurrence is a powerful cue[ ]

[Rabinovich et al. ‘07]

Proposed solutions :

1. Csurka et al. 08 - Hard decision for label estimation

2. Torralba et al. 03 - GIST based unary potential

3. Rabinovich et al. 07 - Full-connected CRF

[ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10

So...



What properties should these global

co-occurence potentials have ?


Desired properties



1. No hard decisions


Desired properties




Incorporation in probabilistic framework

Unlikely possibilities are not completely ruled out


Desired properties




2. Invariance to region size


Desired properties





Cost for occurrence of {people, house, road etc .. }

invariant to image area


Desired properties





The only possible solution :

Local context Global context

Cost defined over the

assigned labels L(x)

L(x)={ , , }


Desired properties




2. Invariance to region size3. Parsimony – simple solutions preferred

L(x)={ building, tree, grass, sky }

L(x)={ aeroplane, tree, flower,

building, boat, grass, sky }


Desired properties





4. Efficiency


Desired properties





4. Efficiency

a) Memory requirements as O(n) with the image size and

number or labels

b) Inference tractable


Previous work



• Torralba et al.(2003) – Gist-based unary potentials

• Rabinovich et al.(2007) - complete pairwise graphs

• Csurka et al.(2008) - hard estimation of labels present


Related work



• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior

• Hoiem et al. 2007 – 3D Layout CRF MDL Prior

• Delong et al. 2010 – label occurence cost

C(x) = K |L(x)|

C(x) = Σ LK

Lδ

L(x)


Related work



• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior

• Hoiem et al. 2007 – 3D Layout CRF MDL Prior

• Delong et al. 2010 – label occurence cost

C(x) = K |L(x)|

C(x) = Σ LK

Lδ

L(x)

All special cases of our model


Inference for Co-occurence

Co-occurence representation



Label indicator functions

Co-occurence representation



Move Energy Cost of current



Move Energylabel set



Move Energy



Decomposition to α-dependent and α-independent part

α-independent α-dependent



Move Energy



Decomposition to α-dependent and α-independent part

Either α or all labels in the image after the move



Move Energy



submodular non-submodular



Move Energy



non-submodular

Non-submodular energy overestimated by E'(t)

•

E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling



Move Energy



non-submodular


•


Occurrence - tight



Move Energy



non-submodular


•


Co-occurrence overestimation



Move Energy



non-submodular


•


General case

[See the paper]



Move Energy



non-submodular


•


Quadratic representation


Potential Training for Co-occurence



Ladický, Russell, Kohli, Torr ICCV09

Generatively trained Co-occurence potential

Approximated by 2nd order representation

Results on MSRC data set

Comparisons of results with and without co-occurence



Input Image

p

Input ImageWithout

Co-occurence

With

Co-occurence

Without

Co-occurence

With

Co-occurence


Pixel CRF Energy for Object-class Segmentation

Pairwise CRF models



Pixel CRF Energy for Object class Segmentation


Input Image Pairwise CRF Result

Shotton et al. ECCV06

Pixel CRF Energy for Object-class Segmentation

Pairwise CRF models



Pixel CRF Energy for Object class Segmentation


• Lacks long range interactions

• Results oversmoothed

• Data term information may be insufficient

Segment CRF Energy for Object-class Segmentation

Pairwise CRF models



Segment CRF nergy for Object class Segmentation


Input Image Unsupervised

Segmentation

Shi, Malik PAMI2000,Comaniciu, Meer PAMI2002,

Felzenschwalb, Huttenlocher, IJCV2004,




Pairwise CRF models



g gy j g


• Allows long range interactions

• Does not distinguish between instances of objects

• Cannot recover from incorrect segmentation


Robust PN approach



g gy j g

Pairwise CRF Segment

consistencyterm

Input Image

Kohli, Ladický, Torr CVPR08

Higher order CRF


Robust PN approach



g gy j g


consistencyterm





Robust PN approach



g gy j g


consistencyterm


Dominant label l


Robust PN approach




consistencyterm


• Enforces consistency as a weak constraint

• Can recover from incorrect segmentation

• Can combine multiple segmentations

• Does not use features at segment level


Robust PN approach




consistencyterm


Transformable to pairwise graph with auxiliary variable


Robust PN approach




consistencyterm

Ladický Russell Kohli Torr ICCV09

where


Robust PN approach




consistencyterm

Input ImageLadický, Russell, Kohli, Torr ICCV09

Higher order CRF

Associative Hierarchical CRFs

Can be generalized to Hierarchical model




• Allows unary potentials for region variables

• Allows pairwise potentials for region variables

• Allows multiple layers and multiple hierarchies


AH CRF Energy for Object-class Segmentation




Pairwise CRF Higher order term







Higher order term recursively defined as

Segmentunary term

Segmentpairwise term

Segment higherorder term







Higher order term recursively defined as

Segmentunary term

Segmentpairwise term

Segment higherorder term

Why is this generalisation useful?


Let us analyze the case with

i




• one segmentation

• potentials only over segment level





t ti






1) Minimum is segment-consistent

2) Cost of every segment-consistent CRF equal to thecost of pairwise CRF over segments



t ti






Equivalent to pairwise CRF over segments!





• Merges information over multiple scales

• Easy to train potentials and learn parameters

• Allows multiple segmentations and hierarchies

• Allows long range interactions

• Limited (?) to associative interlayer connections

Inference for AH-CRF

Graph Cut based move making algorithms [Boykov et al. 01]



α-expansion transformation function for base layer

Ladický (thesis) 2011


Graph Cut based move making algorithms [Boykov et al. 01]



α-expansion transformation function for base layer


α-expansion transformation function for auxiliary layer

(2 binary variables per node)


Interlayer connection between the base layer and the first auxiliary layer:





Interlayer connection between the base layer and the first auxiliary layer:




The move energy is:

where

Inference for AH-CRFThe move energy for interlayer connection between the base and auxiliary layer is:




Can be transformed to binary submodular pairwise function:

Inference for AH-CRFThe move energy for interlayer connection between auxiliary layers is:






Graph constructions





Pairwise potential between auxiliary variables :





Move energy if x = x : c d






Move energy if x ≠ x : c d






Graph constructions




Potential training for Object class Segmentation

Pixel unary potential



Unary likelihoods based on spatial configuration (Shotton et al. ECCV06 )

Classifier trained using boosting


Potential training for Object class Segmentation

Segment unary potential




Classifier trained using boosting (resp. Kernel SVMs)



Results on Corel dataset



Results on VOC 2010 dataset



Input Image Result Input Image Result

VOC 2010-Qualitative



Input Image Result Input Image Result

CamVid-Qualitative

Brostow et al



Our Result

Input Image

GroundTruth

Brostow et al.



Quantitative comparison




MSRC dataset

VOC2009 dataset



Unary Cost dependent on the similarity ofpatches, e.g.cross correlation

Unary Potential

Disparity = 5




Unary Cost dependent on the similarity ofpatches, e.g.cross correlation

Unary Potential

Disparity = 10




• Encourages label consistency in adjacent pixels

• Cost based on the distance of labels

Linear Truncated Quadratic Truncated

Pairwise Potential


Does not work for Road Scenes !



Original Image Dense StereoReconstruction





Patches can be matched to anyother patch for flat surfices

Different brightnessin cameras





Patches can be matched to anyother patch for flat surfices

Different brightnessin cameras

Could object recognition for road scenes help?

Recognition of road scenes is relatively easy

Joint Dense Stereo Reconstruction

and Object class Segmentation

• Object class and 3D location aremutually informative



sky

• Sky always in infinity(disparity = 0)






building carsky


• Cars, buses & pedestrianshave their typical height






building carsky



• Road and pavement on the

ground plane

road






building carsky




ground plane• Buildings and pavement on

the sides

road






building carsky




ground plane• Buildings and pavement on

the sides

• Both problems formulated as CRF• Joint approach possible?

road

Joint Formulation



• Each pixels takes label z i = [ x i y i ] L1 L2

• Dependency of x i and y i encoded as a unary and pairwisepotential, e.g.

• strong correlation between x = road , y = near ground plane

• strong correlation between x = sky, y = 0 • Correlation of edge in object class and disparity domain

Joint Formulation



• Weighted sum of object class, depth and joint potential

• Joint unary potential based on histograms of height

Object layer

Disparity layer

Joint unary links

Unary Potential

Joint Formulation



Pairwise Potential

• Object class and depth edges correlated

• Transitions in depth occur often at the object boundaries

Object layer

Disparity layer

Joint pairwise links

Joint Formulation



Inference



• Standard α-expansion• Each node in each expansion move keeps its old label

or takes a new label [ x L1, y L2],

• Possible in case of metric pairwise potentials

Too many moves! ( |L1| |L2| ) Impractical !

Inference

• Projected move for product label space



• One / Some of the label components remain(s)

constant after the move

• Set of projected moves• α-expansion in the object class projection

• Range-expansion in the depth projection

Dataset

• Leuven Road Scene dataset



• Contained

• 3 sequences

• 643 pairs of images

• We labelled• 50 training + 20 test images

• Object class (7 labels)

• Disparity (100 labels)

• Publicly available

• http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip

Left camera Right camera Object GT Disparity GT

Qualitative Results



• Large improvement for dense stereo estimation

• Minor improvement in object class segmentation

Quantitative Results



Dependency of the ratio of correctly labelled pixels within themaximum allowed error delta

• We proposed :

Summary



p p

• New models enforcing higher order structure

• Label-consistency in segments

• Consistency between scale

• Label-set consistency

• Consistency between different domains

• Graph-Cut based inference methods

• Source code http://cms.brookes.ac.uk/staff/PhilipTorr/ale.htm

There is more to come !

Thank you



Questions?

graph cut based optimization for computer vision

Documents