graph cut based optimization for computer vision
Post on 05-Apr-2018
226 Views
Preview:
TRANSCRIPT
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 1/240
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 2/240
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 3/240
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 4/240
Image Labelling Problems
Image Denoising Geometry Estimation Object Segmentation
Assign a label to each image pixel
Building
Sky
TreeGrass
Depth Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 5/240
Image Labelling Problems
• Labellings highly structured
Possible labelling Unprobable labelling Impossible labelling
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 6/240
Image Labelling Problems
• Labellings highly structured
• Labels highly correlated with very complex dependencies
• Neighbouring pixels tend to take the same label
• Low number of connected components
• Classes present may be seen in one image
• Geometric / Location consistency
• Planarity in depth estimation
• … many others (task dependent)
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 7/240
Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 8/240
Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisationproblem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 9/240
Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisationproblem
• Number of pixels up to millions
• Hard to train complex dependencies• Optimisation problem is hard to infer
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 10/240
Image Labelling Problems
• Labelling highly structured
• Labels highly correlated with very complex dependencies
• Independent label estimation too hard
• Whole labelling should be formulated as one optimisationproblem
• Number of pixels up to millions
• Hard to train complex dependencies• Optimisation problem is hard to infer
Vision is hard !
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 11/240
Image Labelling Problems
• You can
either
• Change the subject from the ComputerVision to the History of Renaissance Art
Vision is hard !
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 12/240
Image Labelling Problems
• You can
either
• Change the subject from the ComputerVision to the History of Renaissance Art
or
• Learn everything about Random Fields and
Graph Cuts
Vision is hard !
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 13/240
Foreground / Background Estimation
Rother et al. SIGGRAPH04
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 14/240
Foreground / Background Estimation
Data term Smoothness term
Data term
Estimated using FG / BG
colour models
Smoothness term
where
Intensity dependent smoothness
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 15/240
Foreground / Background Estimation
Data term Smoothness term
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 16/240
Foreground / Background Estimation
Data term Smoothness term
How to solve this optimisation problem?
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 17/240
Foreground / Background Estimation
Data term Smoothness term
How to solve this optimisation problem?
• Transform into min-cut / max-flow problem
• Solve it using min-cut / max-flow algorithm
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 18/240
Overview
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Applications
• Conclusions
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 19/240
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Task :
Maximize the flow from the sink to the source such that
1) The flow it conserved for each node
2) The flow for each pipe does not exceed the capacity
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 20/240
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 21/240
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
flow from thesourcecapacity
flow from node ito node j
conservation of flow
set of edges
set of nodes
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 22/240
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 23/240
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 24/240
Max-Flow Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 3
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 25/240
Max-Flow Problem
source
sink
9
5
6-3
8-3
42+3
2
2
5-3
3-3
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 3
+3
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 26/240
Max-Flow Problem
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 3
3
l bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 27/240
Max-Flow Problem
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 3
3
l bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 28/240
Max-Flow Problem
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 6
3
M Fl P bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 29/240
Max-Flow Problem
source
sink
9-3
5
3
5-3
45
2
2
2
5
5
3-311
6
2
3-3
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 6
3
+3
+3
M Fl P bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 30/240
Max-Flow Problem
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 6
3
3
3
M Fl P bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 31/240
Max-Flow Problem
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 6
3
3
3
M Fl P bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 32/240
Max-Flow Problem
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 11
3
3
3
M Fl P bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 33/240
Max-Flow Problem
source
sink
6-5
5-5
3
2
45
2
2
2
5-5
5-5
1+51
6
2+5
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 11
3
3
3
M Fl P bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 34/240
Max-Flow Problem
source
sink
1
3
2
45
2
2
2
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 11
3
3
3
M Fl P bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 35/240
Max-Flow Problem
source
sink
1
3
2
45
2
2
2
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 11
3
3
3
M Fl P bl
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 36/240
Max-Flow Problem
source
sink
1
3
2
45
2
2
2
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 13
3
3
3
Max Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 37/240
Max-Flow Problem
source
sink
1
3
2-2
45
2-2
2-2
2-2
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 13
3
3
3
+2
+2
Max Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 38/240
Max-Flow Problem
source
sink
1
3
45
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 13
3
3
3
2
2
Max Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 39/240
Max-Flow Problem
source
sink
1
3
45
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 13
3
3
3
2
2
Max Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 40/240
Max-Flow Problem
source
sink
1
3
45
61
6
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
3
3
2
2
Max Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 41/240
Max-Flow Problem
source
sink
1
3-2
4-25
61
6-2
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
3
3+2
2
2-2
+2
Max Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 42/240
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
3
5
2
2
2
Max Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 43/240
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
3
5
2
2
2
Max Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 44/240
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Find the path from source to sink
While (path exists)
flow += maximum capacity in the path
Build the residual graph (“subtract” the flow)
Find the path in the residual graph
End
flow = 15
3
3
5
2
2
2
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 45/240
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
flow = 15
3
3
5
2
2
2
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 46/240
Max-Flow Problem
source
sink
1
1
5
61
4
7
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in the
residual graph
flow = 15
3
3
5
2
2
2
S
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 47/240
Max-Flow Problem
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in the
residual graph
2. The flow from S to V - S equals to the sum
of capacities from S to V – S
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
S
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 48/240
Max-Flow Problem
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in theresidual graph
2. The flow from S to V - S equals to the sum of
capacities from S to V – S3. The flow from any A to V - A is upper bounded
by the sum of capacities from A to V – A
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
A
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 49/240
Max-Flow Problem
flow = 15
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in theresidual graph
2. The flow from S to V - S equals to the sum of
capacities from S to V – S3. The flow from any A to V - A is upper bounded
by the sum of capacities from A to V – A
4. The solution is globally optimal
source
sink
8/9
5/5
5/6
8/8
2/4
0/2
2/2
0/2
5/5
3/3
5/5
5/5
3/30/10/1
2/6
0/2
3/3
Individual flows obtained by summing up all paths
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 50/240
Max Flow Problem
flow = 15
Ford & Fulkerson algorithm (1956)
Why is the solution globally optimal ?
1. Let S be the set of reachable nodes in theresidual graph
2. The flow from S to V - S equals to the sum of
capacities from S to V – S3. The flow from any A to V - A is upper bounded
by the sum of capacities from A to V – A
4. The solution is globally optimal
source
sink
8/9
5/5
5/6
8/8
2/4
0/2
2/2
0/2
5/5
3/3
5/5
5/5
3/30/10/1
2/6
0/2
3/3
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 51/240
Max Flow Problem
source
sink
1000
1000
Ford & Fulkerson algorithm (1956)
Order does matter
1000
1000
1
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 52/240
Max Flow Problem
source
sink
1000
1000
Ford & Fulkerson algorithm (1956)
Order does matter
1000
1000
1
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 53/240
Max Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
1000
1000
1
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 54/240
Max Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
1000
1000
1
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 55/240
Max Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
999
999
1
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 56/240
Max Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
• Standard algorithm not polynomial
• Breath first leads to O(VE2)
• Path found in O(E)
• At least one edge gets saturated
• The saturated edge distance to thesource has to increase and is atmost V leading to O(VE)
(Edmonds & Karp, Dinic)
999
999
1
Max-Flow Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 57/240
Max Flow Problem
source
sink
999
999
Ford & Fulkerson algorithm (1956)
Order does matter
• Standard algorithm not polynomial
• Breath first leads to O(VE2)
(Edmonds & Karp)
• Various methods use different algorithm
to find the path and vary in complexity
999
999
1
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 58/240
Min Cut Problem
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Task :
Minimize the cost of the cut
1) Each node is either assigned to the source S or sink T
2) The cost of the edge (i, j) is taken if (iS) and (jT)
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 59/240
Cu ob e
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 60/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
source set sink set
edge costs
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 61/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
source set sink set
edge costsS
T
cost = 18
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 62/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
source set sink set
edge costsS
T
cost = 25
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 63/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
source set sink set
edge costsS
T
cost = 23
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 64/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 65/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
transformable into Linear program (LP)
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 66/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
Dual to max-flow problem
transformable into Linear program (LP)
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 67/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
After the substitution of the constraints :
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 68/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x i = 0 x i S x i = 1 x i T
x1x2
x3
x4
x5
x6
source edges sink edges
Edges betweenvariables
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 69/240
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
C( x ) = 5x 1
+ 9x 2
+ 4x 3
+ 3x 3 (1-x
1 ) + 2x
1(1-x
3 )
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 6x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )
+ 8(1-x 5 ) + 5(1-x 6 )
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 70/240
C( x ) = 5x 1
+ 9x 2
+ 4x 3
+ 3x 3 (1-x
1 ) + 2x
1(1-x
3 )
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 6x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )
+ 8(1-x 5 ) + 5(1-x 6 )
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 71/240
C( x ) = 2x 1
+ 9x 2
+ 4x 3
+ 2x 1(1-x
3 )
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )
+ 5(1-x 5 ) + 5(1-x 6 )
+ 3x 1 + 3x 3 (1-x 1 ) + 3x 5 (1-x 3 ) + 3(1-x 5 )
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 72/240
C( x ) = 2x 1
+ 9x 2
+ 4x 3
+ 2x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )
+ 5(1-x 5 ) + 5(1-x 6 )
+ 3x 1 + 3x 3 (1-x 1 ) + 3x 5 (1-x 3 ) + 3(1-x 5 )
3x 1
+ 3x 3 (1-x
1 ) + 3x
5 (1-x
3 ) + 3(1-x
5 )
=
3 + 3x 1(1-x 3 ) + 3x 3 (1-x 5 )
source
sink
9
5
6
8
42
2
2
5
3
5
5
311
6
2
3
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 73/240
C( x ) = 3 + 2x 1
+ 9x 2
+ 4x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )
+ 5(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 )
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
33
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 74/240
C( x ) = 3 + 2x 1
+ 9x 2
+ 4x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 )
+ 5(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 )
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
33
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 75/240
C( x ) = 3 + 2x 1
+ 6x 2
+ 4x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 2x 5 (1-x 4 ) + 6(1-x 4 )
+ 2(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 )
+ 3x 2 + 3x 6 (1-x 2 ) + 3x 5 (1-x 6 ) + 3(1-x 5 )
3x 2
+ 3x 6 (1-x
2 ) + 3x
5 (1-x
6 ) + 3(1-x
5 )
=
3 + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
source
sink
9
5
3
5
45
2
2
2
5
5
311
6
2
33
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 76/240
C( x ) = 6 + 2x 1
+ 6x 2
+ 4x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
+ 2(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 )
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
3
3
3
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 77/240
C( x ) = 6 + 2x 1
+ 6x 2
+ 4x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 )
+ 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
+ 2(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 )
source
sink
6
5
3
2
45
2
2
2
5
5
11
6
2
3
3
3
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 78/240
C( x ) = 11 + 2x 1
+ 1x 2
+ 4x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 )
+ 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
+ 2(1-x 5 ) + 3x 3 (1-x 5 )
source
sink
1
3
2
45
2
2
2
61
6
7
3
3
3
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 79/240
C( x ) = 11 + 2x 1
+ 1x 2
+ 4x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 4 (1-x 1 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 )
+ 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
+ 2(1-x 5 ) + 3x 3 (1-x 5 )
source
sink
1
3
2
45
2
2
2
61
6
7
3
3
3
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 80/240
C( x ) = 13 + 1x 2
+ 4x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 )
+ 2x 4 (1-x 5 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
+ 3x 3 (1-x 5 )
source
sink
1
3
45
61
6
7
3
3
3
2
2
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 81/240
C( x ) = 13 + 1x 2
+ 2x 3
+ 5x 1
(1-x 3
)
+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )
+ 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 )
+ 2x 4 (1-x 5 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
+ 3x 3 (1-x 5 )
source
sink
1
3
45
61
6
7
3
3
3
2
2
x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 82/240
C( x ) = 15 + 1x 2 + 4x 3 + 5x 1(1-x 3 )
+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )
+ 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 )
+ 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
source
sink
1
1
5
61
4
7
3
3
5
2
2
2x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 83/240
C( x ) = 15 + 1x 2 + 4x 3 + 5x 1(1-x 3 )
+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )
+ 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 )
+ 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
source
sink
1
1
5
61
4
7
3
3
5
2
2
2x1x2
x3
x4
x5
x6
Min-Cut Problem
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 84/240
C( x ) = 15 + 1x 2 + 4x 3 + 5x 1(1-x 3 )
+ 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1(1-x 4 )
+ 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 )
+ 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 )
source
sink
1
1
5
61
4
7
3
3
5
2
2
2x1x2
x3
x4
x5
x6 cost = 0
min cut
S
T
cost = 0
• All coefficients positive• Must be global minimum
S – set of reachable nodes from s
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 85/240
Overview
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 86/240
• Motivation
• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation
• Conclusions
Markov and Conditional RF
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 87/240
• Markov / Conditional Random fields modelprobabilistic dependencies of the set of randomvariables
Markov and Conditional RF
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 88/240
• Markov / Conditional Random fields model conditionaldependencies between random variables
• Each variable is conditionally independent of all othervariables given its neighbours
Markov and Conditional RF
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 89/240
• Markov / Conditional Random fields model conditionaldependencies between random variables
• Each variable is conditionally independent of all othervariables given its neighbours
• Posterior probability of the labelling x given data D is :
partition function potential functionscliques
Markov and Conditional RF
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 90/240
• Markov / Conditional Random fields model conditionaldependencies between random variables
• Each variable is conditionally independent of all othervariables given its neighbours
• Posterior probability of the labelling x given data D is :
• Energy of the labelling is defined as :
partition function potential functionscliques
Markov and Conditional RF
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 91/240
• The most probable (Max a Posteriori (MAP)) labellingis defined as:
Markov and Conditional RF
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 92/240
• The most probable (Max a Posteriori (MAP)) labellingis defined as:
• The only distinction (MRF vs. CRF) is that in the CRFthe conditional dependencies between variables
depend also on the data
Markov and Conditional RF
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 93/240
• The most probable (Max a Posteriori (MAP)) labellingis defined as:
• The only distinction (MRF vs. CRF) is that in the CRFthe conditional dependencies between variables
depend also on the data• Typically we define an energy first and then pretend
there is an underlying probabilistic distribution there,but there isn’t really (Pssssst, don’t tell anyone)
Overview
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 94/240
• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems• Recent Advances in Random Field Optimisation
• Conclusions
Pairwise CRF models
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 95/240
Standard CRF Energy
Data term Smoothness term
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 96/240
The energy / potential is called submodular (only) if for everypair of variables :
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 97/240
For 2-label problems x {0, 1} :
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 98/240
For 2-label problems x {0, 1} :
Pairwise potential can be transformed into :
where
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 99/240
For 2-label problems x {0, 1} :
Pairwise potential can be transformed into :
where
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 100/240
For 2-label problems x {0, 1} :
Pairwise potential can be transformed into :
where
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 101/240
After summing up :
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 102/240
After summing up :
Let :
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 103/240
After summing up :
Let : Then :
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 104/240
After summing up :
Let : Then :
Equivalent st-mincut problem is :
Foreground / Background Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 105/240
Data term Smoothness term
Data term
Estimated using FG / BGcolour models
Smoothness term
where
Intensity dependent smoothness
Foreground / Background Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 106/240
source
sink
Rother et al. SIGGRAPH04
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 107/240
Extendable to (some) Multi-label CRFs
• each state of multi-label variable encoded using
multiple binary variables
• the cost of every possible cut must be the same as
the associated energy• the solution obtained by inverting the encoding
Obtain
solutionGraph Cut
Build
Graph
Encoding
multi-label energy solution
Dense Stereo Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 108/240
Left Camera Image Right Camera Image Dense Stereo Result
• For each pixel assigns a disparity label : z• Disparities from the discrete set {0 , 1, .. D }
Dense Stereo Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 109/240
Data term
Left image
feature
Shifted right
image feature
Left Camera Image Right Camera Image
Dense Stereo Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 110/240
Data term
Left image
feature
Shifted right
image feature
Smoothness term
Dense Stereo Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 111/240
Ishikawa PAMI03
Encoding sour
ce
xi0
xi D
x k0
x k D
. .
.
x k1 x k
2
source
sink
.
Dense Stereo Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 112/240
Ishikawa PAMI03
Unary Potential
sour
ce
xi0
xi D
x k0
x k D
. .
.
x k1 x k
2 ∞
∞
∞
∞
∞
∞
∞ ∞
source
sink
.
cost = +
The cost of every cut should beequal to the corresponding
energy under the encoding
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 113/240
Dense Stereo Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 114/240
Ishikawa PAMI03
Unary Potential
sour
ce
xi0
xi D
x k0
x k D
. .
.
x k1 x k
2 ∞
∞
∞
∞
∞
∞
∞ ∞
source
sink
.
cost = ∞
The cost of every cut should beequal to the corresponding
energy under the encoding
Dense Stereo Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 115/240
Ishikawa PAMI03
Pairwise Potential
sour
ce
xi0
xi D
x k0
x k D
. .
.
x k1 x k
2 ∞
∞
∞
∞
∞
∞
∞ ∞
K
K
K
K
source
sink
.
cost = + + K
The cost of every cut should beequal to the corresponding
energy under the encoding
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 116/240
Dense Stereo Estimation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 117/240
See [Ishikawa PAMI03] for more details
sour
ce
xi0
xi D
x k0
x k D
. .
.
x k1 x k
2 ∞
∞
∞
∞
∞
∞
∞ ∞
source
sink
.
Extendable to any convex cost
Convex function
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 118/240
Move making algorithms
• Original problem decomposed into a series of subproblemssolvable with graph cut
• In each subproblem we find the optimal move from the
current solution in a restricted search space
Update
solutionGraph Cut
Build
Graph
Encoding
Propose
move
Initial solutionsolution
Graph Cut based Inference
Example : [B k t l 01]
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 119/240
Example : [Boykov et al. 01]
α-expansion
– each variable either keeps its old label or changes to α
– all α-moves are iteratively performed till convergenceTransformation function :
α-swap
– each variable taking label α or can change its label to α or
– all α-moves are iteratively performed till convergence
Graph Cut based Inference
Sufficient conditions for move making algorithms :
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 120/240
Sufficient conditions for move making algorithms :
(all possible moves are submodular)
α-swap : semi-metricity
Proof:
Graph Cut based Inference
Sufficient conditions for move making algorithms :
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 121/240
Sufficient conditions for move making algorithms :
(all possible moves are submodular)
α-expansion : metricity
Proof:
Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 122/240
Data term Smoothness termData term
Smoothness term
where
Discriminatively trained classifier
Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 123/240
Original Image Initial solution
grass
Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 124/240
Original Image Initial solution Building expansion
grass
grass
building
Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 125/240
Original Image Initial solution Building expansion
Sky expansion
grass
grass
grass
building
building
sky
Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 126/240
Original Image Initial solution Building expansion
Sky expansion Tree expansion
grass
grass
grass grass
building
building building
sky sky
tree
Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 127/240
Original Image Initial solution Building expansion
Sky expansion Tree expansion Final Solution
grass
grass
grass grass grass
building
building buildingbuilding
sky sky sky
tree treeaeroplane
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 128/240
Range-expansion [Kumar, Veksler & Torr 1]
– Expansion version of range swap moves
– Each varibale can change its label to any label in a convexrange or keep its old label
Range-(swap) moves [Veksler 07] – Each variable in the convex range can change its label to any
other label in the convex range
– all range-moves are iteratively performed till convergence
Dense Stereo Reconstruction
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 129/240
Data term
Smoothness term
Same as before
Left Camera Image Right Camera Image Dense Stereo Result
Dense Stereo Reconstruction
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 130/240
Data term
Smoothness term
Same as before
Left Camera Image Right Camera Image Dense Stereo Result
Convex range Truncation
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 131/240
Original Image Initial Solution
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 132/240
Original Image Initial Solution After 1st expansion
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 133/240
Original Image Initial Solution
After 2nd expansion
After 1st expansion
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 134/240
Original Image Initial Solution
After 2nd expansion
After 1st expansion
After 3rd expansion
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 135/240
Original Image Initial Solution
After 2nd expansion
After 1st expansion
After 3rd expansion Final solution
Graph Cut based Inference
E t dibl t t i l f bi hi h d i
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 136/240
• Extendible to certain classes of binary higher order energies
• Higher order terms have to be transformable to the pairwise submodularenergy with binary auxiliary variables z
C
Higher order term Pairwise term
Update
solutionGraph Cut
Transf. to
pairwise subm.
Encoding
Propose
move
Initial solutionsolution
Graph Cut based Inference
E t dibl t t i l f bi hi h d i
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 137/240
• Extendible to certain classes of binary higher order energies
• Higher order terms have to be transformable to the pairwise submodularenergy with binary auxiliary variables z
C
Higher order term Pairwise term
Example :
Kolmogorov ECCV06, Ramalingam et al. DAM12
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 138/240
Enforces label consistency of the clique as a weak constraint
Input image Pairwise CRF Higher order CRF
Kohli et al. CVPR07
Graph Cut based Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 139/240
Update
solutionGraph Cut
Transf. to
pairwise subm.
Encoding
Propose
move
Initial solution solution
If the energy is not submodular / graph-representable – Overestimation by a submodular energy
– Quadratic Pseudo-Boolean Optimisation (QPBO)
Graph Cut based Inference
Overestimation by a submodular energy (convex / concave)
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 140/240
Update
solutionGraph Cut
Overestim. by
pairwise subm.
Encoding
Propose
move
Initial solutionsolution
y gy
– We find an energy E’(t) s.t.
– it is tight in the current solution E’(t0) = E(t0)
– Overestimates E(t) for all t E’(t) ≥ E(t)
– We replace E(t) by E’(t)
–
The moves are not optimal, but quaranteed to converge – The tighter over-estimation, the better
Example :
Graph Cut based Inference
Quadratic Pseudo-Boolean Optimisation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 141/240
Quadratic Pseudo-Boolean Optimisation
–
Each binary variable is encoded using two binaryvariables x i and x i s.t. x i = 1 - x i _ _
Graph Cut based Inference
Quadratic Pseudo-Boolean Optimisation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 142/240
Quadratic Pseudo-Boolean Optimisation
source
sink
xix j
xi x j
_ _ – Energy submodular in x i and x i
– Solved by dropping the constraint x i = 1 - x i
– If x i = 1 - x i the solution for xi is optimal
_
_ _
Overview
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 143/240
• Motivation• Min-Cut / Max-Flow (Graph Cut) Algorithm
• Markov and Conditional Random Fields
• Random Field Optimisation using Graph Cuts
• Submodular vs. Non-Submodular Problems
• Pairwise vs. Higher Order Problems
• 2-Label vs. Multi-Label Problems
• Recent Advances in Random Field Optimisation
• Conclusions
Motivation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 144/240
• To propose formulations enforcing certain structureproperties of the output useful in computer vision
– Label Consistency in segments as a weak constraint
– Label agreement over different scales
– Label-set consistency
– Multiple domains consistency
• To propose feasible Graph-Cut based inference method
Structures in CRF
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 145/240
• Taskar et al. 02 – associative potentials
• Woodford et al. 08 – planarity constraint
• Vicente et al. 08 – connectivity constraint
• Nowozin & Lampert 09 – connectivity constraint
• Roth & Black 09 – field of experts
• Woodford et al. 09 – marginal probability
• Delong et al. 10 – label occurrence costs
Object-class Segmentation Problem
• Aims to assign a class label for each pixel of an image
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 146/240
• Aims to assign a class label for each pixel of an image
• Classifier trained on the training set
• Evaluated on never seen test images
Pairwise CRF models
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 147/240
Standard CRF Energy for Object Segmentation
Cannot encode global consistency of labels!!
Local context
Ladický, Russell, Kohli, Torr ECCV10
Encoding Co-occurrence
C i f l[Heitz et al. '08]
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 148/240
• Thing – Thing
• Stuff - Stuff
• Stuff - Thing
[ Images from Rabinovich et al. 07 ]
Co-occurrence is a powerful cue[ ]
[Rabinovich et al. ‘07]
Ladický, Russell, Kohli, Torr ECCV10
Encoding Co-occurrence
Co occurrence is a powerful cue[Heitz et al. '08]
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 149/240
• Thing – Thing
• Stuff - Stuff
• Stuff - Thing
Co-occurrence is a powerful cue[ ]
[Rabinovich et al. ‘07]
Proposed solutions :
1. Csurka et al. 08 - Hard decision for label estimation
2. Torralba et al. 03 - GIST based unary potential
3. Rabinovich et al. 07 - Full-connected CRF
[ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10
So...
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 150/240
What properties should these global
co-occurence potentials have ?
Ladický, Russell, Kohli, Torr ECCV10
Desired properties
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 151/240
1. No hard decisions
Ladický, Russell, Kohli, Torr ECCV10
Desired properties
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 152/240
1. No hard decisions
Incorporation in probabilistic framework
Unlikely possibilities are not completely ruled out
Ladický, Russell, Kohli, Torr ECCV10
Desired properties
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 153/240
1. No hard decisions
2. Invariance to region size
Ladický, Russell, Kohli, Torr ECCV10
Desired properties
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 154/240
1. No hard decisions
2. Invariance to region size
Cost for occurrence of {people, house, road etc .. }
invariant to image area
Ladický, Russell, Kohli, Torr ECCV10
Desired properties
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 155/240
1. No hard decisions
2. Invariance to region size
The only possible solution :
Local context Global context
Cost defined over the
assigned labels L(x)
L(x)={ , , }
Ladický, Russell, Kohli, Torr ECCV10
Desired properties
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 156/240
1. No hard decisions
2. Invariance to region size3. Parsimony – simple solutions preferred
L(x)={ building, tree, grass, sky }
L(x)={ aeroplane, tree, flower,
building, boat, grass, sky }
Ladický, Russell, Kohli, Torr ECCV10
Desired properties
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 157/240
1. No hard decisions
2. Invariance to region size3. Parsimony – simple solutions preferred
4. Efficiency
Ladický, Russell, Kohli, Torr ECCV10
Desired properties
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 158/240
1. No hard decisions
2. Invariance to region size3. Parsimony – simple solutions preferred
4. Efficiency
a) Memory requirements as O(n) with the image size and
number or labels
b) Inference tractable
Ladický, Russell, Kohli, Torr ECCV10
Previous work
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 159/240
• Torralba et al.(2003) – Gist-based unary potentials
• Rabinovich et al.(2007) - complete pairwise graphs
• Csurka et al.(2008) - hard estimation of labels present
Ladický, Russell, Kohli, Torr ECCV10
Related work
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 160/240
• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior
• Hoiem et al. 2007 – 3D Layout CRF MDL Prior
• Delong et al. 2010 – label occurence cost
C(x) = K |L(x)|
C(x) = Σ LK
Lδ
L(x)
Ladický, Russell, Kohli, Torr ECCV10
Related work
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 161/240
• Zhu & Yuille 1996 – MDL prior• Bleyer et al. 2010 – Surface Stereo MDL prior
• Hoiem et al. 2007 – 3D Layout CRF MDL Prior
• Delong et al. 2010 – label occurence cost
C(x) = K |L(x)|
C(x) = Σ LK
Lδ
L(x)
All special cases of our model
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Co-occurence representation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 162/240
Label indicator functions
Co-occurence representation
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy Cost of current
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 163/240
Move Energylabel set
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 164/240
Decomposition to α-dependent and α-independent part
α-independent α-dependent
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 165/240
Decomposition to α-dependent and α-independent part
Either α or all labels in the image after the move
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 166/240
submodular non-submodular
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 167/240
non-submodular
Non-submodular energy overestimated by E'(t)
•
E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 168/240
non-submodular
Non-submodular energy overestimated by E'(t)
•
E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
Occurrence - tight
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 169/240
non-submodular
Non-submodular energy overestimated by E'(t)
•
E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
Co-occurrence overestimation
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 170/240
non-submodular
Non-submodular energy overestimated by E'(t)
•
E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
General case
[See the paper]
Ladický, Russell, Kohli, Torr ECCV10
Inference for Co-occurence
Move Energy
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 171/240
non-submodular
Non-submodular energy overestimated by E'(t)
•
E'(t) = E(t) for current solution• E'(t) E(t) for any other labelling
Quadratic representation
Ladický, Russell, Kohli, Torr ECCV10
Potential Training for Co-occurence
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 172/240
Ladický, Russell, Kohli, Torr ICCV09
Generatively trained Co-occurence potential
Approximated by 2nd order representation
Results on MSRC data set
Comparisons of results with and without co-occurence
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 173/240
Input Image
p
Input ImageWithout
Co-occurence
With
Co-occurence
Without
Co-occurence
With
Co-occurence
Ladický, Russell, Kohli, Torr ICCV09
Pixel CRF Energy for Object-class Segmentation
Pairwise CRF models
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 174/240
Pixel CRF Energy for Object class Segmentation
Data term Smoothness term
Input Image Pairwise CRF Result
Shotton et al. ECCV06
Pixel CRF Energy for Object-class Segmentation
Pairwise CRF models
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 175/240
Pixel CRF Energy for Object class Segmentation
Data term Smoothness term
• Lacks long range interactions
• Results oversmoothed
• Data term information may be insufficient
Segment CRF Energy for Object-class Segmentation
Pairwise CRF models
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 176/240
Segment CRF nergy for Object class Segmentation
Data term Smoothness term
Input Image Unsupervised
Segmentation
Shi, Malik PAMI2000,Comaniciu, Meer PAMI2002,
Felzenschwalb, Huttenlocher, IJCV2004,
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 177/240
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 178/240
Segment CRF Energy for Object-class Segmentation
Pairwise CRF models
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 179/240
g gy j g
Data term Smoothness term
• Allows long range interactions
• Does not distinguish between instances of objects
• Cannot recover from incorrect segmentation
Segment CRF Energy for Object-class Segmentation
Robust PN approach
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 180/240
g gy j g
Pairwise CRF Segment
consistencyterm
Input Image
Kohli, Ladický, Torr CVPR08
Higher order CRF
Segment CRF Energy for Object-class Segmentation
Robust PN approach
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 181/240
g gy j g
Pairwise CRF Segment
consistencyterm
Kohli, Ladický, Torr CVPR08
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 182/240
Segment CRF Energy for Object-class Segmentation
Robust PN approach
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 183/240
g gy j g
Pairwise CRF Segment
consistencyterm
Kohli, Ladický, Torr CVPR08
Dominant label l
Segment CRF Energy for Object-class Segmentation
Robust PN approach
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 184/240
Pairwise CRF Segment
consistencyterm
Kohli, Ladický, Torr CVPR08
• Enforces consistency as a weak constraint
• Can recover from incorrect segmentation
• Can combine multiple segmentations
• Does not use features at segment level
Segment CRF Energy for Object-class Segmentation
Robust PN approach
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 185/240
Pairwise CRF Segment
consistencyterm
Kohli, Ladický, Torr CVPR08
Transformable to pairwise graph with auxiliary variable
Segment CRF Energy for Object-class Segmentation
Robust PN approach
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 186/240
Pairwise CRF Segment
consistencyterm
Ladický Russell Kohli Torr ICCV09
where
Segment CRF Energy for Object-class Segmentation
Robust PN approach
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 187/240
Pairwise CRF Segment
consistencyterm
Input ImageLadický, Russell, Kohli, Torr ICCV09
Higher order CRF
Associative Hierarchical CRFs
Can be generalized to Hierarchical model
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 188/240
Ladický, Russell, Kohli, Torr ICCV09
• Allows unary potentials for region variables
• Allows pairwise potentials for region variables
• Allows multiple layers and multiple hierarchies
Associative Hierarchical CRFs
AH CRF Energy for Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 189/240
Ladický, Russell, Kohli, Torr ICCV09
Pairwise CRF Higher order term
Associative Hierarchical CRFs
AH CRF Energy for Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 190/240
Ladický, Russell, Kohli, Torr ICCV09
Pairwise CRF Higher order term
Higher order term recursively defined as
Segmentunary term
Segmentpairwise term
Segment higherorder term
Associative Hierarchical CRFs
AH CRF Energy for Object-class Segmentation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 191/240
Ladický, Russell, Kohli, Torr ICCV09
Pairwise CRF Higher order term
Higher order term recursively defined as
Segmentunary term
Segmentpairwise term
Segment higherorder term
Why is this generalisation useful?
Associative Hierarchical CRFs
Let us analyze the case with
i
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 192/240
Ladický, Russell, Kohli, Torr ICCV09
• one segmentation
• potentials only over segment level
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 193/240
Associative Hierarchical CRFs
Let us analyze the case with
t ti
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 194/240
Ladický, Russell, Kohli, Torr ICCV09
• one segmentation
• potentials only over segment level
1) Minimum is segment-consistent
2) Cost of every segment-consistent CRF equal to thecost of pairwise CRF over segments
Associative Hierarchical CRFs
Let us analyze the case with
t ti
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 195/240
Ladický, Russell, Kohli, Torr ICCV09
• one segmentation
• potentials only over segment level
Equivalent to pairwise CRF over segments!
Associative Hierarchical CRFs
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 196/240
Ladický, Russell, Kohli, Torr ICCV09
• Merges information over multiple scales
• Easy to train potentials and learn parameters
• Allows multiple segmentations and hierarchies
• Allows long range interactions
• Limited (?) to associative interlayer connections
Inference for AH-CRF
Graph Cut based move making algorithms [Boykov et al. 01]
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 197/240
α-expansion transformation function for base layer
Ladický (thesis) 2011
Inference for AH-CRF
Graph Cut based move making algorithms [Boykov et al. 01]
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 198/240
α-expansion transformation function for base layer
Ladický (thesis) 2011
α-expansion transformation function for auxiliary layer
(2 binary variables per node)
Inference for AH-CRF
Interlayer connection between the base layer and the first auxiliary layer:
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 199/240
Ladický (thesis) 2011
Inference for AH-CRF
Interlayer connection between the base layer and the first auxiliary layer:
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 200/240
Ladický (thesis) 2011
The move energy is:
where
Inference for AH-CRFThe move energy for interlayer connection between the base and auxiliary layer is:
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 201/240
Ladický (thesis) 2011
Can be transformed to binary submodular pairwise function:
Inference for AH-CRFThe move energy for interlayer connection between auxiliary layers is:
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 202/240
Ladický (thesis) 2011
Can be transformed to binary submodular pairwise function:
Inference for AH-CRF
Graph constructions
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 203/240
Ladický (thesis) 2011
Inference for AH-CRF
Pairwise potential between auxiliary variables :
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 204/240
Ladický (thesis) 2011
Inference for AH-CRF
Move energy if x = x : c d
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 205/240
Ladický (thesis) 2011
Can be transformed to binary submodular pairwise function:
Inference for AH-CRF
Move energy if x ≠ x : c d
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 206/240
Ladický (thesis) 2011
Can be transformed to binary submodular pairwise function:
Inference for AH-CRF
Graph constructions
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 207/240
Ladický (thesis) 2011
Potential training for Object class Segmentation
Pixel unary potential
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 208/240
Unary likelihoods based on spatial configuration (Shotton et al. ECCV06 )
Classifier trained using boosting
Ladický, Russell, Kohli, Torr ICCV09
Potential training for Object class Segmentation
Segment unary potential
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 209/240
Ladický, Russell, Kohli, Torr ICCV09
Classifier trained using boosting (resp. Kernel SVMs)
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 210/240
Results on Corel dataset
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 211/240
Results on VOC 2010 dataset
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 212/240
Input Image Result Input Image Result
VOC 2010-Qualitative
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 213/240
Input Image Result Input Image Result
CamVid-Qualitative
Brostow et al
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 214/240
Our Result
Input Image
GroundTruth
Brostow et al.
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 215/240
Quantitative comparison
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 216/240
Ladický, Russell, Kohli, Torr ICCV09
MSRC dataset
VOC2009 dataset
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 217/240
Dense Stereo Reconstruction
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 218/240
Unary Cost dependent on the similarity ofpatches, e.g.cross correlation
Unary Potential
Disparity = 5
Dense Stereo Reconstruction
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 219/240
Unary Cost dependent on the similarity ofpatches, e.g.cross correlation
Unary Potential
Disparity = 10
Dense Stereo Reconstruction
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 220/240
• Encourages label consistency in adjacent pixels
• Cost based on the distance of labels
Linear Truncated Quadratic Truncated
Pairwise Potential
Dense Stereo Reconstruction
Does not work for Road Scenes !
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 221/240
Original Image Dense StereoReconstruction
Dense Stereo Reconstruction
Does not work for Road Scenes !
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 222/240
Patches can be matched to anyother patch for flat surfices
Different brightnessin cameras
Dense Stereo Reconstruction
Does not work for Road Scenes !
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 223/240
Patches can be matched to anyother patch for flat surfices
Different brightnessin cameras
Could object recognition for road scenes help?
Recognition of road scenes is relatively easy
Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location aremutually informative
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 224/240
sky
• Sky always in infinity(disparity = 0)
Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location aremutually informative
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 225/240
building carsky
• Sky always in infinity(disparity = 0)
• Cars, buses & pedestrianshave their typical height
Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location aremutually informative
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 226/240
building carsky
• Sky always in infinity(disparity = 0)
• Cars, buses & pedestrianshave their typical height
• Road and pavement on the
ground plane
road
Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location aremutually informative
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 227/240
building carsky
• Sky always in infinity(disparity = 0)
• Cars, buses & pedestrianshave their typical height
• Road and pavement on the
ground plane• Buildings and pavement on
the sides
road
Joint Dense Stereo Reconstruction
and Object class Segmentation
• Object class and 3D location aremutually informative
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 228/240
building carsky
• Sky always in infinity(disparity = 0)
• Cars, buses & pedestrianshave their typical height
• Road and pavement on the
ground plane• Buildings and pavement on
the sides
• Both problems formulated as CRF• Joint approach possible?
road
Joint Formulation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 229/240
• Each pixels takes label z i = [ x i y i ] L1 L2
• Dependency of x i and y i encoded as a unary and pairwisepotential, e.g.
• strong correlation between x = road , y = near ground plane
• strong correlation between x = sky, y = 0 • Correlation of edge in object class and disparity domain
Joint Formulation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 230/240
• Weighted sum of object class, depth and joint potential
• Joint unary potential based on histograms of height
Object layer
Disparity layer
Joint unary links
Unary Potential
Joint Formulation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 231/240
Pairwise Potential
• Object class and depth edges correlated
• Transitions in depth occur often at the object boundaries
Object layer
Disparity layer
Joint pairwise links
Joint Formulation
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 232/240
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 233/240
Inference
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 234/240
• Standard α-expansion• Each node in each expansion move keeps its old label
or takes a new label [ x L1, y L2],
• Possible in case of metric pairwise potentials
Too many moves! ( |L1| |L2| ) Impractical !
Inference
• Projected move for product label space
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 235/240
• One / Some of the label components remain(s)
constant after the move
• Set of projected moves• α-expansion in the object class projection
• Range-expansion in the depth projection
Dataset
• Leuven Road Scene dataset
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 236/240
• Contained
• 3 sequences
• 643 pairs of images
• We labelled• 50 training + 20 test images
• Object class (7 labels)
• Disparity (100 labels)
• Publicly available
• http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip
Left camera Right camera Object GT Disparity GT
Qualitative Results
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 237/240
• Large improvement for dense stereo estimation
• Minor improvement in object class segmentation
Quantitative Results
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 238/240
Dependency of the ratio of correctly labelled pixels within themaximum allowed error delta
• We proposed :
Summary
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 239/240
p p
• New models enforcing higher order structure
• Label-consistency in segments
• Consistency between scale
• Label-set consistency
• Consistency between different domains
• Graph-Cut based inference methods
• Source code http://cms.brookes.ac.uk/staff/PhilipTorr/ale.htm
There is more to come !
Thank you
7/31/2019 Graph Cut based Optimization for Computer Vision
http://slidepdf.com/reader/full/graph-cut-based-optimization-for-computer-vision 240/240
Questions?
top related