![Page 1: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/1.jpg)
Solving Markov Random Fields using
Dynamic Graph Cuts &Second Order Cone Programming
Relaxations
M. Pawan Kumar, Pushmeet Kohli
Philip Torr
![Page 2: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/2.jpg)
Talk Outline
• Dynamic Graph Cuts– Fast reestimation of cut– Useful for video– Object specific segmentation
• Estimation of non submodular MRF’s– Relaxations beyond linear!!
![Page 3: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/3.jpg)
Example: Video Segmentation
![Page 4: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/4.jpg)
Model Based Segmentation
Image Segmentation Pose Estimate
[Images courtesy: M. Black, L. Sigal]
![Page 5: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/5.jpg)
Min-Marginals
ImageMAP Solution Belief - Foreground
Lowsmoothness
Highsmoothness
Moderatesmoothness
Colour Scale
1
0
0.5
![Page 6: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/6.jpg)
Uses of Min marginals
• Estimate of true marginals (uncertainty)
• Parameter Learning.
• Get best n solutions easily.
![Page 7: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/7.jpg)
Dynamic Graph Cuts
PB SB
cheaperoperation
computationally
expensive operation
Simplerproblem
PB*
differencesbetweenA and B
A and Bsimilar
PA SA
solve
![Page 8: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/8.jpg)
First segmentation problem MAP solution
Ga
Our Algorithm
Gb
second segmentation problem
Maximum flow
residual graph (Gr)
G`
differencebetween
Ga and Gbupdated residual
graph
![Page 9: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/9.jpg)
• The Max-flow Problem- Edge capacity and flow balance constraints
Computing the st-mincut from Max-flow algorithms
• Notation- Residual capacity (edge capacity – current flow)- Augmenting path
• Simple Augmenting Path based Algorithms- Repeatedly find augmenting paths and push flow.- Saturated edges constitute the st-mincut. [Ford-Fulkerson Theorem]
Sink (1)
Source (0)
a1 a2
2
5
9
42
1
![Page 10: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/10.jpg)
9 + α
4 + α
Adding a constant to both thet-edges of a node does not change the edges constituting the st-mincut.
Key Observation
Sink (1)
Source (0)
a1 a2
2
5
2
1
E (a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
E*(a1,a2 ) = E(a1,a2) + α(a2+ā2)
= E(a1,a2) + α [a2+ā2 =1]
Reparametrization
![Page 11: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/11.jpg)
9 + α
4
All reparametrizations of the graph are sums of these two types.
Other type of reparametrization
Sink (1)
Source (0)
a1 a2
2
5 + α
2 + α
1 - α
Reparametrization, second type
Both maintain the solution and add a constant α to the energy.
![Page 12: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/12.jpg)
Reparametrization
• Nice result (easy to prove)
• All other reparametrizations can be viewed in terms of these two basic operations.
• Proof in Hammer, and also in one of Vlad’s recent papers.
![Page 13: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/13.jpg)
s
Gt
original graph
0/9
0/7
0/5
0/2 0/4
0/1
xi xj
flow/residual capacity
Graph Re-parameterization
![Page 14: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/14.jpg)
s
Gt
original graph
0/9
0/7
0/5
0/2 0/4
0/1
xi xj
flow/residual capacity
Graph Re-parameterization
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0st-mincut
ComputeMaxflow
Gr
Edges cut
![Page 15: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/15.jpg)
Update t-edge Capacities
s
Gr
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
![Page 16: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/16.jpg)
Update t-edge Capacities
s
Gr
t residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
![Page 17: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/17.jpg)
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
5/-1
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
edge capacityconstraint violated!(flow > capacity)
= 5 – 4 = 1
excess flow (e) = flow – new capacity
![Page 18: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/18.jpg)
add e to both t-edges connected to node i
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
3/2
1/0
2/0 4/0
capacitychanges from
7 to 4
edge capacityconstraint violated!(flow > capacity)
= 5 – 4 = 1
excess flow (e) = flow – new capacity
5/-1
![Page 19: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/19.jpg)
Update t-edge Capacities
s
G`t
updated residual graph
xi xj0/12
3/2
1/0
4/0
capacitychanges from
7 to 4
excess flow (e) = flow – new capacity
add e to both t-edges connected to node i
= 5 – 4 = 1
5/0
2/1
edge capacityconstraint violated!(flow > capacity)
![Page 20: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/20.jpg)
Update n-edge Capacities
s
Gr
t
residual graph
xi xj0/12
5/2
3/2
1/0
2/0 4/0
• Capacity changes from 5 to 2
![Page 21: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/21.jpg)
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/12
5/2
3/-1
1/0
2/0 4/0
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
![Page 22: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/22.jpg)
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/12
5/2
3/-1
1/0
2/0 4/0
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint
![Page 23: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/23.jpg)
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/11
5/2
2/0
1/0
2/0 4/0
excess
deficiency
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
![Page 24: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/24.jpg)
Update n-edge Capacities
s
t
Updated residual graph
xi xj0/11
5/2
2/0
1/0
2/0 4/0
deficiency
excess
G`
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to/from the terminals
• Create capacity by adding α = excess to both t-edges.
![Page 25: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/25.jpg)
Update n-edge Capacities
Updated residual graph
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to the terminals
• Create capacity by adding α = excess to both t-edges.
G`
xi xj0/11
5/3
2/0
2/0
3/0 4/1
s
t
![Page 26: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/26.jpg)
Update n-edge Capacities
Updated residual graph
• Capacity changes from 5 to 2- edge capacity constraint violated!
• Reduce flow to satisfy constraint- causes flow imbalance!
• Push excess flow to the terminals
• Create capacity by adding α = excess to both t-edges.
G`
xi xj0/11
5/3
2/0
2/0
3/0 4/1
s
t
![Page 27: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/27.jpg)
Complexity analysis of MRF Update Operations
MRF Energy Operation
Graph Operation Complexity
modifying a unary term
modifying a pair-wise term
adding a latent variable
delete a latent variable
Updating a t-edge capacity
Updating a n-edge capacity
adding a node
set the capacities of all edges of a node zero
O(1)
O(1)
O(1)
O(k)*
*requires k edge update operations where k is degree of the node
![Page 28: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/28.jpg)
• Finding augmenting paths is time consuming.
• Dual-tree maxflow algorithm [Boykov & Kolmogorov PAMI 2004]
- Reuses search trees after each augmentation.
- Empirically shown to be substantially faster.
• Our Idea
– Reuse search trees from previous graph cut computation
– Saves us search tree creation tree time [O(#edges)]
– Search trees have to be modified to make them consistent with new graphs
– Constrain the search of augmenting paths
• New paths must contain at least one updated edge
Improving the Algorithm
![Page 29: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/29.jpg)
Reusing Search Trees
c’ = measure of change in the energy
– Running time
• Dynamic algorithm (c’ + re-create search tree )
• Improved dynamic algorithm (c’)
• Video Segmentation Example
- Duplicate image frames (No time is needed)
![Page 30: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/30.jpg)
Dynamic Graph Cut vs Active Cuts
• Our method flow recycling
• AC cut recycling
• Both methods: Tree recycling
![Page 31: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/31.jpg)
Experimental Analysis
MRF consisting of 2x105 latent variables connected in a 4-neighborhood.
Running time of the dynamic algorithm
![Page 32: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/32.jpg)
Part II SOCP for MRF
![Page 33: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/33.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Labelling m = {1, -1, -1, 1}
Random Variables V = {V1,..,V4}
Label Set L = {-1,1}
![Page 34: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/34.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Cost(m) = 2
![Page 35: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/35.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Cost(m) = 2 + 1
![Page 36: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/36.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Cost(m) = 2 + 1 + 2
![Page 37: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/37.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Cost(m) = 2 + 1 + 2 + 1
![Page 38: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/38.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Cost(m) = 2 + 1 + 2 + 1 + 3
![Page 39: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/39.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Cost(m) = 2 + 1 + 2 + 1 + 3 + 1
![Page 40: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/40.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3
![Page 41: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/41.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13
Minimum Cost Labelling = MAP estimate
Pr(m) exp(-Cost(m))
![Page 42: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/42.jpg)
Aim• Accurate MAP estimation of pairwise Markov random fields
2
5
4
2
6
3
3
7
0
1 1
0
0
2 3
1
1
4 1
0
V1 V2 V3 V4
Label ‘-1’
Label ‘1’
Objectives• Applicable to all types of neighbourhood relationships• Applicable to all forms of pairwise costs• Guaranteed to converge (Convex approximation)
![Page 43: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/43.jpg)
MotivationSubgraph Matching - Torr - 2003, Schellewald et al - 2005
G1
G2
Unary costs are uniform
V2 V3
V1
MRF
ABCD A
BCD
ABCD
![Page 44: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/44.jpg)
MotivationSubgraph Matching - Torr - 2003, Schellewald et al - 2005
G1
G2
| d(mi,mj) - d(Vi,Vj) | <
12
YES NO
Potts Model
Pairwise Costs
![Page 45: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/45.jpg)
Motivation
V2 V3
V1
MRF
ABCD A
BCD
ABCD
Subgraph Matching - Torr - 2003, Schellewald et al - 2005
![Page 46: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/46.jpg)
Motivation
V2 V3
V1
MRF
ABCD A
BCD
ABCD
Subgraph Matching - Torr - 2003, Schellewald et al - 2005
![Page 47: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/47.jpg)
MotivationMatching Pictorial Structures - Felzenszwalb et al - 2001
Part likelihood Spatial Prior
Outline
Texture
Image
P1 P3
P2
(x,y,,)
MRF
![Page 48: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/48.jpg)
Motivation
Image
P1 P3
P2
(x,y,,)
MRF
• Unary potentials are negative log likelihoods
Valid pairwise configuration
Potts Model
Matching Pictorial Structures - Felzenszwalb et al - 2001
12
YES NO
![Page 49: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/49.jpg)
Motivation
P1 P3
P2
(x,y,,)
Pr(Cow)Image
• Unary potentials are negative log likelihoodsMatching Pictorial Structures - Felzenszwalb et al - 2001
Valid pairwise configuration
Potts Model
12
YES NO
![Page 50: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/50.jpg)
Outline• Integer Programming Formulation
• Previous Work
• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model
• Applications– Subgraph Matching– Pictorial Structures
![Page 51: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/51.jpg)
Integer Programming Formulation2
5
4
2
0
1 3
0
V1 V2
Label ‘-1’
Label ‘1’Unary Cost
Unary Cost Vector u = [ 5 2 ; 2 4 ]T
Labelling m = {1 , -1}
Label vector x = [ -1
V1=-1
1
V1 = 1
; 1 -1 ]T
Recall that the aim is to find the optimal x
![Page 52: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/52.jpg)
Integer Programming Formulation2
5
4
2
0
1 3
0
V1 V2
Label ‘-1’
Label ‘1’Unary Cost
Unary Cost Vector u = [ 5 2 ; 2 4 ]T
Labelling m = {1 , -1}
Label vector x = [ -1 1 ; 1 -1 ]T
Sum of Unary Costs = 12
∑i ui (1 + xi)
![Page 53: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/53.jpg)
Integer Programming Formulation2
5
4
2
0
1 3
0
V1 V2
Label ‘-1’
Label ‘1’Pairwise Cost
Labelling m = {1 , -1}
0Cost of V1 = -1 and V1 = -1
0
00
0Cost of V1 = -1 and V2 = -1
3
Cost of V1 = 0-1and V2 = 11 0
00
0 0
10
3 0
Pairwise Cost Matrix P
![Page 54: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/54.jpg)
Integer Programming Formulation2
5
4
2
0
1 3
0
V1 V2
Label ‘-1’
Label ‘1’Pairwise Cost
Labelling m = {1 , -1}
Pairwise Cost Matrix P
0 0
00
0 3
1 0
00
0 0
10
3 0
Sum of Pairwise Costs14
∑ij Pij (1 + xi)(1+xj)
![Page 55: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/55.jpg)
Integer Programming Formulation2
5
4
2
0
1 3
0
V1 V2
Label ‘0’
Label ‘1’Pairwise Cost
Labelling m = {1 , 0}
Pairwise Cost Matrix P
0 0
00
0 3
1 0
00
0 0
10
3 0
Sum of Pairwise Costs14
∑ij Pij (1 + xi +xj + xixj)
14
∑ij Pij (1 + xi + xj + Xij)=
X = x xT Xij = xi xj
![Page 56: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/56.jpg)
Integer Programming FormulationConstraints
• Each variable should be assigned a unique label
∑ xi = 2 - |L|i Va
• Marginalization constraint
∑ Xij = (2 - |L|) xij Vb
![Page 57: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/57.jpg)
Integer Programming FormulationChekuri et al. , SODA 2001
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi {-1,1}
X = x xT
ConvexNon-Convex
![Page 58: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/58.jpg)
Key Point
• In modern optimization the issue is not linear vs non linear but convex vs nonconvex
• We want to find a convex and good relaxation of the integer program.
![Page 59: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/59.jpg)
Outline• Integer Programming Formulation
• Previous Work
• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model
• Applications– Subgraph Matching– Pictorial Structures
![Page 60: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/60.jpg)
Linear Programming Formulation
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi {-1,1}
X = x xT
Chekuri et al. , SODA 2001Retain Convex Part
Relax Non-convex Constraint
![Page 61: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/61.jpg)
Linear Programming Formulation
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi [-1,1]
X = x xT
Chekuri et al. , SODA 2001Retain Convex Part
Relax Non-convex Constraint
![Page 62: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/62.jpg)
Linear Programming Formulation
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi [-1,1]
Chekuri et al. , SODA 2001Retain Convex Part
X becomes a variable to be optimized
![Page 63: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/63.jpg)
Feasible Region (IP) x {-1,1}, X = x2
Linear Programming Formulation
FeasibleRegionfor X.
![Page 64: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/64.jpg)
Feasible Region (IP)Feasible Region (Relaxation 1)
x {-1,1}, X = x2
x [-1,1], X = x2
Linear Programming Formulation
FeasibleRegionfor X.
![Page 65: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/65.jpg)
Feasible Region (IP)Feasible Region (Relaxation 1)Feasible Region (Relaxation 2)
x {-1,1}, X = x2
x [-1,1], X = x2
x [-1,1]
Linear Programming Formulation
FeasibleRegionfor X.
![Page 66: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/66.jpg)
Linear Programming Formulation
• Bounded algorithms proposed by Chekuri et al, SODA 2001
• -expansion - Komodakis and Tziritas, ICCV 2005
• TRW - Wainwright et al., NIPS 2002
• TRW-S - Kolmogorov, AISTATS 2005
• Efficient because it uses Linear Programming
• Not accurate
![Page 67: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/67.jpg)
Semidefinite Programming Formulation
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi {-1,1}
X = x xT
Lovasz and Schrijver, SIAM Optimization, 1990Retain Convex Part
Relax Non-convex Constraint
![Page 68: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/68.jpg)
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi [-1,1]
X = x xT
Semidefinite Programming Formulation
Retain Convex Part
Relax Non-convex Constraint
Lovasz and Schrijver, SIAM Optimization, 1990
![Page 69: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/69.jpg)
Semidefinite Programming Formulation
x1
x2
xn
1
...
1 x1 x2... xn
1 xT
x X
=
Rank = 1
Xii = 1
Positive SemidefiniteConvex
Non-Convex
![Page 70: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/70.jpg)
Semidefinite Programming Formulation
x1
x2
xn
1
...
1 x1 x2... xn
1 xT
x X
=
Xii = 1
Positive SemidefiniteConvex
![Page 71: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/71.jpg)
Schur’s Complement
A B
BT C
=I 0
BTA-1 I
A 0
0 C - BTA-1B
I A-1B
0 I
0
A 0 C -BTA-1B 0
![Page 72: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/72.jpg)
Semidefinite Programming Formulation
X - xxT 0
1 xT
x X
=1 0
x I
1 0
0 X - xxT
I xT
0 1
Schur’s Complement
![Page 73: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/73.jpg)
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi [-1,1]
X = x xT
Semidefinite Programming Formulation
Relax Non-convex Constraint
Retain Convex PartLovasz and Schrijver, SIAM Optimization, 1990
![Page 74: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/74.jpg)
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi [-1,1]
Semidefinite Programming Formulation
Xii = 1 X - xxT 0
Retain Convex PartLovasz and Schrijver, SIAM Optimization, 1990
![Page 75: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/75.jpg)
Feasible Region (IP) x {-1,1}, X = x2
Semidefinite Programming Formulation
FeasibleRegionfor X.
![Page 76: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/76.jpg)
Feasible Region (IP)Feasible Region (Relaxation 1)
x {-1,1}, X = x2
x [-1,1], X = x2
Semidefinite Programming Formulation
FeasibleRegionfor X.
![Page 77: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/77.jpg)
Feasible Region (IP)Feasible Region (Relaxation 1)Feasible Region (Relaxation 2)
x {-1,1}, X = x2
x [-1,1], X = x2
x [-1,1], X x2
Semidefinite Programming Formulation
FeasibleRegionfor X.
![Page 78: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/78.jpg)
Semidefinite Programming Formulation
• Formulated by Lovasz and Schrijver, 1990
• Finds a full X matrix
• Max-cut - Goemans and Williamson, JACM 1995
• Max-k-cut - de Klerk et al, 2000
• Torr AI Stats for labeling problem (2003 TR 2002)
•Accurate, but not efficient •as Semidefinite Programming algorithms slow
![Page 79: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/79.jpg)
Previous Work - Overview
LP SDP
ExamplesTRW-S,
-expansion
Max-k-Cut
Torr 2003
Accuracy Low High
Efficiency High Low
Is there a Middle Path ???
![Page 80: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/80.jpg)
Outline• Integer Programming Formulation
• Previous Work
• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model
• Applications– Subgraph Matching– Pictorial Structures
![Page 81: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/81.jpg)
Second Order Cone Programming
Second Order Cone || v || t OR || v ||2 st
x2 + y2 z2
![Page 82: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/82.jpg)
Minimize fTx
Subject to || Ai x+ bi || <= ciT x + di
i = 1, … , L
Linear Objective Function
Affine mapping of Second Order Cone (SOC)
Constraints are SOC of ni dimensions
Feasible regions are intersections of conic regions
Second Order Cone Programming
![Page 83: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/83.jpg)
Second Order Cone Programming
|| v || t tI v
vT t0
LP SOCP SDP
=1 0
vT I
tI 0
0 t2 - vTv
I v
0 1
Schur’s Complement
![Page 84: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/84.jpg)
Outline• Integer Programming Formulation
• Previous Work
• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model
• Applications– Subgraph Matching– Pictorial Structures
![Page 85: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/85.jpg)
First quick definition:Matrix Dot Product
A B = ∑ij Aij Bij
A11 A12
A21 A22
B11 B12
B21 B22
= A11 B11 + A12 B12 + A21 B21 + A22 B22
![Page 86: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/86.jpg)
SDP Relaxation
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi [-1,1]
Xii = 1 X - xxT 0
We will derive SOCP relaxation from the SDP relaxation
FurtherRelaxation
![Page 87: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/87.jpg)
1-D ExampleX - xxT 0
X - x2 ≥ 0
For two semidefinite matrices, the dot product is non-negative
A A 0
x2 X
SOC of the form || v ||2 st, s is a scalar constant.
![Page 88: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/88.jpg)
Feasible Region (IP)Feasible Region (Relaxation 1)Feasible Region (Relaxation 2)
x {-1,1}, X = x2
x [-1,1], X = x2
x [-1,1], X x2
SOCP Relaxation
For 1D: Same as the SDP formulation
FeasibleRegionfor X.
![Page 89: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/89.jpg)
2-D Example
X11 X12
X21 X22
1 X12
X12 1
=X =
x1x1 x1x2
x2x1 x2x2
xxT =x1
2 x1x2
x1x2
=x2
2
![Page 90: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/90.jpg)
2-D Example(X - xxT)
1 - x12 X12-x1x2. 0
1 0
0 0 X12-x1x2 1 - x22
x12 1
-1 x1 1
C1. 0 C1 0
![Page 91: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/91.jpg)
2-D Example(X - xxT)
1 - x12 X12-x1x2
C2. 0
. 00 0
0 1 X12-x1x2 1 - x22
x22 1
LP Relaxation-1 x2 1
C2 0
![Page 92: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/92.jpg)
2-D Example(X - xxT)
1 - x12 X12-x1x2
C3. 0
. 01 1
1 1 X12-x1x2 1 - x22
(x1 + x2)2 2 + 2X12
SOC of the form || v ||2 st
C3 0
![Page 93: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/93.jpg)
2-D Example(X - xxT)
1 - x12 X12-x1x2
C4. 0
. 01 -1
-1 1 X12-x1x2 1 - x22
(x1 - x2)2 2 - 2X12
SOC of the form || v ||2 st
C4 0
![Page 94: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/94.jpg)
General form of SOC constraints
Consider a matrix C1 = UUT 0
(X - xxT)
||UTx ||2 X . C1
C1 . 0
Continue for C2, C3, … , Cn
SOC of the form || v ||2 st
Kim and Kojima, 2000
![Page 95: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/95.jpg)
SOCP Relaxation
How many constraints for SOCP = SDP ?
Infinite. For all C 0
We specify constraints similar to the 2-D example
![Page 96: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/96.jpg)
SOCP RelaxationMuramatsu and Suzuki, 2001
1 0
0 0
0 0
0 1
1 1
1 1
1 -1
-1 1
Constraints hold for the above semidefinite matrices
![Page 97: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/97.jpg)
SOCP RelaxationMuramatsu and Suzuki, 2001
1 0
0 0
0 0
0 1
1 1
1 1
1 -1
-1 1
a + b
+ c + d
a 0
b 0
c 0
d 0
Constraints hold for the linear combination
![Page 98: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/98.jpg)
SOCP RelaxationMuramatsu and Suzuki, 2001
a+c+d c-d
c-d b+c+d
a 0
b 0
c 0
d 0Includes all semidefinite matrices where
Diagonal elements Off-diagonal elements
![Page 99: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/99.jpg)
SOCP Relaxation - A
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi [-1,1]
Xii = 1 X - xxT 0
![Page 100: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/100.jpg)
SOCP Relaxation - A
x* = argmin 12
∑ ui (1 + xi) + 14
∑ Pij (1 + xi + xj + Xij)
∑ xi = 2 - |L|i Va
∑ Xij = (2 - |L|) xij Vb
xi [-1,1]
(xi + xj)2 2 + 2Xij (xi - xj)2 2 - 2Xij
Specified only when Pij 0 i.e. sparse!!
![Page 101: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/101.jpg)
Triangular Inequality
• At least two of xi, xj and xk have the same sign
• At least one of Xij, Xjk, Xik is equal to one
Xij + Xjk + Xik -1Xij - Xjk - Xik -1-Xij - Xjk + Xik -1-Xij + Xjk - Xik -1
• SOCP-B = SOCP-A + Triangular Inequalities
![Page 102: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/102.jpg)
Outline• Integer Programming Formulation
• Previous Work
• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model
• Applications– Subgraph Matching– Pictorial Structures
![Page 103: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/103.jpg)
Robust Truncated ModelPairwise cost of incompatible labels is truncated
Potts ModelTruncated Linear Model
Truncated Quadratic Model
• Robust to noise
• Widely used in Computer Vision - Segmentation, Stereo
![Page 104: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/104.jpg)
Robust Truncated ModelPairwise Cost Matrix can be made sparse
P = [0.5 0.5 0.3 0.3 0.5]
Q = [0 0 -0.2 -0.2 0]
Reparameterization
Sparse Q matrix Fewer constraints
![Page 105: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/105.jpg)
Compatibility Constraint
Q(ma, mb) < 0 for variables Va and Vb
Relaxation ∑ Qij (1 + xi + xj + Xij) < 0
SOCP-C = SOCP-B + Compatibility Constraints
![Page 106: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/106.jpg)
SOCP Relaxation
• More accurate than LP
• More efficient than SDP
• Time complexity - O( |V|3 |L|3)
• Same as LP
• Approximate algorithms exist for LP relaxation
• We use |V| 10 and |L| 200
![Page 107: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/107.jpg)
Outline• Integer Programming Formulation
• Previous Work
• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model
• Applications– Subgraph Matching– Pictorial Structures
![Page 108: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/108.jpg)
Subgraph MatchingSubgraph Matching - Torr - 2003, Schellewald et al - 2005
G1
G2
Unary costs are uniform
V2 V3
V1
MRF
ABCD A
BCD
ABCD
Pairwise costs form a Potts model
![Page 109: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/109.jpg)
Subgraph Matching
• 1000 pairs of graphs G1 and G2
• #vertices in G2 - between 20 and 30
• #vertices in G1 - 0.25 * #vertices in G2
• 5% noise to the position of vertices
• NP-hard problem
![Page 110: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/110.jpg)
Subgraph MatchingMethod Time
(sec)Accuracy (%)
LP 0.85 6.64
LBP 0.2 78.6
GBP 1.5 85.2
SDP-A 35.0 93.11
SOCP-A 3.0 92.01
SOCP-B 4.5 94.79
SOCP-C 4.8 96.18
![Page 111: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/111.jpg)
Outline• Integer Programming Formulation
• Previous Work
• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model
• Applications– Subgraph Matching– Pictorial Structures
![Page 112: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/112.jpg)
Pictorial Structures
Image
P1 P3
P2
(x,y,,)
MRF
Matching Pictorial Structures - Felzenszwalb et al - 2001
Outline
Texture
![Page 113: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/113.jpg)
Pictorial Structures
Image
P1 P3
P2
(x,y,,)
MRF
Unary costs are negative log likelihoods
Pairwise costs form a Potts model
| V | = 10 | L | = 200
![Page 114: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/114.jpg)
Pictorial StructuresROC Curves for 450 +ve and 2400 -ve images
![Page 115: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/115.jpg)
Pictorial StructuresROC Curves for 450 +ve and 2400 -ve images
![Page 116: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/116.jpg)
Conclusions• We presented an SOCP relaxation to solve MRF
• More efficient than SDP
• More accurate than LP, LBP, GBP
• #variables can be reduced for Robust Truncated Model
• Provides excellent results for subgraph matching and pictorial structures
![Page 117: Solving Markov Random Fields using Dynamic Graph Cuts & Second Order Cone Programming Relaxations M. Pawan Kumar, Pushmeet Kohli Philip Torr](https://reader036.vdocuments.site/reader036/viewer/2022062511/5515eb87550346cf6f8b516c/html5/thumbnails/117.jpg)
Future Work
• Quality of solution– Additive bounds exist– Multiplicative bounds for special cases ??– What are good C’s.
• Message passing algorithm ??– Similar to TRW-S or -expansion– To handle image sized MRF