![Page 1: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/1.jpg)
Happy New Year!
![Page 2: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/2.jpg)
Structured Prediction for Computer VisionMLSS, Sydney 2015
Stephen Gould
19 February 2015
Stephen Gould | MLSS 2015 2/92
![Page 3: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/3.jpg)
Structured Models are Pervasive in Computer Vision
Stephen Gould | MLSS 2015 3/92
![Page 4: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/4.jpg)
Structured Models are Pervasive in Computer Vision
pixel labeling
Stephen Gould | MLSS 2015 3/92
![Page 5: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/5.jpg)
Structured Models are Pervasive in Computer Vision
pixel labelingobject detection,pose estimation
Stephen Gould | MLSS 2015 3/92
![Page 6: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/6.jpg)
Structured Models are Pervasive in Computer Vision
pixel labelingobject detection,pose estimation
scene understanding
Stephen Gould | MLSS 2015 3/92
![Page 7: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/7.jpg)
Structured Models are Pervasive in Computer Vision
pixel labeling object detection,pose estimation
scene understanding
Stephen Gould | MLSS 2015 4/92
![Page 8: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/8.jpg)
Demonstration: Pixel Labeling
[Agarwala et al., 2004]
640 × 480 image ≈ 300k pixels
4 possible labels per pixel
4300,000 label configurations
inference in under 30 seconds (unoptimized code)
Stephen Gould | MLSS 2015 5/92
![Page 9: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/9.jpg)
Demonstration: Pixel Labeling
[Agarwala et al., 2004]
640 × 480 image ≈ 300k pixels
4 possible labels per pixel
4300,000 label configurations
inference in under 30 seconds (unoptimized code)
Stephen Gould | MLSS 2015 5/92
![Page 10: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/10.jpg)
Conditional Markov Random Fields
Also known as:
Markov Networks, Undirected Graphical Models, MRFs,Structured Prediction modelsI make no distinction between these (in this tutorial)
X ∈ X are the observed random variables (always)
Y = (Y1, . . . ,Yn) ∈ Y are the output random variables
Yc are a subset of variables for clique c ⊆ {1, . . . , n}
Define a factored probability distribution
P(Y | X) =1
Z (X)
∏
c
Ψc(Yc ;X)
where Z (X) =∑
Y∈Y
∏
c Ψc(Yc ;X) is the partition function
Main difficulty is the exponential number of configurations
Stephen Gould | MLSS 2015 6/92
![Page 11: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/11.jpg)
Conditional Markov Random Fields
Also known as:
Markov Networks, Undirected Graphical Models, MRFs,Structured Prediction modelsI make no distinction between these (in this tutorial)
X ∈ X are the observed random variables (always)
Y = (Y1, . . . ,Yn) ∈ Y are the output random variables
Yc are a subset of variables for clique c ⊆ {1, . . . , n}
Define a factored probability distribution
P(Y | X) =1
Z (X)
∏
c
Ψc(Yc ;X)
where Z (X) =∑
Y∈Y
∏
c Ψc(Yc ;X) is the partition function
Main difficulty is the exponential number of configurations
Stephen Gould | MLSS 2015 6/92
![Page 12: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/12.jpg)
Conditional Markov Random Fields
Also known as:
Markov Networks, Undirected Graphical Models, MRFs,Structured Prediction modelsI make no distinction between these (in this tutorial)
X ∈ X are the observed random variables (always)
Y = (Y1, . . . ,Yn) ∈ Y are the output random variables
Yc are a subset of variables for clique c ⊆ {1, . . . , n}
Define a factored probability distribution
P(Y | X) =1
Z (X)
∏
c
Ψc(Yc ;X)
where Z (X) =∑
Y∈Y
∏
c Ψc(Yc ;X) is the partition function
Main difficulty is the exponential number of configurations
Stephen Gould | MLSS 2015 6/92
![Page 13: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/13.jpg)
Conditional Markov Random Fields
Also known as:
Markov Networks, Undirected Graphical Models, MRFs,Structured Prediction modelsI make no distinction between these (in this tutorial)
X ∈ X are the observed random variables (always)
Y = (Y1, . . . ,Yn) ∈ Y are the output random variables
Yc are a subset of variables for clique c ⊆ {1, . . . , n}
Define a factored probability distribution
P(Y | X) =1
Z (X)
∏
c
Ψc(Yc ;X)
where Z (X) =∑
Y∈Y
∏
c Ψc(Yc ;X) is the partition function
Main difficulty is the exponential number of configurations
Stephen Gould | MLSS 2015 6/92
![Page 14: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/14.jpg)
Conditional Markov Random Fields
Also known as:
Markov Networks, Undirected Graphical Models, MRFs,Structured Prediction modelsI make no distinction between these (in this tutorial)
X ∈ X are the observed random variables (always)
Y = (Y1, . . . ,Yn) ∈ Y are the output random variables
Yc are a subset of variables for clique c ⊆ {1, . . . , n}
Define a factored probability distribution
P(Y | X) =1
Z (X)
∏
c
Ψc(Yc ;X)
where Z (X) =∑
Y∈Y
∏
c Ψc(Yc ;X) is the partition function
Main difficulty is the exponential number of configurations
Stephen Gould | MLSS 2015 6/92
![Page 15: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/15.jpg)
Conditional Markov Random Fields
Also known as:
Markov Networks, Undirected Graphical Models, MRFs,Structured Prediction modelsI make no distinction between these (in this tutorial)
X ∈ X are the observed random variables (always)
Y = (Y1, . . . ,Yn) ∈ Y are the output random variables
Yc are a subset of variables for clique c ⊆ {1, . . . , n}
Define a factored probability distribution
P(Y | X) =1
Z (X)
∏
c
Ψc(Yc ;X)
where Z (X) =∑
Y∈Y
∏
c Ψc(Yc ;X) is the partition function
Main difficulty is the exponential number of configurations
Stephen Gould | MLSS 2015 6/92
![Page 16: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/16.jpg)
Conditional Markov Random Fields
Also known as:
Markov Networks, Undirected Graphical Models, MRFs,Structured Prediction modelsI make no distinction between these (in this tutorial)
X ∈ X are the observed random variables (always)
Y = (Y1, . . . ,Yn) ∈ Y are the output random variables
Yc are a subset of variables for clique c ⊆ {1, . . . , n}
Define a factored probability distribution
P(Y | X) =1
Z (X)
∏
c
Ψc(Yc ;X)
where Z (X) =∑
Y∈Y
∏
c Ψc(Yc ;X) is the partition function
Main difficulty is the exponential number of configurations
Stephen Gould | MLSS 2015 6/92
![Page 17: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/17.jpg)
Conditional Markov Random Fields
Also known as:
Markov Networks, Undirected Graphical Models, MRFs,Structured Prediction modelsI make no distinction between these (in this tutorial)
X ∈ X are the observed random variables (always)
Y = (Y1, . . . ,Yn) ∈ Y are the output random variables
Yc are a subset of variables for clique c ⊆ {1, . . . , n}
Define a factored probability distribution
P(Y | X) =1
Z (X)
∏
c
Ψc(Yc ;X)
where Z (X) =∑
Y∈Y
∏
c Ψc(Yc ;X) is the partition function
Main difficulty is the exponential number of configurations
Stephen Gould | MLSS 2015 6/92
![Page 18: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/18.jpg)
Machine Learning Tasks
There are two main tasks that we are interested in when talkingabout conditional Markov random fields (machine learning, moregenerally):
Learning: Given data (and a problem specification), how dowe choose the structure and set the parameters of our model?
Inference: Given our model, how do we answer queries aboutinstances of our problem?
Stephen Gould | MLSS 2015 7/92
![Page 19: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/19.jpg)
MAP Inference
We will mainly be interested in maximum a posteriori (MAP)inference
y⋆ = argmaxy∈Y
P(y | x)
= argmaxy∈Y
1
Z (X)
∏
c
Ψc(Yc ;X)
= argmaxy∈Y
log
(
1
Z (X)
∏
c
Ψc(Yc ;X)
)
= argmaxy∈Y
∑
c
log Ψc(Yc ;X)− logZ (X)
= argmaxy∈Y
∑
c
log Ψc(Yc ;X)
Stephen Gould | MLSS 2015 8/92
![Page 20: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/20.jpg)
Energy Functions
Define an energy function
E (Y;X) =∑
c
ψc(Yc ;X)
where ψc (·) = − log Ψc(·)
Then
P(Y | X) =1
Z (X)exp {−E (Y;X)}
Andargmax
y∈YP(y | x) = argmin
y∈YE (y; x)
Stephen Gould | MLSS 2015 9/92
![Page 21: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/21.jpg)
Energy Functions
Define an energy function
E (Y;X) =∑
c
ψc(Yc ;X)
where ψc (·) = − log Ψc(·)
Then
P(Y | X) =1
Z (X)exp {−E (Y;X)}
Andargmax
y∈YP(y | x) = argmin
y∈YE (y; x)
Stephen Gould | MLSS 2015 9/92
![Page 22: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/22.jpg)
Energy Functions
Define an energy function
E (Y;X) =∑
c
ψc(Yc ;X)
where ψc (·) = − log Ψc(·)
Then
P(Y | X) =1
Z (X)exp {−E (Y;X)}
Andargmax
y∈YP(y | x) = argmin
y∈YE (y; x)
Stephen Gould | MLSS 2015 9/92
![Page 23: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/23.jpg)
Energy Functions
Define an energy function
E (Y;X) =∑
c
ψc(Yc ;X)
where ψc (·) = − log Ψc(·)
Then
P(Y | X) =1
Z (X)exp {−E (Y;X)}
Andargmax
y∈YP(y | x) = argmin
y∈YE (y; x)
energy minimization ‘equals’ MAP inference
Stephen Gould | MLSS 2015 9/92
![Page 24: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/24.jpg)
Clique Potentials
A clique potential ψc(yc ; x) defines a mapping from anassignment of the random variables to a real number
ψc : Yc × X → R
The clique potential encodes a preference for assignments tothe random variables (lower value is more preferred)
Often parameterized as
ψc(yc ; x) = wTc φc(yc ; x)
In this tutorial is suffices to think of the clique potentials asbig lookup tables
We will also ignore the explicit conditioning on X
Stephen Gould | MLSS 2015 10/92
![Page 25: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/25.jpg)
Clique Potentials
A clique potential ψc(yc ; x) defines a mapping from anassignment of the random variables to a real number
ψc : Yc × X → R
The clique potential encodes a preference for assignments tothe random variables (lower value is more preferred)
Often parameterized as
ψc(yc ; x) = wTc φc(yc ; x)
In this tutorial is suffices to think of the clique potentials asbig lookup tables
We will also ignore the explicit conditioning on X
Stephen Gould | MLSS 2015 10/92
![Page 26: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/26.jpg)
Clique Potentials
A clique potential ψc(yc ; x) defines a mapping from anassignment of the random variables to a real number
ψc : Yc × X → R
The clique potential encodes a preference for assignments tothe random variables (lower value is more preferred)
Often parameterized as
ψc(yc ; x) = wTc φc(yc ; x)
In this tutorial is suffices to think of the clique potentials asbig lookup tables
We will also ignore the explicit conditioning on X
Stephen Gould | MLSS 2015 10/92
![Page 27: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/27.jpg)
Clique Potentials
A clique potential ψc(yc ; x) defines a mapping from anassignment of the random variables to a real number
ψc : Yc × X → R
The clique potential encodes a preference for assignments tothe random variables (lower value is more preferred)
Often parameterized as
ψc(yc ; x) = wTc φc(yc ; x)
In this tutorial is suffices to think of the clique potentials asbig lookup tables
We will also ignore the explicit conditioning on X
Stephen Gould | MLSS 2015 10/92
![Page 28: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/28.jpg)
Clique Potentials
A clique potential ψc(yc ; x) defines a mapping from anassignment of the random variables to a real number
ψc : Yc × X → R
The clique potential encodes a preference for assignments tothe random variables (lower value is more preferred)
Often parameterized as
ψc(yc ; x) = wTc φc(yc ; x)
In this tutorial is suffices to think of the clique potentials asbig lookup tables
We will also ignore the explicit conditioning on X
Stephen Gould | MLSS 2015 10/92
![Page 29: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/29.jpg)
Clique Potential Arity
E (y; x) =∑
c
ψc(yc ; x)
=∑
i∈V
ψUi (yi ; x)
︸ ︷︷ ︸
unary
+∑
ij∈E
ψPij (yi , yj ; x)
︸ ︷︷ ︸
pairwise
+∑
c∈C
ψHc (yc ; x).
︸ ︷︷ ︸
higher-order
x1 x2 x3
y1 y2 y3
x4 x5 x6
y4 y5 y6
x7 x8 x9
y7 y8 y9
Stephen Gould | MLSS 2015 11/92
![Page 30: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/30.jpg)
Example Energy Functions
Semantic Segm.
Labels: L = {sky, tree, grass, . . .}Unary: classifier, ψU
i(yi = ℓ; x) = logP (φi (x) | ℓ)
Pairwise: contrast-dependent smoothness prior,
ψPij (yi , yj ; x) =
λ0 + λ1 exp
(
−‖xi−xj‖
2
2β
)
, if yi 6= yj
0, otherwise
Stephen Gould | MLSS 2015 12/92
![Page 31: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/31.jpg)
Example Energy Functions
Semantic Segm.
Labels: L = {sky, tree, grass, . . .}Unary: classifier, ψU
i(yi = ℓ; x) = logP (φi (x) | ℓ)
Pairwise: contrast-dependent smoothness prior,
ψPij (yi , yj ; x) =
λ0 + λ1 exp
(
−‖xi−xj‖
2
2β
)
, if yi 6= yj
0, otherwise
Object Detection
Labels: L = [0,W ]× [0,H] × R+
Unary: part detector/filter response, ψUi
= φi (x) ∗ wi (ℓ)Pairwise: deformation cost,
ψPij (yi , yj ; x) =
{
λ‖yi − yj‖22, same scale
∞, otherwise
Stephen Gould | MLSS 2015 12/92
![Page 32: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/32.jpg)
Example Energy Functions
Semantic Segm.
Labels: L = {sky, tree, grass, . . .}Unary: classifier, ψU
i(yi = ℓ; x) = logP (φi (x) | ℓ)
Pairwise: contrast-dependent smoothness prior,
ψPij (yi , yj ; x) =
λ0 + λ1 exp
(
−‖xi−xj‖
2
2β
)
, if yi 6= yj
0, otherwise
Object Detection
Labels: L = [0,W ]× [0,H] × R+
Unary: part detector/filter response, ψUi
= φi (x) ∗ wi (ℓ)Pairwise: deformation cost,
ψPij (yi , yj ; x) =
{
λ‖yi − yj‖22, same scale
∞, otherwise
Photo Montage
Labels: L = {1, 2, . . . ,K}Unary: none!Pairwise: seam penalty
ψPij (yi , yj ; x) = ‖xyi (i)− xyj (i)‖+ ‖xyi (j)− xyj (j)‖
(or edge-normalized variant)
Stephen Gould | MLSS 2015 12/92
![Page 33: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/33.jpg)
Graphical Representation
E (y) = ψ(y1, y2) + ψ(y2, y3) + ψ(y3, y4) + ψ(y4, y1)
Y1 Y2
Y4 Y3
� � � �
Y1 Y2 Y3 Y4
graphical model factor graph
Stephen Gould | MLSS 2015 13/92
![Page 34: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/34.jpg)
Graphical Representation
E (y) =∑
i ,j ψ(yi , yj )
Y1 Y2
Y4 Y3
� � � � � �
Y1 Y2 Y3 Y4
Stephen Gould | MLSS 2015 14/92
![Page 35: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/35.jpg)
Graphical Representation
E (y) = ψ(y1, y2, y3, y4)
Y1 Y2
Y4 Y3
�
Y1 Y2 Y3 Y4
Stephen Gould | MLSS 2015 15/92
![Page 36: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/36.jpg)
Graphical Representation
E (y) = ψ(y1, y2, y3, y4)
Y1 Y2
Y4 Y3
�
Y1 Y2 Y3 Y4
don’t worry too much about the graphical representation,look at the form of the energy function
Stephen Gould | MLSS 2015 15/92
![Page 37: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/37.jpg)
MAP Inference / Energy Minimization
Computing the energy minimizing assignment is NP-hard
argminy∈Y
E (y; x) = argmaxy∈Y
P(y | x)
Some structures admit tractable exact inference algorithms
low treewidth graphs → message passingsubmodular potentials → graph-cuts
Moreover, efficent approximate inference algorithms exist
message passing on general graphsmove making inference (submodular moves)linear programming relaxations
Stephen Gould | MLSS 2015 16/92
![Page 38: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/38.jpg)
MAP Inference / Energy Minimization
Computing the energy minimizing assignment is NP-hard
argminy∈Y
E (y; x) = argmaxy∈Y
P(y | x)
Some structures admit tractable exact inference algorithms
low treewidth graphs → message passingsubmodular potentials → graph-cuts
Moreover, efficent approximate inference algorithms exist
message passing on general graphsmove making inference (submodular moves)linear programming relaxations
Stephen Gould | MLSS 2015 16/92
![Page 39: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/39.jpg)
MAP Inference / Energy Minimization
Computing the energy minimizing assignment is NP-hard
argminy∈Y
E (y; x) = argmaxy∈Y
P(y | x)
Some structures admit tractable exact inference algorithms
low treewidth graphs → message passingsubmodular potentials → graph-cuts
Moreover, efficent approximate inference algorithms exist
message passing on general graphsmove making inference (submodular moves)linear programming relaxations
Stephen Gould | MLSS 2015 16/92
![Page 40: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/40.jpg)
exact inference
Stephen Gould | MLSS 2015 17/92
![Page 41: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/41.jpg)
An Example: Chain Graph
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1 � Y2 � Y3 � Y4
miny
E (y) = miny1,y2,y3,y4
ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
= miny1,y2,y3
ψA(y1, y2) + ψB(y2, y3) + miny4ψC (y3, y4)
︸ ︷︷ ︸
mC→B(y3)
= miny1,y2
ψA(y1, y2) + miny3ψB(y2, y3) +mC→B(y3)
︸ ︷︷ ︸
mB→A(y2)
= miny1,y2
ψA(y1, y2) +mB→A(y2)
Stephen Gould | MLSS 2015 18/92
![Page 42: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/42.jpg)
An Example: Chain Graph
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
miny
E (y) = miny1,y2,y3,y4
ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
= miny1,y2,y3
ψA(y1, y2) + ψB(y2, y3) + miny4ψC (y3, y4)
︸ ︷︷ ︸
mC→B(y3)
= miny1,y2
ψA(y1, y2) + miny3ψB(y2, y3) +mC→B(y3)
︸ ︷︷ ︸
mB→A(y2)
= miny1,y2
ψA(y1, y2) +mB→A(y2)
Stephen Gould | MLSS 2015 18/92
![Page 43: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/43.jpg)
An Example: Chain Graph
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
miny
E (y) = miny1,y2,y3,y4
ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
= miny1,y2,y3
ψA(y1, y2) + ψB(y2, y3) + miny4ψC (y3, y4)
︸ ︷︷ ︸
mC→B(y3)
= miny1,y2
ψA(y1, y2) + miny3ψB(y2, y3) +mC→B(y3)
︸ ︷︷ ︸
mB→A(y2)
= miny1,y2
ψA(y1, y2) +mB→A(y2)
Stephen Gould | MLSS 2015 18/92
![Page 44: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/44.jpg)
An Example: Chain Graph
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
miny
E (y) = miny1,y2,y3,y4
ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
= miny1,y2,y3
ψA(y1, y2) + ψB(y2, y3) + miny4ψC (y3, y4)
︸ ︷︷ ︸
mC→B(y3)
= miny1,y2
ψA(y1, y2) + miny3ψB(y2, y3) +mC→B(y3)
︸ ︷︷ ︸
mB→A(y2)
= miny1,y2
ψA(y1, y2) +mB→A(y2)
Stephen Gould | MLSS 2015 18/92
![Page 45: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/45.jpg)
An Example: Chain Graph
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
miny
E (y) = miny1,y2,y3,y4
ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
= miny1,y2,y3
ψA(y1, y2) + ψB(y2, y3) + miny4ψC (y3, y4)
︸ ︷︷ ︸
mC→B(y3)
= miny1,y2
ψA(y1, y2) + miny3ψB(y2, y3) +mC→B(y3)
︸ ︷︷ ︸
mB→A(y2)
= miny1,y2
ψA(y1, y2) +mB→A(y2)
Stephen Gould | MLSS 2015 18/92
![Page 46: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/46.jpg)
An Example: Chain Graph
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
miny
E (y) = miny1,y2,y3,y4
ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
= miny1,y2,y3
ψA(y1, y2) + ψB(y2, y3) + miny4ψC (y3, y4)
︸ ︷︷ ︸
mC→B(y3)
= miny1,y2
ψA(y1, y2) + miny3ψB(y2, y3) +mC→B(y3)
︸ ︷︷ ︸
mB→A(y2)
= miny1,y2
ψA(y1, y2) +mB→A(y2)
Stephen Gould | MLSS 2015 18/92
![Page 47: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/47.jpg)
Viterbi Decoding
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
The energy minimizing assignment can be decoded as
y⋆1 = argminy1
miny2ψA(y1, y2) +mB→A(y2)
y⋆2 = argminy2
ψA(y⋆1 , y2) +mB→A(y2)
y⋆3 = argminy3
ψB(y⋆2 , y3) +mC→B(y3)
y⋆4 = argminy4
ψC (y⋆3 , y4)
Stephen Gould | MLSS 2015 19/92
![Page 48: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/48.jpg)
Viterbi Decoding
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
The energy minimizing assignment can be decoded as
y⋆1 = argminy1
miny2ψA(y1, y2) +mB→A(y2)
y⋆2 = argminy2
ψA(y⋆1 , y2) +mB→A(y2)
y⋆3 = argminy3
ψB(y⋆2 , y3) +mC→B(y3)
y⋆4 = argminy4
ψC (y⋆3 , y4)
Stephen Gould | MLSS 2015 19/92
![Page 49: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/49.jpg)
Viterbi Decoding
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
The energy minimizing assignment can be decoded as
y⋆1 = argminy1
miny2ψA(y1, y2) +mB→A(y2)
y⋆2 = argminy2
ψA(y⋆1 , y2) +mB→A(y2)
y⋆3 = argminy3
ψB(y⋆2 , y3) +mC→B(y3)
y⋆4 = argminy4
ψC (y⋆3 , y4)
Stephen Gould | MLSS 2015 19/92
![Page 50: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/50.jpg)
Viterbi Decoding
E (y) = ψA(y1, y2) + ψB(y2, y3) + ψC (y3, y4)
Y1,Y2 Y2,Y3 Y3,Y4
The energy minimizing assignment can be decoded as
y⋆1 = argminy1
miny2ψA(y1, y2) +mB→A(y2)
y⋆2 = argminy2
ψA(y⋆1 , y2) +mB→A(y2)
y⋆3 = argminy3
ψB(y⋆2 , y3) +mC→B(y3)
y⋆4 = argminy4
ψC (y⋆3 , y4)
Stephen Gould | MLSS 2015 19/92
![Page 51: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/51.jpg)
What did this cost us?
Y1 � Y2 � · · · � Yn
For a chain of length n with L labels per variable:
Brute force enumeration would cost |Y| = Ln
Viterbi decoding (message passing) costs O(nL2)
The operation minψ(·, ·) +m(·) can be sped up for potentialswith certain structure (e.g., so called convex priors)
Stephen Gould | MLSS 2015 20/92
![Page 52: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/52.jpg)
What did this cost us?
Y1 � Y2 � · · · � Yn
For a chain of length n with L labels per variable:
Brute force enumeration would cost |Y| = Ln
Viterbi decoding (message passing) costs O(nL2)
The operation minψ(·, ·) +m(·) can be sped up for potentialswith certain structure (e.g., so called convex priors)
Stephen Gould | MLSS 2015 20/92
![Page 53: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/53.jpg)
What did this cost us?
Y1 � Y2 � · · · � Yn
For a chain of length n with L labels per variable:
Brute force enumeration would cost |Y| = Ln
Viterbi decoding (message passing) costs O(nL2)
The operation minψ(·, ·) +m(·) can be sped up for potentialswith certain structure (e.g., so called convex priors)
Stephen Gould | MLSS 2015 20/92
![Page 54: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/54.jpg)
Factor Operations
The preceeding inference algorithm was based on two importantoperations defined on factors (clique potentials).
Factor addition creates an outut whose scope is the union ofthe scope of its inputs. Each element of the output is the sumof the corresponding (projected) elements of the inputs.
Yc = Ya ∪ Yb : ψc (yc) = ψa([yc ]a) + ψb([yc ]b)
Factor minimization creates an output where one or moreinput variables are removed. Each element of the output isthe result of minimizing over values of the removed variables.
Yc ⊂ Ya : ψc(yc) = minya\c∈Ya\Yc
ψa({ya\c , yc})
Stephen Gould | MLSS 2015 21/92
![Page 55: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/55.jpg)
Factor Operations
The preceeding inference algorithm was based on two importantoperations defined on factors (clique potentials).
Factor addition creates an outut whose scope is the union ofthe scope of its inputs. Each element of the output is the sumof the corresponding (projected) elements of the inputs.
Yc = Ya ∪ Yb : ψc (yc) = ψa([yc ]a) + ψb([yc ]b)
Factor minimization creates an output where one or moreinput variables are removed. Each element of the output isthe result of minimizing over values of the removed variables.
Yc ⊂ Ya : ψc(yc) = minya\c∈Ya\Yc
ψa({ya\c , yc})
Stephen Gould | MLSS 2015 21/92
![Page 56: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/56.jpg)
Factor Operations Worked Example
y1 y2 ψa
0 0 10 1 41 0 71 1 2
plus
y2 y3 ψb
0 0 50 1 -31 0 11 1 8
=
y1 y2 y3 ψc = ψa + ψb
0 0 0 1 + 5 = 60 0 1 1 - 3 = -20 1 0 4 + 1 = 50 1 1 4 + 8 = 121 0 0 7 + 5 = 121 0 1 7 - 3 = 41 1 0 2 + 1 = 31 1 1 2 + 8 = 10
Stephen Gould | MLSS 2015 22/92
![Page 57: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/57.jpg)
Clique Trees
A clique tree (or tree decomposition) for an energy function E (y)is a pair (C,T ), where C = {C1, . . . ,CM} is a family of subsets of{1, . . . , n} and T is a tree with nodes Cm satisfying:
Family Preserving: if Yc is a clique in E (y) then there mustexist a subset Cm ∈ C with Yc ∈ Cm;
Running Intersection Property: if Cm and Cm′ both containYi then there is a unique path through T between Cm andCm′ such that Yi is in every node along the path.
These properties are sufficient to ensure the message passingcorrectness of message passing.
Stephen Gould | MLSS 2015 23/92
![Page 58: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/58.jpg)
Min-Sum Message Passing on Clique Trees
messages sent in reverse then forward topological orderingmessage from clique i to clique j calculated as
mi→j(Yj ∩Yi) = minYi\Yj
(
ψi (Yi) +∑
k∈N (i)\{j}
mk→i (Yi ∩Yk))
energy minimizing assignment decoded as
y⋆i = argminYi
( min marginal︷ ︸︸ ︷
ψi (Yi) +∑
k∈N (i)
mk→i (Yi ∩ Yk)
)
ties must be decoded consistently
Stephen Gould | MLSS 2015 24/92
![Page 59: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/59.jpg)
Min-Sum Message Passing on Clique Trees
messages sent in reverse then forward topological orderingmessage from clique i to clique j calculated as
mi→j(Yj ∩Yi) = minYi\Yj
(
ψi (Yi) +∑
k∈N (i)\{j}
mk→i (Yi ∩Yk))
energy minimizing assignment decoded as
y⋆i = argminYi
( min marginal︷ ︸︸ ︷
ψi (Yi) +∑
k∈N (i)
mk→i (Yi ∩ Yk)
)
ties must be decoded consistently
Stephen Gould | MLSS 2015 24/92
![Page 60: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/60.jpg)
Min-Sum Message Passing on Clique Trees
messages sent in reverse then forward topological orderingmessage from clique i to clique j calculated as
mi→j(Yj ∩Yi) = minYi\Yj
(
ψi (Yi) +∑
k∈N (i)\{j}
mk→i (Yi ∩Yk))
energy minimizing assignment decoded as
y⋆i = argminYi
( min marginal︷ ︸︸ ︷
ψi (Yi) +∑
k∈N (i)
mk→i (Yi ∩ Yk)
)
ties must be decoded consistently
Stephen Gould | MLSS 2015 24/92
![Page 61: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/61.jpg)
Min-Sum Message Passing on Clique Trees
messages sent in reverse then forward topological orderingmessage from clique i to clique j calculated as
mi→j(Yj ∩Yi) = minYi\Yj
(
ψi (Yi) +∑
k∈N (i)\{j}
mk→i (Yi ∩Yk))
energy minimizing assignment decoded as
y⋆i = argminYi
( min marginal︷ ︸︸ ︷
ψi (Yi) +∑
k∈N (i)
mk→i (Yi ∩ Yk)
)
ties must be decoded consistently
Stephen Gould | MLSS 2015 24/92
![Page 62: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/62.jpg)
Min-Sum Message Passing on Factor Graphs (Trees)
messages from variables to factors
mi→F (yi ) =∑
G∈N (i)\{F}
mG→i (yi )
messages from factors to variables
mF→i (yi ) = miny′F,y ′
i=yi
(
ψF (y′F ) +
∑
j∈N (F )\{i}
mj→F (y′j ))
energy minimizing assignment decoded as
y⋆i = argminyi
∑
F∈N (i)
mF→i(yi )
Stephen Gould | MLSS 2015 25/92
![Page 63: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/63.jpg)
Min-Sum Message Passing on Factor Graphs (Trees)
messages from variables to factors
mi→F (yi ) =∑
G∈N (i)\{F}
mG→i (yi )
messages from factors to variables
mF→i (yi ) = miny′F,y ′
i=yi
(
ψF (y′F ) +
∑
j∈N (F )\{i}
mj→F (y′j ))
energy minimizing assignment decoded as
y⋆i = argminyi
∑
F∈N (i)
mF→i(yi )
Stephen Gould | MLSS 2015 25/92
![Page 64: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/64.jpg)
Min-Sum Message Passing on Factor Graphs (Trees)
messages from variables to factors
mi→F (yi ) =∑
G∈N (i)\{F}
mG→i (yi )
messages from factors to variables
mF→i (yi ) = miny′F,y ′
i=yi
(
ψF (y′F ) +
∑
j∈N (F )\{i}
mj→F (y′j ))
energy minimizing assignment decoded as
y⋆i = argminyi
∑
F∈N (i)
mF→i(yi )
Stephen Gould | MLSS 2015 25/92
![Page 65: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/65.jpg)
Message Passing on General Graphs
Message passing can be generalized to graphs with loops
If the treewidth is small we can still perform exact inference
junction tree algorithm: triangulate the graph and runmessage passing on the resulting tree
Otherwise run message passing anyway
loopy belief propagtaiondifferent message schedules (synchronous/asynchronous,static/dynamic)no convergence or approximation guarantees, in general
Stephen Gould | MLSS 2015 26/92
![Page 66: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/66.jpg)
Message Passing on General Graphs
Message passing can be generalized to graphs with loops
If the treewidth is small we can still perform exact inference
junction tree algorithm: triangulate the graph and runmessage passing on the resulting tree
Otherwise run message passing anyway
loopy belief propagtaiondifferent message schedules (synchronous/asynchronous,static/dynamic)no convergence or approximation guarantees, in general
Stephen Gould | MLSS 2015 26/92
![Page 67: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/67.jpg)
Message Passing on General Graphs
Message passing can be generalized to graphs with loops
If the treewidth is small we can still perform exact inference
junction tree algorithm: triangulate the graph and runmessage passing on the resulting tree
Otherwise run message passing anyway
loopy belief propagtaiondifferent message schedules (synchronous/asynchronous,static/dynamic)no convergence or approximation guarantees, in general
Stephen Gould | MLSS 2015 26/92
![Page 68: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/68.jpg)
Message Passing on General Graphs
Message passing can be generalized to graphs with loops
If the treewidth is small we can still perform exact inference
junction tree algorithm: triangulate the graph and runmessage passing on the resulting tree
Otherwise run message passing anyway
loopy belief propagtaiondifferent message schedules (synchronous/asynchronous,static/dynamic)no convergence or approximation guarantees, in general
Stephen Gould | MLSS 2015 26/92
![Page 69: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/69.jpg)
Message Passing on General Graphs
Message passing can be generalized to graphs with loops
If the treewidth is small we can still perform exact inference
junction tree algorithm: triangulate the graph and runmessage passing on the resulting tree
Otherwise run message passing anyway
loopy belief propagtaiondifferent message schedules (synchronous/asynchronous,static/dynamic)no convergence or approximation guarantees, in general
Stephen Gould | MLSS 2015 26/92
![Page 70: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/70.jpg)
Message Passing on General Graphs
Message passing can be generalized to graphs with loops
If the treewidth is small we can still perform exact inference
junction tree algorithm: triangulate the graph and runmessage passing on the resulting tree
Otherwise run message passing anyway
loopy belief propagtaiondifferent message schedules (synchronous/asynchronous,static/dynamic)no convergence or approximation guarantees, in general
Stephen Gould | MLSS 2015 26/92
![Page 71: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/71.jpg)
Message Passing on General Graphs
Message passing can be generalized to graphs with loops
If the treewidth is small we can still perform exact inference
junction tree algorithm: triangulate the graph and runmessage passing on the resulting tree
Otherwise run message passing anyway
loopy belief propagtaiondifferent message schedules (synchronous/asynchronous,static/dynamic)no convergence or approximation guarantees, in general
Stephen Gould | MLSS 2015 26/92
![Page 72: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/72.jpg)
graph-cut based methods
Stephen Gould | MLSS 2015 27/92
![Page 73: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/73.jpg)
Binary MRF Example
Consider the following energy function fortwo binary random variables, y1 and y2.
01
5
2
01
1
3
01
0 1
0 3
4 0
E (y1, y2) = ψ1(y1) + ψ2(y2) + ψ12(y1, y2)= 5y1 + 2y1︸ ︷︷ ︸
ψ1
+ y2 + 3y2︸ ︷︷ ︸
ψ2
+ 3y1y2 + 4y1y2︸ ︷︷ ︸
ψ12
where y1 = 1− y1 and y2 = 1− y2.
Stephen Gould | MLSS 2015 28/92
![Page 74: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/74.jpg)
Binary MRF Example
Consider the following energy function fortwo binary random variables, y1 and y2.
01
5
2
01
1
3
01
0 1
0 3
4 0
E (y1, y2) = ψ1(y1) + ψ2(y2) + ψ12(y1, y2)= 5y1 + 2y1︸ ︷︷ ︸
ψ1
+ y2 + 3y2︸ ︷︷ ︸
ψ2
+ 3y1y2 + 4y1y2︸ ︷︷ ︸
ψ12
where y1 = 1− y1 and y2 = 1− y2.
Stephen Gould | MLSS 2015 28/92
![Page 75: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/75.jpg)
Binary MRF Example
Consider the following energy function fortwo binary random variables, y1 and y2.
01
5
2
01
1
3
01
0 1
0 3
4 0
E (y1, y2) = ψ1(y1) + ψ2(y2) + ψ12(y1, y2)= 5y1 + 2y1︸ ︷︷ ︸
ψ1
+ y2 + 3y2︸ ︷︷ ︸
ψ2
+ 3y1y2 + 4y1y2︸ ︷︷ ︸
ψ12
where y1 = 1− y1 and y2 = 1− y2.
Graphical Model
y1 y2
Probability Table
y1 y2 E P
0 0 6 0.244
0 1 11 0.002
1 0 7 0.090
1 1 5 0.664
Stephen Gould | MLSS 2015 28/92
![Page 76: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/76.jpg)
Pseudo-boolean Functions [Boros and Hammer, 2001]
Pseudo-boolean Function
A mapping f : {0, 1}n → R is called a pseudo-Boolean function.
Pseudo-boolean functions can be uniquely represented asmulti-linear polynomials, e.g., f (y1, y2) = 6+ y1+5y2− 7y1y2.
Pseudo-boolean functions can also be represented in posiform,e.g., f (y1, y2) = 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2. Thisrepresentation is not unique.
A binary pairwise Markov random field (MRF) is just aquadratic pseudo-Boolean function.
Stephen Gould | MLSS 2015 29/92
![Page 77: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/77.jpg)
Pseudo-boolean Functions [Boros and Hammer, 2001]
Pseudo-boolean Function
A mapping f : {0, 1}n → R is called a pseudo-Boolean function.
Pseudo-boolean functions can be uniquely represented asmulti-linear polynomials, e.g., f (y1, y2) = 6+ y1+5y2− 7y1y2.
Pseudo-boolean functions can also be represented in posiform,e.g., f (y1, y2) = 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2. Thisrepresentation is not unique.
A binary pairwise Markov random field (MRF) is just aquadratic pseudo-Boolean function.
Stephen Gould | MLSS 2015 29/92
![Page 78: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/78.jpg)
Pseudo-boolean Functions [Boros and Hammer, 2001]
Pseudo-boolean Function
A mapping f : {0, 1}n → R is called a pseudo-Boolean function.
Pseudo-boolean functions can be uniquely represented asmulti-linear polynomials, e.g., f (y1, y2) = 6+ y1+5y2− 7y1y2.
Pseudo-boolean functions can also be represented in posiform,e.g., f (y1, y2) = 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2. Thisrepresentation is not unique.
A binary pairwise Markov random field (MRF) is just aquadratic pseudo-Boolean function.
Stephen Gould | MLSS 2015 29/92
![Page 79: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/79.jpg)
Pseudo-boolean Functions [Boros and Hammer, 2001]
Pseudo-boolean Function
A mapping f : {0, 1}n → R is called a pseudo-Boolean function.
Pseudo-boolean functions can be uniquely represented asmulti-linear polynomials, e.g., f (y1, y2) = 6+ y1+5y2− 7y1y2.
Pseudo-boolean functions can also be represented in posiform,e.g., f (y1, y2) = 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2. Thisrepresentation is not unique.
A binary pairwise Markov random field (MRF) is just aquadratic pseudo-Boolean function.
Stephen Gould | MLSS 2015 29/92
![Page 80: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/80.jpg)
Submodular Functions
Submodularity
Let V be a set. A set function f : 2V → R is called submodular iff (X ) + f (Y ) ≥ f (X ∪ Y ) + f (X ∩ Y ) for all subsets X ,Y ⊆ V.
Stephen Gould | MLSS 2015 30/92
![Page 81: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/81.jpg)
Submodular Binary Pairwise MRFs
Submodularity
A pseudo-Boolean function f : {0, 1}n → R is called submodular iff (x) + f (y) ≥ f (x ∨ y) + f (x ∧ y) for all vectors x, y ∈ {0, 1}n .
Submodularity checks for pairwise binary MRFs:
polynomial form (of pseudo-boolean function) has negativecoefficients on all bi-linear terms;
posiform has pairwise terms of the form uv ;
all pairwise potentials satisfy
ψPij (0, 1) + ψP
ij (1, 0) ≥ ψPij (1, 1) + ψP
ij (0, 0)
Stephen Gould | MLSS 2015 31/92
![Page 82: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/82.jpg)
Submodular Binary Pairwise MRFs
Submodularity
A pseudo-Boolean function f : {0, 1}n → R is called submodular iff (x) + f (y) ≥ f (x ∨ y) + f (x ∧ y) for all vectors x, y ∈ {0, 1}n .
Submodularity checks for pairwise binary MRFs:
polynomial form (of pseudo-boolean function) has negativecoefficients on all bi-linear terms;
posiform has pairwise terms of the form uv ;
all pairwise potentials satisfy
ψPij (0, 1) + ψP
ij (1, 0) ≥ ψPij (1, 1) + ψP
ij (0, 0)
Stephen Gould | MLSS 2015 31/92
![Page 83: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/83.jpg)
Submodularity of Binary Pairwise Terms
To see the equivalence of the last two conditions consider thefollowing pairwise potential
01
0 1
α β
γ δ
α +0 0
γ − α γ − α+
0 δ − γ
0 δ − γ+
0 β + γ − α− δ
0 0
E (y1, y2) = α+ (γ − α)y1 + (δ − γ)y2 + (β + γ − α− δ)y1y2
[Kolmogorov and Zabih, 2004]
Stephen Gould | MLSS 2015 32/92
![Page 84: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/84.jpg)
Submodularity of Binary Pairwise Terms
To see the equivalence of the last two conditions consider thefollowing pairwise potential
01
0 1
α β
γ δ
α +0 0
γ − α γ − α+
0 δ − γ
0 δ − γ+
0 β + γ − α− δ
0 0
E (y1, y2) = α+ (γ − α)y1 + (δ − γ)y2 + (β + γ − α− δ)y1y2
[Kolmogorov and Zabih, 2004]
Stephen Gould | MLSS 2015 32/92
![Page 85: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/85.jpg)
Submodularity of Binary Pairwise Terms
To see the equivalence of the last two conditions consider thefollowing pairwise potential
01
0 1
α β
γ δ
α +0 0
γ − α γ − α+
0 δ − γ
0 δ − γ+
0 β + γ − α− δ
0 0
E (y1, y2) = α+ (γ − α)y1 + (δ − γ)y2 + (β + γ − α− δ)y1y2
[Kolmogorov and Zabih, 2004]
Stephen Gould | MLSS 2015 32/92
![Page 86: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/86.jpg)
Minimum-cut Problem
Graph Cut
Let G = 〈V, E〉 be a capacitated digraph with two distinguishedvertices s and t. An st-cut is a partitioning of V into two disjointsets S and T such that s ∈ S and t ∈ T . The cost of the cut isthe sum of edge capacities for all edges going from S to T .
s
u v
t
Stephen Gould | MLSS 2015 33/92
![Page 87: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/87.jpg)
Quadratic Pseudo-boolean Optimization
Main idea:
construct a graph such that every st-cut corresponds to ajoint assignment to the variables y
the cost of the cut should be equal to the energy of theassignment, E (y; x).∗
the minimum-cut then corresponds to the the minimumenergy assignment, y⋆ = argminy E (y; x).
∗Requires non-negative edge weights.Stephen Gould | MLSS 2015 34/92
![Page 88: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/88.jpg)
Quadratic Pseudo-boolean Optimization
Main idea:
construct a graph such that every st-cut corresponds to ajoint assignment to the variables y
the cost of the cut should be equal to the energy of theassignment, E (y; x).∗
the minimum-cut then corresponds to the the minimumenergy assignment, y⋆ = argminy E (y; x).
∗Requires non-negative edge weights.Stephen Gould | MLSS 2015 34/92
![Page 89: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/89.jpg)
Quadratic Pseudo-boolean Optimization
Main idea:
construct a graph such that every st-cut corresponds to ajoint assignment to the variables y
the cost of the cut should be equal to the energy of theassignment, E (y; x).∗
the minimum-cut then corresponds to the the minimumenergy assignment, y⋆ = argminy E (y; x).
∗Requires non-negative edge weights.Stephen Gould | MLSS 2015 34/92
![Page 90: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/90.jpg)
Example st-Graph Construction for Binary MRF
E (y1, y2) = ψ1(y1) + ψ2(y2) + ψij(y1, y2)
= 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2
s
y1 y2
t
1
0
Stephen Gould | MLSS 2015 35/92
![Page 91: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/91.jpg)
Example st-Graph Construction for Binary MRF
E (y1, y2) = ψ1(y1) + ψ2(y2) + ψij(y1, y2)
= 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2
s
y1 y2
t
5
2
1
0
Stephen Gould | MLSS 2015 35/92
![Page 92: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/92.jpg)
Example st-Graph Construction for Binary MRF
E (y1, y2) = ψ1(y1) + ψ2(y2) + ψij(y1, y2)
= 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2
s
y1 y2
t
5
2
1
3
1
0
Stephen Gould | MLSS 2015 35/92
![Page 93: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/93.jpg)
Example st-Graph Construction for Binary MRF
E (y1, y2) = ψ1(y1) + ψ2(y2) + ψij(y1, y2)
= 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2
s
y1 y2
t
5
2
1
33
1
0
Stephen Gould | MLSS 2015 35/92
![Page 94: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/94.jpg)
Example st-Graph Construction for Binary MRF
E (y1, y2) = ψ1(y1) + ψ2(y2) + ψij(y1, y2)
= 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2
s
y1 y2
t
5
2
1
33
4
1
0
Stephen Gould | MLSS 2015 35/92
![Page 95: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/95.jpg)
An Example st-Cut
E (0, 1) = ψ1(0) + ψ2(1) + ψij(0, 1)
= 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2
s
y1 y2
t
5
2
1
33
4
1
0
Stephen Gould | MLSS 2015 36/92
![Page 96: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/96.jpg)
Another st-Cut
E (1, 1) = ψ1(1) + ψ2(1) + ψij(1, 1)
= 2y1 + 5y1 + 3y2 + y2 + 3y1y2 + 4y1y2
s
y1 y2
t
5
2
1
33
4
1
0
Stephen Gould | MLSS 2015 37/92
![Page 97: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/97.jpg)
Invalid st-Cut
This is not a valid cut, since it does not correspond to apartitioning of the nodes into two sets—one containing s and onecontaining t.
s
y1 y2
t
5
2
1
33
4
1
0
Stephen Gould | MLSS 2015 38/92
![Page 98: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/98.jpg)
Alternative st-Graph Construction
Sometimes you will see the roles of s and t switched.
s
y1 y2
t
5
2
1
33
4
1
0
s
y1 y2
t
2
5
3
1
4
3
0
1
These graphs represent the same energy function.
Stephen Gould | MLSS 2015 39/92
![Page 99: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/99.jpg)
Big Picture: Where are we?
We can now formulate inference in a submodular binarypairwise MRF as a minimum-cut problem.
{0, 1}n → R
How do we solve the minimum-cut problem?
Stephen Gould | MLSS 2015 40/92
![Page 100: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/100.jpg)
Max-flow/Min-cut Theorem
Max-flow/Min-cut Theorem [Fulkerson, 1956]
The maximum flow f from vertex s to vertex t is equal to theminimum cost st-cut.
s
u v
t
Stephen Gould | MLSS 2015 41/92
![Page 101: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/101.jpg)
Maximum Flow Example
s
a b
c d
t
5 3
3
5 2
1
3 5
Stephen Gould | MLSS 2015 42/92
![Page 102: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/102.jpg)
Maximum Flow Example (Augmenting Path)
s
a b
c d
t
0/5 0/30/3
0/5 0/2
0/1
0/3 0/5
flow
0
notation
u vf /c
edge with capacity c ,and current flow f .
Stephen Gould | MLSS 2015 43/92
![Page 103: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/103.jpg)
Maximum Flow Example (Augmenting Path)
s
a b
c d
t
0/5 0/30/3
0/5 0/2
0/1
0/3 0/5
flow
0
notation
u vf /c
edge with capacity c ,and current flow f .
Stephen Gould | MLSS 2015 43/92
![Page 104: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/104.jpg)
Maximum Flow Example (Augmenting Path)
s
a b
c d
t
3/5 0/30/3
3/5 0/2
0/1
3/3 0/5
flow
3
notation
u vf /c
edge with capacity c ,and current flow f .
Stephen Gould | MLSS 2015 43/92
![Page 105: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/105.jpg)
Maximum Flow Example (Augmenting Path)
s
a b
c d
t
3/5 0/30/3
3/5 0/2
0/1
3/3 0/5
flow
3
notation
u vf /c
edge with capacity c ,and current flow f .
Stephen Gould | MLSS 2015 43/92
![Page 106: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/106.jpg)
Maximum Flow Example (Augmenting Path)
s
a b
c d
t
5/5 0/32/3
3/5 2/2
0/1
3/3 2/5
flow
5
notation
u vf /c
edge with capacity c ,and current flow f .
Stephen Gould | MLSS 2015 43/92
![Page 107: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/107.jpg)
Maximum Flow Example (Augmenting Path)
s
a b
c d
t
5/5 0/32/3
3/5 2/20/1
3/3 2/5
flow
5
notation
u vf /c
edge with capacity c ,and current flow f .
Stephen Gould | MLSS 2015 43/92
![Page 108: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/108.jpg)
Maximum Flow Example (Augmenting Path)
s
a b
c d
t
5/5 1/31/3
4/5 2/2
1/1
3/3 3/5
flow
6
notation
u vf /c
edge with capacity c ,and current flow f .
Stephen Gould | MLSS 2015 43/92
![Page 109: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/109.jpg)
Maximum Flow Example (Augmenting Path)
s
a b
c d
t
5/5 1/31/3
4/5 2/2
1/1
3/3 3/5
flow
6
notation
u vf /c
edge with capacity c ,and current flow f .
Stephen Gould | MLSS 2015 44/92
![Page 110: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/110.jpg)
Augmenting Path Algorithm Summary
while an augmenting path exists (directed path with positivecapacity between the source and sink)
send flow along the augmenting path updating edgecapacities to produce a residual graph
put all nodes reachable from the source in S
put all nodes that can reach the sink in T
Stephen Gould | MLSS 2015 45/92
![Page 111: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/111.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
0/5 0/30/3
0/5 0/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 0 0b 0 0c 0 0d 0 0t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 112: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/112.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
0/5 0/30/3
0/5 0/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 0 0b 0 0c 0 0d 0 0t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 113: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/113.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
0/5 0/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 0 5b 0 3c 0 0d 0 0t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 114: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/114.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
0/5 0/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 1 5b 0 3c 0 0d 0 0t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 115: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/115.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
0/5 0/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 1 5b 0 3c 0 0d 0 0t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 116: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/116.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 0/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 0 6c 0 2d 0 0t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 117: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/117.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 0/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 1 6c 0 2d 0 0t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 118: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/118.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 0/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 1 6c 0 2d 0 0t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 119: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/119.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 1 4c 0 2d 0 2t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 120: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/120.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/2
0/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 1 4c 1 2d 0 2t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 121: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/121.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/20/1
0/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 1 4c 1 2d 0 2t 0 0
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 122: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/122.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/2
1/1
1/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 1 4c 1 0d 0 3t 0 1
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 123: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/123.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/2
1/1
1/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 1 4c 1 0d 1 3t 0 1
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 124: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/124.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/2
1/1
1/3 0/5
state
h(·) e(·)s 6 ∞a 1 0b 1 4c 1 0d 1 3t 0 1
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 125: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/125.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/2
1/1
1/3 3/5
state
h(·) e(·)s 6 ∞a 1 0b 1 4c 1 0d 1 0t 0 4
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 126: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/126.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/2
1/1
1/3 3/5
state
h(·) e(·)s 6 ∞a 1 0b 2 4c 1 0d 1 0t 0 4
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 127: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/127.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/33/3
2/5 2/2
1/1
1/3 3/5
state
h(·) e(·)s 6 ∞a 1 0b 2 4c 1 0d 1 0t 0 4
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 128: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/128.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
2/5 2/2
1/1
1/3 3/5
state
h(·) e(·)s 6 ∞a 1 3b 2 1c 1 0d 1 0t 0 4
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 129: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/129.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
2/5 2/2
1/1
1/3 3/5
state
h(·) e(·)s 6 ∞a 2 3b 2 1c 1 0d 1 0t 0 4
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 130: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/130.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
2/5 2/2
1/1
1/3 3/5
state
h(·) e(·)s 6 ∞a 2 3b 2 1c 1 0d 1 0t 0 4
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 131: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/131.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
5/5 2/2
1/1
1/3 3/5
state
h(·) e(·)s 6 ∞a 2 0b 2 1c 1 3d 1 0t 0 4
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 132: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/132.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
5/5 2/2
1/1
1/3 3/5
state
h(·) e(·)s 6 ∞a 2 0b 2 1c 1 3d 1 0t 0 4
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 133: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/133.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 2 0b 2 1c 1 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 134: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/134.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 2 0b 7 1c 1 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 135: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/135.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 3/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 2 0b 7 1c 1 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 136: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/136.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 2 0b 7 0c 1 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 137: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/137.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 2 0b 7 0c 3 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 138: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/138.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 2 0b 7 0c 3 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 139: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/139.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 2 1b 7 0c 3 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 140: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/140.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 4 1b 7 0c 3 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 141: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/141.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 4 1b 7 0c 3 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 142: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/142.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 4 0b 7 0c 3 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 143: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/143.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 4 0b 7 0c 5 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 144: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/144.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 4 0b 7 0c 5 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 145: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/145.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 4 1b 7 0c 5 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 146: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/146.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 6 1b 7 0c 5 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 147: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/147.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 6 1b 7 0c 5 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 148: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/148.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 6 0b 7 0c 5 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 149: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/149.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 6 0b 7 0c 7 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 150: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/150.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
5/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 6 0b 7 0c 7 1d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 151: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/151.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 6 1b 7 0c 7 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 152: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/152.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 7 1b 7 0c 7 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 153: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/153.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
5/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 7 1b 7 0c 7 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 154: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/154.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
4/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 7 0b 7 0c 7 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 46/92
![Page 155: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/155.jpg)
Maximum Flow Example (Push-Relabel)
s
a b
c d
t
4/5 2/30/3
4/5 2/2
1/1
3/3 3/5
state
h(·) e(·)s 6 ∞a 7 0b 7 0c 7 0d 1 0t 0 6
notation
u vf /c
edge with capacity c ,current flow f .
Stephen Gould | MLSS 2015 47/92
![Page 156: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/156.jpg)
Push-Relabel Algorithm Summary
Initialize: set height of s to number of nodes in the graph;set excess for all nodes to zero.
Push: for a node with excess capacity, push as much flow aspossible onto neighbours with lower height
Relabel: for a node with excess capacity and no neighbourswith lower height, increase its height to one more than itslowest neighbour (with residual capacity).
Stephen Gould | MLSS 2015 48/92
![Page 157: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/157.jpg)
Comparison of Maximum Flow Algorithms
Current state-of-the-art algorithm for exact minimization of generalsubmodular pseudo-Boolean functions is O(n5T + n6), where T isthe time taken to evaluate the function [Orlin, 2009].
†assumes integer capacitiesStephen Gould | MLSS 2015 49/92
![Page 158: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/158.jpg)
Comparison of Maximum Flow Algorithms
Current state-of-the-art algorithm for exact minimization of generalsubmodular pseudo-Boolean functions is O(n5T + n6), where T isthe time taken to evaluate the function [Orlin, 2009].
Algorithm Complexity
Ford-Fulkerson O(E max f )†
Edmonds-Karp (BFS) O(VE 2)
Push-relabel O(V 3)
Boykov-Kolmogorov O(V 2E max f )(∼ O(V ) in practice)
†assumes integer capacitiesStephen Gould | MLSS 2015 49/92
![Page 159: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/159.jpg)
Maximum Flow (Boykov-Kolmogorov, PAMI 2004)
Stephen Gould | MLSS 2015 50/92
![Page 160: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/160.jpg)
Maximum Flow (Boykov-Kolmogorov, PAMI 2004)
growth stage
search trees from s
and t grow untilthey touch
Stephen Gould | MLSS 2015 51/92
![Page 161: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/161.jpg)
Maximum Flow (Boykov-Kolmogorov, PAMI 2004)
growth stage
search trees from s
and t grow untilthey touch
augmentation stage
the path found isaugmented
Stephen Gould | MLSS 2015 52/92
![Page 162: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/162.jpg)
Maximum Flow (Boykov-Kolmogorov, PAMI 2004)
growth stage
search trees from s
and t grow untilthey touch
augmentation stage
the path found isaugmented; treesbreak into forests
Stephen Gould | MLSS 2015 53/92
![Page 163: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/163.jpg)
Maximum Flow (Boykov-Kolmogorov, PAMI 2004)
growth stage
search trees from s
and t grow untilthey touch
augmentation stage
the path found isaugmented; treesbreak into forests
adoption stage
trees are restored
Stephen Gould | MLSS 2015 54/92
![Page 164: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/164.jpg)
Reparameterization of Energy Functions
E (y1, y2) = 2y1+5y1+3y2+y2
+ 3y1y2 + 4y1y2
s
y1 y2
t
5
2
1
33
4
1
0
E (y1, y2) = 6y1+5y2+7y1y2
s
y1 y2
t
6
5
7
1
0
Stephen Gould | MLSS 2015 55/92
![Page 165: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/165.jpg)
Big Picture: Where are we now?
We can perform inference in submodular binary pairwiseMarkov random fields exactly.
{0, 1}n → R
What about...
non-submodular binary pairwise Markov random fields?
multi-label Markov random fields?
higher-order Markov random fields?
Stephen Gould | MLSS 2015 56/92
![Page 166: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/166.jpg)
Big Picture: Where are we now?
We can perform inference in submodular binary pairwiseMarkov random fields exactly.
{0, 1}n → R
What about...
non-submodular binary pairwise Markov random fields?
multi-label Markov random fields?
higher-order Markov random fields?
Stephen Gould | MLSS 2015 56/92
![Page 167: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/167.jpg)
Non-submodular Binary Pairwise MRFs
Non-submodular binary pairwise MRFs have potentials that do notsatisfy ψP
ij (0, 1) + ψPij (1, 0) ≥ ψ
Pij (1, 1) + ψP
ij (0, 0).
They are often handled in one of the following ways:
approximate the energy function by one that is submodular(i.e., project onto the space of submodular functions);
solve a relaxation of the problem using QPBO (Rother et al.,2007) or dual-decomposition (Komodakis et al., 2007).
Stephen Gould | MLSS 2015 57/92
![Page 168: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/168.jpg)
Non-submodular Binary Pairwise MRFs
Non-submodular binary pairwise MRFs have potentials that do notsatisfy ψP
ij (0, 1) + ψPij (1, 0) ≥ ψ
Pij (1, 1) + ψP
ij (0, 0).
They are often handled in one of the following ways:
approximate the energy function by one that is submodular(i.e., project onto the space of submodular functions);
solve a relaxation of the problem using QPBO (Rother et al.,2007) or dual-decomposition (Komodakis et al., 2007).
Stephen Gould | MLSS 2015 57/92
![Page 169: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/169.jpg)
Approximating Non-submodular Binary Pairwise MRFs
Consider the non-submodular potentialA B
C Dwith
A+ D > B + C .
We can project onto a submodular potential by modifying thecoefficients as follows:
∆ = A+ D − C − B
A← A−∆
3
C ← C +∆
3
B ← B +∆
3
Stephen Gould | MLSS 2015 58/92
![Page 170: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/170.jpg)
QPBO (Roof Duality) [Rother et al., 2007]
Consider the energy function
E (y) =∑
i∈V
ψUi (yi ) +
∑
ij∈E
ψPij (yi , yj )
︸ ︷︷ ︸
submodular
+∑
ij∈E
ψPij (yi , yj)
︸ ︷︷ ︸
non-submodular
We can introduce duplicate variables yi into the energy function,and write
E ′(y, y) =∑
i∈V
ψUi (yi) + ψU
i (1− yi )
2
+∑
ij∈E
ψPij (yi , yj) + ψP
ij (1− yi , 1− yj)
2
+∑
ij∈E
ψPij (yi , 1− yj) + ψP
ij (1− yi , yj)
2
Stephen Gould | MLSS 2015 59/92
![Page 171: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/171.jpg)
QPBO (Roof Duality) [Rother et al., 2007]
Consider the energy function
E (y) =∑
i∈V
ψUi (yi ) +
∑
ij∈E
ψPij (yi , yj )
︸ ︷︷ ︸
submodular
+∑
ij∈E
ψPij (yi , yj)
︸ ︷︷ ︸
non-submodular
We can introduce duplicate variables yi into the energy function,and write
E ′(y, y) =∑
i∈V
ψUi (yi) + ψU
i (1− yi )
2
+∑
ij∈E
ψPij (yi , yj) + ψP
ij (1− yi , 1− yj)
2
+∑
ij∈E
ψPij (yi , 1− yj) + ψP
ij (1− yi , yj)
2
Stephen Gould | MLSS 2015 59/92
![Page 172: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/172.jpg)
QPBO (Roof Duality)
E ′(y, y) =∑
i∈V
12ψUi (yi ) +
12ψUi (1− yi)
+∑
ij∈E
12ψPij (yi , yj) +
12ψPij (1− yi , 1− yj)
+∑
ij∈E
12ψPij (yi , 1− yj) + 1
2ψPij (1− yi , yj)
Observations
if yi = 1− yi for all i , then E (y) = E ′(y, y).E ′(y, y) is submodular.
Ignore the constraint on yi and solve anyway. Result satisfiespartial optimality: if yi = 1− yi then yi is the optimal label.
Stephen Gould | MLSS 2015 60/92
![Page 173: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/173.jpg)
QPBO (Roof Duality)
E ′(y, y) =∑
i∈V
12ψUi (yi ) +
12ψUi (1− yi)
+∑
ij∈E
12ψPij (yi , yj) +
12ψPij (1− yi , 1− yj)
+∑
ij∈E
12ψPij (yi , 1− yj) + 1
2ψPij (1− yi , yj)
Observations
if yi = 1− yi for all i , then E (y) = E ′(y, y).E ′(y, y) is submodular.
Ignore the constraint on yi and solve anyway. Result satisfiespartial optimality: if yi = 1− yi then yi is the optimal label.
Stephen Gould | MLSS 2015 60/92
![Page 174: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/174.jpg)
Multi-label Markov Random Fields
The quadratic pseudo-Boolean optimization techniques describedabove cannot be applied directly to multi-label MRFs.
However...
...for certain MRFs we can transform the multi-label probleminto a binary one exactly.
...we can project the multi-label problem onto a series ofbinary problems in a so-called move-making algorithm.
Stephen Gould | MLSS 2015 61/92
![Page 175: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/175.jpg)
Multi-label Markov Random Fields
The quadratic pseudo-Boolean optimization techniques describedabove cannot be applied directly to multi-label MRFs.
However...
...for certain MRFs we can transform the multi-label probleminto a binary one exactly.
...we can project the multi-label problem onto a series ofbinary problems in a so-called move-making algorithm.
Stephen Gould | MLSS 2015 61/92
![Page 176: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/176.jpg)
The “Battleship” Transform [Ishikawa, 2003]
If the multi-label MRFs has pairwise potentials that are convexfunctions over the label differences, i.e., ψP
ij (yi , yj ) = g(|yi − yj |)where g(·) is convex, then we can transform the energy functioninto an equivalent binary one.
y = 1⇔ z = (0, 0, 0)
y = 2⇔ z = (1, 0, 0)
y = 3⇔ z = (1, 1, 0)
y = 4⇔ z = (1, 1, 1)
s
1 1
2 2
3 3
t
∞
∞
∞
∞
Stephen Gould | MLSS 2015 62/92
![Page 177: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/177.jpg)
Move-making Inference
Idea:
initialize yprev to any valid assignment
restrict the label-space of each variable yi from L to Yi ⊆ L(with y
prev
i ∈ Yi)
transform E : Ln → R to E : Y1 × · · · × Yn → R
find the optimal assignment y for E and repeat
each move results in an assignment with lower energy
Stephen Gould | MLSS 2015 63/92
![Page 178: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/178.jpg)
Iterated Conditional Modes [Besag, 1986]
Reduce multi-variate inference to solving a series ofunivariate inference problems.
ICM move
For one of the variables yi , set Yi = L. Set Yj = {yprev
j } for allj 6= i (i.e., hold all other variables fixed).
can be used for arbitrary energy functions
Stephen Gould | MLSS 2015 64/92
![Page 179: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/179.jpg)
Iterated Conditional Modes [Besag, 1986]
Reduce multi-variate inference to solving a series ofunivariate inference problems.
ICM move
For one of the variables yi , set Yi = L. Set Yj = {yprev
j } for allj 6= i (i.e., hold all other variables fixed).
can be used for arbitrary energy functions
Stephen Gould | MLSS 2015 64/92
![Page 180: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/180.jpg)
Alpha Expansion and Alpha-Beta Swap [Boykov et al., 2001]
Reduce multi-label inference to solving a series of binary(submodular) inference problems.
α-expansion move
Choose some α ∈ L. Then for all variables, set Yi = {α, yprev
i }.
ψPij (·, ·) must be metric for the resulting move to be submodular
αβ-swap move
Choose two labels α, β ∈ L. Then for each variable yi such thatyprev
i ∈ {α, β}, set Yi = {α, β}. Otherwise set Yi = {yprev
i }.
ψPij (·, ·) must be semi-metric
Stephen Gould | MLSS 2015 65/92
![Page 181: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/181.jpg)
Alpha Expansion Potential Construction
ynexti =
{
yprevi if ti = 1
α if ti = 0
E (t) =∑
i
ψi (α)ti + ψi (yprevi )ti +
∑
ij
ψij(α,α)ti tj
+ ψij(α, yprevj )ti tj + ψij(y
previ , α)ti tj + ψij (y
previ , yprevj )ti tj
Stephen Gould | MLSS 2015 66/92
![Page 182: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/182.jpg)
Alpha Expansion Potential Construction
ynexti =
{
yprevi if ti = 1
α if ti = 0
E (t) =∑
i
ψi (α)ti + ψi (yprevi )ti +
∑
ij
ψij(α,α)ti tj
+ ψij(α, yprevj )ti tj + ψij(y
previ , α)ti tj + ψij (y
previ , yprevj )ti tj
Stephen Gould | MLSS 2015 66/92
![Page 183: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/183.jpg)
A Note on Higher-Order Models
Order reduction. [Ishikawa, 2009]
Replace −∏n
i=1 yi with z +n∑
i=1
yiz
︸ ︷︷ ︸
*
− 1.
Special forms. E.g., lower-linear envelopes [Gould, 2011]
ψHc (yc) , min
k
{
ak∑
i∈c
yi + bk
}
= mink{fk(yc)}
Assume sorted on ak . Then replace above with
f1(yc) +∑
k
zk (fk+1(yc)− fk(yc))
︸ ︷︷ ︸
* submodular binary pairwise
Stephen Gould | MLSS 2015 67/92
![Page 184: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/184.jpg)
A Note on Higher-Order Models
Order reduction. [Ishikawa, 2009]
Replace −∏n
i=1 yi with z +n∑
i=1
yiz
︸ ︷︷ ︸
*
− 1.
Special forms. E.g., lower-linear envelopes [Gould, 2011]
ψHc (yc) , min
k
{
ak∑
i∈c
yi + bk
}
= mink{fk(yc)}
Assume sorted on ak . Then replace above with
f1(yc) +∑
k
zk (fk+1(yc)− fk(yc))
︸ ︷︷ ︸
* submodular binary pairwise
Stephen Gould | MLSS 2015 67/92
![Page 185: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/185.jpg)
A Note on Higher-Order Models
Order reduction. [Ishikawa, 2009]
Replace −∏n
i=1 yi with z +n∑
i=1
yiz
︸ ︷︷ ︸
*
− 1.
Special forms. E.g., lower-linear envelopes [Gould, 2011]
ψHc (yc) , min
k
{
ak∑
i∈c
yi + bk
}
= mink{fk(yc)}
Assume sorted on ak . Then replace above with
f1(yc) +∑
k
zk (fk+1(yc)− fk(yc))
︸ ︷︷ ︸
* submodular binary pairwise
Stephen Gould | MLSS 2015 67/92
![Page 186: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/186.jpg)
relaxations and dual decomposition
Stephen Gould | MLSS 2015 68/92
![Page 187: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/187.jpg)
Mathematical Programming Formulation
Let θc,yc , ψc (yc) and let µc,yc ,
{
1, if Yc = yc
0, otherwise
argminy∈Y
∑
c
ψc(yc)
m
minimize (over µ) θTµ
subject to µc,yc ∈ {0, 1}, ∀c , yc ∈ Yc∑
ycµc,yc = 1, ∀c
∑
yc\yiµc,yc = µi ,yi , ∀i ∈ c , yi ∈ Yi
Stephen Gould | MLSS 2015 69/92
![Page 188: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/188.jpg)
Mathematical Programming Formulation
Let θc,yc , ψc (yc) and let µc,yc ,
{
1, if Yc = yc
0, otherwise
argminy∈Y
∑
c
ψc(yc)
m
minimize (over µ) θTµ
subject to µc,yc ∈ {0, 1}, ∀c , yc ∈ Yc∑
ycµc,yc = 1, ∀c
∑
yc\yiµc,yc = µi ,yi , ∀i ∈ c , yi ∈ Yi
Stephen Gould | MLSS 2015 69/92
![Page 189: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/189.jpg)
Binary Integer Program: Example
Consider energy function E (y1, y2) = ψ1(y1) +ψ12(y1, y2) + ψ2(y2)for binary variables y1 and y2.
Y1 Y1,Y2 Y2
θ =
ψ1(0)ψ1(1)ψ2(0)ψ2(1)
ψ12(0, 0)ψ12(1, 0)ψ12(0, 1)ψ12(1, 1)
µ =
µ1,0µ1,1µ2,0µ2,1µ12,00µ12,10µ12,01µ12,11
s.t.
µ1,0 + µ1,1 = 1µ2,0 + µ2,1 = 1
µ12,00 + µ12,10+ µ12,01 + µ12,11 = 1µ12,00 + µ12,01 = µ1,0µ12,10 + µ12,11 = µ1,1µ12,00 + µ12,10 = µ2,0µ12,01 + µ12,11 = µ2,1
Stephen Gould | MLSS 2015 70/92
![Page 190: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/190.jpg)
Binary Integer Program: Example
Consider energy function E (y1, y2) = ψ1(y1) +ψ12(y1, y2) + ψ2(y2)for binary variables y1 and y2.
Y1 Y1,Y2 Y2
θ =
ψ1(0)ψ1(1)ψ2(0)ψ2(1)
ψ12(0, 0)ψ12(1, 0)ψ12(0, 1)ψ12(1, 1)
µ =
µ1,0µ1,1µ2,0µ2,1µ12,00µ12,10µ12,01µ12,11
s.t.
µ1,0 + µ1,1 = 1µ2,0 + µ2,1 = 1
µ12,00 + µ12,10+ µ12,01 + µ12,11 = 1µ12,00 + µ12,01 = µ1,0µ12,10 + µ12,11 = µ1,1µ12,00 + µ12,10 = µ2,0µ12,01 + µ12,11 = µ2,1
Stephen Gould | MLSS 2015 70/92
![Page 191: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/191.jpg)
Binary Integer Program: Example
Consider energy function E (y1, y2) = ψ1(y1) +ψ12(y1, y2) + ψ2(y2)for binary variables y1 and y2.
Y1 Y1,Y2 Y2
θ =
ψ1(0)ψ1(1)ψ2(0)ψ2(1)
ψ12(0, 0)ψ12(1, 0)ψ12(0, 1)ψ12(1, 1)
µ =
µ1,0µ1,1µ2,0µ2,1µ12,00µ12,10µ12,01µ12,11
s.t.
µ1,0 + µ1,1 = 1µ2,0 + µ2,1 = 1
µ12,00 + µ12,10+ µ12,01 + µ12,11 = 1µ12,00 + µ12,01 = µ1,0µ12,10 + µ12,11 = µ1,1µ12,00 + µ12,10 = µ2,0µ12,01 + µ12,11 = µ2,1
Stephen Gould | MLSS 2015 70/92
![Page 192: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/192.jpg)
Binary Integer Program: Example
Let y1 = 1 and y2 = 0. Then
µ =
µ1,0µ1,1µ2,0µ2,1µ12,00µ12,10µ12,01µ12,11
=
01100100
· θ =
ψ1(0)ψ1(1)ψ2(0)ψ2(1)
ψ12(0, 0)ψ12(1, 0)ψ12(0, 1)ψ12(1, 1)
So θTµ = ψ1(1) + ψ2(0) + ψ12(1, 0).
Stephen Gould | MLSS 2015 71/92
![Page 193: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/193.jpg)
Local Marginal Polytope
M =
{
µ ≥ 0
∣∣∣∣∣
∑
yiµi ,yi = 1, ∀i
∑
yc\yiµc,yc = µi ,yi , ∀i ∈ c , yi ∈ Yi
}
M is tight if factor graph is a tree
for cyclic graphsM may contain fractional vertices
for submodular energies, factional solutions are never optimal
Stephen Gould | MLSS 2015 72/92
![Page 194: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/194.jpg)
Local Marginal Polytope
M =
{
µ ≥ 0
∣∣∣∣∣
∑
yiµi ,yi = 1, ∀i
∑
yc\yiµc,yc = µi ,yi , ∀i ∈ c , yi ∈ Yi
}
M is tight if factor graph is a tree
for cyclic graphsM may contain fractional vertices
for submodular energies, factional solutions are never optimal
Stephen Gould | MLSS 2015 72/92
![Page 195: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/195.jpg)
Linear Programming (LP) Relaxation
Binary integer program
minimize (over µ) θTµ
subject to µc,yc ∈ {0, 1}µ ∈ M
Linear program
minimize (over µ) θTµ
subject to µc,yc ∈ [0, 1]µ ∈ M
Solution by standard LP solvers typically infeasible due tolarge number of variables and constraints
More easily solved via coordinate ascent of the dual
Solutions need to be rounded or decodedStephen Gould | MLSS 2015 73/92
![Page 196: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/196.jpg)
Linear Programming (LP) Relaxation
Binary integer program
minimize (over µ) θTµ
subject to µc,yc ∈ {0, 1}µ ∈ M
Linear program
minimize (over µ) θTµ
subject to µc,yc ∈ [0, 1]µ ∈ M
Solution by standard LP solvers typically infeasible due tolarge number of variables and constraints
More easily solved via coordinate ascent of the dual
Solutions need to be rounded or decodedStephen Gould | MLSS 2015 73/92
![Page 197: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/197.jpg)
Linear Programming (LP) Relaxation
Binary integer program
minimize (over µ) θTµ
subject to µc,yc ∈ {0, 1}µ ∈ M
Linear program
minimize (over µ) θTµ
subject to µc,yc ∈ [0, 1]µ ∈ M
Solution by standard LP solvers typically infeasible due tolarge number of variables and constraints
More easily solved via coordinate ascent of the dual
Solutions need to be rounded or decodedStephen Gould | MLSS 2015 73/92
![Page 198: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/198.jpg)
Linear Programming (LP) Relaxation
Binary integer program
minimize (over µ) θTµ
subject to µc,yc ∈ {0, 1}µ ∈ M
Linear program
minimize (over µ) θTµ
subject to µc,yc ∈ [0, 1]µ ∈ M
Solution by standard LP solvers typically infeasible due tolarge number of variables and constraints
More easily solved via coordinate ascent of the dual
Solutions need to be rounded or decodedStephen Gould | MLSS 2015 73/92
![Page 199: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/199.jpg)
Linear Programming (LP) Relaxation
Binary integer program
minimize (over µ) θTµ
subject to µc,yc ∈ {0, 1}µ ∈ M
Linear program
minimize (over µ) θTµ
subject to µc,yc ∈ [0, 1]µ ∈ M
Solution by standard LP solvers typically infeasible due tolarge number of variables and constraints
More easily solved via coordinate ascent of the dual
Solutions need to be rounded or decodedStephen Gould | MLSS 2015 73/92
![Page 200: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/200.jpg)
Dual Decomposition: Rewriting the Primal
minimize (over µ)∑
c θTc µc
subject to µ ∈ M
m (pad θc)
minimize (over µ)∑
c θT
c µ
subject to µ ∈M
m (introduce copies of µ)
minimize (over µ, {µc})∑
c θT
c µc
subject to µc = µ
µ ∈ M
Stephen Gould | MLSS 2015 74/92
![Page 201: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/201.jpg)
Dual Decomposition: Rewriting the Primal
minimize (over µ)∑
c θTc µc
subject to µ ∈ M
m (pad θc)
minimize (over µ)∑
c θT
c µ
subject to µ ∈M
m (introduce copies of µ)
minimize (over µ, {µc})∑
c θT
c µc
subject to µc = µ
µ ∈ M
Stephen Gould | MLSS 2015 74/92
![Page 202: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/202.jpg)
Dual Decomposition: Rewriting the Primal
minimize (over µ)∑
c θTc µc
subject to µ ∈ M
m (pad θc)
minimize (over µ)∑
c θT
c µ
subject to µ ∈M
m (introduce copies of µ)
minimize (over µ, {µc})∑
c θT
c µc
subject to µc = µ
µ ∈ M
Stephen Gould | MLSS 2015 74/92
![Page 203: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/203.jpg)
Dual Decomposition: Forming the Dual
Primal problem
minimize (over µ, {µc})∑
c θT
c µc
subject to µc = µ
µ ∈ M
Introducing dual variables λc we have Lagrangian
L(µ, {µc}, {λc}) =∑
c
θT
c µc +
∑
c
λTc (µc − µ)
=∑
c
(θc + λc)Tµc −
∑
c
λTc µ
Stephen Gould | MLSS 2015 75/92
![Page 204: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/204.jpg)
Dual Decomposition: Forming the Dual
Primal problem
minimize (over µ, {µc})∑
c θT
c µc
subject to µc = µ
µ ∈ M
Introducing dual variables λc we have Lagrangian
L(µ, {µc}, {λc}) =∑
c
θT
c µc +
∑
c
λTc (µc − µ)
=∑
c
(θc + λc)Tµc −
∑
c
λTc µ
Stephen Gould | MLSS 2015 75/92
![Page 205: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/205.jpg)
Dual Decomposition: Forming the Dual
Primal problem
minimize (over µ, {µc})∑
c θT
c µc
subject to µc = µ
µ ∈ M
Introducing dual variables λc we have Lagrangian
L(µ, {µc}, {λc}) =∑
c
θT
c µc +
∑
c
λTc (µc − µ)
=∑
c
(θc + λc)Tµc −
∑
c
λTc µ
Stephen Gould | MLSS 2015 75/92
![Page 206: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/206.jpg)
Dual Decomposition
maximize{λc}
min{µc}
∑
c
(θc + λc)Tµc
subject to∑
c λc = 0
m
maximize{λc}
∑
c
minµc
(θc + λc)Tµc
subject to∑
c λc = 0
m
maximize{λc}
∑
c
minyc
ψc(yc) + λc(yc)
subject to∑
c λc = 0
Stephen Gould | MLSS 2015 76/92
![Page 207: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/207.jpg)
Dual Decomposition
maximize{λc}
min{µc}
∑
c
(θc + λc)Tµc
subject to∑
c λc = 0
m
maximize{λc}
∑
c
minµc
(θc + λc)Tµc
subject to∑
c λc = 0
m
maximize{λc}
∑
c
minyc
ψc(yc) + λc(yc)
subject to∑
c λc = 0
Stephen Gould | MLSS 2015 76/92
![Page 208: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/208.jpg)
Dual Decomposition
maximize{λc}
min{µc}
∑
c
(θc + λc)Tµc
subject to∑
c λc = 0
m
maximize{λc}
∑
c
minµc
(θc + λc)Tµc
subject to∑
c λc = 0
m
maximize{λc}
∑
c
minyc
ψc(yc) + λc(yc)
subject to∑
c λc = 0
Stephen Gould | MLSS 2015 76/92
![Page 209: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/209.jpg)
Dual Lower Bound
E (y) =∑
c
ψc (yc)
=∑
c
ψc (yc) + λc(yc)
(
iff∑
c
λc(yc) = 0
)
miny
E (y) ≥∑
c
minyc
ψc(yc) + λc(yc)
miny
E (y) ≥ max{λc}:
∑
c λc=0
∑
c
minyc
ψc(yc) + λc(yc)
Stephen Gould | MLSS 2015 77/92
![Page 210: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/210.jpg)
Dual Lower Bound
E (y) =∑
c
ψc (yc)
=∑
c
ψc (yc) + λc(yc)
(
iff∑
c
λc(yc) = 0
)
miny
E (y) ≥∑
c
minyc
ψc(yc) + λc(yc)
miny
E (y) ≥ max{λc}:
∑
c λc=0
∑
c
minyc
ψc(yc) + λc(yc)
Stephen Gould | MLSS 2015 77/92
![Page 211: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/211.jpg)
Dual Lower Bound
E (y) =∑
c
ψc (yc)
=∑
c
ψc (yc) + λc(yc)
(
iff∑
c
λc(yc) = 0
)
miny
E (y) ≥∑
c
minyc
ψc(yc) + λc(yc)
miny
E (y) ≥ max{λc}:
∑
c λc=0
∑
c
minyc
ψc(yc) + λc(yc)
Stephen Gould | MLSS 2015 77/92
![Page 212: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/212.jpg)
Subgradients
Subgradient
A subgradient of a function f at x is any vector g satisfying
f (y) ≥ f (x) + gT (y − x) for all y
Stephen Gould | MLSS 2015 78/92
![Page 213: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/213.jpg)
Subgradient Method
The basic subgradient method is a algorithm for minimizing anondifferentiable convex function f : Rn → R.
x(k+1) = x(k) − αkg(k)
x(k) is the k-th iterate
g (k) is any subgradient of f at x(k)
αk > 0 is the k-th step size
It is possible that −g (k) is not a descent direction for f at x(k), sowe keep track of the best point found so far
f(k)best = min
{
f(k−1)best , f (x(k))
}
Stephen Gould | MLSS 2015 79/92
![Page 214: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/214.jpg)
Subgradient Method
The basic subgradient method is a algorithm for minimizing anondifferentiable convex function f : Rn → R.
x(k+1) = x(k) − αkg(k)
x(k) is the k-th iterate
g (k) is any subgradient of f at x(k)
αk > 0 is the k-th step size
It is possible that −g (k) is not a descent direction for f at x(k), sowe keep track of the best point found so far
f(k)best = min
{
f(k−1)best , f (x(k))
}
Stephen Gould | MLSS 2015 79/92
![Page 215: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/215.jpg)
Step Size Rules
Step sizes are chosen ahead of time (unlike line search is ordinarygradient methods). A few common step size schedules are:
constant step size: αk = α
constant step length: αk = γ
‖g (k)‖2
square summable but not summable:∑∞
k=1 α2k <∞,
∑∞k=1 αk =∞
nonsummable diminishing:
limk→∞
αk = 0,∑∞
k=1 αk =∞
nonsummable diminishing step lengths: αk = γk‖g (k)‖2
limk→∞
γk = 0,∑∞
k=1 γk =∞
Stephen Gould | MLSS 2015 80/92
![Page 216: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/216.jpg)
Step Size Rules
Step sizes are chosen ahead of time (unlike line search is ordinarygradient methods). A few common step size schedules are:
constant step size: αk = α
constant step length: αk = γ
‖g (k)‖2
square summable but not summable:∑∞
k=1 α2k <∞,
∑∞k=1 αk =∞
nonsummable diminishing:
limk→∞
αk = 0,∑∞
k=1 αk =∞
nonsummable diminishing step lengths: αk = γk‖g (k)‖2
limk→∞
γk = 0,∑∞
k=1 γk =∞
Stephen Gould | MLSS 2015 80/92
![Page 217: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/217.jpg)
Step Size Rules
Step sizes are chosen ahead of time (unlike line search is ordinarygradient methods). A few common step size schedules are:
constant step size: αk = α
constant step length: αk = γ
‖g (k)‖2
square summable but not summable:∑∞
k=1 α2k <∞,
∑∞k=1 αk =∞
nonsummable diminishing:
limk→∞
αk = 0,∑∞
k=1 αk =∞
nonsummable diminishing step lengths: αk = γk‖g (k)‖2
limk→∞
γk = 0,∑∞
k=1 γk =∞
Stephen Gould | MLSS 2015 80/92
![Page 218: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/218.jpg)
Step Size Rules
Step sizes are chosen ahead of time (unlike line search is ordinarygradient methods). A few common step size schedules are:
constant step size: αk = α
constant step length: αk = γ
‖g (k)‖2
square summable but not summable:∑∞
k=1 α2k <∞,
∑∞k=1 αk =∞
nonsummable diminishing:
limk→∞
αk = 0,∑∞
k=1 αk =∞
nonsummable diminishing step lengths: αk = γk‖g (k)‖2
limk→∞
γk = 0,∑∞
k=1 γk =∞
Stephen Gould | MLSS 2015 80/92
![Page 219: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/219.jpg)
Step Size Rules
Step sizes are chosen ahead of time (unlike line search is ordinarygradient methods). A few common step size schedules are:
constant step size: αk = α
constant step length: αk = γ
‖g (k)‖2
square summable but not summable:∑∞
k=1 α2k <∞,
∑∞k=1 αk =∞
nonsummable diminishing:
limk→∞
αk = 0,∑∞
k=1 αk =∞
nonsummable diminishing step lengths: αk = γk‖g (k)‖2
limk→∞
γk = 0,∑∞
k=1 γk =∞
Stephen Gould | MLSS 2015 80/92
![Page 220: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/220.jpg)
Convergence Results
For constant step size and constant step length, the subgradientalgorithm will converge to within some range of the optimal value,
limk→∞
f(k)best < f ⋆ + ǫ
For the diminishing step size and step length rules the algorithmconverges to the optimal value,
limk→∞
f(k)best = f ⋆
but may take a very long time to converge.
Stephen Gould | MLSS 2015 81/92
![Page 221: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/221.jpg)
Optimal Step Size for Known f⋆
Assume we know f ⋆ (we just don’t know x⋆). Then
αk =f (x(k))− f ⋆
‖g (k)‖22
is an optimal step size in some sense. Called the Polyak step size.
A good approximation when f ⋆ is not known (but non-negative) is
αk =f (x(k))− γ · f
(k−1)best
‖g (k)‖22
where 0 < γ < 1.
Stephen Gould | MLSS 2015 82/92
![Page 222: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/222.jpg)
Optimal Step Size for Known f⋆
Assume we know f ⋆ (we just don’t know x⋆). Then
αk =f (x(k))− f ⋆
‖g (k)‖22
is an optimal step size in some sense. Called the Polyak step size.
A good approximation when f ⋆ is not known (but non-negative) is
αk =f (x(k))− γ · f
(k−1)best
‖g (k)‖22
where 0 < γ < 1.
Stephen Gould | MLSS 2015 82/92
![Page 223: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/223.jpg)
Projected Subgradient Method
One extension of the subgradient method is the projectedsubgradient method which solves problems of the form
minimize f (x)subject to x ∈ C
Here the updates are
x(k+1) = PC
(
x(k) − αkg(k))
The projected subgradient method has similar convergenceguarantees to the subgradient method.
Stephen Gould | MLSS 2015 83/92
![Page 224: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/224.jpg)
Projected Subgradient Method
One extension of the subgradient method is the projectedsubgradient method which solves problems of the form
minimize f (x)subject to x ∈ C
Here the updates are
x(k+1) = PC
(
x(k) − αkg(k))
The projected subgradient method has similar convergenceguarantees to the subgradient method.
Stephen Gould | MLSS 2015 83/92
![Page 225: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/225.jpg)
Projected Subgradient Method
One extension of the subgradient method is the projectedsubgradient method which solves problems of the form
minimize f (x)subject to x ∈ C
Here the updates are
x(k+1) = PC
(
x(k) − αkg(k))
The projected subgradient method has similar convergenceguarantees to the subgradient method.
Stephen Gould | MLSS 2015 83/92
![Page 226: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/226.jpg)
Supergradient of mini{aTi x + bi}
Consider f (x) = mini{aTi x+ bi} and let I (x) = argmini{a
Ti x + bi}.
Then for any i ∈ I (x), g = ai is a supergradient of f at x.
f (x) + gT (z− x) = f (x)− aTi (z− x) i ∈ I (x)
= f (x)− aTi x− bi + aTi z+ bi
= aTi z+ bi
≥ f (z)
Stephen Gould | MLSS 2015 84/92
![Page 227: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/227.jpg)
Supergradient of mini{aTi x + bi}
Consider f (x) = mini{aTi x+ bi} and let I (x) = argmini{a
Ti x + bi}.
Then for any i ∈ I (x), g = ai is a supergradient of f at x.
f (x) + gT (z− x) = f (x)− aTi (z− x) i ∈ I (x)
= f (x)− aTi x− bi + aTi z+ bi
= aTi z+ bi
≥ f (z)
Stephen Gould | MLSS 2015 84/92
![Page 228: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/228.jpg)
Dual Decomposition Inference [Komodakis et al., 2010]
initialize λc = 0
loopslaves solve minyc ψc(yc) + λc(yc) (to get µ⋆
c )master updates λc as
λc ← λc + α
(
µ⋆
c −1
C
∑
c′
µ⋆
c′
)
until convergence
Stephen Gould | MLSS 2015 85/92
![Page 229: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/229.jpg)
Dual Decomposition Inference [Komodakis et al., 2010]
initialize λc = 0
loopslaves solve minyc ψc(yc) + λc(yc) (to get µ⋆
c )master updates λc as
λc ← λc + α
(
µ⋆
c −1
C
∑
c′
µ⋆
c′
)
until convergence
Stephen Gould | MLSS 2015 85/92
![Page 230: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/230.jpg)
parameter learning
Stephen Gould | MLSS 2015 86/92
![Page 231: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/231.jpg)
Max-Margin Learning
Assume we have an energy function which is linear in itsparameters, Ew(y; x) = wTφ(y; x).
Let D = {(yt , xt)}Tt=1 be our set of training examples.
Our goal in learning is to find a parameter setting x⋆ so thatfor each training example Ew(yt ; xt) is lower than the energyof any other assignment Ew(y; xt) by some margin.
We formalise the notion of margin by defining a loss function∆(yt , y), which is zero when y = yt and positive otherwise.
For simplicity let us assume we only have a single trainingexample (y†, x†).
Stephen Gould | MLSS 2015 87/92
![Page 232: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/232.jpg)
Max-Margin Learning
Assume we have an energy function which is linear in itsparameters, Ew(y; x) = wTφ(y; x).
Let D = {(yt , xt)}Tt=1 be our set of training examples.
Our goal in learning is to find a parameter setting x⋆ so thatfor each training example Ew(yt ; xt) is lower than the energyof any other assignment Ew(y; xt) by some margin.
We formalise the notion of margin by defining a loss function∆(yt , y), which is zero when y = yt and positive otherwise.
For simplicity let us assume we only have a single trainingexample (y†, x†).
Stephen Gould | MLSS 2015 87/92
![Page 233: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/233.jpg)
Max-Margin Learning
Assume we have an energy function which is linear in itsparameters, Ew(y; x) = wTφ(y; x).
Let D = {(yt , xt)}Tt=1 be our set of training examples.
Our goal in learning is to find a parameter setting x⋆ so thatfor each training example Ew(yt ; xt) is lower than the energyof any other assignment Ew(y; xt) by some margin.
We formalise the notion of margin by defining a loss function∆(yt , y), which is zero when y = yt and positive otherwise.
For simplicity let us assume we only have a single trainingexample (y†, x†).
Stephen Gould | MLSS 2015 87/92
![Page 234: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/234.jpg)
Max-Margin Learning
Assume we have an energy function which is linear in itsparameters, Ew(y; x) = wTφ(y; x).
Let D = {(yt , xt)}Tt=1 be our set of training examples.
Our goal in learning is to find a parameter setting x⋆ so thatfor each training example Ew(yt ; xt) is lower than the energyof any other assignment Ew(y; xt) by some margin.
We formalise the notion of margin by defining a loss function∆(yt , y), which is zero when y = yt and positive otherwise.
For simplicity let us assume we only have a single trainingexample (y†, x†).
Stephen Gould | MLSS 2015 87/92
![Page 235: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/235.jpg)
Max-Margin Learning
Assume we have an energy function which is linear in itsparameters, Ew(y; x) = wTφ(y; x).
Let D = {(yt , xt)}Tt=1 be our set of training examples.
Our goal in learning is to find a parameter setting x⋆ so thatfor each training example Ew(yt ; xt) is lower than the energyof any other assignment Ew(y; xt) by some margin.
We formalise the notion of margin by defining a loss function∆(yt , y), which is zero when y = yt and positive otherwise.
For simplicity let us assume we only have a single trainingexample (y†, x†).
Stephen Gould | MLSS 2015 87/92
![Page 236: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/236.jpg)
Max-Margin Quadratic Program
Learning goal: Find w such that Ew(y)− Ew(y†) ≥ ∆(y†, y).
Relaxed and regularized learning goal:
minimize
regularization︷ ︸︸ ︷
1
2‖w‖22 +
slack︷︸︸︷
Cξ
subject to wTφ(y)− wTφ(y†)︸ ︷︷ ︸
energy difference
≥ ∆(y, y†)− ξ︸ ︷︷ ︸
rescaled margin
,
very large︷ ︸︸ ︷
∀y ∈ Y
ξ ≥ 0
Stephen Gould | MLSS 2015 88/92
![Page 237: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/237.jpg)
Max-Margin Quadratic Program
Learning goal: Find w such that Ew(y)− Ew(y†) ≥ ∆(y†, y).
Relaxed and regularized learning goal:
minimize
regularization︷ ︸︸ ︷
1
2‖w‖22 +
slack︷︸︸︷
Cξ
subject to wTφ(y)− wTφ(y†)︸ ︷︷ ︸
energy difference
≥ ∆(y, y†)− ξ︸ ︷︷ ︸
rescaled margin
,
very large︷ ︸︸ ︷
∀y ∈ Y
ξ ≥ 0
Stephen Gould | MLSS 2015 88/92
![Page 238: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/238.jpg)
Re-writing Margin Constraints
Recognize that wTφ(y)− wTφ(y†) ≥ ∆(y, y†
)− ξ for all y so, in
particular, it must hold for the worst case y.
minimize 12‖w‖
22 + Cξ
subject to ξ ≥ maxy∈Y
{
∆(y, y†)− wTφ(y)}
︸ ︷︷ ︸
loss-augmented inference (for given w)
+wTφ(y†)
ξ ≥ 0
As long as ∆(y, yt) decomposes over cliques of E we can useinference to find the most violated constraint (for a fixed w).
Stephen Gould | MLSS 2015 89/92
![Page 239: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/239.jpg)
Re-writing Margin Constraints
Recognize that wTφ(y)− wTφ(y†) ≥ ∆(y, y†
)− ξ for all y so, in
particular, it must hold for the worst case y.
minimize 12‖w‖
22 + Cξ
subject to ξ ≥ maxy∈Y
{
∆(y, y†)− wTφ(y)}
︸ ︷︷ ︸
loss-augmented inference (for given w)
+wTφ(y†)
ξ ≥ 0
As long as ∆(y, yt) decomposes over cliques of E we can useinference to find the most violated constraint (for a fixed w).
Stephen Gould | MLSS 2015 89/92
![Page 240: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/240.jpg)
Re-writing Margin Constraints
Recognize that wTφ(y)− wTφ(y†) ≥ ∆(y, y†
)− ξ for all y so, in
particular, it must hold for the worst case y.
minimize 12‖w‖
22 + Cξ
subject to ξ ≥ maxy∈Y
{
∆(y, y†)− wTφ(y)}
︸ ︷︷ ︸
loss-augmented inference (for given w)
+wTφ(y†)
ξ ≥ 0
As long as ∆(y, yt) decomposes over cliques of E we can useinference to find the most violated constraint (for a fixed w).
Stephen Gould | MLSS 2015 89/92
![Page 241: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/241.jpg)
Cutting-Plane Max-Margin Learning
Start with active set A = {}.
Solve for w and ξ
minimize 12‖w‖
22 + Cξ
subject to wTφ(y)− wTφ(y†) ≥ ∆(y, y†)− ξ, ∀y ∈ Aξ ≥ 0
Find the most violated constraint,
y⋆ ∈ argminy∈Y
{
wTφ(y)−∆(y, y†)}
Add y⋆ to active set A and repeat.
Stephen Gould | MLSS 2015 90/92
![Page 242: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/242.jpg)
Subgradient Descent Max-Margin Learning
Recognize that ξ⋆ = maxy∈Y{∆(y, y†)− wTφ(y)
}. So rewrite the
max-margin QP as the non-smooth optimization problem
minimize1
2‖w‖22 + C max
y∈Y
{
∆(y, y†)− wTφ(y)}
︸ ︷︷ ︸
family of linear functions
which we can solve by the subgradient method.
Stephen Gould | MLSS 2015 91/92
![Page 243: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/243.jpg)
Tutorial Summary
Structured prediction models, or energy functions, arepervasive in computer vision (and other fields).
Often we are interested in finding the energy minimizingassignment.
Exact and approximate inference algorithms exploit structure:
message passing for low treewidth graphsgraph-cuts for submodular energiesdual decomposition for decomposeable energies
Parameter learning within a max-margin setting.
Still very active research in inference and learning.
Stephen Gould | MLSS 2015 92/92
![Page 244: Happy New Year!users.cecs.anu.edu.au/~sgould/teaching/mlss-2015.pdf · Happy New Year! Structured Prediction for Computer Vision MLSS, Sydney 2015 Stephen Gould 19 February 2015 Stephen](https://reader034.vdocuments.site/reader034/viewer/2022052023/60389afaca3e3d59d949591c/html5/thumbnails/244.jpg)
Tutorial Summary
Structured prediction models, or energy functions, arepervasive in computer vision (and other fields).
Often we are interested in finding the energy minimizingassignment.
Exact and approximate inference algorithms exploit structure:
message passing for low treewidth graphsgraph-cuts for submodular energiesdual decomposition for decomposeable energies
Parameter learning within a max-margin setting.
Still very active research in inference and learning.
Stephen Gould | MLSS 2015 92/92