rewriting our energy function - artificial...
TRANSCRIPT
![Page 1: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/1.jpg)
Rewriting our energy function
Is a function of labelings and segment models, but our models are functions of the labelings, so we rewrite to be a function of ONLY the labelings.
![Page 2: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/2.jpg)
5 2 3= [0.5 0.2 0.3]
2 4 4= [0.2 0.4 0.4]
Background Pixels Foreground Pixels
Rewriting our energy function
![Page 3: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/3.jpg)
5 2 3= [0.5 0.2 0.3]
2 4 4= [0.2 0.4 0.4]
Background Pixels Foreground Pixels
Rewriting our energy function
![Page 4: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/4.jpg)
5 2 3= [0.5 0.2 0.3]
2 4 4= [0.2 0.4 0.4]
Background Pixels Foreground Pixels
Rewriting our energy function
![Page 5: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/5.jpg)
Rewriting our energy function
Concave. Minimized when pixels in a bucket are all of the same label.
Convex. Minimized when all pixels are split evenly between foreground and background.
![Page 6: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/6.jpg)
Some intuition
When every pixel in its own bucket, then for all buckets:
Then is constant w.r.t labelings.
Bias towards splitting pixels evenly between foreground and background.
![Page 7: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/7.jpg)
Some intuition
When every pixel in its own bucket, then for all buckets:
Then is constant w.r.t labelings.
Bias towards splitting pixels evenly between foreground and background.
Consider other extreme, where all pixels are together in one bucket.
Then and , so, convex and concave parts cancel.
No bias.
![Page 8: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/8.jpg)
Some intuition
If no pixels in a bucket are neighbors, then there is an optimal solution where all pixels in that bucket are assigned together.
![Page 9: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/9.jpg)
So how do we optimize this?
![Page 10: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/10.jpg)
![Page 11: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/11.jpg)
An aside.......... (in paper)
![Page 12: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/12.jpg)
What is st-mincut?
SOURCE
SINK
(Background Pixels)
(Foreground Pixels)
![Page 13: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/13.jpg)
Mapping unary and pairwise potential to mincut
SOURCE
SINK
(Background Pixels)
(Foreground Pixels)
How do we map the cliques onto this?
![Page 14: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/14.jpg)
Mapping clique potential to mincut
Recall intuition about
![Page 15: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/15.jpg)
Mapping clique potential to mincut
Recall intuition about
= +
+
...
![Page 16: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/16.jpg)
Mapping clique potential to mincut
Recall intuition about
=
SOURCE
(Foreground Pixels)
A lot of edges!
![Page 17: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/17.jpg)
Mapping clique potential to mincut
![Page 18: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/18.jpg)
Mapping clique potential to mincut
![Page 19: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/19.jpg)
Mapping clique potential to mincut
![Page 20: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/20.jpg)
Back to the original problem...
Recall: we have figured out how to optimize the Dual Decomposition of our joint optimization problem.
![Page 21: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/21.jpg)
Back to the original problem...
This can perform poorly.
Combine approaches:
is our segmentation from dual decomposition, which gives energy
Compute models and empirically
Do EM by minimizing (not joint)
![Page 22: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/22.jpg)
Final formulationMinimize w.r.t. segmentation labelings:
Solution cannot be worse than just the EM step, or just the joint minimization approximation, if we can find it.
DD works well for small
![Page 23: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/23.jpg)
Experimentation (add weight term)-163 histogram buckets evenly covering RGB space-Neighbors defined as all 8 surrounding pixels
-
-~10 minutes per image
![Page 24: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/24.jpg)
Results: Dual Decomposition Only
Can get global optimum from DD.
![Page 25: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/25.jpg)
Results: DD+EM MethodSolution 1:
Solution 2:
% times better than EM Optimum on
![Page 26: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/26.jpg)
Results
1. Find solution using EM and DD separately, choose best.
2. Run DD+EM with = 0.75
3. Run DD+EM with = 0.5
4. Run DD+EM with = 0.25
5. Run DD+EM with = 0
7.2% error rate on GrabCut dataset.
8.1% error rate on GrabCut with histogram model
![Page 27: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/27.jpg)
Extensions
Reduces error on globally optimal set from 4.1% to 3%
![Page 28: Rewriting our energy function - Artificial Intelligencevision.stanford.edu/.../dmandle_presentation.pdf · Some intuition When every pixel in its own bucket, then for all buckets:](https://reader033.vdocuments.site/reader033/viewer/2022043008/5f99d151aa0f57766e44de77/html5/thumbnails/28.jpg)
Summary
● Reformulated energy for simultaneous optimization of labeling and color model
● Derived dual decomposition so we could try to find optimal solution
● Optimized dual decomposition in polynomial time
● [Reviewed min-st cut]
● Adjusted algorithm to improve results