robust higher order potentials for enforcing label consistency pushmeet kohli microsoft research...
Post on 19-Dec-2015
220 views
TRANSCRIPT
![Page 1: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/1.jpg)
Robust Higher Order Potentials Robust Higher Order Potentials For Enforcing Label ConsistencyFor Enforcing Label Consistency
Pushmeet KohliPushmeet KohliMicrosoft Research CambridgeMicrosoft Research Cambridge
Lubor LadickyLubor Ladicky Philip TorrPhilip TorrOxford Brookes University, OxfordOxford Brookes University, Oxford
CVPR 2008CVPR 2008
![Page 2: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/2.jpg)
Image labelling Problems
Image Denoising
Geometry Estimation
Object Segmentation
Assign a label to each image Assign a label to each image pixelpixel
Building
Sky
Tree Grass
![Page 3: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/3.jpg)
Object Segmentation using CRFs
Unary potentials based on
Colour, Location and Texture
features
CRF Energy
(Shotton et al. ECCV 2006)
Encourages label consistency in adjacent pixels
![Page 4: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/4.jpg)
Limitations of Pairwise CRFs
Image
Unary Potential
MAP-CRF
Solution
• Encourages short boundaries (Shrinkage bias)
• Can only enforce label consistency in adjacent pixels
• Inability to incorporate region based features
![Page 5: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/5.jpg)
Label Consistency in Image Regions
• Pixels constituting some regions belong to- Same plane (Orientation)
(Hoiem, Efros, & Herbert, ICCV’05)- Same object
(Russel, Efros, Sivic, Freeman, & Zisserman, CVPR06)
Image (MSRC) Segmentation (Mean shift)
![Page 6: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/6.jpg)
Image labelling using segments
Image Object Labelling
UnsupervisedSegmentation
• Geometric Context [Hoiem et al, ICCV05]
• Object Segmentation[He et al. ECCV06, Yang et al. CVPR07, Rabinovich et al. ICCV07, Batra et al. CVPR08]
• Interactive Video Segmentation[Wang, SIGGRAPH 2005 ]
Not robust to Inconsistent Segments!
![Page 7: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/7.jpg)
Our Higher Order CRF Model
cMultiple Segmentation
s
Encourages label consistency in
regions
![Page 8: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/8.jpg)
Label Consistency in Segments
• Encourages consistency within super-pixels
• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]
c
![Page 9: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/9.jpg)
Label Consistency in Segments
Cost: 0
c
• Encourages consistency within super-pixels
• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]
![Page 10: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/10.jpg)
Label Consistency in Segments
Cost: f (|c|)
c
• Encourages consistency within super-pixels
• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]
![Page 11: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/11.jpg)
• Encourages consistency within super-pixels
• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]
Label Consistency in Segments
Cost: f (|c|)
c
Does not distinguish between Good/Bad
Segments !
![Page 12: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/12.jpg)
Quality based Label Consistency
How to measure quality G(c)?[Ren and Malik ICCV03, Rabinovich et al. ICCV07, many others]
• Colour and Texture Similarity• Contour Energy
Label inconsistency cost depends on segment quality
Measure quality from variance in feature responses
Higher order generalization of contrast-sensitive pairwise potential
![Page 13: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/13.jpg)
Quality based Label Consistency
MSRC imageMean shift
segmentation
Segment Quality(darker is better)
![Page 14: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/14.jpg)
Robust Consistency Potentials
Inconsistent Pixels
max
0 1
PN Potts
Too Rigid!
max
0 1 T
Robust
Inconsistent Pixels
![Page 15: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/15.jpg)
Robust Consistency Potentials
Number of Inconsistent
Pixels
SlopeMaximum
Inconsistency Cost
max
0 1 T
Robust
Inconsistent Pixels
![Page 16: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/16.jpg)
Minimizing Higher order Energy Functions
• Message passing is computationally expensive– High runtime and space complexity - O(LN) – L = Number of Labels, N = Size of Clique
• Efficient BP for Higher Order MRFs [Lan et al. ECCV 06, Potetz CVPR 2007]
– 2x2 clique potentials for Image Denoising – Take minutes per iteration (Hours to
converge)
![Page 17: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/17.jpg)
Minimizing Higher order Energy Functions
• Graph Cut based move making algorithm[Kohli et al. CVPR 2007]
– Can handle very high order energy functions – Extremely efficient: computation time in the order of
seconds– Only applicable to some classes of functions (PN Potts)– Cannot handle robust consistency potential
• This paper– Can minimize a much larger class of higher order energy
functions– Same time complexity as [Kohli et al. CVPR 2007]
![Page 18: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/18.jpg)
Move making algorithms
Expansion and Swap move algorithms[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)• Each move results in a solution with smaller energy
Current SolutionCurrent Solution
Generate pseudo-boolean move
function
Generate pseudo-boolean move
function
Minimize move function to get optimal move
Minimize move function to get optimal move
Move to new
solution
Move to new
solution
How to minimize
move functions?
![Page 19: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/19.jpg)
Minimizing Move Functions using Graph Cuts
st-mincut
(Positive weights)
st-mincut
(Positive weights)
Second orderPseudo-boolean
Function Minimization (submodular)
Second orderPseudo-boolean
Function Minimization (submodular)
Optimal moves can be found extremely efficiently using graphs cuts
Most pairwise CRF models used in Computer Vision lead to submodular
move functions
![Page 20: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/20.jpg)
Minimizing Higher Order Energy Functions: Our results
• Result 1– We show that a large class of higher order potentials
lead to higher order submodular move functions– Can be minimized in polynomial time
• Submodular Function Minimization– Minimizing general submodular functions is
computationally expensive– Complexity O(n6)– Cannot handle large problems!
Details in Technical ReportDetails in Technical Report
![Page 21: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/21.jpg)
Minimizing Higher Order Energy Functions: Our results
• Minimizing Higher order functions using Graph cuts– Higher order functions can be transformed to second
order functions by adding auxillary variables– Exponential number of auxillary variables needed in
general
• Result 2– Our higher order functions can be transformed to
second-order functions using ≤2 auxillary variables per potential.
– Can be minimized extremely efficiently – Complexity << O(n6) Details in Technical ReportDetails in Technical Report
![Page 22: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/22.jpg)
Overview of our Method
Unary Potentials [Shotton et al. ECCV
2006]
+Contrast Sensitive Pairwise
Potentials
Higher Order Potentials
(Multiple Segmentations)
Segmentation Solution
+
Higher Order Energy
Energy Minimizatio
n
![Page 23: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/23.jpg)
Experimental results
Datasets: MSRC (21), Sowerby (7) [Shotton et al. ECCV 2006] [He et al. CVPR 04]
![Page 24: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/24.jpg)
Qualitative Results
Image(MSRC-
21)
Pairwise CRF
Higher order CRF
Ground Truth
SheepSheep
GrassGrass
![Page 25: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/25.jpg)
Qualitative Results (Contd..)
Image(MSRC-
21)
Pairwise CRF
Higher order CRF
Ground Truth
Results can be improved using image specific colour models Rother et al. SIGGRAPH 2004 Shotton et al. ECCV
2006
![Page 26: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/26.jpg)
Quantitative Results: Problems
Rough ground truth segmentations
Fine structures have small influence on overall pixel accuracy
![Page 27: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/27.jpg)
Generating Accurate Segmentations
Image(MSRC-21)
OriginalSegmentation
NewSegmentation
• Generated accurate segmentation of 27 images• 30 minutes per image
![Page 28: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/28.jpg)
Relationship between Qualitative and Quantitative
Results
Image(MSRC-21)
Pairwise CRF
Higher order CRF
Ground Truth
95.8%Overall Pixel
Accuracy
98.7%
Small changes in pixel accuracy can lead to large improvements in segmentation results.
![Page 29: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/29.jpg)
Quantitative Accuracy• Measure accuracy in labelling boundary pixels.• Accuracy evaluated in boundary bands of
variable width
Image(MSRC-21)
Hand-labelled Segmentation
Trimap(8-pixels)
Trimap(16-pixels)
![Page 30: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/30.jpg)
Quantitative Accuracy
• Measure accuracy in labelling boundary pixels.
• Accuracy evaluated in boundary bands of variable width
![Page 31: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/31.jpg)
Conclusions
• Method to enforce label consistency in image regions
• Generalization of the commonly used Pairwise CRF model
• Allows integration of pixel and region level features for image labelling problems
![Page 32: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/32.jpg)
Thanks
![Page 33: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/33.jpg)
Running Time ResultsTim
e (
sec)
Number of Segmentations
Inconsistency Cost
![Page 34: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/34.jpg)
Qualitative Results (Contd..)
Image(MSRC-
21)
Pairwise CRF
Higher order CRF
Ground Truth
![Page 35: Robust Higher Order Potentials For Enforcing Label Consistency Pushmeet Kohli Microsoft Research Cambridge Lubor LadickyPhilip Torr Oxford Brookes University,](https://reader030.vdocuments.site/reader030/viewer/2022032800/56649d375503460f94a1016a/html5/thumbnails/35.jpg)
Transformation to second order functions
cAuxiliary variables