shu kong - university of california, irvineskong2/slides/pixel_embedding_for... · 2017-11-20 ·...
TRANSCRIPT
![Page 1: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/1.jpg)
Shu KongCS, ICS, UCI
Recurrent Pixel Embedding for Grouping
![Page 2: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/2.jpg)
1. Problem Statement -- Pixel Grouping
2. Pixel-Pair Spherical Max-Margin Embedding
3. Recurrent Mean Shift Grouping
4. Experiment
5. Conclusion and Extension
Outline
Note: the slides were made before paper submission, please treat them as supplemental material and refer to the paper for updated content.
![Page 3: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/3.jpg)
Pixel Labeling
Tasks diving into pixels --
![Page 4: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/4.jpg)
Pixel Labeling: Low-Level Vision
Tasks diving into pixels --
Low-level vision:
edge, boundary, contour
![Page 5: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/5.jpg)
Pixel Labeling: Mid-Level Vision
Tasks diving into pixels --
Low-level vision:
edge, boundary, contour
Mid-level vision:
object proposal
![Page 6: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/6.jpg)
Pixel Labeling: High-Level Vision
Tasks diving into pixels --
Low-level vision:
edge, boundary, contour
Mid-level vision:
object proposal
High-level vision:
semantic segmentation
instance-level semantic segmentation
![Page 7: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/7.jpg)
Pixel Labeling: Learning
Tasks diving into pixels --
Low-level vision:
edge, boundary, contour
Mid-level vision:
object proposal
High-level vision:
semantic segmentation
instance-level semantic segmentation
logistic loss
logistic loss for score
regression for location
logistic loss for mask&score
cross-entropy for category
![Page 8: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/8.jpg)
Pixel Labeling: New Framework
A new framework consisting of two novel modules --
![Page 9: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/9.jpg)
Pixel Labeling: New Framework
A new framework consisting of two novel modules -- This framework is agnostic to architecture, so ignore deep learning for now!
![Page 10: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/10.jpg)
Pixel Labeling: New Framework
A new framework consisting of two novel modules --
1. pixel-pair spherical max-margin regression
2. recurrent mean shift grouping
![Page 11: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/11.jpg)
Pixel Labeling: New Framework
A new framework consisting of two novel modules --
1. pixel-pair spherical max-margin regressionl learning an embedding space on the hyper-sphere such that
• if meeting the pair-wise criterion, learn to push pixels to be close to each other,
e.g. both are boundaries, from same instance;
• if not, learn to pull them apart.
2. recurrent mean shift grouping
![Page 12: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/12.jpg)
Pixel Labeling: New Framework
A new framework consisting of two novel modules --
1. pixel-pair spherical max-margin regressionl learning an embedding space on the hyper-sphere such that
• if meeting the pair-wise criterion, learn to push pixels to be close to each other;
e.g. both are boundaries, from same instance;
• if not, learn to pull them apart.
2. recurrent mean shift groupingl iteratively group the pixels into discrete clusters, such as criteria:
boundary vs. non-boundary; object proposals; semantic segments
![Page 13: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/13.jpg)
Pixel-Pair Spherical Max-Margin Regression
![Page 14: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/14.jpg)
Pixel-Pair Spherical Max-Margin Regression
date back to Fisher Linear discriminant analysis (LDA)
![Page 15: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/15.jpg)
Pixel-Pair Spherical Max-Margin Regression
date back to Fisher Linear discriminant analysis (LDA)
To utilize the label information in finding informative projection, maximizing the following objective
where
![Page 16: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/16.jpg)
Pixel-Pair Spherical Max-Margin Regression
What loss functions can we use at pixel-level?
![Page 17: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/17.jpg)
Pixel-Pair Spherical Max-Margin Regression
What loss functions can we use at pixel-level?
Principle --
1. for positive pairs of pixels (meeting the criterion), minimizing the pair-wise discrepancy/distance;
2. for negative pairs, minimizing the similarity.
![Page 18: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/18.jpg)
Pixel-Pair Spherical Max-Margin Regression
What loss functions can we use at pixel-level?
Principle --
1. for positive pairs of pixels (meeting the criterion), minimizing the pair-wise discrepancy/distance;
2. for negative pairs, minimizing the similarity.
Bert De Brabandere, Davy Neven, Luc Van GoolSemantic Instance Segmentation with a Discriminative Loss Function, arxiv, 2017
![Page 19: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/19.jpg)
Pixel-Pair Spherical Max-Margin Regression
What loss functions can we use at pixel-level?
Principle --
1. for positive pairs of pixels (meeting the criterion), minimizing the pair-wise discrepancy/distance;
2. for negative pairs, minimizing the similarity.
for example:
Euclidean distance between pixel feature vectors for measuring distance.
Its inverse, or Gaussian transform, can measure the similarity.
.....
Bert De Brabandere, Davy Neven, Luc Van GoolSemantic Instance Segmentation with a Discriminative Loss Function, arxiv, 2017Alejandro Newell, Jia Deng, Associative Embedding: End-to-End Learning for Joint Detection and Grouping, NIPS, 2017Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, Kevin P. Murphy, Semantic Instance Segmentation via Deep Metric Learning
![Page 20: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/20.jpg)
Pixel-Pair Spherical Max-Margin Regression
We propose the module to learn a hyper-sphere (embedding space), such that
positive pairs have high cosine similarity;
negative pairs have low cosine similarity.
![Page 21: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/21.jpg)
Pixel-Pair Spherical Max-Margin Regression
Why cosine similarity?
E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997.L. Lovisolo ; E.A.B. da Silva, uniform distribution of points on a hyper-sphere with applications to vector bit-plane encoding, IEE Proc. Vision, Image and Signal Processing, 2001
![Page 22: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/22.jpg)
Pixel-Pair Spherical Max-Margin Regression
Why cosine similarity?
1. scale-invariant to the length of feature vector;
E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997.L. Lovisolo ; E.A.B. da Silva, uniform distribution of points on a hyper-sphere with applications to vector bit-plane encoding, IEE Proc. Vision, Image and Signal Processing, 2001
![Page 23: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/23.jpg)
Pixel-Pair Spherical Max-Margin Regression
Why cosine similarity?
1. scale-invariant to the length of feature vector;
2. easy to analyze how to set hyper-parameters;
E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997.L. Lovisolo ; E.A.B. da Silva, uniform distribution of points on a hyper-sphere with applications to vector bit-plane encoding, IEE Proc. Vision, Image and Signal Processing, 2001
![Page 24: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/24.jpg)
Pixel-Pair Spherical Max-Margin Regression
Why cosine similarity?
1. scale-invariant to the length of feature vector;
2. easy to analyze how to set hyper-parameters;
E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997.L. Lovisolo ; E.A.B. da Silva, uniform distribution of points on a hyper-sphere with applications to vector bit-plane encoding, IEE Proc. Vision, Image and Signal Processing, 2001
![Page 25: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/25.jpg)
Pixel-Pair Spherical Max-Margin Regression
We use the calibrated cosine similarity as below
![Page 26: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/26.jpg)
Pixel-Pair Spherical Max-Margin Regression
We use the calibrated cosine similarity as below
loss function contains postive and negative pairs
![Page 27: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/27.jpg)
Pixel-Pair Spherical Max-Margin Regression
We use the calibrated cosine similarity as below
loss function contains postive and negative pairs
alpha is the margin, hyper parameter to be set.
![Page 28: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/28.jpg)
Pixel-Pair Spherical Max-Margin Regression
We use the calibrated cosine similarity as below
loss function contains postive and negative pairs
alpha is the margin, hyper parameter to be set.
Gradient is one, didn't penalize hard pixels in sensitive regions, say nearby boundary, segments, etc.
![Page 29: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/29.jpg)
Pixel-Pair Spherical Max-Margin Regression
Important theories
1. the loss has a lower bound, minimum;
2. the lower bound does not depend on the dimension of the embedding
space.
![Page 30: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/30.jpg)
Pixel-Pair Spherical Max-Margin Regression
2D case
![Page 31: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/31.jpg)
Pixel-Pair Spherical Max-Margin Regression
3D case
![Page 32: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/32.jpg)
Pixel-Pair Spherical Max-Margin Regression
https://en.wikipedia.org/wiki/N-sphere
![Page 33: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/33.jpg)
Pixel-Pair Spherical Max-Margin Regression
https://en.wikipedia.org/wiki/N-sphere
![Page 34: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/34.jpg)
Pixel-Pair Spherical Max-Margin Regression
https://en.wikipedia.org/wiki/N-sphere
![Page 35: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/35.jpg)
Pixel-Pair Spherical Max-Margin Regression
One more
![Page 36: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/36.jpg)
Pixel-Pair Spherical Max-Margin Regression
Last one -- Combination-aware Weighting
![Page 37: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/37.jpg)
Recurrent Mean Shift Grouping
From good embedding space to pixel labeling
How to get the instances?
How to group the pixels?
![Page 38: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/38.jpg)
Recurrent Mean Shift Grouping
From good embedding space to pixel labeling
How to get the instances?
How to group the pixels?
k-means, k-medoids?
![Page 39: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/39.jpg)
Recurrent Mean Shift Grouping
From good embedding space to pixel labeling
How to get the instances?
How to group the pixels?
k-means, k-medoids?
mean shift
![Page 40: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/40.jpg)
Recurrent Mean Shift Grouping
mean shift
R.Collins, CSE, PSU, CSE598G Spring 2006
![Page 41: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/41.jpg)
Recurrent Mean Shift Grouping
mean shift
R.Collins, CSE, PSU, CSE598G Spring 2006
![Page 42: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/42.jpg)
Recurrent Mean Shift Grouping
mean shift
K. Fukunaga, L. Hostetler, The Estimation of the Gradient of a Density Function, with Applications in Pattern - Recognition, PAMI, 1975
![Page 43: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/43.jpg)
Recurrent Mean Shift Grouping
mean shift
Other than estimating the PDF directly, estimating the gradient --
![Page 44: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/44.jpg)
Recurrent Mean Shift Grouping
mean shift
then
![Page 45: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/45.jpg)
Recurrent Mean Shift Grouping
mean shift: iteratively updating by shifting the data by such an amount
![Page 46: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/46.jpg)
Recurrent Mean Shift Grouping
mean shift: iteratively updating by shifting the data by such an amount
![Page 47: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/47.jpg)
Recurrent Mean Shift Grouping
mean shift: iteratively updating by shifting the data by such an amount
Gaussian blurring mean-shift (GBMS) algorithm
the new iterate is the data average under the posterior probabilities given the current iterate:
![Page 48: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/48.jpg)
Recurrent Mean Shift Grouping
Gaussian blurring mean-shift (GBMS) algorithm
Miguel A. Carreira-Perpinán, Fast Nonparametric Clustering with Gaussian Blurring Mean-Shift, ICML, 2006
![Page 49: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/49.jpg)
Recurrent Mean Shift Grouping
Gaussian blurring mean-shift (GBMS) algorithm
It's guaranteed to converge without gradient vanishing or exploding
Miguel A. Carreira-Perpinán, Fast Nonparametric Clustering with Gaussian Blurring Mean-Shift, ICML, 2006
![Page 50: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/50.jpg)
Recurrent Mean Shift Grouping
Gaussian blurring mean-shift (GBMS) algorithm
Miguel A. Carreira-Perpinán, Fast Nonparametric Clustering with Gaussian Blurring Mean-Shift, ICML, 2006
It's guaranteed to converge without gradient vanishing or exploding
But, are the updated data still on the sphere?
![Page 51: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/51.jpg)
Recurrent Mean Shift Grouping
L2 normalization in the loop
Takumi Kobayashi, Nobuyuki Otsu, Von Mises-Fisher Mean Shift for Clustering on a Hypersphere, ICPR, 2010
It's guaranteed to converge without gradient vanishing or exploding.
![Page 52: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/52.jpg)
Recurrent Mean Shift Grouping
running the von-Mises Fisher mean shift offline
![Page 53: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/53.jpg)
Recurrent Mean Shift Grouping
mean shift as recurrent module
![Page 54: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/54.jpg)
Recurrent Mean Shift Grouping
mean shift as recurrent module
![Page 55: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/55.jpg)
Recurrent Mean Shift Grouping
mean shift as recurrent module
![Page 56: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/56.jpg)
Recurrent Mean Shift Grouping
mean shift grouping in the loop
![Page 57: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/57.jpg)
Recurrent Mean Shift Grouping
What does it mean by mean shift gradient?
![Page 58: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/58.jpg)
Recurrent Mean Shift Grouping
What does it mean by mean shift gradient?
![Page 59: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/59.jpg)
Recurrent Mean Shift Grouping
mean shift grouping in the loop
input image loop-0 loop-5
![Page 60: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/60.jpg)
Learning to Group
Low-level vision:
edge, boundary, contour
Mid-level vision:
object proposal
High-level vision:
semantic segmentation
instance-level semantic segmentation
End-to-end trainable from data;
with the cross-entropy loss.
![Page 61: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/61.jpg)
Backbone
K He, X Zhang, S Ren, J Sun, Deep Residual Learning for Image Recognition, CVPR, 2016
architecture agnostic -- we use ResNet
![Page 62: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/62.jpg)
Experiment: Boundary Detection
boundary detection
one of most imbalanced problem, 85% pixels are non-boundary
1. learn the embedding space of 3-dim with our loss;
2. after convergence, adding logistic loss and fine-tuning;
3. averaging multiple outputs at resBlock2~5 followed by thinning (NMS).
![Page 63: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/63.jpg)
Experiment: Boundary Detection
visualize the 3-dim embedding maps as rgb image
before&after fine-tuning with logistic loss
![Page 64: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/64.jpg)
Experiment: Boundary Detection
visualize the 3-dim embedding maps as rgb image
before&after fine-tuning with logistic loss
![Page 65: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/65.jpg)
Experiment: Boundary Detection
quantitative comparison
![Page 66: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/66.jpg)
Experiment: Boundary Detection
test image
input image
![Page 67: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/67.jpg)
Experiment: Boundary Detection
test image aesthetically colorful
input image res2 res3
res4 res5rand-proj
![Page 68: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/68.jpg)
Experiment: Boundary Detection
[zoom-in] encoding orientation, distance transform
![Page 69: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/69.jpg)
Experiment: Boundary Detection
[zoom-in] encoding orientation, distance transform
the Mobius strip
![Page 70: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/70.jpg)
Experiment: Object Proposal Detection
object proposal detection
class-agnostic
reduce search space for subsequential tasks, e.g. object detection
the proposal framework is particularity suitable for this tasks
How suitable?
![Page 71: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/71.jpg)
Experiment: Object Proposal Detection
object proposal detection -- How suitable?
Achieving very high average recall (AR) with a dozen proposals per image!
Average Recall: averaging recall at IoU=[0.5:0.05:0.95]
![Page 72: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/72.jpg)
Experiment: Object Proposal Detection
Quanlitatively: Ours vs. SharpMask
![Page 73: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/73.jpg)
Experiment: Semantic Segmentation
semantic segmentation
with cross-entropy loss
The pixel pair loss can fill in
the “holes”.
![Page 74: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/74.jpg)
Experiment: Semantic Instance Segmentation
Instance-level semantic segmentation
Using the semantic segmentation result to vote for the semantic label within each object proposal
![Page 75: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/75.jpg)
Experiment: Semantic Instance Segmentation
Instance-level semantic segmentation
Using the semantic segmentation result to vote for the semantic label within each object proposal
![Page 76: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/76.jpg)
Experiment: Semantic Instance Segmentation
Instance-level semantic segmentation
Using the semantic segmentation result to vote for the semantic label within each object proposal
![Page 77: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/77.jpg)
Experiment: Semantic Instance Segmentation
Quanlitatively
div8 vs div4
![Page 78: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/78.jpg)
Conclusion and Extension
• The framework is architecture-wise agnostic, conceptually simple, computationally efficient, practically effective, and theoretically abundant;
![Page 79: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/79.jpg)
Conclusion and Extension
• The framework is architecture-wise agnostic, conceptually simple, computationally efficient, practically effective, and theoretically abundant;
• it can be purposed for boundary detection, object proposal detection, generic and instance-level segmentation, spanning low-, mid- and high-level vision tasks.
![Page 80: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/80.jpg)
Conclusion and Extension
• The framework is architecture-wise agnostic, conceptually simple, computationally efficient, practically effective, and theoretically abundant;
• it can be purposed for boundary detection, object proposal detection, generic and instance-level segmentation, spanning low-, mid- and high-level vision tasks.
• Experiments demonstrate that the new framework achieves state-of-the-art performance on all these tasks.
![Page 81: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/81.jpg)
1. R. Fisher. Dispersion on a sphere. In Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, volume 217, pages 295–305. The Royal Society, 1953.
2. E. B. Saff and A. B. Kuijlaars. Distributing many points on a sphere. The mathematical intelligencer, 19(1):5–11, 1997
3. K. Fukunaga and L. Hostetler. The estimation of the gradient of a density function, with applications in pattern recogni- tion. IEEE Transactions on information theory, 21(1):32–40, 1975
4. V. A. Epanechnikov. Non-parametric estimation of a multi-variate probability density. Theory of Probability & Its Applications, 14(1):153–158, 1969
Reference
![Page 82: Shu Kong - University of California, Irvineskong2/slides/pixel_embedding_for... · 2017-11-20 · Shu Kong CS, ICS, UCI Recurrent Pixel Embedding for Grouping. 1. Problem Statement](https://reader035.vdocuments.site/reader035/viewer/2022070805/5f038d957e708231d4099edf/html5/thumbnails/82.jpg)
Thanks