EECS 274 Computer Vision

Segmentation by Clustering II

Segmentation and Grouping

• Watershed• Mean shift filtering and segmentation

• Reading: FP Chapter 14, S Chapter 5

Watershed Segmentation Visualize an image in 3D: spatial coordinates and gray

levels. In such a topographic interpretation, there are 3 types of

points Points belonging to a regional minimum (basin) Points at which a drop of water would fall to a single minimum.

(The catchment basin or watershed of that minimum.) Points at which a drop of water would be equally likely to fall to

more than one minimum. (The divide lines or watershed lines.)

Watershed lines

Watershed Segmentation The objective is to find watershed lines (via thresholding) The idea is simple:

Suppose that a hole is punched in each regional minimum and that the entire topography is flooded from below by letting water rise through the holes at a uniform rate.

When rising water in distinct catchment basins is about the merge, a dam is built to prevent merging. These dam boundaries correspond to the watershed lines.

Watershed Segmentation

• Start with all pixels with the lowest possible value– These form the basis for initial watersheds

• For each intensity level k:– For each group of pixels of intensity k

• If adjacent to exactly one existing region, add these pixels to that region• Else if adjacent to more than one existing regions, mark as boundary• Else start a new region

– Implemented with breadth-first search

Watershed Segmentation

Watershed Segmentation

Watershed algorithm might be used on the gradient image instead of the original image.

Watershed Segmentation

Due to noise and other local irregularities of the gradient, over segmentation might occur.

Watershed Segmentation

A solution is to limit the number of regional minima. Use markers to specify the only allowed regional minima.

Watershed Segmentation

A solution is to limit the number of regional minima. Use markers to specify the only allowed regional minima. (For example, gray-level values might be used as a marker.)

Often leads to over-segmentationUseful for interactive segmentation

Mean Shift

• Non-parametric model• Efficiently finding peaks in

the high-dimensional data distribution without computing the complete function explicitly

• See– K. Fukunaga and L. Hostetler,

“The estimation of the gradient of a density function with applications in pattern recognition”, IEEE Trans. Information Theory, 1975

– Y. Cheng, “Mean shift, mode seeking, and clustering”, PAMI 1995

Non-parametric density function• Smooth the data by convolving it with a

fixed kernel of width h

• Known as kernel density estimation or Parzen window technique

• Instead of using brute force approach, mean shift uses a variant of multiple restart gradient descent technique

)(

)()(2

2

rk

h

xxkxxKxf

i i

ii

is the kernel function or Parzen window

1D mean shift

Starting at some initial guess for a local maximum, yk, which can beRandom input data pint xi, mean shift computes the gradient of density estimate f(x) at yk and takes uphill step

Mean shift algorithm

• Compute the gradient of the density estimate f(x) at yk

• Rewrite the gradient function

• Mean shift is the difference between the weighted mean of neighbors xi around x and the current value of x

)(')(where

)()()()(2

2

rkrg

h

xxgxxxxGxxxf

i

ii

iii

mean shiftxxxG

xxGxxm

xmxxGxf

ii

iii

ii

thecalled is)(

)()(where

)()()(

Modes

• The mode yk at iteration k is replaced by locally weighted mean

• Comaniciu and Meer show that this algorithm converges to a local maximum of f(x) under some reasonably weak condition on kernel k(r)

• Two kind of kernels:

• The corresponding derivative kernel g(r) are a unit ball and another is Gaussian, respectively

iik

iiki

kkk xyG

xyGxymyy

)(

)()(1

)exp()(

)1,0max()(

21 rrk

rrk

N

E

Epanechnikov kernel

Gaussian kernel

Finding modes

• Simplest way– start a separate mean shift mode

estimate y at every input point xi

– and iterate for a fixed number of steps until the magnitude is below a threshold

• Faster way – randomly subsample the input points xi

– and find the nearest mode sample evolution path to the remaining points

Mean shift segmentation

• Better results can be obtained with a combination of kernels, i.e., joint domain of color, r, and location, s

• Clustering of {xi} is achieved by placing a seed yi at each xi in n-d feature space (e.g., n=5)

2

2

2

2

)(s

s

r

rj

h

xk

h

xkxK

Regions containing less than M pixels are discarded

Mean shift filtering and segmentation

Mean Shift Segmentation

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

Mean Shift AlgorithmMean Shift Algorithm

1. Choose a search window size.2. Choose the initial location of the search window.3. Compute the mean location (centroid of the data) in the search window.4. Center the search window at the mean location computed in Step 3.5. Repeat Steps 3 and 4 until convergence.

The mean shift algorithm seeks the “mode” or point of highest density of a data distribution:

Mean Shift Segmentation Algorithm1. Convert the image into tokens (via color, gradients, texture measures etc).2. Choose initial search window locations uniformly in the data.3. Compute the mean shift window location for each initial position.4. Merge windows that end up on the same “peak” or mode.5. The data these merged windows traversed are clustered together.

*Image From: Dorin Comaniciu and Peter Meer, Distribution Free Decomposition of Multivariate Data, Pattern Analysis & Applications (1999)2:22–30

Mean Shift Segmentation

Mean Shift Segmentation

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html