chapter 7 – segmentation ii 7.1. mean shift segmentation 7.2. active contour models – snakes...

Chapter 7 – Segmentation II

7.1. Mean Shift Segmentation

7.2. Active Contour Models – Snakes

7.3. Geometric Deformable Models – Level

Sets and Geodesic Active Contours

7.4. Fuzzy Connectivity

7.5. Towards 3D Graph-Based Image Segmentation

7.6. Graph Cut-Segmentation

7.7. Optimal Single and Multiple Surface

Segmentation

7-1

1. For each data point

Fix a window around the data point.

Compute the mean of data within the window.

2. Shift data points to their computed means

simultaneously

3. Repeat till convergence.

7.1. Mean Shift Segmentation

Idea of mean shift:

7-2

Window

Center ofmass

Mean Shiftvector

7-3

7-4

7-5

7-6

7-7

Density distribution

Data points

Find modes

Densityestimator ,

1

1( ) ( )

ni

h K di

f Khnh

x x

x

1 , ,{ } , di i n i R x x

Find clusters

Objective:

Density kernel K(x)

Uniform

21( ) exp( )

2x xNK c

2(1 ) if 1

( )0 otherwise

x xxE

cK

1 if 1( )

0 if 1

xx

xUK

7-8

Normal

Epanechnikov

In practice, radially symmetric kernels are used, i.e.,2

( ) ( )x xK ck where k : profile of kernel K

7-9

Uniform1 if 1

( )0 if 1

U

xk x

x

Profile of kernel K(x)

Epanechnikov

Normal

7-10

Given n data points in d-D space , the density

estimation at point x : ,1

1( ) ( )

ni

h K di

f Khnh

x x

x

,1

1( ) ( )

x xx

ni

h K di

f Khnh

2

( ) ( )x x x xi i

kK c kh h

We are interested in locating x for which , ( ) 0h Kf x

2

,1

2

21

( ) ( )

2 ( ) ( )

nk i

h K di

nk i

idi

cf k

hnh

ck

hnh

x xx

x xx x

Use profile, i.e.,

The density gradient:

(See next page)

7-11

( ( )) ( ) ( ) ( )( ( )) ( )

dk f dk f df dfk f k f

d df d d

x x xx

x x x2

( ) ifh

x xxLet

Then2 2 2

i i idk k

h h d h

x x x x x x

x

2

2

( ) ( )

2 2 ( )

i i iT

ii

d d

d h d h h

h h h

x x x x x x

x x

x xx x

2 2

2

2( )i i

ik kh hh

x x x xx x

7-12

2( ( ) / )x xi ig g h let

Assume the derivative of k is ( ) ( )k x g x 2

, 21

2

21

2( ) ( ) ( ( ) / )

2 ( ) ( ( ) / )

nk

h K i idi

nk

i idi

cf k h

nh

cg h

nh

x x x x x

x x x x

2 21 1

21 1

2 2= ( ) = ( )

2( )

n nk k

i i i i id di i

n nk

i i idi i

c cg g g

nh nh

cg g

nh

x x x x

x x

7-13

21 1 1

2( ) ( ) /( )

n n nk

i i i idi i i

cg g g

nh

x x

Let g(x) be the profile of kernel G(x), i.e.,

2( ) ( )gG c gx x

The first term is proportional to a density

estimator 2

,1

( ) ( )n

g ih G d

i

cf g

hnh

x x

x computed with

kernel ( )G x

21

2 nk

idi

cg

nh

7-14

The second term is the mean

shift vector 1 1

( ) /( )x xn n

i i ii i

g g

The successive location of the kernel G are2 2

11 1

( ) / ( )y x y x

y xn n

j i j ij i

i i

g gh h

2

1 1, 2

1 1

( )

( )

( )

x xx x

m x x xx x

n ni

i i ii i

h G n ni

ii i

g gh

g gh

7-15

7-16

Color image

(L u v)- color space

Example: Mode Detection

Feature vector of pixel i, ( )i L u vx

7-17

Example: Image Segmentation

Feature vector of pixel i,

where : spatial-domain part, : range-domain part

( )s ri x x x

sx rxe.g.,

7-18

Two steps of mean-shift image segmentation:

1) Discontinuity Preserving Filtering

-- Preserves discontinuities of images

7-19

Let : pixels of the original image

: pixels of the filtered imageix

iz

2) Mean Shift Clustering

Given ( , ),i jMz : the mode associated withjM izknown from Step 1.

-- Regularizes regions

7-20

BOA : Basin Of Attraction, the set of all locations that converge to the same mode

where : segmentation label of pixel i : convergence points known from Step 1

iL,i cony

7-21

Examples: (1) Image Segmentation

(2) Image Smoothing and Segmentation

Original image Smoothed image Segmentation image

Gary levels of original image The path of mean shift

Gary levels after segmentationGary levels after smoothing7-22

7.2. Active Contour Models – Snakes

7-23

-- An energy-minimization approach

-- (i) A snake is a deformable model whose energy

depends on its shape and location in the image

(ii) Local minima of the energy correspond to

desired image properties.

(iii) Initial snake position should be provided.

7-24

7-25

7.2.1 Traditional Snakes and Balloons

Snake:

The energy functional of the snake, which is to be

minimized is a weighted combination of internal

and external forces

( ) [ ( ), ( )], [0,1]v s x s y s s

The internal forces emanate from the shape of the snake

The external forces come from the image and

constraints

7-26

The image forces may come from lines, edges, and

terminations

3/ 2/g g g

where ( , , ( , )),

( , ) : Gaussian smoothed ( , )

x y g x y

g x y f x y

g

7-27

The constraint forces come from user

specified properties to be imposed on the snake,

e.g., smmothness.

7-28

2 2

2 20

v vvext

d d d ds s E s

ds ds ds ds

written this equation in an evolution form

0,s

dE E

ds v v

To minimize

This leads to the Euler-Lagrange motion equation

, let 0.dE

ds

Substitute int extE E E into the equation

where , , s s

s

E d EE E

ds

v v

vv

v v

22

2 2

, , ,

,ext

s t s t s ts s

t s s s s

E s t

v v v

vStop when( , )

0s t

t

v

7-29

A balloon, which can be inflated, is extended from

the snake by including an additional pressure force

so that it can overcome small isolated barriers

7-30

7.2.3 Gradient Vector Flow Snakes

Difficulties of previous approaches:

(i) initialization , (ii) concaves of boundaries

GVF: An external force field points toward

boundaries when in their proximity

7-31

2 22 2 2 2x y x yE u y v v f f dxdy g

GVF is derived from image by minimizing an energy functional

( , ) ( ( , ), ( , ))g x y u x y v x y

2 2 2 0,x x yu u f f f 2 2 2 0y x yv v f f f

g can be obtained by solving the following Euler equations, which are diffusion equations

Rewrite the above equations in an evolution form

2

2 2

, , , , ( , ,

, )( , , )

t

x x y

u x y t u x y t u x y t

f x y f x y f x y

2

2 2

, , , , ( , ,

, )( , , )

t

y x y

v x y t v x y t v x y t

f x y f x y f x y

7-32

Let in Eqs. (7.15) and (7.22) forming the

GVF snake equation

extE g

, , ,v v v gt s t s t s t

7-33

2 2 2

dE f f d

g g x

22 0,f f g g

22t f f g g g

3D GVF:

7-34

7-35

7.2.2 ExtensionsDeformable models, Fourier deformable model,Finite element snakes, B-snakes, united snakes.

7.3. Geometric Deformable Models (GDM) – Level Sets and Geodesic Active Contours

7-36

1. Parametric deformable model: Borders are represented in a parametric form, e.g., snakes2. Geometric deformable model : Borders are represented by partial differential equations, e.g., level sets, geodesic active contours

2 main groups of deformable models:

Advantages of GDM:(i) Curves are evolved using only geometric

computations, independent of any parameterization

7-37

[ ( ), ( )]C x t y tLet a curve be denoted by

(ii) Curves are represented as level sets of higher dimensional functions yielding seamless treatment of topological changes(iii) Multiple object can be detected simultaneously

( ) ( )t

as t d F

( ) ( )s

t V tt

F

( ) ( ) ( )t x t y t F i j

The associated length function:

positional vector

: speed function

or by

[ ( ( )), ( ( ))] [ ( ), ( )] ( )C x t s y t s C X s Y s s X

7-38

Assume the curve moves only in a direction normal

(N) to itself, i.e.

( )( )

sV t s

t

X

N

Assume also the speed function is a function of

curvature c, ( )( )

sV c s

t

X

N

( ) ( ) ( ) ( ) ( )[ , ] [ , ]

[ ( ), ( )] ( )

s X s Y s X s s Y s s

t t t s t s ts

X s Y s s V tt

X

X

( ) ( )s s X N

7-39

Constant Deformation Equation

-- Deformation is similar to inflation balloon force and

may introduce singularities (e.g., sharp corners)

0/X Nt V

/X Nt c Curvature Deformation Equation

7-40

The above geometric deformable model starts with an initial curve and evolves its shape using a speed function. Such an evolution can be implemented using level sets technique.

Combined constant and curvature deformation:

0( ) .k c Vt

X

N

7-41

1

.1

kG I

Let

Then, k -> 0 when the curve approaches edges. Strong edges will stop the curve propagation.

7-42

Level set function is a higher-dimensional

function, which represents a boundary as a level set.

The curve at time t is the set of image points, for

which the value of the level set function at time t

is equal to zero , 0x y

,x y

7-43

Using the level set representation of a curve allows its evolution by updating the level set function.

7-44

( )0,

V c

t

( )V c

t

2 2

32 2 2

2

( )

xx y x y xy yy x

x y

c

A level set function with contour

as its zero-level set

Differentiate w.r.t. t

,x y sX

0s X

0X

t t

,

N

( )( ),

sV c s

t

X

N ( ),

sV c

t

X

1tI tI

Experiment:

7-45

0 if

( ) if

if in

out

x

x x

x( , )

0t t

t

t t t

( ) x : the initial contour

Level set function:

( )t F F

n

( ) 0t tF

max{0, },int extF F F ,intF ( )extF E

( )E : the edge magnitude

7-46

7.4. Fuzzy Connectivity

7-47

Uncertainties for image segmentation: Noise, uneven illumination, limited spatial resolution, partial occlusion, etc.

Fuzzy connectivity segmentation keeps considering the likelihood (i.e., hanging togetherness) of whether nearby pixels belong to the same object

Fuzzy affinity : a strength of hanging togetherness of nearby pixels, which is a function of distance between two adjacent pixels.

e.g., 2

( , )( , )

1 ( ) ( )

c dc d

k f c f d

: fuzzy adjacency

f : image properties

7-48

2

12 21 1

11 if 2

( , ) 1 ( )

0 otherwise

i ii

i ii

c dc d k c d

Fuzzy adjacency :

Fuzzy connectedness :

7-49

Connectedness map : c : seed; : any pixel

( , )ic d id

7-50

2

12 21 1

11 if 2

( , ) (7.51)1 ( )

0 otherwise

i ii

i ii

c dc d k c d

2

( , )( , ) (7.54)

1 ( ) ( )

c dc d

k f c f d

( ) ( 1)

0 1( ) min ( , )

( , ) max ( )

n n

n N

M

c c

c d

7-51

7-52

Thresholding yields the segmentation result.

7-53

Extensions: Relative fuzzy connectivity,

7-54

Multi-seeded fuzzy connectivity

Scale-based fuzzy connectivity

7.6. Graph Cut-Segmentation

7-55

Idea:Seeds may be

interactively or

automatically

identified.

7-56

(1) Construct a graph with n+2 nodes

(2) Each node has 4 n-links, connected to neighbors. Each link is assigned a cost derived from smoothness term , where f: image valuesmooth ( )C f

Steps: Given an image of size n

(3) Each node has 2 t-links, connected to source S and sink T. Each link is assigned a cost derived from data term data ( )C f

7-57

(5) Find the minimum cut that minimizes

data smooth( ) ( ) ( )C f C f C f

(4) Identify seed pixels and define “hard constraint” for their links

(6) Obtain the segmentation result.

7-58

7-59

( ) ( )p pp I

R R L

L

( , )( , )

( ) ( , )p q p qp q N

B B L L

L

1 if ( , )

0 otherwisep q

p q

L LL L

: the resulting labeling vector

Let I : the set of pixels

Example: Given an image

N: a set of neighborhood pixel pairs ( , ), ,p q p q I

1 2( , , , ),IL L L L {object, background}iL

L is obtained by finding the cut that minimizes the cost where

: regional property term

: boundary property term

( ) ( ) ( ),C R B L L L

7-60

The costs of arcs are defined in the following table

e.g.,

2

2

(object) ln ( | ), (background) ln ( | )

( )1( , ) exp

, 2

p p p p

p q

R P I O R P I B

I IB p q

p q

7-61

Example:

7-62