image representation and processing with level setsarantxa.ii.uam.es/.../20071120psalembier.pdf ·...

Image representation and processing with level sets UAM: 20th of November 2007

1

Image representation and processing with level sets

Philippe SalembierPhilippe SalembierImageImage and and VideoVideo ProcessingProcessing GroupGroupTSC TSC DeptDept

UPCUPCUPC

Outline

Image representation with level sets

Level set representation of images

Curve parametrization and evolution

Image filtering

Curve evolution and image diffusion

Classical diffusion processes

Image segmentation

Introduction to segmentation

Segmentation with active contours


2

Outline




Image filtering



Image segmentation



Level set representation of images (I)

Consider the image as the stacking of geometrical primitives: its level sets


3

Level set representation of images (II)

Decomposition of images in its level set:

Reconstruction of an image from its level sets (stacking):

{ }λλ ≥= ),(),( jixjiX

{ }λλ Xjijix ∈∨= ),(),(

Level set representation of images (III)

The representation highlights the geometrical aspects of:The signal = the signal is the stacking of geometrical primitives.

The processing = binary, one can process the level sets and reconstruct the images.

Threshold decomposition

Binary processingof level sets Stacking

In the sequel: process images through the deformation of its level sets ⇔ Partial differential equations (PDE)


4

Outline




Image filtering



Image segmentation



Curve parametrization

Continuous image model: i,j ∈ R

Curve parametrization:

Arc-length parametrization:

Tangent and Normal:

i

j( ) [ ]{ }bappjpipC ,,)(),()( ∈=

C(a)

C(a+ε)

Tangent

Normal

( ) [ ]{ }LengthssjsisCdds

,0,)(),()(

1sC that such

∈=

=

2

2

)(,)(

dsCdNk

dssdj

dssdi

dsdCT

=

⎟⎠⎞

⎜⎝⎛==

Circle of radius: 1/k (k: curvature)

Nkds

CdT

dsCd

dsdC

dsdC

dsdC

=⊥

=⇒=

2

2

2

2

0,1,


5

Level set and image features (I)

Level set and level curve:

The normal to the level curve and the image gradient are collinear:

xxN

∇∇

−=

0,

:rule chain theusing

0)( , Along

=∇=∂∂

+∂∂

=

=⇒=

Txdsdj

jx

dsdi

ix

dsdx

dsdxsxC λλ

x∇

NT

{ }{ },),(),(

,),(),(

λ

λ

λ

λ

==

≥=

jixjiC

jixjiX

Level set and image features (II)

Curvature of level curve: It can be shown that the curvature of level sets can also be computed directly on the image:

32

2

222

2

2

2 xdidx

djxd

didjxd

djdx

didx

djdx

dixd

xxdivk

∇⎟⎟⎠

⎞⎜⎜⎝

⎛+−=

⎟⎟⎠

⎞⎜⎜⎝

⎛

∇∇

= ( )djda

didaadiv += : "divergence"

xxT

xxdivk

xxN

∇∇

−=⎟⎟⎠

⎞⎜⎜⎝

⎛

∇∇

=∇∇

−=⊥

,,


6

Outline




Image filtering



Image segmentation



Curve evolution and image diffusion (I)

Curve evolution through PDE:

α simply modifies the curve

parametrization. If we are interested in the evolution of the geometry, the most general evolution is given by:

i

j

NTVtC βα +==∂∂

V

Tα

Nβ

NtC β=∂∂


7

Curve evolution and image diffusion (II)

The fundamental level set equation:

Level set propagation with normal velocity β is equivalent to

image diffusion by a PDEProof:

xtxN

tC

∇=∂∂

⇔=∂∂ ββ

xxxx

xxx

tx

xxNNxNx

dtdCx

tx

tx

dtdj

jx

dtdi

ix

dttjidxtjixC

∇=∇∇

∇=∇∇

−∇−=∂∂

∇∇

−=∇−=∇−=∇−=∂∂

=∂∂

+∂∂

+∂∂

=⇒=

βββ

ββ

λ

λ

λ

,,

but ,,,,

0);,();,( , Along

Outline




Image filtering



Image segmentation




8


Flows of constant velocity

Smoothing flows:

Isotropic diffusion

Anisotropic diffusion

Sharpening flow

Flows of constant velocity (I)

β = 1, flow in the direction of the level curve normal and of magnitude 1:

Equivalent to a dilation with a circle:

Isotropic expansion of constant velocityTopological changes

xtxN

tC

∇=∂∂

⇔=∂∂

NtC=

∂∂

Structuring element


9

Flows of constant velocity (II)

xtxN

tC

∇=∂∂

⇔=∂∂

Flows of constant velocity (IV)

Inverse flow: xtxN

tC

∇−=∂∂

⇔−=∂∂

Erosion


10

Isotropic smoothing flow (I)

In 1960 Gabor observed that the difference between a signal and its smoothed version is proportional to the laplacian of the signal.

Heat equation:

The solution of the heat equation is known analytically: convolution with a Gaussian impulse response.

( )xdivjx

ix

tx

∇=∂∂

+∂∂

=∂∂

2

2

2

2

Isotropic smoothing flow (II)

( )

( ) xx

xdiv

xdivtx

∇∇∇

=

∇=∂∂

Level curves move according to:

Smoothing is isotropic and goes across edges!

( )x

xdiv∇∇


11

Goal: Smooth in a direction parallel to the edges

Smoothing is stopped when curvature of level lines is low (edge presence):

From isotropic to anisotropic flow (I)

( ) ( )

xkxxxdiv

tx

xx

xdivxdivtx

∇=∇⎟⎟⎠

⎞⎜⎜⎝

⎛

∇∇

=∂∂

∇∇∇

=∇=∂∂ : tochanged

NktCxk

tx

=∂∂

⇔∇=∂∂

From isotropic to anisotropic flow (II)

( ) ( ) xx

xdivxdiv ∇∇∇

=∇ xkxxxdiv ∇=∇⎟⎟⎠

⎞⎜⎜⎝

⎛

∇∇

x∇x


12

From isotropic to anisotropic flow (III)

( ) ( ) xx

xdivxdiv ∇∇∇

=∇ xkxxxdiv ∇=∇⎟⎟⎠

⎞⎜⎜⎝

⎛

∇∇

x∇x

Anisotropic diffusion: Curvature motion

Curvature motion is defined by the geometric heat flow:

Non convex shapes become convex (and then disappear):

Shape inclusion is preserved:

2

2

dsCdNk

tC

==∂∂

Original curve T=500 T=1000 T=1500 T=4000

Original curve T=500


13

Anisotropic diffusion: image smoothing (I)

Isotropic diffusion Anisotropic diffusion

Low curvature = reduced smoothing

Anisotropic diffusion: image smoothing (II)

Isotropic diffusion Anisotropic diffusion


14

Sharpening flow (I)

Basic idea: The heat equation smoothes the signal. Can we invert it?

( ) ( )xdivtxxdiv

tx

∇−=∂∂

⇒∇=∂∂

Level curves at

200

Smoothed versionOriginal Inverse heat

equ.: 50Inverse heat

equ.: 100

Unstable process!

Sharpening flow (II)

Alternative strategy:

( ) ( )

( )

( )⎪⎩

⎪⎨

⎧

<∇∇=∂∂

>∇∇−=∂∂

∇∇−=∂∂

⇒∇=∂∂

0 when,

0 when,

:isThat

)(

xdivxtx

xdivxtx

xxdivsigntxxdiv

tx

Image

Laplacian

Positive Laplacian⇒ Erosion

Negative Laplacian⇒ Dilation

Propagation stops when Laplacian = 0 (inflection points)

Shock filters


15

Sharpening flow (III)

Level curves at

140

Smoothed versionOriginal Shock filter

Sharpening flow (IV)

Smoothed versionOriginal Shock filter


16

Summary on diffusion process

Flow of constant velocity ⇔ Erosion, Dilation

Isotropic diffusion ⇔ Heat equation (on the image)


⇔ Geometrical heat equation (on the level curve)

⇔ Curvature motion

Shock filters

⇔ “Stable” inverse of the heat equation

⇔ Combination of erosion / dilation depending on the laplacian sign

Outline




Image filtering



Image segmentation




17

Objects and superposition model

Object: ShapeTexture (Gray level, color)

3D world Camera Image

The contour-texture model

Original

Segmentation

Regions map

Model

Hierarchicalrepresentation


18

Ideal definition of segmentation

• Create a partition of image I: {Ri}

... of connected regions Ri, Rj such that: (C: criterion)

⎩⎨⎧

∀=∪∀=

jiRRCirueRC

ji

i

, False,)( ,T)(

⎩⎨⎧

∀∅==

jiRRR

ji

i

, , Image

∩∪

Evaluation of the segmentation quality

• (Extremely) difficult

• Application dependant

• Often deceiving


19

Manual segmentation

Best segmentation?

General segmentation scheme

Simplification Featureextraction Decision

• Remove useless or annoying information

• Preserve shape information

• Feature space:Gray levelColorVarianceFrequency component

• Pixels belonging to the same regions should correspond to homogeneous features

• Partition definition:→ Transition→ Homogeneity


20

Segmentation: transition-based

SimplificationSimplification FeatureextractionFeature

extraction DecisionDecision

Transition detection

Region detection

Transition detection

Transitionestimation

Partitioncreation

Originalimage

Transitionimage Partition

Enhancecontours

Define thepartition


21

Transition estimation

Which modelWhich model ?

•Space-frequency High frequencies Simple differences, Prewitt, Sobel, etc.

• Geometric Vertical transitions Morphological gradients

0

5

10

02

46

810

0

5

108

10

12

14

16

Original image

Spectrum

Examples of gradient estimation

Original image

Morphological

(x ⊕ b) - (x Θ b)

-1-0.5

00.5

1

-1-0.5

00.5

10

0.5

1

1.5

2

FxFy

Mag

nitu

de

Simple differences Sobel

Frequency response

||∇x||=Sqrt(∇horx2+∇verx2)

-1-0.5

00.5

1

-1-0.5

00.5

10

2

4

6

8

FxFy

Mag

nitu

de


22

Partition creation

Straightforward strategy: Gradient binarization

Difficulty to define the threshold

Noise may appear: Cleaning with opening

Contours may not be closed: Closing or gradient extrapolation

Contours may be thick: remove pixels without breaking the contours

Not very robust!

Outline




Image filtering



Image segmentation




23

Partition creation

Robust strategy: Active contours (Snakes)Evolution of a close curve towards the points of high gradients

How to define the curve evolution?

How to implement it?

How to define the initial curve?

Active contour (snakes): curve evolution (I)

Basic idea: Minimize the “length” of the curveIf s is the arc-length parametrization of the curve C:

Assume g is low when the gradient is high and g∈[0,1], ex:

of function is g where,)( length Geodesic

Length

xdsxg

ds

∇∇=

=

∫∫

1

1)( pxxg

∇+=∇


24

Active contour: curve evolution (II)

minimum is )(

that such)( curve theFind

∫ ∇ dsxg

sC

Gradient: ∇x

3D view of ∇x 3D view of g(∇x)

Active contour: curve evolution (III)

How to find the minimum of a function?:Compute the gradient of the function:

Gradient descent strategy: iterative search of i ⇒ i(t). It leads

to the following PDE:

0)()( of Minimum =⇒di

idEiE

iiE

tti

∂∂

−=∂∂ )()(

i

E(i)

)0(ididEtii Δ−= )0()1(

0<didE


25

Active contour: curve evolution (IV)

Using as criterion the geodesic length:

NNgNgkCE

tCdsxgCE ,)()( ∇−=

∂∂

−=∂∂

⇒∇= ∫

Internal force:• Curvature motion• Smoothing

External force:• The evolution stops when the gradient of

g is perpendicular to the curve normal

N∇g

NNgNckgtC ,)( ∇−+=∂∂

Add erosion/dilation to search the space:

Active contour: implementation (I)

Explicit implementation is complex:Resample the curve when it shrinks

Handle complex topological changes

Implicit implementation is much simpler:Create a (any) function u which has C(s) as zero level set.

Use the level set result:

utuN

tC

∇=∂∂

⇔=∂∂ ββ

No curve sampling, no curve resampling, natural handling of topological changes


26

Active contour: implementation (II)

Classical approach: u is the distance function of C(s)

( )⎩⎨⎧

∇∇=∇∇−=

∇∇∇

∇+∇+=∂∂

uudivkuuN

uuuguckg

tu and ,

,,)(

NNgNckgtC ,)( ∇−+=∂∂

⇓

Evolution of distance function ⇔ Evolution of the curve

Active contour: example

Original image x Gradient ∇x g(∇x) Initial contour and distance function u

Final distance function u Final contour


27

Summary (I)

Level set representation of images:Fundamental geometric representation leading to

Mathematical morphology (historical evolution)

Diffusion process on images:

Evolution of level curve ⇔ PDE proportional to image gradient

Classical diffusion processes:Flow of constant velocity ⇔ Erosion, Dilation

Isotropic diffusion ⇔ Heat equation (on the image)


⇔ Geometrical heat equation (on the level curve)

⇔ Curvature motion

Shock filters

⇔ “Stable” inverse of the heat equation

⇔ Combination of erosion / dilation depending on the laplacian sign

Summary: Transition-based segmentation

3 main steps for segmentation:Simplification, feature extraction and decision

Decision can be based on discontinuity (transition-based) or homogeneity (region-based)

Transition-based segmentation:Estimate transitions: gradient (may be linear or not)

Decision:Thresholding: difficult because of threshold definition, noise and open contour

Active contours:

Curve evolution to minimize the length “weighted” by the gradient

Gradient descent leads to a combination of internal (smoothing) forces and external force (attraction to data)

Use an implicit representation of the curve: level set implementation

image representation and processing with level setsarantxa.ii.uam.es/.../20071120psalembier.pdf ·...

Documents