image analysis active contour models (snakes)

University of Ioannina - Department of Computer Science and Engineering

Christophoros Nikou

[email protected]

Images taken from:

Computer Vision course by Kristen Grauman, University of Texas at Austin.

Image Analysis

Active contour models (snakes)

mailto:[email protected]

2

C. Nikou – Image Analysis (T-14)

a.k.a. active contours, snakes

Given: initial contour (model) near desired object

[Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]

Figure credit: Yuri Boykov

Deformable contours

3


Given: initial contour (model) near desired object

Goal: evolve the contour to fit exact object boundary

Main idea: elastic band is

iteratively adjusted so as to

• be near image positions with

high gradients, and

• satisfy shape “preferences” or

contour priors

[Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]

Deformable contours (cont.)

4


Image from http://www.healthline.com/blogs/exercise_fitness/uploaded_images/HandBand2-795868.JPG

Deformable contours (cont.)

5


initial intermediate final

Both are useful for shape fitting but they differ signifficanlty:

Hough

Rigid model shape

Single voting pass can

detect multiple instances

Deformable contours

Prior on shape types, but shape

iteratively adjusted (deforms)

Requires initialization nearby

One optimization “pass” to fit a

single contour

Deformable contours vs Hough

6


• Some objects have similar basic form but

some variety in the contour shape.

Deformable contours (why?)

7


• Non-rigid,

deformable

objects can

change their

shape over

time, e.g. lips,

hands…


8


• Non-rigid,

deformable

objects can

change their

shape over

time, e.g. lips,

hands…


9


• Non-rigid, deformable objects can change their shape

over time.


10


• Representation of the contours

• Defining the energy functions

– External

– Internal

• Minimizing the energy function

• Extensions:

– Tracking

– Interactive segmentation

Aspects to consider

11




– External

– Internal


• Extensions:

– Tracking


Aspects to consider

12


• We’ll consider a discrete representation of the contour,

consisting of a list of 2D point positions (“vertices”).

),,( iii yx

1,,1,0 ni for

• At each iteration, we’ll have the

option to move each vertex to

another nearby location (“state”).

),( 00 yx

),( 1919 yx

Representation

13


initial intermediate final

How should we adjust the current contour to form the new

contour at each iteration?

• Define a cost function (“energy” function) that says how

good a candidate configuration is.

• Seek next configuration that minimizes that cost function.

Fitting deformable contours

14




– External

– Internal


• Extensions:

– Tracking


Aspects to consider

15


The total energy (cost) of the current snake is

defined as:

externalinternaltotal EEE

A good fit between the current deformable contour

and the target shape in the image will yield a low

value for this cost function.

Internal energy: encourages prior shape preferences:

e.g., smoothness, elasticity, particular known shape.

External energy (“image” energy): encourages contour

to fit on places where image structures exist, e.g., edges.

Energy function

16


• It measures how well the curve matches

the image data

• It “Attracts” the curve toward different

image features

– Edges, lines, texture gradient, etc.

External energy

17


Magnitude of gradient - (Magnitude of gradient)

2 2( ) ( )x yG I G I

How do edges affect the shape

of the rubber band?

Think of external energy from

image as gravitational pull

towards areas of high contrast

External energy (cont.)

2 2( ) ( )x yG I G I

18


• Gradient images and

• External energy at a point on the curve is:

• External energy for the whole curve:

),( yxGx ),( yxGy

)|)(||)(|()( 22 yxexternal GGE

21

0

2 |),(||),(| iiy

n

i

iixexternal yxGyxGE

External energy (cont.)

19


• External image can instead be taken from the distance

transform of the edge image.

original - gradient

Distance transform

edges

Value at (x,y) indicates how far that position is

from the nearest edge point

Distance transform

20


What are the underlying

boundaries in this fragmented

edge image?

And in this one?

Internal energy: intuition

21


A priori, we want to favor smooth shapes, contours with

low curvature, contours similar to a known shape, etc.

to balance what is actually observed (i.e., in the gradient

image).

Internal energy: intuition (cont.)

22


For a continuous curve, a common internal energy term

is the “bending energy”.

At some point v(s) on the curve, this is:

Tension,Elasticity

Stiffness,Curvature

sd

d

ds

dsEinternal 2

2

))((

22

Internal energy

23


• For our discrete representation,

• Internal energy for the whole curve:

10),( niyx iii

i1ivds

d

11112

2

2)()( iiiiiiids

d

1

0

2

11

2

1 2n

i

iiiiiinternalE

…

Note these are derivatives relative to position---not spatial

image gradients.

Why do these reflect tension and curvature?

Internal energy

24


(1,1) (1,1)

(2,2)

(3,1)(3,1)

(2,5)

2

11 2)( iiiicurvature vE 2

11

2

11 )2()2( iiiiii yyyxxx

Example (curvature)

25


• Current elastic energy definition uses a discrete estimate

of the derivative:

What is the possible problem

with this definition?

2

1

1

0

2

1 )()( ii

n

i

ii yyxx

1

0

2

1

n

i

iielasticE

Penalize elasticity

26


• Current elastic energy definition uses a discrete estimate

of the derivative:

1

0

2

1

n

i

iielasticE

21

0

2

1

2

1 )()(

n

i

iiii dyyxx

where d is the average distance between

pairs of points – updated at each iteration.

Instead, we may

employ:

Penalize elasticity (cont.)

27


• The preferences for low-curvature, smoothness help

deal with missing data:

Illusory contours found!

Missing data

28


• If the object is a smooth variation of a known shape, we can use a term that will penalize deviation from that shape:

where are the points of the known shape.

1

0

2)ˆ(n

i

iiinternalE

}ˆ{ i

Extend the internal energy by a

shape prior

29


externalinternaltotal EEE

1

0

2

11

2

1 2n

i

iiiiiinternal dE

21

0

2 |),(||),(| iiy

n

i

iixexternal yxGyxGE

Total energy

30


large small medium

• e.g., weight controls the penalty for internal elasticity

Fig from Y. Boykov

Total energy (cont.)

31


• A simple elastic snake is defined by:

– A set of n points,

– An internal energy term (tension,

bending, plus optional shape prior)

– An external energy term (gradient-

based)

• To segment an object:

– Initialize in the vicinity of the object

– Modify the points to minimize the total

energy

Recap

32




– External

– Internal


• Extensions:

– Tracking


Aspects to consider

33


• Several algorithms have been proposed to

fit deformable contours.

• We’ll look at:

– Greedy search

– Dynamic programming (for 2D snakes)

– Calculus of variations

Energy minimization

34


• For each point, define a search

window around it and move to where

energy function is minimal

– Typical window size, e.g., 5 x 5 pixels

• Stop when a predefined number of

points have not changed in last

iteration, or after a maximum number

of iterations

• Note:

– Convergence not guaranteed

– Need decent initialization

Energy minimization (greedy)

35


1v

2v3v

4v6v

5v

• The Viterbi algorithm.

• Iterate until optimal position for each point is the center

of the box, i.e., the snake is optimal in the local search

space constrained by boxes.

[Amini, Weymouth, Jain, 1990]

Energy minimization (dynamic

programming)

36


1

1

11 ),(),,(n

i

iiintotal EE

• Possible because snake energy can be rewritten as a

sum of pair-wise interaction potentials:

• Or sum of triple-interaction potentials.

1

1

111 ),,(),,(n

i

iiiintotal EE

Dynamic programming

37


21

1

2

11 |),(||),(|),,,,,( iiy

n

i

iixnntotal yxGyxGyyxxE

2

1

1

1

2

1 )()( ii

n

i

ii yyxx

1

1

2

1 ||)(||),,(n

i

intotal GE

1

1

2

1 ||||n

i

ii

),(...),(),(),,( 113222111 nnnntotal vvEvvEvvEE

2

1

2

1 ||||||)(||),( iiiiii GE where

Re-writing the above with : iii yxv ,

Snake energy: pair-wise interactions

38


),(...),(),( 11322211 nnntotal vvEvvEvvEE

),( 322 vvE),( 211 vvE

Main idea: determine the optimal position (state) of

predecessor, for each possible position of self.

Then backtrack from best state for last vertex.

1v 2v 3v 4v 5v

Viterbi algorithm

4 4 5( , )E v v3 3 4( , )E v v

39


),( 44 nvvE),( 433 vvE

)3(3E

)(3 mE )(4 mE

)3(4E

)2(4E

)1(4E

)(mEn

)3(nE

)2(nE

)1(nE

)2(3E

)1(3E

)(2 mE

)3(2E

),(...),(),( 11322211 nnntotal vvEvvEvvEE

),( 322 vvE

)1(2E

)2(2E

),( 211 vvE

0)1(1 E

0)2(1 E

0)3(1 E

0)(1 mE

states

1

2

…

m

vert

ices

1v 2v 3v 4v nv

)( 2nmOComplexity: vs. brute force search ____?

Viterbi algorithm (cont.)

40


),(...),(),( 11322211 nnn vvEvvEvvE

DP can be applied to optimize an open ended snake

For a closed snake, a “loop” is introduced into the total energy.

1n

),(),(...),(),( 111322211 vvEvvEvvEvvE nnnnn

1

n

2

1n

3 4

Work around:

1) Fix v1 and solve for rest .

2) Fix an intermediate node at

its position found in (1),

solve for rest.

Viterbi algorithm (cont.)

41


Calculus of variations

• Given a 1D function ( ) :[0,1] ,u x

the basic problem is to minimize a given

energy (functional) w.r.t u(x):

1

0( ) ( , , )E u F x u u dx

with boundary conditions: (0) , (1)u a u b

2:F and given by the problem.

42


Calculus of variations (cont.)

• First variation:

0 0 ( ) ( ) 0,E

E E u v E uu

:[0,1] , (0) , (1)v v a v b

v being a small increment satisfying the

boundary conditions.

1 1

0 0( ) ( ) ( , , ) ( , , )E u v E u F x u v u v dx F x u u dx

1

0[ ( , , ) ( , , )]F x u v u v F x u u dx

43



• Using Taylor series approximation:

1

00 [ ( , , ) ( , , )] 0E F x u v u v F x u u dx

( , , ) ( , , ) ...F F

F x u v u v F x u u v vu u

the energy function becomes:

1

00 0

F FE v v dx

u u

44



• Integrating the second integral by parts:

Since this holds for all v, the Euler equation is:

1 1

0 00 0

F FE v dx v dx

u u

11 1 1

0 0 00

0 0F F d F F d F

v dx v v dx v dxu u dx u u dx u

0F d F

u dx u

45



• In a similar form, for the energy:

we obtain the Euler equation:

1

0( ) ( , , , )E u F x u u u dx

2

20

F d F d F

u dx u dx u

46



• Analogous is the 2D problem:

whose Euler equation is

2 21 1

2 20 0( ) ( , , , , , )

u u u uE u F x y dxdy

x y x y

2 2

2 20

x y xx yy

F d F d F d F d F

u dx u dy u dx u dy u

47


Back to the snake energy

and the Euler equation is:

221 2

2

0

( ) ( ) ( ) ( )internal image

dv d vE v E E a b cg s ds

ds ds

The image force is of the form:2

1( )

1 ( )g s

I s

2 4

2 40

d v d va b c g

ds ds

48


Iterative scheme

• There is no closed form solution.

• We consider that we have an initial

solution.

• An iterative Newton-type scheme is

applied until stability is achieved:

2 4

2 4

dv d v d va b c g

dt ds ds

49


Iterative scheme (cont.)

• Temporal discretization:

,

Tt t t t t tdv v v x x y y

dt

• Spatial discretization: ,

T

dv v v

ds x y

2 4

1 1 2 1 1 22 4, 2 , 4 6 4 6i i i i i i i i i i i

v v vx x x x x x x x x x

x x x

2 4

1 1 2 1 1 22 4, 2 , 4 6 4 6i i i i i i i i i i i

v v vy y y y y y y y y y

y y y

50


Iterative scheme (cont.)

1

1

t t T

x

t t T

y

I A c

I A c

x x g

y y g

• Plugging the previous results into the main

equation, the problem is solved independently

in each spatial direction:

0 1 0 1 0 0 0 0,... , ,... , ( , )... , ( , )...x y

N N

T T T Tt t t t t t t t t t t t t t

x yx x y y g x y g x y

x y g g

circulant 6 2 (4 2 ) 0 0 ... 0 (4 2 )N N

A b a b a b b b a

51




– External

– Internal


• Extensions:

– Tracking


Aspects to consider

52


1. Use final contour/model extracted at frame t as

an initial solution for frame t+1

2. Evolve initial contour to fit exact object boundary

at frame t+1

3. Repeat, initializing with most recent frame.

Tracking Heart Ventricles (multiple frames)

Tracking by deformable contours

53


Visual Dynamics Group, Dept. Engineering Science, University of Oxford.

Traffic monitoringHuman-computer interactionAnimationSurveillanceComputer assisted diagnosis in medical imaging

Applications:

Tracking by deformable contours

(cont.)

http://www.robots.ox.ac.uk/~vdg/~vdg/

54


3D active contours

55


• May over-smooth the boundary

• It cannot follow topological changes of

objects

Limitations

56


• External energy: snake does not really “see” object

boundaries in the image unless it gets very close to it.

image gradientsare large only directly on the boundary

I

Limitations (cont.)

• Solution: Balloon force or gradient

vector flow snake

57


Gradient vector flow

• Diffusion of the edge force.

• Compute a new force field u(x,y), v(x,y)

minimizing the energy:

2 2 2 2 2 2 2 21( , ) ( ) ( ) ( ) ( )

2x y x y x y x xE u v u u v v g g u g u g dxdy

• Euler equations:

2 2 2

2 2 2

( )( ) 0

( )( ) 0

x y x

x y y

u g g u g

v g g v g

58


Traditional external force field v/s GVF field

Traditional force

GVF force

(Diagrams courtesy “Snakes, shapes, gradient vector flow”, Xu, Prince)

59

C. Nikou – Image Analysis (T-14)5-59

Diffusing Image Gradients

image gradients diffused via Gradient Vector Flow (GVF)

Chenyang Xu and Jerry Prince, 98 http://iacl.ece.jhu.edu/projects/gv

f/

http://iacl.ece.jhu.edu/projects/gvf/

http://iacl.ece.jhu.edu/projects/gvf/

60


More examples

61


3D surfaces

62


Pros:

• Useful to track and fit non-rigid shapes

• Contour remains connected

• Possible to fill in “subjective” contours

• Flexibility in how energy function is defined, weighted.

Cons:

• Must have decent initialization near true boundary, may

get stuck in local minimum

• Parameters of energy function must be set well based on

prior information

Pros and cons

image analysis active contour models (snakes)

Documents