1 ece 738 paper presentation paper: active appearance models author: t.f.cootes, g.j. edwards and...

1

ECE 738 Paper presentation

Paper: Active Appearance Models

Author: T.F.Cootes, G.J. Edwards and C.J.Taylor

Student: Zhaozheng Yin

Instructor: Dr. Yuhen Hu

Date: Feb. 14 2005

Note: some slides copyrighted by the original authors

2

Papers

• T.F.Cootes, G.J. Edwards and C.J.Taylor. "Active Appearance Models", IEEE PAMI, Vol.23, No.6, pp.681-685, 2001

• T.F.Cootes, G.J. Edwards and C.J.Taylor. "Active Appearance Models", in Proc. European Conference on Computer Vision 1998 Vol. 2, pp. 484-498, Springer, 1998. (Best paper prize)

3

Flexible models

• Statistical Shape Models• Active Shape Models (ASM)• Combined Appearance Models• Active Appearance Models (AAM)

Shape model ASM AAM

4

Flexible models• Shape

Shape is the geometric information invariant to a particular class of transformations (translation+rotation+scaling)

• Appearance

5

Applications

• Flexible models can be used to:– Locate examples of structures in new images– Classify objects found in images– Filter images to pick out interesting features

• Practical problems:Face recognition, industrial inspection and medical image analysis

6

Flexible models

• Statistical Shape Models• Active Shape Models (ASM)• Combined Appearance Models• Active Appearance Models (AAM)

Shape model ASM AAM

7

Statistical Shape Models

•Given sets of training images build a statistical shape model•Each shape in the training set is represented by a set of n labeled landmark points, which must be consistent from one shape to the next. Ex. The outline of a hand is represented by 72 labeled points

1

23

4

5

6

8


•Each shape is represented by a 2n*1 vector

•Using Principal Component Analysis (PCA) or eigen analysis, the shape model is

where P is a 2n*t matrix whose columns are unit vectors along principle axes or basis vectorb is a t*1 vector of shape parameters or weight

Ex. Vary the first three parameters of the shape vector, b, one at a time

),...,,...,( 1,1 nn yyxxX

bPXX

9

Aligning Two Shapes

• Procrustes analysis:– Find transformation which minimizes

– Resulting shapes have • approximately the same scale and orientation

221 |)(| xx T

10

Aligning a Set of Shapes

• Generalized Procrustes Analysis– Find the transformations Ti which minimise

– Where

– Under the constraint that

2|)(| iiT xm

)(1

iiTn

xm

1|| m

11

Dimensionality Reduction

11bpxx

1b

xx

1p

12

Dimensionality Reduction

• Data lies in subspace of reduced dim.

• However, for some t,

i

i

nnbb ppxx 11

tjb j if 0

t

) is of (Variance jjb

13


•Another example

Shape of the facial structures with 68 points

14

Flexible models

• Statistical Shape Models

• Active Shape Models (ASM)

• Combined Appearance Models

• Active Appearance Models (AAM)

15

Active Shape Models

• Suppose we have a statistical shape model– Trained from sets of examples

• How do we use it to interpret new images?

• Use an “Active Shape Model”

• Iterative method of matching model to image

16

Active Shape Models (ASM)

• Assume we have an initial estimate for the pose and shape parameters (eg the mean shape).

bPXX

17

Active Shape Models (ASM)• Iterative algorithm

– Look along normals through each model point to find the best local match for the model of the image appearance at that point (eg strongest nearby edge)

– Update the pose and shape parameters to best fit the model instance to the found points

– Repeat until convergence

Initial pos 5th iterations convergence

18

ASM Search Overview

• Local optimization

• Initialize near target– Search along profiles for best match,X’– Update parameters to match to X’.

),( ii YX

19

Active Shape Models (ASM)• Performance improvement

(Multi-resolution implementation/coarse-to-fine approach)

we start searching on a coarse level of a Gaussian image pyramid, and progressively refine. This leads to much faster, more accurate and more robust search.

20

Flexible models





21

Combined Appearance Models

• Idea:

Statistical Shape Model models the shape change of an object construct a similar statistical model to represented the intensity variation across a region

(Think: skeleton and muscle)

22


• Method:

Given a set of training images, labeled with land mark points, we can use image warping to deform each image so that the object has the mean shape, then build a statistical model of the grey-levels across the object.

Ex. The central image is the mean

23

Building Appearance Models

• For each example extract shape vector

• Build statistical shape model,

Shape, x = (x1,y1, … , xn, yn)T

ssbPxx

24

Building Appearance Models

• For each example, extract texture vector

Shape, x = (x1,y1, … , xn, yn)T

Texture, gWarp tomeanshape

25

Warping texture

• Problem:– Given corresponding points in two images,

how do we warp one into the other?

• Two common solutions1. Piece-wise linear using triangle mesh

2. Thin-plate spline interpolation

26

Interpolation using Triangles

Region of interest enclosed by triangles.

Moving nodes changes each triangle

Just need to map regions between two triangles

),( :points Control ii yx )','( :points Warped ii yx

27

Barycentric Co-ordinates

cbax

a b

c

x

''' cbax '

'a

'b

'c

'x

1

10 and 10

if triangle theinside is

βα

x

28

Building Texture Models

• For each example, extract texture vector

• Normalise vectors (as for eigenfaces)

• Build eigen-model

Texture, g

Warp tomeanshape

ggbPgg

29

Combined Models

• Shape and texture often correlated– When smile, shadows change (texture) and

shape changes

• Learning this correlation leads to more compact (and specific) model

30


ggbPgg ssbPxx

cQgg

cQxx

g

s

Varying c changes both

shape and texture

In this paper:

31

Flexible models





32

Active Appearance Models

• Suppose we have a statistical appearance model– Trained from sets of examples

• How do we use it to interpret new images?

• Use an “Active Appearance Model”

• Iterative method of matching model to image

Interpreting Images

Place model in image

Measure Difference

Update Model

Iterate

34

Active Appearance Models (AAM)

• AAM vs. ASM

The Active Appearance Model (AAM) is a generalization of the widely used Active Shape Model approach, but uses all the information in the image region covered by the target object, rather than just that near modeled edges.

35

Quality of Match

• Residual difference:

• p : all parameters, eg

• Ideally find and optimize p(p|r)

• Cannot usually know p(r)

)()()( pIpIpr imm

)()()( prprp TE

),,,,( sYX cccp

)(

)()|()|( :rule Bayes

r

pprrp

p

ppp

36

Quality of Match

• Usually attempt to maximize

(1)

• This is equivalent to maximizing

(2)

• Which is equivalent to minimizing

(3)

)()|( ppr pp

)(log)|)((log pppr pp

)(log))((log)( pprp ppE

37

Quality of Match

• Assuming independent Gaussian noise:

(1)

(2)

(3)

22

)()(exp))((

r

T

p

prprpr

constpr

T

22

)()())((log

prpr

pr

constpEr

)(log2

|)(|)(

2

2

ppr

p

38

Quality of Match

• If we assume all parameters equally likely (within certain limits)

(1)• Thus we need to find the parameters which

minimize the sum of squares of residuals,(2)

constEr

2

2

2

|)(|)(

pr

p

constp )(p

2|)(|)( prp E

39

Learning the Relationship

• For each of a training set

– find best fit given landmarks, p

– randomly perturb p by p and measure

(in model

frame)

)()()( ppIpIppr imm

40

More Analytic Approach

pp

rprppr δ

)()(

rrEpp Tor , minimize To )E(

TT

p

r

p

r

p

rR

1

)(δ pRrp

Taylor expansion:

Final result in the paper:

where

41

AAM Algorithm

• Initial estimate Im(p)

• Start at coarse resolution

• At each resolution

– Measure residual error, r(p)

– predict correction p = -Rr

– p p - p

– repeat to convergence

42

Active Appearance Models (AAM)

• ExampleA face model built from 400 images. The figure below shows frames from an AAM search for a new face, each starting with the mean model displaced from the true face centre.

Figure: Multi-Resolution search from displaced position

43

Problems

• Automatic Model Building– Require correspondences across a set– Hard to achieve reliably

• Reliable measure of quality of fit– Necessary for good matching– Essential for detection

• Model initialization– Getting good initial estimate can be hard– 10% percent of the image size and scale

44

AAM Summery

• ParametersAn AAM contains a statistical model of the shape and grey-level appearance of the object of interest.

• Goals

Matching to an image involves finding model parameters which minimize the difference between the image and a synthesized model example, projected into the image. The potentially large number of parameters makes this a difficult problem.

45

AAM Summery

• Iterations

We observe that displacing each model parameter from the correct value induces a particular pattern in the residuals. In a training phase, the AAM learns a linear model of the relationship between parameter displacements and the induced residuals. During search it measures the residuals and uses this model to correct the current parameters, leading to a better fit.

46

Some other group’s AAM research

• Simon Baker and Iain Matthews at CMU are doing some wonderful work on analyzing and improving the AAM update algorithm. They have gone on to develop fast tracking, model building and 3D reconstruction algorithms. Awesome.

1 ece 738 paper presentation paper: active appearance models author: t.f.cootes, g.j. edwards and...

Documents