cse527-l14

This lecture: models of shape and texture

Active Shape Models Eigenfaces Active Appearance Models Class Projects

1 Automatic Erosion Detection in RA Monitoring | Georg Langs 1 1. Active Shape Models

Adjust to texture Fit to shape model

Example: bone contours

2 2

Task: localize anatomical structures

Task: Analyze a hand radiograph

3 Automatic Erosion Detection in RA Monitoring | Georg Langs

Task: Analyze a hand radiograph


Assume: we are looking for proximal phalanx 2

PP3

PP2

PP4

PP5

MC2

MC3

MC4

MC5

MP5

MP4

MP3

MP2

Analyzing a hand radiograph


We have a priori knowledge about the typical appearance: e.g. bone shapes and texture

PP2

How can we represent this knowledge? How can we exploit it?

How to capture a priori knowledge?


Simple shape representation: shape consists of a set of points defined by their individual coordinates very powerful very general very unspecific

Alternative: capture common properties of the bone find a representation that is restricted to plausible bones.

Prior knowledge: first works

Snakes

Deformable templates

7

A.L.Yuille, D. Cohen, and P.Hallinan, Feature extraction from faces using deformable templates, CVPR 89, IJCV 92.

Data-driven

Prior knowledge

M. Kass, A. Witkin and D. Terzopoulos, `Snakes, ICCV 1987, IJCV 88

Current work: Statistical Shape Models

8 1. Active Shape Models

Each example is represented by a vector containing the coordinates of the landmarks.

Learning: Model Acquisition Inference: Model Fitting

Model learning

How can we learn what a bone looks like?

Objectives: Expressiveness (model can account for all bones) Compactness (as simple as possible)

Method: Principal Components Analysis (PCA)

9

To build a statistical model of the data we need many examples.

Bone shapes: vectors in

Goal: project data onto a low-dimensional linear

subspace that best explains their variation.

The space of all bone shapes

PCA: feature space


PCA: probability distribution


Example with high probability

Example with low probability

New subspace: `better coordinate system


New coordinates reflect the distribution of the data. Few coordinates suffice to represent a high dimensional vector They can be viewed as parameters of a model

Mean

Principal Component Analysis

Given: N data points x1, ,xN in Rd We want to find a new set of features that are linear

combinations of original ones:

u(xi) = uT(xi )

(: mean of data points)

What unit vector u in Rd captures the most variance of the data?

Principal Component Analysis Variance of projection on u:

Projection of data point

Covariance matrix of data

The direction that maximizes the variance: the eigenvector associated with the largest eigenvalue of

N

N

Direction: Unit norm vector

Principal component analysis

The direction that captures the maximum covariance of the data is the eigenvector corresponding to the largest eigenvalue of the data covariance matrix

Furthermore, the top k orthogonal directions that capture the most variance of the data are the k eigenvectors corresponding to the k largest eigenvalues

Covariance matrix:

Uncorrelated coordinates: diagonal covariance

Covariance matrix reminder

Height, Income Height, Weight

PCA: decorrelation of random variables

PCA: projection onto eigenvectors of covariance matrix

18

Dimensionality reduction by using only leading eigenvectors

PCA: step by step

1. Compute the empirical mean and subtract it from the data, and compute empirical covariance matrix

2. Compute eigenvalue decomposition of the covariance matrix resulting in

3. Retain only the k eigenvectors with highest corresponding eigenvalues.

1. Active Shape Models

PCA: step by step

4. Given a new input vector

5. project it into the eigenspace

6. resulting in the coefficient vector or parameter vector


Only first k eigenvectors

Using PCA to model shape


+ + + = +

Active shape models (ASM)

A set of training examples (images) A set of landmarks, that are present on

all images Build a statistical model of shape

variation (PCA) Build a statistical model of the local

texture (PCA) Use the model for the search in a new

image


ASM: model building

23

m Landmarks represented as


Training set: Instances vary in - shape - texture - orientation - scale - position

pose

ASM: alignment of examples


s1

s2

scale center (translate) rotate center of

gravity

The alignment removes rotation, scale and translation differences Minimizes Procrustes distance:

ASM: alignment of examples

Center all training shapes Choose one example as initial mean shape and scale

Align all shapes to by minimizing the Procrustes distance

Recalculate the mean shape from the aligned set.

Iterate until the mean shape converges


Given: a set of aligned shapes Perform PCA on the set of shapes, resulting in

a set of Eigenvectors and corresponding Eigenvalues

Each shape can be reconstructed by

ASM: shape model


ASM search


Adjust to texture Fit to shape model

Initialize

Update the instance according to shape model and pose

Initialize

Generate model pose instance Find pose parameters to map to Project into the model coordinates Update model parameters to match Apply constraints

ASM search: shape


ASM search

29 1. Active Shape Models 29

Modelling the appearance

To peform a search with the shape model, we have to capture the appearance, too. Local appearance at the landmark positions:

Active shape model (ASM)

Appearance of the enclosed texture Active appearance model (AAM)





= +

+ w1u1+w2u2+w3u3+w4u4+ ^ x =

Face Recognition Problem

Face Recognition Problem

[Sinha and Poggio 2002]

Challenges: Image Variability

Expression Illumination Pose

Short Term

Facial Hair Makeup Eyewear

Hairstyle Piercings Aging

Long Term

Invariance Problem

Images as Points in Euclidean Space

Let an n-pixel image to be a point in an n-D space, x Rn.

Each pixel value is a coordinate of x.

x 1

x 2

x n

Face Recognition: Euclidean Distances

D2 > 0

D3 = 0

D1 > 0

~

Face Recognition: Euclidean Distances

D2 > 0

D3 > D1 or 2

D1 > 0

[Hallinan 1994] [Adini, Moses, and Ullman 1994]

Nearest Neighbor Classifier

Variations on distance function (e.g. L1, robust distances)

Multiple templates per class- perhaps many training images per class.

Expensive to compute k distances, especially when each image is big (N dimensional).

May not generalize well to unseen examples. Some solutions:

Bayesian classification Dimensionality reduction

Image Variability: Appearance Manifolds

x2

x1

xn

Lighting x Pose [Murase and Nayar 1993]

Image Variability: From Few to Many

x2

x1

xn

Lighting x Pose

[Georghiades, Belhumeur, and Kriegman 1999]

Real

Synthetic

Modeling Image Variability

Can we model illumination and pose variability in images of a face?

Yes, if we can determine the shape and texture of the face. But how?

Appearance modelling for faces When viewed as vectors of pixel values, face images are

extremely high-dimensional 100x100 image = 10,000 dimensions

Very few vectors correspond to valid face images

Original coordinates are not revealing about face properties We want to model the subspace of face images Same story as shape modelling.

Eigenfaces: Key idea Assume that most face images lie on

a low-dimensional subspace determined by the first k (k

Eigenfaces example Training images x1,,xN

Eigenfaces example Top eigenvectors: u1,uk

Mean:

Eigenfaces example Principal component (eigenvector) uk

+ 3kuk

3kuk

Eigenfaces example

Face x in face space coordinates:

=

Eigenfaces example

Face x in face space coordinates:

Reconstruction:

= +

+ w1u1+w2u2+w3u3+w4u4+

=

^ x =

Recognition with eigenfaces Process labeled training images:

Find mean and covariance matrix Find k principal components (eigenvectors of ) u1,uk Project each training image xi onto subspace spanned by

principal components: (wi1,,wik) = (u1T(xi ), , ukT(xi ))

Given novel image x: Project onto subspace:

(w1,,wk) = (u1T(x ), , ukT(x )) Check reconstruction error x x to determine whether image

is really a face Classify as closest training face in k-dimensional subspace

^

M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991

Challenges addressed by Eigenfaces


Short Term



Long Term

Limitations

Global appearance method: not robust to misalignment, background variation

Linear Discriminant Analysis

We wish to choose linear functions of the features that allow good discrimination. Assume class-conditional covariances are the same Want linear feature that maximises the spread of class means for

a fixed within-class variance



Short Term



Long Term

DArcy Thompson, On Growth and Form (1917)

Modelling Shape Variability via Deformations

Modelling Shape Variability via Deformations

DArcy Thompson, On Growth and Form (1917)

Active Appearance Models

Appearance = Shape + Texture Shape: deformation from `template domain to

instance domain

Texture: intensity (or color) patch of an image in the template domain.

I(S (x)) = T (x)

X S (X )

T emplate Ins tance

Shape-free texture

An attempt to eliminate texture variation due to shape

Given instance I and a reference template T we warp our image so that points of I move into the corresponding points of T

warped to becomes

T I

AAM training

Training set: annotated grayscale images

Shape: displacements

from mean shape

Texture (shape-free)

Model of shape Model of texture

PCA PCA Optional: Interpolate to recover dense deformation field

Active Appearance Models

Shape:

Appearance:

Synthesis:

I(S (x)) = T (x)

X S (X )

T emplate Ins tance

AAM model Shape Eigenvectors

Texture Eigenvectors

Playing with the Parameters

First two modes of shape variation First two modes of gray-level variation

First four modes of appearance variation

Active Appearance Model fitting

Minimize reconstruction error

How? Alternate between estimating s and t

For t: projection of deformed image onto PCA basis For s?

Given: 1) an appearance model, 2) a new image, 3) a starting approximation

Find: the best matching synthetic image

AAM parameter estimation: shape

Iterative scheme (Newton-Raphson minimization)

72

Active Appearance Model Search (Results)

Model fitting + Image segmentation AAM fitting

Variational Segmentation

EM formulation E-step: Assign data to models M-step: Estimate model parameters

Joint segmentation & recognition E-step: segmentation M-step: deformable model fitting

M-step E-step Atomic Regions

Model fitting + Image segmentation



Short Term



Long Term

Modeling Image Variability: 3-D Faces

Laser Range Scanners Stereo Cameras Structured Light

Photometric Stereo

What can we do with 3d shape?

[Blanz and Vetter 1999, 2003]

Building a Morphable Face Model


3-D Morphable Models


3D Morphable models

Recover Shape

Synthesize new views

Synthesize new expressions

82

Rough manual initialization Gradient descent to minimize reconstruction

error functional

And then

3-D Morphable Models fitting

83

3D AAM for face tracking

CMU group: I. Matthews, S. Baker, R. Gross (230 Frames per second)

84

3D AAM for face tracking

Playing with Facial Attributes Several classes of attributes are modeled: Facial expressions (smile, frown) Individual characteristics (double chin, hooked

nose, maleness) Distinctiveness

Manipulating Facial Attributes via Deformations For each face in the database, two scans are recorded:

Sneutral, and Sexpression. The difference vector DS = Sexpression - Sneutral is saved and

later on simply added to the 3D reconstruction of the input image.

cse527-l14

Documents

models of shape

active shape models

active shape models

shape model example

statistical shape models

project data

data covariance matrix

automatic erosion detection