Face Alignment Using Cascaded Boosted Regression

Active Shape Models

Michael Dixon

Faces in computer vision

• What problems do people work on?– Detection– Alignment– High-level analysis

• Face recognition• Facial expression

recognition• Face tracking

2

Face alignment

• Given an image of a face and an initial guess, localize key facial features

• Approaches– Active Shape Model,

1992– Boosted Regression

ASM, 2007

3

Training data

• Given many examples, learn a model

1500 hand-labeled face images

4

The Active Shape Model framework

5Shape FeaturesInput image

The Active Shape Model framework

6Shape FeaturesInput image

Shape model

• Given many examples of a shape

• Learn a set of constraints on allowable shapes

7

Learning a shape model

• Represent as a linear subspace

Mean face shape

Principal variations from the mean8

The Active Shape Model framework

Shape FeaturesInput image9

Feature model

• Given a patch near a facial feature, predict the correct position of that feature

Given Predict

10

Learning a feature model

• Generate training examples with known feature positions

• Train a regression model to predict the correct displacement

11

Boosted regression

• Goal: Learn a function to predict a set of target values

• Boosting builds a strong regression model from many weak models– Evaluate a large pool of possible weak regression

functions– Select the function with the lowest error and add

it to the strong regression model– Update the target values and repeat

12

Weak regression model

13

bthaf mm Weak regression functionHaar wavelet features

hm =The sum of all pixel values under the white box minus the sum of all pixel values under the black box

Haar wavelet response

Weak regression example

bthaf mm

fit weak regression function to data

disp

lace

men

thm

a = -0.027b = 0.012t = 21.7

disp

lace

men

thm

14

Strong regression model

15

Predicted displacement

Gro

und-

trut

h di

spla

cem

ent

25 weak regression functions combined into a strong regression function

The Active Shape Model framework

• Combining the shape and feature models

Shape Features Alignment16

Fitting using Boosted Regression ASM

• Initialize the feature positions

• Iteratively– Predict feature

positions using regression model

– Constrain to fit the shape model

– Update feature positions

17

Limitations of the previous work

• How often does the boosted regression feature model improve on the initial estimate?

Some improvement

Significantimprovement

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

10

20

30

40

50

60

70

80

90

100

Displacement (in pixels)

Perc

ent t

hat i

mpr

oved

Improved by at least 50%

Any improvement

18

Predicted position vs. actual position

Accuracy trade-off

• Regression model can’t accurately predict both large and small displacements

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

10

20

30

40

50

60

70

80

90

100

Displacement (in pixels)

Some improvement

Significantimprovement

Perc

ent t

hat i

mpr

oved

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

10

20

30

40

50

60

70

80

90

100

Displacement (in pixels)

Some improvement

Significantimprovement

Perc

ent t

hat i

mpr

oved

Model trained on large displacements Model trained on small displacements

19

Proposed solution

• Train multiple models (coarse to fine) and apply them in sequence

Coarse regression model

Fine regression model

Displacement (in pixels)

Perc

ent t

hat i

mpr

oved

20

Cascaded Boosted Regression ASM

21

FaceDetector

FaceDetector

Boosted Regression ASM15 iterations

Stage 15 iterations

Stage 25 iterations

Stage 35 iterations

Cascaded Boosted Regression ASM

Image

Image

Alignment

Alignment

Learning an alignment cascade• Train a new stage of the

cascade using the output of the previous stage– Use a face detector as the

initial stage

• For each stage– Measure error distribution of

each feature– Generate training examples

from the error distribution– Train new feature models– Align all images using the

updated model to get a new error distribution

22

Qualitative comparisonBo

oste

d Re

gres

sion

ASM

Casc

aded

Boo

sted

Re

gres

sion

ASM

23

Quantitative evaluation• Error metric:

where:– di is the distance between the estimated position

and the ground truth position of the ith point– s is the inter-ocular distance

• An alignment is only as good as its worst point

s

de i

i 20,,1max

Alignment vs.Ground-truth

24

Results

• Evaluated on 500 unseen test images

0 0.05 0.1 0.15 0.2 0.25 0.30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Cum

ulati

ve e

rror

dis

trib

ution

Alignment error

CascadedStandardAverage face

25

73%

19%

3%

Results

• Alignment accuracy after each stage

26

0 0.05 0.1 0.15 0.2 0.25 0.30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Stage 1Stage 2Stage 3Cu

mul

ative

err

or d

istr

ibuti

on

Alignment error Stage

0 1 2 30

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Med

ian

alig

nmen

t err

or

Conclusions

• Boosted Regression ASMs are a newly proposed method for performing face alignment

• Training a cascade of Boosted Regression ASMs can significantly improve alignment accuracy

27

Questions?

28

29