RESEARCH POSTER PRESENTATION DESIGN © 2011

www.PosterPresentations.com

Discriminative Deep Face Shape Model for Facial Point Detection

In this paper, we address the problem of facial

point detection under varying facial expressions

and poses by proposing a discriminative deep

face shape model that is constructed from the

Restricted Boltzmann Machine and its variants.

1. Problem

Yue Wu and Qiang JiRensselaer Polytechnic Institute

Figure 1. Facial point detection. a. The Facial points that

define the face shapes. b. Facial images with detected facial

points.

2. Motivation

Observation:

(1) There exist patterns of face shapes.

(2) The face shape depends on the facial

expressions and head poses.

Motivation:

To increase the accuracy and robustness of

facial feature detection algorithm, a face

shape model that captures the face shape

patterns with varying facial expressions and

poses should be utilized.

3. Discriminative Deep Face Shape Model

Goal:

Build a model to captures the conditional

joint probability 𝑝(𝑥|𝑚) of the ground truth

facial point location 𝑥, given their

measurements 𝑚 from local point detectors.

4. Facial Point Detection Using the Face Shape Model

Model:

• A discriminative model based on Restricted

Boltzmann Machines (Fig. 2(a) and Fig. 3).

• Bottom layer: 𝑚, measurements of point

locations from local pint detectors.

• Middle layer: 𝑥, face shape under varying

expressions and poses. 𝑦, frontal face

shape under corresponding expressions for

the same subjects. (see Fig. 2(b) for some

examples).

• Top layer: sets of hidden nodes ℎ1 and ℎ2.

(a) (b)

Figure 2. a. The proposed discriminative deep face shape

model. It consist of a factorized three-way RBM connecting

nodes 𝑥, 𝑦, and ℎ1. It also include two RBMs that model the

connections among 𝑥, ℎ1 and 𝑚, 𝑥. b. Corresponding frontal

and non-frontal images for the same subjects and expression.

(a) (b)

(c) (d)

Figure 3. Graphical depiction about different parts of the

model. a.b. Factored three way RBM. c.d. RBM models.

Model training:

• Input: Complete data {𝑥𝑐 , 𝑦𝑐 , 𝑚𝑐}𝑐=1𝑁𝐶

, including

the face shape in arbitrary pose and

expression, its measurement, and its

corresponding frontal shape with same

expression. Incomplete data {𝑥𝑖 , 𝑚𝑖}𝑖=1𝑁𝐼

without the frontal face shape.

• Output: Model parameters 𝜃.

• Method:

• Maximal Likelihood learning

• Gradient ascent algorithm. The gradient:

• Use mean-field fixed point equations to

estimate the data dependent terms. Use

the Persistent Markov Chains to estimate

the model dependent term.

𝜃∗ = 𝑎𝑟𝑔𝑚𝑎𝑥𝜃 𝐿 𝜃; 𝐷𝑎𝑡𝑎𝐶 + 𝐿(𝜃; 𝐷𝑎𝑡𝑎𝐼)

𝜕𝐿(𝜃)

𝜕𝜃= −

𝜕𝐸

𝜕𝜃𝑃𝑑𝑎𝑡𝑎𝐶

−𝜕𝐸

𝜕𝜃𝑃𝑑𝑎𝑡𝑎𝐼

−𝜕𝐸

𝜕𝜃𝑃𝑚𝑜𝑑𝑒𝑙

Model inference:

• Input: the measurements 𝑚𝑡 from local point

detectors, and the model parameters 𝜃 that

defines 𝑝(𝑥|𝑚; 𝜃).

• Output: the inferred facial point locations 𝑥∗

• Method: Gibbs sampling.

𝑥∗ = 𝑎𝑟𝑔𝑚𝑎𝑥𝑥𝑝(𝑥|𝑚𝑡)

Figure 3.

Diagram

illustration of

the facial point

detection

algorithm using

the face shape

model. Local

point detection

and shape

refinements

using the

proposed model

are performed

iteratively.

5. Experiments

• Evaluate the proposed facial point detection algorithm.

•

• Comparison with other state-of-the-art works.

Figure 5. Comparing local

detectors with different feature

descriptors and classifiers.

Figure 6. Comparing different

variants of the proposed face

shape models.

Figure 7. Detection error (mean and std) for each point across four

testing databases.

(a) MultiPie (b) Helen (c) LFPW

Figure 8.

Detection

results on

samples images

from four

databases.

(d) AFW