Expression-invariant Face Recognition using Geodesic Distance Isometries

Kerry Widder

A Review of ‘Robust expression-invariant face recognition

from partially missing data’ and other works by

A. Bronstein, M. Bronstein and R. Kimmel

© 2006 by Kerry R. Widder 2

Outline• Problem statement• Earlier work (Canonical Form)

– Model– Novel idea– Implementation– Issues

• Recent work (Partial Embedding)– Motivation– New idea– Implementation– Issues– Results– Future work

© 2006 by Kerry R. Widder 3

Problem Statement

Goal: Expression-invariant face recognition using range data images.

© 2006 by Kerry R. Widder 4

Earlier Work – “Canonical Forms” (CF)

• Model the face as a Riemannian manifold, S• Use minimal Geodesic distances between points on the

face as a representation of the manifold– Intuition: skin on a face moves with expression, but the geodesic

distances between points on the face remain almost constant, thus changes in expression are isometric transformations.

– Geodesic distances between points show the intrinsic geometry of the face (i.e., the identity of the face)

– Euclidean distances between points show the extrinsic geometry of the face (i.e., the expression of the face)

– Experimental evidence: study done on one subject - data shows geodesic distances are isometric across expressions (is one subject sufficient to make the claim???)

Model

© 2006 by Kerry R. Widder 5

Earlier Work – “Canonical Forms” (CF)

• Need a distance function for comparison purposes – want d(S, f(S)) = 0 for all functions f which are isometries of S.

• Direct use of Geodesic distances has issues due to data being a sampled version of S:– Sample locations may be different between S & f(S) – Number of samples may be different between S & f(S)– Order of samples may be different between S & f(S)

• Solution: Represent S as a subset of Rm with the intrinsic geometry approximately preserved – ‘isometric embedding’– The image produced is called a Canonical Form (CF)

Novel idea – Isometric Embedding

© 2006 by Kerry R. Widder 6

Earlier Work – “Canonical Forms” (CF)

• Compute the Geodesic distances using the Fast Marching Method (FMM) – requires O(N2) or greater to compute.

• Error criteria: Raw Stress (won’t be exact, resulting in embedding error)– Equation:

– Minimize using Multidimensional Scaling (MDS) - O(N2) to compute• Perform alignment of the CF image• Matching – compare probe CF image with gallery CF images

– Higher order moments

Implementation

jijiSjiSraw ssdxxdX m

2,,;

© 2006 by Kerry R. Widder 7

3D recognition via geometric invariants

Range camera acquires facial surface (I).

The surface is smoothed (II), subsampled and cropped (III).

Fast marching computes geodesic distances on the surface.

Facial surface is flattened via MDS (IV). Rigid surface matching using the canonical surfaces (V).

A. Bronstein, M. Bronstein and R. Kimmel, “3D face recognition using geometric invariants“, 2003

I II III IV V

3,4,...X R

(X)arg min ij ijij

d

© 2006 by Kerry R. Widder 8

Earlier Work – “Canonical Forms” (CF)

• Strength – can pre-compute the signatures• Limitations

– Sensitivity to the definition of the boundaries.– No partial matching (including occlusions).– Computational complexity O(N2) – requires ≥ 2500 samples for face recognition

Issues

© 2006 by Kerry R. Widder 9

Recent Work – “Partial Embedding” (PE)

• Reduce the error/distortion due to the embedding of a face into the canonical form

• Expand the domain of faces that can be handled to include:– Partial faces.– Faces with occlusions.

Motivation

© 2006 by Kerry R. Widder 10

Recent Work – “Partial Embedding” (PE)

• Embed the probe face directly into the target face from the gallery– If the two faces are isometric, the error will be zero.– If the two faces are not isometric, the error will give a

measure of their similarity.

• Definition: Partial Embedding (PE)– A mapping, φ, of probe face manifold Q onto gallery

target face manifold S.

New Idea

Nji

qqdtopossibleascloseasisqqdthatsuch

SssQqq

jiSjiQ

SNQN

,...,1,

,_______,__

,,,...,,,...,: 11

© 2006 by Kerry R. Widder 11

Recent Work – “Partial Embedding” (PE)

• Computation of the PE uses a new procedure called ‘Generalized Multidimensional Scaling (GMDS)

• Error criteria – generalized stress:–

– Where ui, i=1,…,M denote the vectors of parametric coordinates of s i, and W = (wij) is a symmetric matrix of non-negative weights.

– The weights are set to 0 or 1 – if the whole surface is being matched, all are set to 1; if only part of the surface is being matched, the appropriate weights are set to 0.

– Minimization of the error is done iteratively• Distance function

–

– Allows partial matching of faces– Issues with points on the boundary of an occlusion

Implementation

ji

jiQjiSijSQgen qqduudwdWU 2,,,,;

jiij

SQgenUPE w

dWUQSd

,,;minarg,

© 2006 by Kerry R. Widder 12

GMDS

Bronstein Bronstein Kimmel 2006

( )

( ( ), ( )) ( , )

( ( ), ( )) ( , )

arg min

arg min

Q Si j i j

ij

Q Si j i j

ijS Q

s s s s

s s s s

© 2006 by Kerry R. Widder 13

Partial matching problem. Shown in blue dotted is a geodesic between the points q1, q2 Q; the corresponding inconsistent

geodesic on Q’ is shown in black.

Bronstein Bronstein Kimmel 2006

dQ (q1,q2)

dQ’ (q1,q2)

Q’ Q

q2

q1

© 2006 by Kerry R. Widder 14

Recent Work – “Partial Embedding” (PE)

• Strengths– Reduced error/distortion.– Can handle partial data.– Not sensitive to preprocessing steps.– No alignment step.

• Limitations– Boundary issues for partial matching.– Can’t pre-compute distance function (can minimize this

impact using a hierarchical search scheme).

Issues

© 2006 by Kerry R. Widder 15

Recent Work – “Partial Embedding” (PE)

• Database– FRGC data – used 30 subjects – 1 neutral expression (gallery), 5

moderate expression (probes).– Gallery images were sub-sampled at 2500 points– Probe images were sub-sampled at 53 points– Two sets of probe images created – 1. face cropped with narrow

geodesic mask, 2. Severe occlusions

• Results– Mild occlusion – EER of 3.1%– Severe occlusion – EER of 5.5%– 100% rank one recognition rate– Computation time: 1/5 sec. per comparison

Results

© 2006 by Kerry R. Widder 16

Bronstein Bronstein Kimmel 2006

© 2006 by Kerry R. Widder 17

Bronstein Bronstein Kimmel 2006

© 2006 by Kerry R. Widder 18

Future Work – “Partial Embedding” (PE)

• Larger data set• More severe expressions• Extend to texture

© 2006 by Kerry R. Widder 19

Selected References• A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Robust expression-

invariant face recognition from partially missing data. Proc. 9th European Conf. on Computer Vision, May 2006.

• A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Three-dimensional face recognition. Intl. J. Computer Vision, 64(1):5-30, August 2005.

• A. Elad and R. Kimmel. On bending invariant signatures for surfaces. IEEE Trans. PAMI, 25(10):1285-1295, 2003.

• A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proc. Nat. Acad. Sci., 103(5):1168-1172, January 2006.

• A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Expression-invariant representations for human faces. Technical Report CIS-2005-01, Dept. of Computer Science, Technion, Israel, June 2005.