1

Facial Processing Projects at the Intelligent Systems Lab

Qiang Ji

Intelligent Systems Laboratory (ISL) Department of Electrical, Computer, and System Eng.

Rensselaer Polytechnic Institutejiq@rpi.edu

Image Formation and Processing group, Beckman Institute, UIUC, Sept. 7th, 2007

2

Talk outline

Overview of research at ISLFace related projects at ISLSummary and future research

3

Research at ISL

Object tracking, image segmentation, pose estimation, object recognition, performance evaluation

HCI, Transportation, Biometrics, Biology, Medicine, Entertainment, etc..

Probabilistic Graphical Models Computer Vision

Applications

Model learning, active and efficient inference, and mixed graphical models

4

Facial Processing Projects at ISL

Multi-view face and eye detection and trackingFacial feature trackingRigid and non-rigid facial motion separationFacial expression recognitionSpontaneous facial action units recognitionEye gaze trackingFace Recognition (IEEE TIP, Zou&Ji, in press)Performance Evaluation of FR system (Wang&Ji, PAMI07)

Applications

5

Multi-view face and eye detection (Wang&Ji, CVPR05)

Perform face and eye detection and tracking under varying posePropose a recursive Nonparametric Discriminant Analysis (NDA) analysis approach for face and eye detection under different poses

6

Features for Multi-View Face Detection

Pixel intensity: raw data

Haar wavelet features: Haar features essentially are geometric block features.

Linear discriminant featureFisher discriminant analysis (FDA)Nonparametric discriminant analysis (NDA)

Some Haar features

Extracting a vector from an image

face image(20*20)

face vector (400*1)

xAy T=

7

Discriminant Features Extraction

Fisher discriminant analysis (FDA): find a linear feature that best separates different classes.

Disadvantage: It is only optimal for Gaussian distributions assuming equal priors of different classes; only one effective feature is extracted since the rank of between-class scatter matrix is 1 for a two-class problem.

Nonparametric discriminant analysis(NDA): the full rank intra- and extra-class scatter matrices are calculated from the intra-class nearest neighbors and extra-class nearest neighbors .

The mapping matrix A can then be obtained by solving the generalized eigen-value problem: Disadvantage: time consuming and needs many training samples to accurately locate the NNs.

xAy T=

αα

αα

γIE

IE

xxxxx

xxxx

−+−

−−=

),min([ ]TEExxb xxxxES ))(( −−= γ

[ ]TIIxxw xxxxES ))(( −−= γ

ExIx

AASS bw λ=− )( 1

8

We propose to apply a recursive strategy in NDA:Search nearest neighbors in transformed feature space.

Recursive Nonparametric Discriminant Analysis (RNDA)

Bin i Bin j y

Intra-class NNsof class 1

Intra-class NNs ofclass 2

Extra-class NNsClass 1

Class 2

Recursively update NNs and discriminant feature until the estimated error rate converges.

Searching nearest neighbors at the transformed feature space

9

Recursive Nonparametric Discriminant Analysis (RNDA)

RNDA AlgorithmBegin with the Fisher discriminant analysis result , for i = 0,1, 2, …

Search nearest neighbors at feature space instead of the original x space.Compute the nonparametric scatter matrices based on the nearest neighbors.Calculate the new discriminant projection based on updated NNs.Continue the above procedure until the error rate converges.

Advantage: RNDA relaxes the Gaussian assumption in Fisher discriminant analysis and reduces the computational complexity of traditional nonparametric discriminant analysis.

xAy Ti=

0A

1+iA

10

Feature Selection and Combination with AdaBoost

Multiple RNDA features are selected and combined withAdaBoost

Extract RNDA feature from training data. Feature histograms are used to represent class distribution:

Probabilistic classifier is constructed based on class distributions.

AdaBoost iteratively updates the weights of training samples,

From the updated weights, more features and classifiers are learned. Finally, we combine all the individual classifiers to form a composite classifier:

∑=t

t xhxH )()(

)( xhgxx

txeww −←

)|()|( Ω=≈Ω xAyPxP T

xAy T=

tTt

Tt

t TxAyPxAyP

xPxPxh +

Ω=Ω=

=

ΩΩ

=)|()|(log

21

)|()|(log

21)(

2

1

2

1

11

Training a Multi-View Face Detector

More than 10,000 face images are collected from various sources

Many more non-face images are collected from website.

frontal facedetector

rightprofile facedetector

left profileface

detector

left fullprofileface

detector

left halfprofileface

detector

right halfprofileface

detector

right fullprofileface

detector

nonface

nonface nonfacenonfacenonface

multi-viewfaces

The structure of the multi-view face detector

12

Multi-View Face Detection Results

Some multi-view face detection results

13

Eye Localization Results

2.67%1.35%1.31%1.96%2.04%Normalized error

6.40162.69273.16524.58084.9914Pixel error

(std)(mean)(std)(mean)

Euclidean distance

verticalhorizontal

Eye localization accuracy on FRGC database

Face and eye detection results

Validate eye localization on above 5,000 2D images in FRGC V1.0.

Above 99.0% eyes are automatically detected from the detected face

14

More Eye Detection Results

More eye detection results under different environments

15

Face and Eye Detection Demos

16

Facial Feature Detection and Tracking

(Tong&Ji, PRJ07, Zhi&Ji, ICPR06)

Twenty-eight facial features around mouth, nose, eyes and eyebrows are selected.

Facial feature detection

Facial Feature tracking

17

Facial Feature Detection

o Face and eye are first detected in a frontal faceo Image is normalized, based on which mean face model is scaled and

superimeimposed on the face image, producing the initial feature locations

o Gabor wavelet jets are used to refine each feature position via the fast phase-based Gabor Wavelet matching.

A face-guided facial feature detection algorithm is developed:

Approximation Refinement

Mean Face Mesh

18

Facial Feature Tracking

Stage one (online information):o Kalman Filtering is used to model the dynamics of each facial feature.o Given a the model for each feature point, the fast phase-based

displacement estimation is used to locate each facial feature automatically. o Each facial feature model is updated in each frame dynamically.

Issues:(1) It will drift away due to the accumulated error under the significant

appearance changes.(2) No effective measurement to understand the tracking failure situation.

19

Facial Feature Correction

Stage two (offline information or prior information):

o A feature model that is the most similar to the tracking model in the first stage is selected from a training set for each facial feature collected offline.

o A new position is estimated via the fast phase-based displacement estimation by using the selected patch as a model for each facial feature.

Stage three (Correction using appearance information):Probabilistically combine the results from online and offline information:

offlineonline xxx rrr⋅−+⋅= )1( αα

onlineS --- Similarity measurement in the first stage

--- Similarity measurement in the second stage

offlineonline

online

SSS+

=αwhere

offlineS

20

Shape Constraints

In order to correct those geometrically violated facial features that deviate far from their actual positions, the geometry constraint among them can be imposed.

• Using Active Shape Model, local and global shape models are constructed to constrain the global face shape and the shape ofeach facial component for frontal face

• Face pose is estimated using a subset of tracked rigid points through RANSAC method

• For non-frontal face, the ASM models are corrected using the estimated face pose

• The pose-corrected shape constraints are then imposed on the facial features

21

Facial Feature Tracking Demo

22

Rigid and Non-rigid Face Motion Separation

(zhu and Ji, CVPR 06)

The motion of the face is the sum of two independent motions:(1) The rigid motion (face pose)(2) The non-rigid motion (facial expression)

Issue: Both motions are nonlinearly coupled in the face image, and they need be separated to perform facial expression analysis. The goal of this research is to recover both the 3D rigid and the non-rigid facial motion for facial expression analysis.

(a) Rigid motion (b) Non-rigid motion (c) Coupled motion

23

X is a vector of 3D points and is the facial deformation under facial expressions with respect to the neural face XN

3D Facial Expression Model

Given a 3D neutral face model , it will vary under facial expressions as follows:

XXX N ∆+=

X∆

NX

24

Facial Modeling via PCA

o Method: the facial expression is represented by a linear combination of a set of basis facial deformation vectors:

are the 3D basis facial deformation vectors, and are the deformation coefficients. kjQj ,...,1, =∆

∑ ∆≈∆=

k

j jj QX1α

kjj ,...,1, =α

Facial expression is revealed from the movements of a small set of facial features.

o Issue: still too many parameters ( )

These basis facial deformation vectors are learned from a set oftraining samples via PCA analysis and represented as follows:

Tjl

jl

jjj

jlzyxjzyxQ

∆∆∆∆∆∆=∆ L

111kj ,...,1, =

l×3

25

Facial Expression Model Integration

∑= ∆

∆∆

+=k

i zyx

zyx

i

i

i

iN

N

N

Mvu

1α

By integrating the obtained facial expression model with the image projection model M, a projection model can be derived as follows:

The model describes how the effects of face pose (M) and facial expression ( ) are combined together to yield the face image (u,v).iα

26

Motion Decomposition

The recovery of the pose and expression parameters is formulated as the following minimization process:

2

min1, 1∑ ∑

∆∆∆

−−

= =

l

jiM

k

i ij

ij

ij

iNj

Nj

Nj

j

j

zyx

Mzyx

Mvu

αα

Once the parameters are recovered, face pose information (M) and the facial deformation (∆X) can be derived, based on which we can perform facial expression analysis.

Subject to 2313 MM = 0'2313 =×MMand

27

Facial Motion Extraction Demo

28

Facial Expression Analysis

Given the rigid and the non-rigid facial motions, we want to recognize six basic facial expressions, based on Ekman’sFacial Action Coding Systems (FACS)

29

Facial Action Coding Systems (FACS)

FACS is a method for measuring facial behaviors. It defines expressions as one of 46 "Action Units (AU)", each of which describes a contraction or relaxation of one or more facial muscles.

FACS defines the relations between action units and facial expressions

30

FACS (cont’d)

FACS is deterministic FACS is mostly staticFACS is qualitative with respect to the AU relationsFACS is defined with respect to facial muscles. Measurements are often done through image or through image sensors.

31

Probabilistic Facial Expression Modeling

(zhang and Ji, ICCV03)

The six basic facial expressions can be modeled and recognized using the Dynamic Bayesian Networks:

o Reformulate Facial Action Coding System (FACS) in a temporal andprobabilistic framework to model the facial expression by accounting for

(1) spatial dependency(2) dynamics (temporal behavior)(3) uncertainties with facial feature measurement and facial expression.

o Associate facial motion (rigid and non-rigid) measurements with FACS Action Units (AUs).

32

AU-based Facial Expression Analysis

Grouping AUs as primary AUs and auxiliary AUsfor a facial expressionPrimary AUs are the AUs or AU combinations that can be unambiguously classified as belonging to one of the six expressions

An auxiliary AUs provides supplementary support to a facial expression.

33

AU Measurements

Most AUs are measured by the positions and changes of the facial features, i.e., ∆X, the non-rigid facial motion

Other AUs are quantified by head movements (the rigid facial motion).

Other AUs are measured by facial wrinkles detected via edge analysis Example of wrinkle detection

Geometricalrelationship of facial feature points

34

Facial Expression Modeling with Dynamic Bayesian Networks

35

Probabilistic Facial Expression Modeling

Using the model , the six prototypic facial expressions can be recognized under arbitrary face orientations via Dynamic Bayesian Networks (Zhang&Ji, PAMI05).

36

Spontaneous Facial Action Unit Recognition (Tong&Ji, CVPR06&07, and PAMI07)

Facial actions act in a coordinated way to produce meaningful expressionsFacial actions dynamically evolve and relate to each otherFacial actions are accompanied with head movements

37

Existing Work

Most work is for posed expressions for frontal faces, therefore not spontaneous expressionMost work ignore the spatial and dynamic relationships among Aus

38

Causal Relationships Among Facial Components

•2D facial shape could be viewed as a stochastic process generated by three hidden causes: head pose, 3D facial shape, and non-rigid facialmuscular movements.• 3D facial shape characterizes the intrinsic properties of a subject• Non-rigid facial muscular movements represented by facial action units cause the 3D shape deformation of the facial surface

• 3D head pose characterizes the overall head movement

39

Spatial Relationships Among Action Units

In a spontaneous facial behavior, there are some relations among AUs:

•Groups of AUs often appear together to show meaningful expression, e.g. AU6 (cheek raiser) +AU12 (lip corner puller) represents happy

•Some AUs would appear simultaneously, such as AU1 (inner brow raiser) and AU2 (outer brow raiser)

•Some AU combinations are nearly impossible, e.g. AU23 (lip tighten) and AU27 (mouth stretch)

Muscular anatomy of upper face AUs

40

Dynamic Relationships among AUsIn a spontaneous facial activity, multiple AUs often proceed in sequence to represent different naturalistic facial expressions.

There are two types of temporal relationships among AUs:Intra-AU: AUi at time t-1 to AUi at time t represents the self

development of each AU Inter-AU: AUi at time t-1 to AUj (i≠j) at time t represents the

dynamic dependencies among AUs

For example, in a spontaneous smile, AU12 (lip corner puller) is first activated to express a slight emotion; then, with the increasing of emotion intensity, AU6 (cheek raiser) is activated; and after both reach their apexes simultaneously, AU6 is relaxed, and next AU12 is released.

41

Proposed Solution

Propose to use the Dynamic Bayesian Network

to systematically represent the uncertainties of AU observations, the spatial and dynamic dependencies among AUsto represent relationships among AUs, head poses, and their measurementsto recognize facial actions through probabilistic inference

42

A DBN for Facial Activity Modeling

The DBN model is employed for modeling:

the effect of head motion on 2D global shape;

the relationship between 2D global shape and local component shapes

the relationship between AUs and 2D local shapes

relationships among AUsmeasurement uncertainty

(shaded nodes)dynamic evolution of the

temporal variables (self arrows) and dynamic dependencies among AUs (links from t-1 to t)First layer: the global constraint

Second layer: a set of 2D local component shapesthird layer: a set of facial action units

43

Facial Activity Recognition through Probabilistic Inference

3 1 1

13 1

1 3, ,

3 3 3

( , | , , , , ) ( ) ( )

( | ) ( | , )[ ( | ( ))][ ( | ( ))] ( | )

[ ( | ( ))] ( | )[ ( | )][ ( | )

M N

KD g l M

lj i

N S P Sg Sl AU DCS S S

M K

S D g D lj lj k k Sg gj k

N M N

i i P S lj AU ii j i

p pose AU O O O O O c p pose p S

p O S p S S pose p S pa S p C pa C p O S

p AU pa AU p O pose p O S p O AU

= ∑∫

∏ ∏

∏ ∏

L L

LL

L

]∏

Given the model, the true joint states of head pose and the AUs can be inferred simultaneously given the measurements of the 3D face, head pose, the 2D global shape, the 2D local shapes, and the AUs by finding the most probable explanation (MPE) of the evidence.

Based on the conditional independence encoded in the DBN, the inference could be factorized as below:

44

AU Recognition on Spontaneous Facial Expressions

False positive rate Positive recognition rate

45

Experimental results under real-world condition

46

Eye Gaze Tracking

•Gaze is important for HCI. It often represents a person’s desire or intent. Gaze estimation is, however, often ignored by the computer vision community

•Develop a real time non-intrusive eye gaze tracking system under natural head movement with minimum personal calibration

Objectives :

47

Eye model

Gaze is the line of sight or visual axis. The intersection ofthe visual axis with the object is the gaze point or point ofregard.

48

Gaze Estimation Techniques

Gaze can be estimated with differentcamera and light configurations

A single camera and a single lightA single camera with two lightsMultiple cameras with multiple lights

49

Eye Gaze Tracking with one Camera and Two Lights (Zhu&Ji, CVPR06)

Principle of eye gaze estimation

1) Detect pupil and estimate its center 2) Detect the cornea reflection of the lights3) Estimate the cornea center4) Determine the optical axis5) Determine the visual axis through a one-time personal

calibration6) Intersect the visual axis with the

screen to produce the gaze point

50

Captured image

51

Compute virtual pupil using one camera

( )pk= + −p o o v

Given the cornea center (c), the virtual pupil (p) can be solvedusing the following 2 equations

(5)

(6)K− =p cK is a constant for each subject, which can be obtained through a 9 point subject calibration. (The assumption: virtual pupil also on optical axis is validated in Appendix D of the attached paper)

52

Compute cornea center using one camera

If there are only one light, there are 7 unknowns, 6 equations.

If there are N lights,

there are 4N+4 unknowns, 5N+3 equations.

53

Transfer optical axis to visual axis

αAdd Kappa ( horizontal angle and vertical angle )β

54

Subject calibration

There are 4 subject-depended parameters (R, K, , ) in our algorithm. They can be obtained through a subject calibration procedure.

During calibration, the subject is asked to fixate at 9 points on the screen sequentially.

α β

55

System Overview

56

Gaze Tracking Demo

57

Eye Gaze Demo 2: Eye Mouse

58

Applications: Driver Fatigue monitoring

59

Emotion Modeling and Recognition

E m o tio na l M ouse

V isua l S enso r PressureSensor

Photo Sensor

TemperatureSensor

GSR Sensor

60

Biometric: Facial Recognition

61

Facial Motion Capture and Animation

Facial motion includes eye movement tracking, facial muscle movement tracking, and head movement tracking

62

Summary and Future work

Summarize the recent face related projects at ISL. Additional details may be found at http://www.ecse.rpi.edu/homepages/qji/Face/face.html

We also create an image (mostly face) databse at http://www.ecse.rpi.edu/homepages/cvrl/database/database.html

Future work:Focus on developing real time and non-intrusive system for spontaneous facial activity understanding systemCombine computer vision with graphical models for robust and consistent visual understanding and interpretation Apply to different applications human computer interaction (e.g.emotion recognition), transportation, security, medical diagnosis, learning, games, polygraph, entertainment, etc..