An Infant Facial Expression Recognition SystemBased on Moment Feature Extraction

C. Y. Fang, H. W. Lin, S. W. Chen

Department of Computer Science and Information Engineering

National Taiwan Normal University

Taipei, Taiwan

2

Outline

IntroductionSystem Flowchart

Infant Face DetectionFeature ExtractionCorrelation Coefficient Calculation Infant Facial Expression Classification

Experimental ResultsConclusions and Future Work

3

Introduction Infants can not protect themselves generally. Vision-based surveillance systems can be used for infant care.

Warn the baby sitter Avoid dangerous situations

This paper presents a vision-based infant facial expression recognition system for infant safety surveillance.

camera

4

The classes of infant expressions

Five infant facial expressions: crying, gazing, laughing, yawning and vomiting

Three poses of the infant head: front, turn left and turn right

Total classes: 15 classes

crying gazing laughing

yawning vomiting

frontturn right

turn left

5

System Flowchart

Infant face detection: to remove the noises and to reduce the effects of lights and shadows to segment the image based on the skin color information

Feature extraction: to extract three types of moments as features, including Hu moments, R

moments, and Zernike moments Feature correlation calculation:

to calculate the correlation coefficients between two moments of the same type for each 15-frame sequence

Classification: to construct the decision trees to classify the infant facial expressions

In fan t F ace D etection F eatu re E x traction

C las s ification F eatu re C orrelationC alcu lation

Im ag e S eq u en ces

6

Infant Face Detection

Lighting compensation To make the skin color detection correctly

Infant face extraction

Step1: Skin color detection Using three bands

S of HSI Cb of YCrCb U of LUX

Step2: Noise reduction Using 10x10 median filter

Step3: Infant face identification Using temporal information

Lighting compensation

Skin color detection

Noise reduction

7

Infant Face Detection Step 3: Infant face identification

1tI tI

1 Case 2 Case

1tI tI

tItI

8

Moments

To calculate three types of moments Hu moment [Hu1962] R moment [Liu2008] Zernike moment [Zhi2008]

Given an image I and let f be an image function. The digital (p, q)th moment of I is given by

The central (p, q)th moments of I can be defined as

where and

The normalized central moments of I

where

.),( )(),(

Iyx

qppq yxfyxIm

).,()()( )(),(

00 yxfyyxxIIyx

qppq

00

pqpq

00

100 m

mx

00

010 m

my

12

qp

9

Hu Moment

• Hu moments are translation, scale, and rotation invariant.

])()(3)[)(3(

])(3))[()(3(

))((4

])())[((

])()(3)[)(3(

])(3))[()(3(

)()(

)3()3(

4)(

20321

2123003213012

20321

21230123003217

0321123011

20321

2123002206

20321

2123003210321

20321

21230123012305

20321

212304

20321

212303

211

202202

02201

H

H

H

H

H

H

H normalized central moments

10

Example: Hu Momentscrying

11

Example: Hu Moments

yawning

12

Example: Hu Moments

yawningcrying

• If the infant facial expressions are different then the values of Hu moments are also different.

13

R Moment

• Liu (2008) proposed ten R moments which can improve the scale invariability of Hu moments.

43

510

52

69

23

68

51

67

31

66

5

45

5

34

4

33

21

212

1

21

||

||

||

||

||

||

||

||

||

HH

HR

HH

HR

HH

HR

HH

HR

HH

HR

H

HR

H

HR

H

HR

HH

HHR

H

HR

Hu moments

14

Example: R Momentscrying

Hu moments

• R moments and Hu moments may have different properties.

15

Zernike Moment

Zernike moments of order p with repetition q for an image function f is

where

To simplify the index, we use Z1, Z2,…, Z10 to represent Z80, Z82,…, Z99, respectively.

),(4

cos)/2(22

2/

1

8

12

vsfs

qvNsR

N

pC

N

s

s

vpqpq

),(4

sin)/2(22

2/

1

8

12

vsfs

qvNsR

N

pS

N

s

s

vpqpq

2/122 )( pqpqpq SCZ

sp

qp

spq r

qspqsps

sprR 2

2/|)|(

0

! 2/||2 ! 2/||2 !

)!()1()(

real part

imaginary part

16

Example: Zernike Momentscrying

17

Correlation Coefficients• A facial expression is a sequential change of the values of the moments. • The correlation coefficients of two moments may be used to represent the

facial expressions.• Let Ai = , i = 1, 2,…, m,

indicates the ith moment Ai of the frame Ik, k = 1, 2,…, n.

The correlation coefficients between Ai and Aj can be defined as

where and

ji

ji

ji SS

Sr

AA

AAAA

n

kiiI AA

nS

ki

1

22 )(1

1A

n

kjjIiiI AAAA

nS

kkji

1

)])([(1

1AA

iA : the mean of the elements in Ai

},...,,{21 niIiIiI AAA

kiIA

18

Correlation Coefficients

• The correlation coefficients between seven Hu moment sequences.

H1 H2 H3 H4 H5 H6 H7

H1 1 0.8778 0.9481 -0.033 -0.571 -0.8052 0.8907H2 1 0.9474 0.1887 -0.4389 -0.8749 0.9241H3 1 0.1410 -0.6336 -0.9044 0.9719H4 1 0.0568 -0.3431 0.2995H5 1 0.7138 -0.6869H6 1 -0.9727H7 1

yawning

Decision Tree• Decision trees are used to classify the infant facial expressions.

19

H1H2 H1H3 H2H3

- + +

+ + -

+ + +

+ + -

- + +

- - +

- - -

+ - -

- - -

+ - +

H1H3>0

Yes No

correlation coefficients

20

Decision Tree

• The correlation coefficients between two attributes Ai and Aj are used to split the training instances.

• Let the training instances in S be split into two subsets S1 and S2 by the correlation coefficient, then the measure function is

• The best correlation coefficient selected by the system is

K

h

K

h S

hS

S

hS

S

hS

S

hS

r N

N

N

N

N

N

N

NSE

ji1 1

22 ,loglog)(2

2

2

2

1

1

1

1

AA

)(minarg)( ,

**SESr

jiji r

ji AAAA

21

Decision tree construction

Step 1: Initially, put all the training instances into the root SR, regard SR as an internal

decision node and input SR into a decision node queue.

Step 2: Select an internal decision node S from the decision node queue calculate the entropy of node S.

If the entropy of node S larger then a threshold Ts, then goto Step 3, else label

node S as a leaf node, goto Step 4.

Step 3: Find the best correlation coefficient to split the training instances in node S.

Split the training instances in S into two nodes S1 and S2 by correlation

coefficients and add S1, S2 into the decision node queue. Goto Step 2.

Step 4: If the queue is not empty, then goto Step 2, else stop the algorithm.

22

Experimental Results

Training: 59 sequences Testing: 30 sequences Five infant facial expressions: crying, laughing, dazing, yawning, vomiting Three different poses of infant head: front, turn left, and turn right Fifteen classes are classified.

crying laughing dazing yawning vomiting

Turnleft

Front

Turn right

Feature type: Hu momentsInternal nodes: 16Leaf nodes: 17Height: 8

076 HH

rHS9

yes no

041 HH

r

HS10crying

yes no

062 HH

rHS11 yawning

yes no

061 HH

rHS12 crying

yes no

yawningHS13

075 HH

r

021 HH

rHS16

yes noHS14

043 HH

r

no

laughing

no

dazing dazing

yesyes

HS15

031 HH

r

no

cryingdazing

yes

HS2

yes no

076 HH

r

HS3

061 HH

r

HS6

yes no

063 HH

r

HS5vomiting

yes no

063 HH

r

HS5 vomiting

yes no

laughingvomiting

yes no

051 HH

rHS7 yawning

yes no

031 HH

rHS8 crying

yes no

laughingcrying

yes no

HS1054

HHr

053 HH

r

Experimental Results

Testing sequences Classification results

24

laughing

laughing

dazing

vomiting

• The classification results of the Hu-moment decision tree

25

Feature type: R momentsInternal nodes: 15Leaf nodes: 17Height: 10

no

yes

vomiting

yes no075

RRr

yawning

no021

RRr

yes

074 RR

ryes

0105 RR

r

dazing

no

crying

vomiting

yes no064

RRr

laughingno

014 RR

ryes

no0101

RRr

yes

no0105

RRr

yes

021 RR

ryes

nocrying

dazing laughing

no

051 RR

r

yes

no

041 RR

r

yes

081 RR

r

yes nodazing

crying vomiting

no0109

RRr

yes

071 RR

r 061 RR

r

031 RR

r

yes

yes no

laughingdazing

no

vomiting

yes no

laughingcrying

26

Experimental Results

Testing sequences Classification results

vomiting

dazing

yawning

dazing

• The classification results of the R-moment decision tree

Feature type: Zernike momentsInternal nodes: 19Leaf nodes: 20Height: 7

no

yes no062

ZZr

no084

ZZryes

063 ZZ

ryes

crying

vomiting

laughing

dazing yawning

yes no0109

ZZr

yesno

075 ZZ

r

no

076 ZZ

r

yesyes

no

0107 ZZ

r

cryingyes no

076 ZZ

r

no

071 ZZ

r

yes

crying

no

041 ZZ

r

yes

no

051 ZZ

r

yes

no

071 ZZ

r

yes

crying laughing

021 ZZ

r

yesno

081 ZZ

r

vomiting

yes no

031 ZZ

r

crying

yes no

041 ZZ

r

dazing yawning

yes no

laughingdazing

dazing032 ZZ

r

yes

021 ZZ

r

no

vomitingyes

no

laughing dazing

no

091 ZZ

r

yawning

yes

28

Experimental Results

Testing sequences Classification results

crying

crying

vomiting

crying

• The classification results of the Zernike-moment decision tree

29

Conclusions

The comparison of the results

The correlation coefficients of the moments are useful attributes to classify the infant facial expressions.

The classification tree created by the Hu moments has less height and number of node, but higher classification rate.

Height of the decision

tree

Number of nodes

Number of training

sequences

Number of testing

sequences

Classification Rate

Hu moments 8 16+17 59 30 90%

R moments 10 15+17 59 30 80%

Zernike moments 7 19+20 59 30 87%

30

Conclusions and Future Work

Conclusion A vision-based infant facial expression recognition system

Infant face detection Moment features extraction Correlation coefficient calculation Decision tree classification

Future work To collect more experimental data To fuzzify the decision tree

Binary decision trees may have less noise tolerant ability. If the correlation coefficients are close to zero, the noises will greatly affect the

classification results.

31