university of murcia (spain) artificial perception and pattern recognition group refining face...

UNIVERSITY OF MURCIA (SPAIN)

ARTIFICIAL PERCEPTION AND PATTERN

RECOGNITION GROUP

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS

REFINING FACE TRACKING WITH INTEGRAL PROJECTIONS

Ginés García MateosDept. de Informática y Sistemas

University of Murcia - SPAIN

2

REFINING FACE

TRACKING WITH

INTEGRAL PROJECTIONS

Ginés García Mateos

AVBPA’2003GUILDFORDJUNE, 2003

Introduction Introduction

• Main objective: develop a new technique to track human faces and facial features:– Working in real-time: fast processing

– Under realistic conditions: robust to facial expressions, lighting conditions, 3D head pose and movements

– With high location accuracy: facial features location (eyes, nose, mouth)

3

REFINING FACE

TRACKING WITH




IntroductionIntroduction

• Index of the presentation:– Face integral projections

– Integral projection models

– Alignment of projections

– The tracking process

– Experimental results

– Conclusions

4

REFINING FACE

TRACKING WITH




Face integral projectionsFace integral projectionsDefinition. Let i(x,y) be an image, and R(i) a region in it

– Vertical integral projection

PVR : {ymin, ..., ymax} R

PVR(y) = i(x,y); (x,y) R(i)

– Horizontal integral projection

PHR : {xmin, ..., xmax} R

PHR(x) = i(x,y); (x,y) R(i) FACE PVFACE(y)

y

EYES

PHEYES(x)

x

5

REFINING FACE

TRACKING WITH




Face integral projectionsFace integral projections

• Dimensionality reduction: 3D world 2D images 1D integral projections

• Advantages:– Fast to compute and to process

• Disadvantages:– Loss of information. Is it relevant for

the problem?

• What happens when applied to human faces?

6

REFINING FACE

TRACKING WITH





• Different individuals

7

REFINING FACE

TRACKING WITH





• Different facial expressions

8

REFINING FACE

TRACKING WITH





• Different segmented regions

9

REFINING FACE

TRACKING WITH




Integral project. modelsIntegral project. models

• Face integral projec. is an interesting and robust feature for tracking

• It has been applied using heuristic analysis: max-min search, fuzzy logic, thresholding projections

• Proposal: define and work with adaptable projection models

• How to model a variety of projection patterns?

10

REFINING FACE

TRACKING WITH




Integral project. modelsIntegral project. models• A projection model is a pair:

M : {mmin, ..., mmax} R (Mean)

V : {mmin, ..., mmax} R (Variance)

11

REFINING FACE

TRACKING WITH




Integral project. modelsIntegral project. models• Advantages of working with explicit

projection models:– The model is learnt from examples. In

tracking, it is adapted to tracked faces– We can define a signal to model

distance:

12

REFINING FACE

TRACKING WITH




Integral project. modelsIntegral project. models• Advantages of working with explicit

projection models:– The model can be reprojected

Reprojection (by outer product) using 1 vertical IP and 2 horizontal IP

13

REFINING FACE

TRACKING WITH




Alignment of projectionsAlignment of projections• Corresponding facial features should

be projected on the same locations

Before alignment

After alignment

14

REFINING FACE

TRACKING WITH




Alignment of projectionsAlignment of projections• Alignment with respect to a model• Problem formulation:

– Let S: {smin, ..., smax} R be a signal

– Let M,V: {mmin, ..., mmax} R be a model

– Let S’ be a family of scale and translations alignments of S:

– Find parameters (a,b,c,d,e) which minimize:

15

REFINING FACE

TRACKING WITH




The tracking processThe tracking process• Tracking is based on the alignment of integral projections

• Main steps:

1. Prediction and segmentation

2. Vertical alignment

3. Horizontal alignment

4. Orientation estimation

16

REFINING FACE

TRACKING WITH




The tracking processThe tracking process• Features to track

• Input to the tracker – Bounding ellipse– Facial features: eyes

and mouth

– Face model – State of tracking in frame t-1– Frame t

17

REFINING FACE

TRACKING WITH




The tracking processThe tracking process

1. Prediction and segmentation• Null predictor: locations in frame t-1

are used to extract the face in frame t

Predicted location Wrapped Segmented

18

REFINING FACE

TRACKING WITH





2. Vertical alignment• Using the vertical projection of the face,

and the model, align the face vertically

Segmented

Model

PVFACE(y)Align PVFACE to MVFACE

y

Align using the obtained parameters (a,b,c,d,e)

19

REFINING FACE

TRACKING WITH





3. Horizontal alignment• Using the horizontal projection of the

eyes’ region, align the face horizontally

Segmentedafter step 2

Model

PHEYES(x) Align PHEYES to MHEYES

y

Align using the obtained parameters (a,b,c,d,e)

x

20

REFINING FACE

TRACKING WITH





4. Orientation estimation• Using vertical projections of each eye,

estimate the orientation of the faceSegmentedafter step 3

Model

PVEYE1, PVEYE2 Align PVEYEi to MVEYEi

y

21

REFINING FACE

TRACKING WITH





• Global structure of the tracker

1. Prediction and segmentation

2. Vertical alignment

3. Horizontal alignment

4. Orientation est.

22

REFINING FACE

TRACKING WITH




Experimental resultsExperimental results

• Experiments:– Location accuracy

– Execution time per frame

– Robustness to facial expressions, 3D pose, lighting conditions

• Different sources: TV, video-conference camera and DVD

• Compared with CamShift algorithm

23

REFINING FACE

TRACKING WITH




Experimental resultsExperimental results• FILE NAME: tl5-02.avi SOURCE: TV• FORMAT: 640x480 (25 fps) LENGTH: 280 frames

• MODEL SIZE (pixels): 97x123

• AVG/MAX ERROR (mm): 3.41 / 12.9

• TIME/FRAME (ms): 4.02

24

REFINING FACE

TRACKING WITH




Experimental resultsExperimental results• FILE NAME: a3-05.avi SOURCE: TV• FORMAT: 640x480 (25 fps) LENGTH: 541 frames


• AVG/MAX ERROR (mm): 1.95 / 9.76


25

REFINING FACE

TRACKING WITH




Experimental resultsExperimental results• FILE NAME: a3-2.avi SOURCE: TV• FORMAT: 320x240 (25 fps) LENGTH: 440 frames


• AVG/MAX ERROR (mm): -


26

REFINING FACE

TRACKING WITH




Experimental resultsExperimental results• FILE NAME: ggm2.avi SOURCE: QuickCam• FORMAT: 320x240 (25 fps) LENGTH: 655 frames


• AVG/MAX ERROR (mm): 1.83 / 9.29


27

REFINING FACE

TRACKING WITH




Experimental resultsExperimental results• FILE NAME: sw2-1.avi SOURCE: DVD• FORMAT: 320x240 (30 fps) LENGTH: 427 frames


• AVG/MAX ERROR (mm): -


28

REFINING FACE

TRACKING WITH





• Location accuracy:– Errors in mm (in the face plane) using a

ground-truth location of facial features– Average error below 4 mm, maximum

error 14 mm– With CamShift: average error over 10 mm,

maximum error 30 mm

29

REFINING FACE

TRACKING WITH





• Execution time, per frame:– Off-the-self PC: AMD Athlon at 1.2 GHz– Average time below 5 ms, with 640x480

resolution, face size 100x120 pixels – With CamShift: average time about 10 ms,

unable to work in one video sequence

30

REFINING FACE

TRACKING WITH




ConclusionsConclusions

• The tracking problem is decomposed into three main independent steps:– Vertical alignment– Horizontal alignment– Orientation estimation

• The process is fast, accurate and robust in the tested conditions

• It is exclusively based on integral projections

31

REFINING FACE

TRACKING WITH




ConclusionsConclusions

• The tracker is not affected by background distractors

• It can be applied either in color and grey-scale images

• Main limitation: maximum allowed movement (approx. 1 m/s, at 25 fps)

• Future work: improve the prediction step, e.g. with Kalman filters

32

REFINING FACE

TRACKING WITH




LastLast

• This work has been supported by Spanish CICYT project DPI-2001-0469-C03-01

• Demo videos:

http://dis.um.es/~ginesgm/fip

• Grupo PARP web page:

http://dis.um.es/parp

Thank you very much

university of murcia (spain) artificial perception and pattern recognition group refining face...

Documents