Automatic Face Recognition: State of the Art

Anil K. Jain(with Unsang Park, Brendan Klare, Hyun-Cheol Choi)

Department of Brain & Cognitive EngineeringKorea University

Department of Computer Science & EngineeringMichigan State University

Cameras Everywhere

1M CCTV cameras in London & 4M in U.K.; average Briton is seen by 300 cameras/day; 400Kcameras in Beijing provide 100% coverage of public places; 150K cameras in Seoul

Given a query face (probe), identify it from a target population (gallery)

MATCH

Probe Gallery

Automated Face Recognition

1:1 vs. 1 to N matching

Computing Similarity

"This recognition problem is made difficultby the great variability in head rotation andtilt, lighting intensity and angle, facialexpression, aging, etc.” Bledsoe, Chan and Bisson (1964-66)

Used 20 inter-point distances for matching

Pose, lighting, expression (PIE)

Occlusion

Aging

Intra-class Variability

www.marykateandashley.com news.bbc.co.uk/hi/english/in_depth/americas/2000/us_elections

Inter-class Similarity

• Why face recognition?• Applications• Matching algorithms• State of the art performance• Current research

Outline

Why Face?

• Face recognition: common human experience

• Social interaction: expression, emotion, intent, age

• Multidisciplinary nature: Cognitive science, HCI,

graphics, computer vision,..

• Easy to capture: covert acquisition

• Applications: surveillance; border crossing;

deduplication; entertainment,…

1,130 papers with “face recognition” in the title published in 2009 alone

Bertillon System (1882)

H.T. F. Rhodes, Alphonse Bertillon: Father of Scientific Detection, Harrap, 1956

Value of photographing prisoners was recognised by the Habitual Criminal Act, U.K., 1869

Matching 700K faces against 51M gallery (Florida DMV) found 5K duplicates

Detecting Multiple Enrollment

Border Crossing

SmartGate, Australia

HK-Schenzen border crossing

http://www.flir.com/US/

Surveillance

Entertainment

Determine viewer demographics

http://www.tstore.co.kr/userpoc/game/viewProduct.omp?insDpCatNo=DP03002&insProdId=0000028419

&prodGrdCd=PD004401&t_top=DP000503Virtual makeover

Tae Hee Kim

How Automated FR Works

Face Detection

Feature Extraction Matching

Image Normalization

Visible to Shortwave Infrared (SWIR) Spectrum (Bourlai et al., 2010)

Face Sensing

2D (still, video), 3D (shape, texture), multipsectral

Face Detection

*Theo Pavlidis, http://home.att.net/~t.pavlidis/comphumans/comphuman.htm

Challenge: Representation

How to learn salient features?

Pose-dependent

Algorithms

Pose-invariant

Pose-dependency

Matching features

Appearance-based -- Elastic Bunch Graph Matching

Local feature-based

Hybrid

Viewer-centered Images

-- Active Appearance Model

Object-centered Models

-- Morphable Model

Face representation

PCA, LDA LFA

Taxonomy of Face Recognition

LBP, Gabor

Holistic Features

…EigenFaces

Fisherfaces

Reconstructed face

Input face

…

PCA LDA

Minimize reconstruction error Maximize between-class to within-class scatter

56.4 38.6 -19.7 9.8 -45.9 19.6 -21.4 14.2

18.3 35.6 -17.5 -27.6 60.6 -20.8 41.9 -9.6

• Local Binary Patterns– Represent a local face

region as distribution of LBP features • normalized histogram

– Performs better than holistic methods

Local Features

0

5

1

5

2

5

Histogram of LBP feature values

Ojala et al., “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” IEEE TPAMI, 2002

Face Descriptors

Figure 1: An attribute classifier is trained torecognize the presence or absence of adescribable aspect (65) of visual appearance.Responses for 13 attribute classifiers are shownfor a pair of images of Halle Berry.

Figure 2: A number of simile classifiers aretrained to recognize the similarities of parts offaces to 60 reference people (Rj). Theresponses to 13 simile classifiers are shown fora pair of images of Harrison Ford.

N. Kumar et al., “Attribute and Simile Classifiers for Face Verification,” ICCV, 2009

Amazon Mechanical Turk used to obtain ~1000 training examples/attribute; $5k

Performance: State-of-Art

Best performance: high resolution 2D (controlled lighting) & 3D images

• Goal: GAR=98% @ FAR=0.1%, an order of magnitude better than FRVT2002

• Four input face modalities

• Large image database (up to 100K)

Face Recognition Vendor Test (FRVT 2006)

P.J. Phillips, “FRVT 2006 and ICE 2006, Large-Scale Results,” March, 2007. http://www.frvt.org/frvt2006/

High res., controlled lighting, neutral

Controlled lighting, smiling (400 IPD)

Uncontrolled lighting, smiling (190 IPD) 3D shape + texture

Video Surveillance Trial

~60% true ID: German Federal Police at Mainz Train Station (2007)

Human vs. Machine• O’Toole (2007) compared humans and 7 algorithms

on face pairs; 3 algorithms surpassed avg. human performance on difficult pairs; six on easy pairs

• Ding (2010): TH algorithm was better than 4500 customs inspectors on easy pairs in operational data

Easy Pair Difficult Pair

O’Toole et al., “Face recog. alg. surpass humans matching faces over changes in illumination,” TPAMI, 2007Ding et al. “Computers do better than experts matching faces in a large population”, IEEE ICCI, 2010

Current Research

Face MarksPeriocular Age Invariance

Sketch RecognitionFace Individuality

IR Face Recognition Avatar Recognition

Face at a Distance

Baseline performance: FaceVACS from Cognitec

Age Invariance

• Facial shape and texture change over time • Applications

– Age specific access control (vending machines)

– Missing children, multiple enrollment

Age 18 Age 31 Age 29Age 17

Databases: FG-NET and MORPH

Age Invariant Face Recognition

Training set(age-separated images)

Feature extraction & subspace learning

Learn appearance aging pattern

3D aging model

……

……

Build ensemble of subspaces: Minimize within-subject to between-subject variation

},,,{' 10 Nϕϕϕ =Φ

Φ

ΦΦ

Φ

)(

)1()(

)1(

M

M

MLBP

MLBP

SIFT

SIFT

Approach #1: aging invariant subspace learning

Approach #2: appearance aging model

28Aging simulation

Input

+

Age

i

Gallery

Im

ag

es

Pro

be I

mag

es

FaceVACS and generative model fail; discriminative

approach succeeds

Discriminative approach fails; FaceVACS and generative model

succeed

Matching Results

All three methods fail; fusion of generative and discriminative models

succeeds

Park, Tong & Jain, "Age Invariant Face Recognition", IEEE Trans. PAMI, 2010

Facial Sketch

30

Sketch to Mug shot Matching

Sketch drawn based on eye witness description

Klare, Li, and Jain, "Matching Forensic Sketches to Mug shot Photos," IEEE Trans. PAMI, 2010 (To Appear)

Mug shots in the Michigan police database

TRAINING

(sketch, photo) pairs

Patch featuresGroup patch vectors

into slices

12 34 . N...56

Learndiscriminantprojection for

each slice

0

5

1

5

2

5

Overlapping patches

MATCHINGProbeSketch

GalleryPhotos

0

5

1

5

2

5

Discriminantprojection

Feature extraction and grouping into slices

Nearest neighbor classifier

Matching Sketch to Photo

Forensic Sketch Matching

Successful Matches Failed Matches

Periocular

Periocular region(220x225)

Park, Jillela, Ross and Jain, " Periocular Biometrics in the Visible Spectrum", IEEE Trans. Inf. Forensics & Security 2010

• Performance on FRGC (3,400 images of 568 subjects)• No occlusion: 88% vs. 99.8% for FaceVACS• With occlusion: 81% vs. 40% for FaceVACS

1700x2270

Face Marks

Some marks are distinctive & permanent

Large birth mark Large birth mark Gang tattoo

Recognition with Face Marks

FaceVACS fails at rank-1 Recognition succeeds with FaceVACS + face marks

Matching with face marks

⊗Score fusion

≠ =

≠ =

Park & Jain, "Face Matching and Retrieval Using Soft Biometrics," IEEE Trans. on Inf. Forensics and Security, 2010

Face Recognition At A Distance• PTZ camera system

– Acquires high resolution face images (up to 10m)– Two static cameras control the PTZ camera– 96% recognition accuracy (20 probe and 10K gallery subjects)

PTZ view

Tracking

Motion segmentation

Choi, Park, and Jain, " PTZ Camera Assisted Face Acquisition, Tracking & Recognition," BTAS, 2010

Apsaras of Angkor Wat

Hindu temple built in 1,150AD; French explorers discovered the hidden ruins ~1890

Do they represent different ethnicities?

140 landmarks

Facial components allow domain expertsto assign different weights

Facial Landmarks

Klare, Mallapragada, Jain, Davis, "Clustering Face Carving: Exploring the Devatas of Angkor Wat", ICPR, 2010

Procustes Alignment

Remove variations in translation, rotation and scaling; fit a Point Distribution model (PDM) to each facial component; compute weighted sum of component similarities

Generating Clusters

Faces in Virtual World• Rise in criminal activity in Second Life

– Al-Qaeda recruitment [1], cyber crime [2], identity theft

• Search virtual world for avatars on watch list

Nood and Attema, The Second Life of Virtual Reality, http://www.epn.net/interrealiteit/EPN-REPORT-The_Second_Life_of_VR.pdfCole, Osama bin Laden's “SecondLife", Salon, http://www.salon.com/opinion/feature/2008/02/25/avatars/Yampolskiy, Klare and Jain, “Face recognition in virtual world”, working paper, 2010

Human-Like Faces

CourtesyFrank Hegel

Summary

• Face recognition is a topic of great interest to several disciplines

• Progress in automatic face recognition driven by: searching large face databases in real-time with high accuracy and low cost; humans are not necessarily the best for this task

• Excellent performance in constrained environments: frontal pose, neutral expression, controlled illumination & background, small age gap

• Unconstrained FR will require: better sensing & modeling, additional cues, contextual information,..