Data Insufficiency in Sketch Versus Photo Face Recognition

Jonghyun  ChoiAbhishek  Sharma,  David  W.  Jacobs,  Larry  S.  Davis

Ins=tute  of  Advanced  Computer  StudiesUniversity  of  Maryland,  College  Park

CVPR  Workshop  in  Biometrics  2012

17  June  2012

Sketch-Photo Face Recognition•Why  is  it  important?-­‐ Automated  criminal  search  by  forensic  sketch  can  reduce  the  8me  of  crime  inves8ga8on

Matching

GalleryProbe

Image Courtesy by B. Klare from “Matching Forensic Sketches to Mug Shot Photos”, PAMI 2011

Popular Benchmark in Literature

• CUFS  dataset[1]

-­‐ Public  benchmark  dataset-­‐ Promo8ng  the  ini8al  research‣ A  controlled  dataset

✓Well  lit  photos,  neutral  expression,  frontal  poses

-­‐ Many  approaches  evaluated  so  far

[1]  Tang  and  Wang,  “Face  Photo  Recogni8on  using  Sketch”,  ICIP  2002

Timeline of Research

2002

First CUFS dataset released (188) Baseline

ICIP2003

CUFS dataset expanded (608),

Bayesian Approach

ICCV

Non-linear approach

CVPR

Baseline

T.CSVT

Journal Extension

2006ECCV

Common Discriminant

Feature Extraction

2007 2008 2010 2011 2012ICASSP

E-HMM and selected ensemble

T.CSVT

E-HMM and selected ensemble

Journal Extension

ECCV

Lighting and Pose Invariant

CUFS dataset expanded (1,800),

Coupled Information

Theoretic Encoding

CVPR

Coupled Spectral Regression

2009CVPR

20052004

Local Feature based Discriminant

Analysis (LFDA)

PAMI

• On the CUFS Dataset

65

76.667

88.333

100

Accuracy

Summary of Previous ResultsApproach #Train #Test Rate (%)

Sketch SynthesisTang and Wang 88 100 71Tang and Wang 306 300 81.3

Nonlinear 306 300 87.67E-HMM 306 300 95.24

MS MRF+LDA 306 300 96.3MS MRF+LDA 88 100 96

MS MRF+W.PCA 88 100 99Modelling Modality Gap

PLS-subspace 88 100 93.6Klare et al. 306 300 99.47

CITP 306 300 99.87

Nearly perfect result!

Is the problem solved?

Yes

Yes CUFS Datasetfor

CUFS Dataset• 606  photo-­‐sketch  pairs-­‐ Random  par88oning  for  training/tes8ng

• Combined  with  CUFSF  (2011)  from  FERET-­‐ Total  1,800  photo-­‐sketch  pairs• viewed  sketch  dataset-­‐ Provides  well-­‐aligned  photo-­‐sketch  pairs‣ Good  for  analysis  of  difference  in  sketch  

and  photo  domains  without  any  other  factors’  interven8ons

• viewed  sketch  dataset

CUFS Dataset (Cont’d)

• viewed  sketch  dataset-­‐  Sketch  is  drawn  by  ar8sts‣  Capturing  subtle  edge  similarity  (e.g.  hair  style)-­‐  Pre-­‐processed  to  make  them  well  aligned  as  well

CUFS Dataset (Cont’d)

• viewed  sketch  dataset-­‐  Sketch  is  drawn  by  ar8sts‣  Capturing  subtle  edge  similarity  (e.g.  hair  style)-­‐  Pre-­‐processed  to  make  them  well  aligned  as  well

Real-World Scenario for Sketch-Photo Face Recognition1. Eye-­‐witness  describes  the  criminal’s  

facial  traits  verbally2. Forensic  ar;sts  draw  the  sketch  

according  to  the  verbal  descrip;on• Forensic  sketch[1]

[1]  B.  Klare  et  al.,”Matching  Forensic  Sketches  to  Mug  Shot  Photos”,  PAMI  2011

➡ Viewed sketch is not realistic

*Images  courtesy  from  B.  Klare

Insufficiency of the CUFS Dataset• Well-­‐alignment  of  CUFS  dataset-­‐ Good  for  ini8al  research  on  domain  difference  

(photo-­‐sketch)  w/o  interven8on  of  other  factors-­‐ But  simplifies  the  problem  too  much‣ Simple  edge  matching  techniques  might  work

‣ Ignores  true  variability  of  sketch-­‐face  recogni8on:

✓ Mis-­‐alignment  of  fiducial  components

✓ Seman8c  descrip8on  of  shapes  of  fiducial  component

✓ No  precise  descrip8on  for  subtle  difference  (e.g.  hair  style)

Today, We Show• To  obtain  good  result  on  CUFS  dataset-­‐ Discrimina8ve  edge  matching  technique  is  

enough‣ Outperforms  state  of  the  art  in  face  

iden8fica8on  sebng-­‐ No  effort  in  reducing  modality  gap  is  required‣ Thus  no  training  set  except  gallery  set  is  

required

‣ But  even  outperforms  state  of  the  art  in  bigger  set

Discriminative Edge Matching

• Edge  features-­‐ Gabor  wavelet  response‣ Blurred  edge  tendency:  Macro  edge

-­‐ CCS-­‐POP[1]

‣Micro-­‐edgelet

• Discrimina8ve  weight  on  the  feature-­‐ Build  a  one-­‐vs-­‐all  PLS  model

[1]  Choi  et  al.,”A  Complementary  Local  Feature  for  Face  Iden8fica8on”,  WACV  2012

Partial Least Sqaures• A  supervised  dimension  reduc8on  technique  by  

maximizing  covariance  of  weighted  independent  variable  (X)  and  weighted  dependent  variable  (Y)

• Using  NIPALS  algorithm[1]  to  obtain  the  regression  solu8on  from  X  to  Y

[1]  H.  Wold,  Par8al  Least  Squares,  1985

feature label

w = max

|w|=1cov(Xw, Y )

2

Overall System Diagram[1]

GalleryBuild  “One-­‐vs-­‐All”  PLS  regression  models

Posi=ve  Samples

Nega=ve  Samples

…Model

A…

Posi=ve  Samples

Nega=ve  Samples

…

ModelZ

…

…

A B

DE

F

G

Z

Tes=ng Regression  responses

Model  A Model  B Model  C Model  D Model  Z

…

C

Iden=fica=on  Result

…

Model  Building  (Training)

PLSR

PLSR

ProbeRegression

C

[1]  Schwartz  et  al.,  “A  Robust  and  Scalable  Approach  to  Face  Iden8fica8on”,  ECCV  2010

Experimental Setup• CUFS+CUFSF  (CUFS)  dataset-­‐ 1,800  pairs  of  sketch-­‐face-­‐ No  extra  training  set:  Use  all  pairs  for  test-­‐ Photo  to  Sketch  /  Sketch  to  Photo  experiments

• Comparison  to  previous  work• Various  image  cropping-­‐ Tight/Loose  crop-­‐ Horizontal/Ver8cal  strip  crop-­‐ Fiducial  component  crop

Experimental ResultsApproach #Train #Test Rate (%)

Sketch SynthesisTang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3

Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24

MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99

Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87

Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90

Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50

Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94

• Test  with  100  samples

70

77.5

85

92.5

100

71

9699

93.6

100

2002 [24]2006 [26] 2010 [31]

2010 [31]2011

Ours

Accuracy (%)

Approach #Train #Test Rate (%)Sketch Synthesis

Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3

Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24

MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99

Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87

Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90

Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50

Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94Our  Method

Experimental Results

Experimental Results• Test  with  300  samples

80

85

90

95

100

81.3

87.67

95.24 96.3

99.47 99.87 100

2003 [25]2005 [13]

2007,8 [4,35]2009 [30]

2011 [9]2011 [32]

Ours (G+C)

Accuracy (%)

Approach #Train #Test Rate (%)Sketch Synthesis

Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3

Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24

MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99

Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87

Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90

Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50

Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94Our  Method

Experimental Results• Test  with  1,800  samples-­‐ Compared  to  the  results  tested  with  300  samples

99.4

99.55

99.7

99.85

100

99.47

99.8799.94

2011 [9] w/3002011 [32] w/300

Ours (G+C) w/1800

Accuracy (%)

Approach #Train #Test Rate (%)Sketch Synthesis

Tang and Wang [24,26] 88 100 71Tang and Wang [25] 306 300 81.3

Nonlinear [13] 306 300 87.67E-HMM [4,35] 306 300 95.24

MS MRF+LDA [30] 306 300 96.3MS MRF+LDA (from [31]) 88 100 96MS MRF+W.PCA [31] 88 100 99

Modelling Modality GapPLS-subspace [21] 88 100 93.6Klareet al. [9] 306 300 99.47CITP [32] 306 300 99.87

Ours (Gabor only) 0 300 99.80 ± 0.44Ours (CCS-POP only) 0 300 95.53 ± 0.90

Ours(CCS-POP+Gabor) 0 300 100Ours (Gabor only) 0 1,800 99.50

Ours (CCS-POP only) 0 1,800 96.28Ours(CCS-POP+Gabor) 0 1,800 99.94

Our  Method

Different Cropping

• Tight/Loose  Crop

• Horizontal/Ver8cal  Strip  Crop

• Fiducial  Component  Crop

Results on Tight/Loose Cropping

97

97.75

98.5

99.25

100

97.6

98.8

99.94

TightMedium

Loose

Accuracy (%)

Results on Fiducial Component Cropping

77

82.75

88.5

94.25

10095.11

77.67 79.585.22

Ocular Nose MouthHair

Accuracy

Results on Strip Cropping

77

82.75

88.5

94.25

100

1 2 3 4

H V Hc Vc

Discussion & Conclusion

• A  simple  discrimina8ve  edge  analysis  can  perform  well  in  overly-­‐reduced  problem  of  sketch-­‐photo  matching

• Now  is  the  8me  to  move  on  to  more  challenging  dataset

•We  suggest  a  guideline  for  new  dataset  (Please  refer  to  our  paper)

Q/AThank you!

Authors are supported byMURI Grant N00014-08-10638 from Office of Naval Research