CVPR SLAM 2007
Using Group Prior to Identify People In Consumer Images
Andrew C. GallagherTsuhan Chen
Carnegie Mellon University Eastman Kodak Company
June 18, 2007
CVPR SLAM 2007
The ProblemConsumer image collections are growing exponentially each year.
Consumers want to search for images based on whom the image contains. And they don’t like to label images!
This is more than a face recognition problem. To best understand the semantics of who is in the images, we need to understand the people in the images.
CVPR SLAM 2007
Traditional Face Recognition
Determines the assignment of each person independently
Extract facial features Build a classifier that finds the most likely name, given the features.
But this method does not take full advantage of the available information!
CVPR SLAM 2007
The Group Prior for Learning The Semantics of People in
ImagesDetermine the joint assignment of all people in the image to names, using the group prior.
By the unique object constraint (UOC), an individual can appear only once in the image. The group prior characterizes the prior probability of certain groups of people appearing together in an image.
CVPR SLAM 2007
System Diagram
Ambiguous LabelResolution
Images (Faces)
Ambiguous Labels
Classifier Training
Labeled Faces
RecognizePeople
Annotated Image
HannahJonahHolly
AndyJonah
HollyJonah
Jonah Holly Jonah
Holly JonahAndyHannah
Unlabeled Image
Group Prior
Hannah
Holly
CVPR SLAM 2007
Recognizing a Person
When a single person is in the image:
: the set of all unique names
: a member of the set
: the features from person image
Posterior Probability
Individual PriorLikelihood
CVPR SLAM 2007
Recognizing Multiple People
The graph model represents the features and people in an image.
The graph encodes the independence assumptions of our model.
E.g. given the identity of a person, their features are independent of others in the image.
p1
f1
p2
f2
pM
fM
…
…
The Group Prior
CVPR SLAM 2007
Recognizing Multiple People
The joint probability function:
p1
f1
p2
f2
pM
fM
…
…
The Group Prior
: an index over the people in the image
: the set of all features for all people
: the set of people in the image
: a subset of ; a particular assignmentof a name to each person in .
The Group PriorLikelihood
CVPR SLAM 2007
Estimating the Group Prior
For pairs of names, the group prior is estimated by counting the number of images the pair appears, then normalizing.
The group prior for 3 or more people is estimated according to the group prior pairwise graphical model.
The Group Prior
The Individual Prior
CVPR SLAM 2007
System Diagram
Ambiguous LabelResolution
Images (Faces)
Ambiguous Labels
Classifier Training
Labeled Faces
RecognizePeople
Annotated Image
HannahJonahHolly
AndyJonah
HollyJonah
Jonah Holly Jonah
Holly JonahAndyHannah
Unlabeled Image
Group Prior
Hannah
Holly
CVPR SLAM 2007
Ambiguous Labels
Ambiguous labels indicate who is in the image, but not which person is which name.
A constrained clustering algorithm is used to ‘resolve’ the labels.
The resolved labels are used to learn the feature distribution for each name.
Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly
Hannah
Jonah Holly Jonah AndyJonah Andy
HannahJonah Andy
Hannah
Jonah Holly
Hannah
Hannah AndyHannah HollyJonah Andy
Hannah JonahHolly
Hannah AndyHannah Andy
-1.5 -1 -0.5 0 0.5 1-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5Feature Space
Feature 1
Fea
ture
2
Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly
Hannah
Jonah Holly Jonah AndyJonah Andy
HannahJonah Andy
Hannah
Jonah Holly
Hannah
Hannah AndyHannah HollyJonah Andy
Hannah JonahHolly
Hannah AndyJonah Andy
-1.5 -1 -0.5 0 0.5 1-0.5
0
0.5Feature Space
Feature 1
Fe
atu
re 2
Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly
Hannah
Jonah Holly Jonah AndyJonah Andy
HannahJonah Andy
Hannah
Jonah Holly
Hannah
Hannah AndyHannah HollyJonah Andy
Hannah JonahHolly
Hannah AndyJonah Andy
-1.5 -1 -0.5 0 0.5 1-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5Feature Space
Feature 1
Fe
atu
re 2
Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly
Hannah
Jonah Holly Jonah AndyJonah Andy
HannahJonah Andy
Hannah
Jonah Holly
Hannah
Hannah AndyHannah HollyJonah Andy
Hannah JonahHolly
Hannah AndyJonah Andy
-1.5 -1 -0.5 0 0.5 1-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5Feature Space
Feature 1
Fe
atu
re 2
Andy Hannah Andy HannahJonah HannahJonah Hannah Jonah Holly
Hannah
Jonah Holly Jonah AndyJonah Andy
HannahJonah Andy
Hannah
Jonah Holly
Hannah
Hannah AndyHannah HollyJonah Andy
Hannah JonahHolly
Hannah AndyJonah Andy
-1.5 -1 -0.5 0 0.5 1-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5Feature Space
Feature 1
Fe
atu
re 2
CVPR SLAM 2007
Classification with Group Prior
From the joint pdf, inference questions can be answered:Most Probable Explanation MAP
MAP- Most probable assignment of a particular person .
CVPR SLAM 2007
Experiment
The image collection:
Facial Features: Active Shape Model [Cootes95] based features, then PCA reduces to 5D.
Images 1197
Images with multiple people
188
No. Faces in these images
420
Individuals 5
• Ambiguously label a portion of the image collection, classify the identities of all the rest.
• Compare 4 Priors:• Group Prior (GP)• UOC Prior A binary
version of the GP that respects the UOC.
• The individual prior. • No Prior
• The performance is quantified for:• MAP • MPE
CVPR SLAM 2007
ResultsGroup Prior produces a large benefit.
Note: All images were ambiguously labeled; no people were explicitly labeled.
Example Classification
(from 10 labeled images)
Individual Prior / no Prior
Ethan Ethan
Unique Object Constraint
Hannah
Ethan
Group Prior Holly Ethan
0 0.2 0.4 0.6 0.8 10.2
0.4
0.6
0.8
1
Portion of Images Labeled
Cla
ssifi
catio
n R
ate
MAP
GroupUOCIndivnone
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
Portion of Images Labeled
Cla
ssifi
catio
n R
ate
MPE
GroupUOCIndivnone
CVPR SLAM 2007
Prior Work
Many face recognition methods- most ignore the issue of prior probabilities. [Zhao03]
Face recognition methods have been used to assist the labeling of image collections. [Zhang04]
In news photos, names from captions have been assigned to faces. [Berg04]
The co-occurrence of people in images has been studied, but not combined with image features. [Naaman05]
CVPR SLAM 2007
Conclusions
The group prior models the social relationships between individuals.
We learn feature distributions and relationships between the labels (people).
By using the group prior, recognition accuracy is significantly improved!