seeing people in social context - university of washington...seeing people in social context:...
TRANSCRIPT
![Page 1: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/1.jpg)
Seeing People in Social Context: Recognizing People and Social Relationships
Gang Wang1, Andrew Gallagher2, Jiebo Luo2, and David Forsyth1
1 University of Illinois at Urbana-Champaign, Urbana, IL2 Kodak Research Laboratories, Rochester, NY
ECCV 2010
Presented by Aditya SankarCSE 590V
![Page 2: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/2.jpg)
Traditional Approach
![Page 3: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/3.jpg)
Traditional Approach
![Page 4: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/4.jpg)
Traditional Approach
Judy
John
Noah
Andrew
![Page 5: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/5.jpg)
Traditional Approach
Construct Appearance Model
Judy
John
Noah
Andrew
![Page 6: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/6.jpg)
Traditional Approach
Construct Appearance Model Recognize
Noah
Judy
JohnAndrew
Judy
John
Noah
Andrew
![Page 7: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/7.jpg)
Traditional Approach
Construct Appearance Model Recognize
Noah
Judy
JohnAndrew
Does not work on weakly labeled data sets
Judy
John
Noah
Andrew
![Page 8: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/8.jpg)
Weak Labeling
Judy, John, Noah and Andrew in the UK John, Judy and the kids at Eric’s wedding
Photo albums, news captions, Flickr tags etc.
Label ambiguity increases learning difficulty
![Page 9: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/9.jpg)
People in personal image collections are generally not strangers
Social relationships often exhibit certain visual patterns
![Page 10: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/10.jpg)
People in personal image collections are generally not strangers
Social relationships often exhibit certain visual patternsIn this case:
- Husband and wife are in close proximity- Husband is taller
![Page 11: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/11.jpg)
Can we improve face recognition by considering these social relationships?
Daisy, Noah
Edward, Daisy & Noah
![Page 12: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/12.jpg)
Can we improve face recognition by considering these social relationships?
Daisy, Noah
Edward, Daisy & Noah
![Page 13: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/13.jpg)
Can we improve face recognition by considering these social relationships?
Daisy, Noah
Edward, Daisy & Noah
![Page 14: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/14.jpg)
Related Work
Berg et al. “Names and Faces”, CVPR 2004
Stone et al. “Autotagging Facebook”, CVPR 2008
Divvala et al. “An Empirical Study of Context in Object Detection”,
CVPR 2008
Automatic Face Annotation Weakly Labeled Images
Contextual Features
![Page 15: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/15.jpg)
Representing Social Relationships
rij: social relationship between ith and jth person
fij: social relationship ‘features’ between ith and jth face
- Height difference- Face size ratio- Closeness- Age difference (appearance based)- Gender (appearance based)
![Page 16: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/16.jpg)
Representing Social Relationships
rij: social relationship between ith and jth person
fij: social relationship ‘features’ between ith and jth face
![Page 17: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/17.jpg)
Model
pi: the ith person namewAi: facial features associated with pi
rij: social relationship between ith and jth personti: age of the ith personfij: social relationship ‘features’ between ith and jth face
![Page 18: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/18.jpg)
Model
pi: the ith person namewAi: facial features associated with pi
rij: social relationship between ith and jth personti: age of the ith personfij: social relationship ‘features’ between ith and jth face
Appearance term represented with a discriminative model.wAi denotes facial features associated with pi
![Page 19: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/19.jpg)
Model
pi: the ith person namewAi: facial features associated with pi
rij: social relationship between ith and jth personti: age of the ith personfij: social relationship ‘features’ between ith and jth face
Relationship term represented with a generative model.fAiAj denotes social relationship ‘features’ between faces Ai and Aj
rij denotes the discrete social relationship between ith and jth personA is a hidden variable that relates names and faces
![Page 20: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/20.jpg)
Model
pi: the ith person namewAi: facial features associated with pi
rij: social relationship between ith and jth personti: age of the ith personfij: social relationship ‘features’ between ith and jth face
Since relationships are annotated p( rij | pi, pj) = 1
![Page 21: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/21.jpg)
Learning
![Page 22: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/22.jpg)
LearningLearn using EM
![Page 23: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/23.jpg)
LearningLearn using EM
Parameter: Simplifications: System initialized with parameters produced by the baseline model (omits social relationships)
![Page 24: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/24.jpg)
LearningLearn using EM
Parameter: Simplifications: System initialized with parameters produced by the baseline model (omits social relationships)
E Step:
Simplifications: Prior distribution of A treated as uniform distribution. Only assign one pi to a wj when p(pi | wj) is bigger than a threshold.
![Page 25: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/25.jpg)
LearningLearn using EM
Parameter: Simplifications: System initialized with parameters produced by the baseline model (omits social relationships)
E Step:
Simplifications: Prior distribution of A treated as uniform distribution. Only assign one pi to a wj when p(pi | wj) is bigger than a threshold.
M Step: Maximize by updating p(p | w) and p(f | r, t) separately.
![Page 26: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/26.jpg)
Inference
Input
- No name labels- Extract facial features (W) and relationship features (F)
![Page 27: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/27.jpg)
Inference
Input
- No name labels- Extract facial features (W) and relationship features (F)
Inference
![Page 28: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/28.jpg)
Inference
Input
- No name labels- Extract facial features (W) and relationship features (F)
Inference
Output
- Tagged faces (P)
Noah
Judy
John Andrew
![Page 29: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/29.jpg)
Experiment A
Training Test
Collection 1:1,125 images47 people600 training examples
Training Test
Collection 2:1,123 images34 people600 training examples
Training Test
Collection 3:1,117 images152 people600 training examples
Recognizing people with social relationships
Annotate subset of data Train model
Infer name assignment for
test image
Data
Procedure
![Page 30: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/30.jpg)
Results - Experiment ARecognizing people with social relationships
Green - Correctly recognized with relationship modelingRed - Incorrectly recognized with relationship modeling
![Page 31: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/31.jpg)
Results - Experiment ARecognizing people with social relationships
Average improvement:5.4%
![Page 32: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/32.jpg)
Experiment BRecognizing social relationships in novel image sets
Training
Collection 1:1,125 images47 people600 training examples
Test
Collection 2:1,123 images34 people
Test
Public dataset [1]:5,080 images28,231 people
Data
[1] A. Gallagher and T. Chen. Understanding Images of Groups of People. In Proc. CVPR, 2009.
Train relationship model on Collection 1
Classify social relationships on previously unseen image
Procedure
![Page 33: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/33.jpg)
Results - Experiment BRecognizing social relationships in novel image sets
Mother-Child
Husband-Wife
Siblings
![Page 34: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/34.jpg)
Results - Experiment BRecognizing social relationships in novel image sets
Confusion Matrices
Test on Collection 2 Test on Public Collection
Random assignment = 14.3%Average Performance = 50.8%
Random assignment = 20%Average Performance = 52.7%
![Page 35: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/35.jpg)
Discussion• Relatively large no. of training examples (50% of collection).
What is the actual overhead of relationship labeling?
• Can we add more appearance based features?
• Eg. Husband skin tone is darker than wife’s *
• Performance of classifier in exceptional cases
• Wife taller than husband
• Same-sex couple
• Marginal improvement - 5.4%
• They use Fisher subspace features (weak). Will the gain reduce if we include more attributes?
• Limited to family photos. Other applications?
* Manyam et. al. “Two faces are better than one”, IJCB 2011
![Page 36: Seeing People in Social Context - University of Washington...Seeing People in Social Context: Recognizing People and Social Relationships Gang Wang1, Andrew Gallagher2, Jiebo Luo2,](https://reader034.vdocuments.site/reader034/viewer/2022052020/6033bd8ce633df4d30662dc9/html5/thumbnails/36.jpg)
Thanks!