![Page 1: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/1.jpg)
Lecture 16 -Fei-Fei Li
Lecture 16:
Object recognition:
Part-based generative models
Professor Fei-Fei Li
Stanford Vision Lab
28-Nov-111
![Page 2: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/2.jpg)
Lecture 16 -Fei-Fei Li
What we will learn today?
• Introduction
• Constellation model
– Weakly supervised training
– One-shot learning
• (Problem Set 4 (Q1))
28-Nov-112
![Page 3: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/3.jpg)
Lecture 16 -Fei-Fei Li
Challenges: intra-class variation
28-Nov-113
![Page 4: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/4.jpg)
Lecture 16 -Fei-Fei Li
Usual Challenges:
Variability due to:
• View point
• Illumination
• Occlusions
28-Nov-114
![Page 5: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/5.jpg)
Lecture 16 -Fei-Fei Li
Basic issues
• Representation
– 2D Bag of Words (BoW) models;
– Part-based models;
– Multi-view models;
• Learning
– Generative & Discriminative BoW models
– Generative models
– Probabilistic Hough voting
• Recognition
– Classification with BoW
– Classification with Part-based models
28-Nov-115
![Page 6: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/6.jpg)
Lecture 16 -Fei-Fei Li
Basic issues
• Representation
– 2D Bag of Words (BoW) models;
– Part-based models;
– Multi-view models;
• Learning
– Generative & Discriminative BoW models
– Generative models
– Probabilistic Hough voting
• Recognition
– Classification with BoW
– Classification with Part-based models
28-Nov-116
![Page 7: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/7.jpg)
Lecture 16 -Fei-Fei Li
Basic issues
• Representation
– 2D Bag of Words (BoW) models;
– Part-based models;
– Multi-view models (Lecture #19);
• Learning
– Generative & Discriminative BoW models
– Generative models
– Probabilistic Hough voting
• Recognition
– Classification with BoW
– Classification with Part-based models
28-Nov-117
![Page 8: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/8.jpg)
Lecture 16 -Fei-Fei Li
Problem with bag-of-words
• All have equal probability for bag-of-words methods
• Location information is important
28-Nov-118
![Page 9: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/9.jpg)
Lecture 16 -Fei-Fei Li
Model: Parts and Structure
28-Nov-119
![Page 10: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/10.jpg)
Lecture 16 -Fei-Fei Li
• Fischler & Elschlager 1973
• Yuille ‘91• Brunelli & Poggio ‘93• Lades, v.d. Malsburg et al. ‘93• Cootes, Lanitis, Taylor et al. ‘95• Amit & Geman ‘95, ‘99 • et al. Perona ‘95, ‘96, ’98, ’00, ‘03• Huttenlocher et al. ’00• Agarwal & Roth ’02
etc…
Parts and Structure Literature
28-Nov-1110
![Page 11: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/11.jpg)
Lecture 16 -Fei-Fei Li
The Constellation ModelT. Leung
M. Burl
Representation
Detection
Shape statistics – F&G ’95Affine invariant shape – CVPR ‘98
CVPR ‘96ECCV ‘98
M. WeberM. Welling
Unsupervised LearningECCV ‘00Multiple views - F&G ’00 Discovering categories - CVPR ’00
R. Fergus
L. Fei-Fei
Joint shape & appearance learningGeneric feature detectors
One-Shot LearningIncremental learning
CVPR ’03Polluted datasets - ECCV ‘04
ICCV ’03CVPR ‘04
28-Nov-1111
![Page 12: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/12.jpg)
Lecture 16 -Fei-Fei Li
A B
DC
Deformations
28-Nov-1112
![Page 13: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/13.jpg)
Lecture 16 -Fei-Fei Li
Presence / Absence of Features
occlusion
28-Nov-1113
![Page 14: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/14.jpg)
Lecture 16 -Fei-Fei Li
Background clutter
28-Nov-1114
![Page 15: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/15.jpg)
Lecture 16 -Fei-Fei Li
Foreground modelGenerative probabilistic model
Gaussian shape pdf
Clutter modelUniform shape pdfProb. of detection
0.8 0.75
0.9
# detections
pPoisson(N2|λλλλ2)
pPoisson(N1|λλλλ1)
pPoisson(N3|λλλλ3)
Assumptions: (a) Clutter independent of foreground detections(b) Clutter detections independent of each other
Example1. Object Part Positions
3a. N false detect2. Part Absence
N1
N2
3b. Position f. detect
N3
28-Nov-1115
![Page 16: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/16.jpg)
Lecture 16 -Fei-Fei Li
Learning Models `Manually’
• Obtain set of training images
• Label parts by hand, train detectors
• Learn model from labeled parts
• Choose parts
28-Nov-1116
![Page 17: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/17.jpg)
Lecture 16 -Fei-Fei Li
Recognition1. Run part detectors exhaustively over image
=
=
2
0
3
2
e.g.
0
0
0
0
4
3
2
1
h
N
N
N
N
h
K
K
K
K
1
2
3
3
2
41
1
2 3
1
2
2. Try different combinations of detections in model- Allow detections to be missing (occlusion)
3. Pick hypothesis which maximizes:
4. If ratio is above threshold then, instance detected
),|(
),|(
HypClutterDatap
HypObjectDatap
28-Nov-1117
![Page 18: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/18.jpg)
Lecture 16 -Fei-Fei Li
So far…..• Representation
– Joint model of part locations– Ability to deal with background clutter and occlusions
• Learning– Manual construction of part detectors– Estimate parameters of shape density
• Recognition– Run part detectors over image– Try combinations of features in model– Use efficient search techniques to make fast
28-Nov-1118
![Page 19: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/19.jpg)
Lecture 16 -Fei-Fei Li
Unsupervised LearningWeber & Welling et. al.
28-Nov-1119
![Page 20: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/20.jpg)
Lecture 16 -Fei-Fei Li
(Semi) Unsupervised learning
•Know if image contains object or not•But no segmentation of object or manual selection of features
28-Nov-1120
![Page 21: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/21.jpg)
Lecture 16 -Fei-Fei Li
Unsupervised detector training - 1
• Highly textured neighborhoods are selected automatically• produces 100-1000 patterns per image
10
10
28-Nov-1121
![Page 22: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/22.jpg)
Lecture 16 -Fei-Fei Li
Unsupervised detector training - 2
“Pattern Space” (100+ dimensions)
28-Nov-1122
![Page 23: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/23.jpg)
Lecture 16 -Fei-Fei Li
Unsupervised detector training - 3
100-1000 images ~100 detectors
28-Nov-1123
![Page 24: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/24.jpg)
Lecture 16 -Fei-Fei Li
• Task: Estimation of model parameters
Learning
• Let the assignments be a hidden variable and use EM algorithm to learn them and the model parameters
• Chicken and Egg type problem, since we initially know neither:
- Model parameters
- Assignment of regions to foreground / background
• Take training images. Pick set of detectors. Apply detectors.
28-Nov-1124
![Page 25: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/25.jpg)
Lecture 16 -Fei-Fei Li
ML using EM1. Current estimate
...
Image 1 Image 2 Image i
2. Assign probabilities to constellations
Large P
Small P
3. Use probabilities as weights to re-estimate parameters. Example: µµµµ
Large P x + Small P x
new estimate of µµµµ
+ … =
28-Nov-1125
![Page 26: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/26.jpg)
Lecture 16 -Fei-Fei Li
Detector Selection
ParameterEstimation
Choice 1
Choice 2ParameterEstimation
Model 1
Model 2
Predict / measure model performance(validation set or directly from model)
Detectors (≈100)
•Try out different combinations of detectors (Greedy search)
28-Nov-1126
![Page 27: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/27.jpg)
Lecture 16 -Fei-Fei Li
Frontal Views of Faces
• 200 Images (100 training, 100 testing)
• 30 people, different for training and testing
28-Nov-1127
![Page 28: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/28.jpg)
Lecture 16 -Fei-Fei Li
Learned face modelPre-selected Parts
Model Foreground pdf
Sample Detection
Parts in Model
Test Error: 6% (4 Parts)
28-Nov-1128
![Page 29: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/29.jpg)
Lecture 16 -Fei-Fei Li
Face images
28-Nov-1129
![Page 30: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/30.jpg)
Lecture 16 -Fei-Fei Li
Background images
28-Nov-1130
![Page 31: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/31.jpg)
Lecture 16 -Fei-Fei Li
Preselected Parts
Model Foreground pdf
Sample Detection
Parts in Model
Car from RearTest Error: 13% (5 Parts)
28-Nov-1131
![Page 32: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/32.jpg)
Lecture 16 -Fei-Fei Li
Detections of Cars
28-Nov-1132
![Page 33: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/33.jpg)
Lecture 16 -Fei-Fei Li
Background Images
28-Nov-1133
![Page 34: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/34.jpg)
Lecture 16 -Fei-Fei Li
3D Object recognition – Multiple mixture components
28-Nov-1134
![Page 35: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/35.jpg)
Lecture 16 -Fei-Fei Li
3D Orientation Tuning
Frontal Profile
0 20 40 60 80 10050
55
60
65
70
75
80
85
90
95
100Orientation Tuning
angle in degrees
% C
orr
ect
% C
orr
ect
28-Nov-1135
![Page 36: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/36.jpg)
Lecture 16 -Fei-Fei Li
So far (2)…..• Representation
– Multiple mixture components for different viewpoints
• Learning– Now semi-unsupervised– Automatic construction and selection of part detectors– Estimation of parameters using EM
• Recognition– As before
• Issues:-Learning is slow (many combinations of detectors)-Appearance learnt first, then shape
28-Nov-1136
![Page 37: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/37.jpg)
Lecture 16 -Fei-Fei Li
Issues• Speed of learning
– Slow (many combinations of detectors)
• Appearance learnt first, then shape– Difficult to learn part that has stable location but
variable appearance– Each detector is used as a cross-correlation filter,
giving a hard definition of the part’s appearance
• Would like a fully probabilistic representation of the object
28-Nov-1137
![Page 38: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/38.jpg)
Lecture 16 -Fei-Fei Li
Object categorization
Fergus et. al.
CVPR ’03, IJCV ‘0628-Nov-1138
![Page 39: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/39.jpg)
Lecture 16 -Fei-Fei Li
Detection & Representation of regions
Appearance
Location
Scale
(x,y) coords. of region centre
Radius of region (pixels)
11x11 patchNormalizeProjection onto
PCA basis
c1
c2
c15
……
…..
Gives representation of appearance in low-dimensional vector space
• Find regions within image
• Use salient region operator(Kadir & Brady 01)
28-Nov-1139
![Page 40: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/40.jpg)
Lecture 16 -Fei-Fei Li
Motorbikes example•Kadir & Brady saliency region detector
28-Nov-1140
![Page 41: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/41.jpg)
Lecture 16 -Fei-Fei Li
Foreground modelGaussian shape pdf
Poission pdf on # detections
Uniform shape pdf
Generative probabilistic model (2)
Clutter model
Gaussian part appearance pdf
Gaussian background appearance pdf
Prob. of detection
0.8 0.75 0.9
Gaussian relative scale pdf
log(scale)
Uniformrelative scale pdf
log(scale)
based on Burl, Weber et al. [ECCV ’98, ’00]
28-Nov-1141
![Page 42: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/42.jpg)
Lecture 16 -Fei-Fei Li
MotorbikesSamples from appearance model
28-Nov-1142
![Page 43: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/43.jpg)
Lecture 16 -Fei-Fei Li
Recognized Motorbikes
28-Nov-1143
![Page 44: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/44.jpg)
Lecture 16 -Fei-Fei Li
Background images evaluated with motorbike model
28-Nov-1144
![Page 45: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/45.jpg)
Lecture 16 -Fei-Fei Li
Frontal faces
28-Nov-1145
![Page 46: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/46.jpg)
Lecture 16 -Fei-Fei Li
Airplanes
28-Nov-1146
![Page 47: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/47.jpg)
Lecture 16 -Fei-Fei Li
Spotted cats
28-Nov-1147
![Page 48: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/48.jpg)
Lecture 16 -Fei-Fei Li
Summary of results
DatasetFixed scale experiment
Scale invariant experiment
Motorbikes 7.5 6.7
Faces 4.6 4.6
Airplanes 9.8 7.0
Cars (Rear) 15.2 9.7
Spotted cats 10.0 10.0
% equal error rateNote: Within each series, same settings used for all datasets
28-Nov-1148
![Page 49: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/49.jpg)
Lecture 16 -Fei-Fei Li
Comparison to other methods
Dataset Ours Others
Motorbikes 7.5 16.0Weber et al. [ECCV ‘00]
Faces 4.6 6.0 Weber
Airplanes 9.8 32.0 Weber
Cars (Side) 11.5 21.0Agarwal
Roth [ECCV ’02]
�% equal error rate
28-Nov-1149
![Page 50: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/50.jpg)
Lecture 16 -Fei-Fei Li
Why this design?
• Generic features seem to well in finding consistent parts of the object
• Some categories perform badly – different feature types needed
• Why PCA representation?– Tried ICA, FLD, Oriented filter responses etc.– But PCA worked best
• Fully probabilistic representation lets us use tools from machine learning community
28-Nov-1150
![Page 51: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/51.jpg)
Lecture 16 -Fei-Fei Li
S. Savarese, 2003
28-Nov-1151
![Page 52: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/52.jpg)
Lecture 16 -Fei-Fei Li P. Buegel, 156228-Nov-1152
![Page 53: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/53.jpg)
Lecture 16 -Fei-Fei Li
One-Shot learningFei-Fei et. al.
ICCV ’03, PAMI ‘0628-Nov-1153
![Page 54: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/54.jpg)
Lecture 16 -Fei-Fei Li
AlgorithmTraining
ExamplesCategories
Burl, et al. Weber, et al. Fergus, et al.
200 ~ 400Faces, Motorbikes,
Spotted cats, Airplanes, Cars
Viola et al. ~10,000 Faces
Schneiderman, et al. ~2,000 Faces, Cars
Rowley et al.
~500 Faces
28-Nov-1154
![Page 55: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/55.jpg)
Lecture 16 -Fei-Fei Li
1 2 3 4 5 6 7 8 90
10
20
30
40
50
60
log2 (Training images)
Cla
ssifi
catio
n er
ror
(%)
Generalisation performance
TestTrain
Number of training examples
Previously
6 part Motorbike model
28-Nov-1155
![Page 56: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/56.jpg)
Lecture 16 -Fei-Fei Li
How do we do better than what statisticians have told us?
• Intuition 1: use Prior information
• Intuition 2: make best use of training information
28-Nov-1156
![Page 57: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/57.jpg)
Lecture 16 -Fei-Fei Li
Prior knowledge: means
ShapeAppearance
28-Nov-1157
![Page 58: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/58.jpg)
Lecture 16 -Fei-Fei Li
Bayesian framework
P(object | test, train) vs. P(clutter | test, train)
)object()trainobject,|test( pp
Bayes Rule
θθθ dpp∫ )trainobject,|()object,|test(
Expansion by parametrization
28-Nov-1158
![Page 59: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/59.jpg)
Lecture 16 -Fei-Fei Li
( )MLθδPrevious Work:
Bayesian framework
P(object | test, train) vs. P(clutter | test, train)
)object()trainobject,|test( pp
Bayes Rule
θθθ dpp∫ )trainobject,|()object,|test(
Expansion by parametrization
28-Nov-1159
![Page 60: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/60.jpg)
Lecture 16 -Fei-Fei Li
One-Shot learning: ( ) ( )θθ pp object,train
Bayesian framework
P(object | test, train) vs. P(clutter | test, train)
)object()trainobject,|test( pp
Bayes Rule
θθθ dpp∫ )trainobject,|()object,|test(
Expansion by parametrization
28-Nov-1160
![Page 61: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/61.jpg)
Lecture 16 -Fei-Fei Li
θθθθ1
θθθθ2θθθθn
model ( θθθθ) space
Each object model θθθθ
Gaussian shape pdfGaussian part
appearance pdf
Model Structure
28-Nov-1161
![Page 62: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/62.jpg)
Lecture 16 -Fei-Fei Li
θθθθ2θθθθn
model distribution: p( θθθθ)• conjugate distribution of p(train| θθθθ,object)
θθθθ1
model ( θθθθ) space
Each object model θθθθ
Gaussian shape pdfGaussian part
appearance pdf
Model Structure
28-Nov-1162
![Page 63: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/63.jpg)
Lecture 16 -Fei-Fei Li
Learning Model Distribution
• use Prior information
• Bayesian learning
• marginalize over theta
� Variational EM (Attias, Hinton, Minka, etc.)
( ) ( ) ( )θθθ ppp object ,train trainobject, ∝
28-Nov-1163
![Page 64: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/64.jpg)
Lecture 16 -Fei-Fei Li
E-Step
Random initializationVariational EM
prior knowledge of p(θθθθ)
new estimate of p( θθθθ|train)
M-Step
new θθθθ’s
28-Nov-1164
![Page 65: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/65.jpg)
Lecture 16 -Fei-Fei Li
Experiments
Training:
1- 6 randomly
drawn images
Testing:
50 fg/ 50 bg images
object present/absent
Datasets
spotted catsairplanes motorbikesfaces
28-Nov-1165
![Page 66: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/66.jpg)
Lecture 16 -Fei-Fei Li
Faces
Airplanes
Motorbikes
Spotted cats
28-Nov-1166
![Page 67: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/67.jpg)
Lecture 16 -Fei-Fei Li
Experiments: obtaining priors
spotted cats
airplanes
motorbikes
faces
model ( θθθθ) space
28-Nov-1167
![Page 68: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/68.jpg)
Lecture 16 -Fei-Fei Li
Experiments: obtaining priors
spotted cats
faces
airplanes
motorbikes
model ( θθθθ) space
28-Nov-1168
![Page 69: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/69.jpg)
Lecture 16 -Fei-Fei Li
Number of training examples
28-Nov-1169
![Page 70: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/70.jpg)
Lecture 16 -Fei-Fei Li
Number of training examples
28-Nov-1170
![Page 71: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/71.jpg)
Lecture 16 -Fei-Fei Li
Number of training examples
28-Nov-1171
![Page 72: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/72.jpg)
Lecture 16 -Fei-Fei Li
Number of training examples
28-Nov-1172
![Page 73: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/73.jpg)
Lecture 16 -Fei-Fei Li
AlgorithmTraining
ExamplesCategories
Results(error)
Burl, et al. Weber, et al. Fergus, et al.
200 ~ 400Faces, Motorbikes,
Spotted cats, Airplanes, Cars
5.6 - 10 %
Viola et al. ~10,000 Faces 7-21%
Schneiderman, et al. ~2,000 Faces, Cars 5.6 – 17%
Rowley et al.
~500 Faces7.5 –
24.1%
BayesianOne-Shot 1 ~ 5 Faces, Motorbikes,
Spotted cats, Airplanes8 –
15 %
28-Nov-1173
![Page 74: Lecture 16: Object recognition: Part-based …vision.stanford.edu/teaching/cs231a_autumn1213_internal/...Fei-Fei Li Lecture 16 - The Constellation Model T. Leung M. Burl Representation](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ed815b2cba89e334c673799/html5/thumbnails/74.jpg)
Lecture 16 -Fei-Fei Li
What we have learned today?
28-Nov-1174
• Introduction
• Constellation model
– Weakly supervised training
– One-shot learning
• (Problem Set 4 (Q1))