features-based object recognition pierre moreels california institute of technology thesis defense,...
Post on 15-Jan-2016
218 views
TRANSCRIPT
![Page 1: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/1.jpg)
Features-based Object Recognition
Pierre MoreelsCalifornia Institute of Technology
Thesis defense, Sept. 24, 2007
![Page 2: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/2.jpg)
2
The recognition continuumvariab
ility
Individual objects
means of transportation
BMW logo
Categories
cars
![Page 3: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/3.jpg)
Applications
Autonomousnavigation
Identification, Security.
Help Daiki find his toys !
![Page 4: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/4.jpg)
4
• Problem setup
• Features
• Coarse-to-fine algorithm
• Probabilistic model
• Experiments
• Conclusion
Outline
![Page 5: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/5.jpg)
5
…
The detection problem
New scene (test image)
Models fromdatabase
Find models and their pose (location, orientation…)
![Page 6: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/6.jpg)
6
…
Hypotheses – models + positions
New scene (test image)
Models fromdatabase
1
2
Θ = affine transformation
![Page 7: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/7.jpg)
7
…
Matching features
Models fromdatabase
New scene (test image)
Set of correspondences = assignment vector
![Page 8: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/8.jpg)
8
Features detection
![Page 9: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/9.jpg)
9
Image characterization by features
• Features = high information content
‘locations in the image where the signal changes two-dimensionally’ C.Schmid
• Reduce the volume of information
edge strength map
features
– [Sobel 68]– Diff of Gaussians [Crowley84]– [Harris 88]– [Foerstner94]– Entropy [Kadir&Brady01]
![Page 10: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/10.jpg)
10
Correct vs incorrect descriptors matches
Mutual Euclidean distances in appearance space of descriptors
12
34
5
6
7
8
- Pixels intensity within a patch- Steerable filters [Freeman1991]- SIFT [Lowe1999,2004]- Shape context [Belongie2002]- Spin [Johnson1999]- HOG [Dalal2005]
![Page 11: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/11.jpg)
11
Stability with respect to nuisances
Which detector / descriptor
combination is best for recognition ?
![Page 12: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/12.jpg)
Past work on evaluation of features• Use of flat surfaces, ground truth easily established• In 3D images appearance changes more !
[Schmid&Mohr00] [Mikolajczyk&Schmid 03,05,05]
![Page 13: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/13.jpg)
13
Database : 100 3D objects
![Page 14: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/14.jpg)
14
Testing setup
[Moreels&Perona ICCV05, IJCV07]
Used by [Winder, CVPR07]
![Page 15: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/15.jpg)
Results – viewpoint change M
ahal
anob
is d
ista
nce
No
‘bac
kgro
und’
imag
es
![Page 16: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/16.jpg)
2D vs. 3D
Ranking of detectors/descriptorscombinations are modified whenswitching from 2D to 3D objects
![Page 17: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/17.jpg)
17
Features matching algorithm
![Page 18: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/18.jpg)
18
Features assignments
models from database
New scene (test image)
. . .
Interpretation
. . .
![Page 19: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/19.jpg)
19
Coarse-to-fine strategy• We do it every day !
Search for my place : Los Angeles area – Pasadena – Loma Vista - 1351
my car
![Page 20: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/20.jpg)
Coarse-to-fine example
[Fleuret & Geman 2001,2002]
Face identification in complex scenes
Coarse resolution
Intermediate resolution
Fine resolution
![Page 21: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/21.jpg)
21
• Progressively narrow down focus on correct region of hypothesis space
• Reject with little computation cost irrelevant regions of search space
• Use first information that is easy to obtain
• Simple building blocks organized in a cascade
• Probabilistic interpretation of each step
Coarse-to-Fine detection
![Page 22: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/22.jpg)
22
Coarse data : prior knowledge
• Which objects are likely to be there, which pose are they likely to have ?
unlikelysituations
![Page 23: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/23.jpg)
23
New scene (test image)…
Models fromdatabase
4 votes
2 votes
0 vote
Model voting
Search tree (appearance space – leaves = database features)
![Page 24: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/24.jpg)
24
(x1,y1,s1,1)
(x2,y2,s2,2)
Transform predicted by this match: x = x2-x1
y = y2-y1
s = s2 / s1
= 2 - 1
Each match is represented by a dot in
the space of 2D similarities (Hough space)
x
y
s
Use of rich geometric information
[Lowe1999,2004]
![Page 25: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/25.jpg)
• Prediction of position of model center after transform
• The space of transform parameters is discretized into ‘bins’
• Coarse bins to limit boundary issues and have a low false-alarm rate for this stage
• We count the number of votes collected by each bin.
Coarse Hough transform
N~
Model
Test scene
correct transformation
![Page 26: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/26.jpg)
26Output of PROSAC : pose transformation
+ set of features correspondences
Correspondence or clutter ? PROSAC
• Similar to RANSAC – robust statistic for parameter estimation
• Priority to candidates with good quality of appearance match
• 2D affine transform : 6 parameters
each sample contains 3 candidate correspondences.
d
d
d
[Fischler 1973] [Chum&Matas 2005]
![Page 27: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/27.jpg)
27
Probabilistic model
![Page 28: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/28.jpg)
28
Generative model
![Page 29: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/29.jpg)
29
Recognition steps
![Page 30: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/30.jpg)
Score of an extended hypothesis
Hypothesis:model + position
observed featuresgeometry + appearance
database of models
constant
Consistency(after PROSAC)Prior on model
and poses
Featuresassignments
Votes per model Votes per model pose bin(Hough transform)
Prior on assignments(before actual observations)
![Page 31: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/31.jpg)
ConsistencyConsistency between observations and predictions from hypothesis
model m
position of model m
Common-frame approximation : parts are conditionally independent once reference position of the object is fixed. [Lowe1999,Huttenlocher90,Moreels04]
Con
stel
latio
n m
odel
Com
mon
-fra
me
![Page 32: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/32.jpg)
32
foreground features ‘null’ assignments
geometry geometryappearance appearance
Consistency - appearance Consistency - geometry
ConsistencyConsistency between observations and predictions from hypothesis
![Page 33: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/33.jpg)
Learning foreground & background densities
• Ground truth pairs of matches are collected
• Gaussian densities, centered on the nomimal value that appearance / pose should have according to H
• Learning background densities is easy: match to random images.
[Moreels&Perona, IJCV, 2007]
![Page 34: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/34.jpg)
34
Experiments
![Page 35: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/35.jpg)
An example
Model votin
g
Hough
bins
![Page 36: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/36.jpg)
36
An example
After
PROSAC
Probabilistic
scores
![Page 37: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/37.jpg)
37
Efficiency of coarse-to-fine processing
![Page 38: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/38.jpg)
38
Giuseppe Toys database – Models
61 objects, 1-2 views/object
![Page 39: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/39.jpg)
Giuseppe Toys database – Test scenes
141 test scenes
![Page 40: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/40.jpg)
40
Home objects database – Models
49 objects, 1-2 views/object
![Page 41: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/41.jpg)
41
Home objects database – Test scenes
141 test scenes
![Page 42: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/42.jpg)
42
Results – Giuseppe Toys database
Lowe’99,’04
Lower false alarmrate- more systematic verification of geometry consistency- more consistent verification of geometric consistency
undetected objects: features with poor appearance distinctivenessindex to incorrect models
-
+
![Page 43: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/43.jpg)
43
Results – Home objects database
![Page 44: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/44.jpg)
44
Failure modeTest image hand-labeledbefore the experiments
![Page 45: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/45.jpg)
45
Test – Text and graphics
![Page 46: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/46.jpg)
46
Test – no texture
![Page 47: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/47.jpg)
Test – Clutter
![Page 48: Features-based Object Recognition Pierre Moreels California Institute of Technology Thesis defense, Sept. 24, 2007](https://reader036.vdocuments.site/reader036/viewer/2022062322/56649d445503460f94a213d1/html5/thumbnails/48.jpg)
48
• Coarse-to-fine strategy prunes irrelevant search branches at early stages.
• Probabilistic interpretation of each step.
• Higher performance than Lowe, especially in cluttered environment.
• Front end (features) needs more work for smooth or shiny surfaces.
Conclusions