Shape Matching
and Object Recognition
using Low Distortion Correspondence
Alexander C. Berg, Tamara L. Berg, Jitendra Malik
U.C. Berkeley
Object Category Recognition
Deformable Template Matching with Exemplars for Recognition
● Use exemplars as deformable templates
● Find a correspondence between the query image and each template
Query
Image
Database of
Templates
Deformable Template Matching with Exemplars for Recognition
● Use exemplars as deformable templates
● Find a correspondence between the query image and each template
Query
Image
Database of
Templates
Best matching template is a helicopter
Correspondence for Deformable Template Matching
● Evaluate correspondence based on:
– Similarity of appearance near feature points
– Similarity in configuration of the feature points
QueryTemplate
Correspondence for Deformable Template Matching
● Evaluate correspondence based on:
– Similarity of appearance near feature points
– Similarity in configuration of the feature points
QueryTemplate
Correspondence for Deformable Template Matching
● Evaluate correspondence based on:
– Similarity of appearance near feature points
– Similarity in configuration of the feature points
QueryTemplate
Correspondence for Deformable Template Matching
● Evaluate correspondence based on:
– Similarity of appearance near feature points
– Similarity in configuration of the feature points
QueryTemplate
Correspondence Result
QueryTemplate
Interpolated CorrespondenceUsing Thin Plate Splines
QueryTemplate
Correspondence for Deformable Template Matching
If i i' and j j' then rijri'j'
If i i' then pipi'
i i'
j j'
i i'
j j'QueryTemplate r
ijr
i'j'
Geometric Blur(Local Appearance Descriptor)
Geometric Blur Descriptor
~
Compute sparse
channels from image
Extract a patch
in each channel
Apply spatially varying
blur and sub-sample
(Idealized signal)
Descriptor is robust to small affine distortions
Berg & Malik '01
Geometric Blur(Local Appearance Similarity)
Geometric Blur Descriptor
Geometric Blur Descriptor~
Are Features Enough?
Not Quite...
Color indicates
similarity using
Geometric Blur
Descriptor
Measuring Distortion(Similarity in Configuration)
i i'
j j'
i i'
j j'QueryTemplate R
ij Si'j'
Measure distortion in vectors between pairs of feature points
- R and S same length for rotations
- R and S same direction for scalings
Cost Function as IQP
Appearance costif i -> j
Distortion cost ifi -> j and k -> l
Integer Quadratic Programming Problem...
iff template point i maps to query point j, If binary vector x represents a correspondence
cf. Maciel & Costeira '03
Optimization
● Integer Quadratic Programming is NP hard
● The instances we generate seem easy
● Using a linear bound to initialize gradient descent provides good results
– (In fact better than the guarantee of Goemans &
Williamson's randomized algorithm)
● Varying the linear constraints on x allows
– one-one, one-many, or fixed number of outliers, etc.
Correspondence Result
Interpolated CorrespondenceUsing Thin Plate Splines
Quadratic Assignment(Using IQP)
Linear Assignment(e.g. Hungarian)
Correspondence Examples (Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Correspondence Examples(Shape Matching)
Application to RecognitionCaltech 101
– 101 classes of objects + “background”
● Large Scale
● Roughly aligned
● Large intra-class variation
● Fei-Fei, Fergus, Perona '04
Caltech 101 Recognition Results
Chance ~1%
N.N. whole image 16%
Discriminative version ofConstellation Model 27%
N.N. Geometric BlurDescriptors 38%
Low Distortion Correspondence (GB+IQP) 45%
102 way Alternative Forced Choice test
(15 training examples per class)
102 way confusion matrix
100%
0%
Model Building for Segmentation
Rough correspondenceto each example image
Average quality of alignment
Threshhold
Automatic vs Hand Segmentation
Application to RecognitionFaces
● Face dataset from Berg et al '03
– Medium to large scale faces
– AP News photos
● 20 face exemplars
● Same methodology as Caltech 101, but multiple objects / image
– After one face is identified its features are removed and the
search continues
● Compared to a detector from Mikolajczyk based on
Schneiderman & Kanade, that is quite successful on this dataset
Application to RecognitionFaces
Conclusion
● Use rich descriptors that are insensitive to typical
transformations
– Geometric Blur
● Enforce relationship constraints among corresponding
features
– Integer Quadratic Programming
● Estimate smooth transform
– Thin Plate Splines
Thank You
Acknowledgements
Charless FowlkesXiaofeng RenDavid Forsyth