shape matching and object recognition using low distortion correspondence alexander c. berg, tamara...
TRANSCRIPT
![Page 1: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/1.jpg)
Shape Matching
and Object Recognition
using Low Distortion Correspondence
Alexander C. Berg, Tamara L. Berg, Jitendra Malik
U.C. Berkeley
![Page 2: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/2.jpg)
Object Category Recognition
![Page 3: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/3.jpg)
Deformable Template Matching with Exemplars for Recognition
● Use exemplars as deformable templates
● Find a correspondence between the query image and each template
Query
Image
Database of
Templates
![Page 4: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/4.jpg)
Deformable Template Matching with Exemplars for Recognition
● Use exemplars as deformable templates
● Find a correspondence between the query image and each template
Query
Image
Database of
Templates
Best matching template is a helicopter
![Page 5: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/5.jpg)
Correspondence for Deformable Template Matching
● Evaluate correspondence based on:
– Similarity of appearance near feature points
– Similarity in configuration of the feature points
QueryTemplate
![Page 6: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/6.jpg)
Correspondence for Deformable Template Matching
● Evaluate correspondence based on:
– Similarity of appearance near feature points
– Similarity in configuration of the feature points
QueryTemplate
![Page 7: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/7.jpg)
Correspondence for Deformable Template Matching
● Evaluate correspondence based on:
– Similarity of appearance near feature points
– Similarity in configuration of the feature points
QueryTemplate
![Page 8: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/8.jpg)
Correspondence for Deformable Template Matching
● Evaluate correspondence based on:
– Similarity of appearance near feature points
– Similarity in configuration of the feature points
QueryTemplate
![Page 9: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/9.jpg)
Correspondence Result
QueryTemplate
![Page 10: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/10.jpg)
Interpolated CorrespondenceUsing Thin Plate Splines
QueryTemplate
![Page 11: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/11.jpg)
Correspondence for Deformable Template Matching
If i i' and j j' then rijri'j'
If i i' then pipi'
i i'
j j'
i i'
j j'QueryTemplate r
ijr
i'j'
![Page 12: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/12.jpg)
Geometric Blur(Local Appearance Descriptor)
Geometric Blur Descriptor
~
Compute sparse
channels from image
Extract a patch
in each channel
Apply spatially varying
blur and sub-sample
(Idealized signal)
Descriptor is robust to small affine distortions
Berg & Malik '01
![Page 13: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/13.jpg)
Geometric Blur(Local Appearance Similarity)
Geometric Blur Descriptor
Geometric Blur Descriptor~
![Page 14: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/14.jpg)
Are Features Enough?
Not Quite...
Color indicates
similarity using
Geometric Blur
Descriptor
![Page 15: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/15.jpg)
Measuring Distortion(Similarity in Configuration)
i i'
j j'
i i'
j j'QueryTemplate R
ij Si'j'
Measure distortion in vectors between pairs of feature points
- R and S same length for rotations
- R and S same direction for scalings
![Page 16: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/16.jpg)
Cost Function as IQP
Appearance costif i -> j
Distortion cost ifi -> j and k -> l
Integer Quadratic Programming Problem...
iff template point i maps to query point j, If binary vector x represents a correspondence
cf. Maciel & Costeira '03
![Page 17: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/17.jpg)
Optimization
● Integer Quadratic Programming is NP hard
● The instances we generate seem easy
● Using a linear bound to initialize gradient descent provides good results
– (In fact better than the guarantee of Goemans &
Williamson's randomized algorithm)
● Varying the linear constraints on x allows
– one-one, one-many, or fixed number of outliers, etc.
![Page 18: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/18.jpg)
Correspondence Result
![Page 19: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/19.jpg)
Interpolated CorrespondenceUsing Thin Plate Splines
![Page 20: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/20.jpg)
Quadratic Assignment(Using IQP)
![Page 21: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/21.jpg)
Linear Assignment(e.g. Hungarian)
![Page 22: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/22.jpg)
Correspondence Examples (Shape Matching)
![Page 23: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/23.jpg)
Correspondence Examples(Shape Matching)
![Page 24: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/24.jpg)
Correspondence Examples(Shape Matching)
![Page 25: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/25.jpg)
Correspondence Examples(Shape Matching)
![Page 26: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/26.jpg)
Correspondence Examples(Shape Matching)
![Page 27: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/27.jpg)
Correspondence Examples(Shape Matching)
![Page 28: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/28.jpg)
Correspondence Examples(Shape Matching)
![Page 29: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/29.jpg)
Correspondence Examples(Shape Matching)
![Page 30: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/30.jpg)
Correspondence Examples(Shape Matching)
![Page 31: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/31.jpg)
Correspondence Examples(Shape Matching)
![Page 32: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/32.jpg)
Correspondence Examples(Shape Matching)
![Page 33: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/33.jpg)
Application to RecognitionCaltech 101
– 101 classes of objects + “background”
● Large Scale
● Roughly aligned
● Large intra-class variation
● Fei-Fei, Fergus, Perona '04
![Page 34: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/34.jpg)
Caltech 101 Recognition Results
Chance ~1%
N.N. whole image 16%
Discriminative version ofConstellation Model 27%
N.N. Geometric BlurDescriptors 38%
Low Distortion Correspondence (GB+IQP) 45%
102 way Alternative Forced Choice test
(15 training examples per class)
102 way confusion matrix
100%
0%
![Page 35: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/35.jpg)
Model Building for Segmentation
Rough correspondenceto each example image
Average quality of alignment
Threshhold
![Page 36: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/36.jpg)
Automatic vs Hand Segmentation
![Page 37: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/37.jpg)
Application to RecognitionFaces
● Face dataset from Berg et al '03
– Medium to large scale faces
– AP News photos
● 20 face exemplars
● Same methodology as Caltech 101, but multiple objects / image
– After one face is identified its features are removed and the
search continues
● Compared to a detector from Mikolajczyk based on
Schneiderman & Kanade, that is quite successful on this dataset
![Page 38: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/38.jpg)
Application to RecognitionFaces
![Page 39: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/39.jpg)
Conclusion
● Use rich descriptors that are insensitive to typical
transformations
– Geometric Blur
● Enforce relationship constraints among corresponding
features
– Integer Quadratic Programming
● Estimate smooth transform
– Thin Plate Splines
![Page 40: Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley](https://reader035.vdocuments.site/reader035/viewer/2022062421/56649b4a550346318e8c0e2f/html5/thumbnails/40.jpg)
Thank You
Acknowledgements
Charless FowlkesXiaofeng RenDavid Forsyth