![Page 1: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/1.jpg)
Efficiently searching for similar images (Kristen Grauman)
Universidad Católica San Pablo
Cristina Patricia Cáceres Jáuregui
![Page 2: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/2.jpg)
Motivation
Fast image search is a useful component for a number of vision problems.
Plenty of nuisance parameters (lighting, pose, background clutter, etc.)
![Page 3: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/3.jpg)
Nuisance parameters
![Page 4: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/4.jpg)
Outline
Scalable image search
• Fast correspondence-based search with local features
• Fast similarity search for learned metrics
![Page 5: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/5.jpg)
Local image features
![Page 6: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/6.jpg)
How to handle sets of features?
Want to compare, index, cluster, etc. local representations, but:
• Each instance is unordered set of vectors• Varying number of vectors per instance
![Page 7: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/7.jpg)
Comparing sets of local features Previous strategies:
• Match features individually, vote on small sets to verify
• Explicit search for one-to-one correspondences
• Bag-of-words: Compare frequencies of prototype features
![Page 8: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/8.jpg)
Pyramid match kernel
optimal partial matching
Optimal match: O(m3)Pyramid match: O(mL)
m = # featuresL = # levels in pyramid
![Page 9: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/9.jpg)
Pyramid match: main idea
descriptor space
Feature space partitions serve to “match” the local descriptors within successively wider regions.
![Page 10: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/10.jpg)
Pyramid match: main idea
Histogram intersection counts number of possible matches at a given partitioning.
![Page 11: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/11.jpg)
Image search with matching-sensitive hash functions
• Main idea:– Map point sets to a vector space in such a
way that a dot product reflects partial match similarity (normalized PMK value).
– Exploit random hyperplane properties to construct matching-sensitive hash functions.
– Perform approximate similarity search on hashed examples.
![Page 12: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/12.jpg)
Locality Sensitive Hashing (LSH)
Q111101
110111
110101
h r1…rkXi
N
h r1…rk
<< N
Q
Guarantee “approximate”-nearest neighbors in sub-linear time, given appropriate hash functions.
Randomized LSHfunctions
![Page 13: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/13.jpg)
LSH functions for dot productsThe probability that a random hyperplane separates two unit vectors depends on the angle between them:
A)High dot product: unlikely to split
B)Lower dot product: likely to split
Corresponding hash function:
![Page 14: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/14.jpg)
Metric learning
There are various ways to judge appearance/shape similarity…
but often we know more about (some) data than just their appearance.
![Page 15: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/15.jpg)
Metric learning
• Exploit partially labeled data and/or (dis)similarity constraints to construct more useful distance function
• Can dramatically boost performance on clustering, indexing, classification tasks.
• Various existing techniques
![Page 16: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/16.jpg)
Fast similarity search for learned metrics
• Goal: – Maintain query time guarantees while performing approximate search with a learned metric
• Main idea:– Learn Mahalanobis distance parameterization– Use it to affect distribution from which random hash functions are selected
• LSH functions that preserve the learned metric
• Approximate NN search with existing methods
![Page 17: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/17.jpg)
Fast Image Search for Learned Metrics
It should be unlikely that a hash function will split examples like those having similarity constraints…
…but likely that it splits those having dissimilarity constraints.
h( ) = h( ) h( ) ≠ h( )
Learn a Malhanobis metric for LSH
![Page 18: Universidad Católica San Pablo Cristina Patricia Cáceres Jáuregui cristina.caceres.jauregui@ucsp.edu.pe](https://reader036.vdocuments.site/reader036/viewer/2022062518/56649cab5503460f9496c82c/html5/thumbnails/18.jpg)
• Local image features useful, important to handle efficiently
• Introduced scalable methods to allow fast similarity search methods with
– Local feature matching– Learned Mahalanobis metrics
• Key idea: design hash functions that encode matching process, or the constraints provided
Summary