local features: detection and description
DESCRIPTION
Local features: detection and description . Monday March 7 Prof. Kristen Grauman UT-Austin. Midterm Wed. Covers material up until 3/1 Solutions to practice exam handed out today Bring a 8.5”x11” sheet of notes if you want - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/1.jpg)
Local features:detection and description
Monday March 7Prof. Kristen Grauman
UT-Austin
![Page 2: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/2.jpg)
Midterm Wed.
– Covers material up until 3/1– Solutions to practice exam handed out today– Bring a 8.5”x11” sheet of notes if you want– Review the outlines and notes on course website,
accompanying reading in textbook
![Page 3: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/3.jpg)
Last time
• Image warping based on homography• Detecting corner-like points in an image
![Page 4: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/4.jpg)
Today
• Local invariant features– Detection of interest points
• (Harris corner detection)• Scale invariant blob detection: LoG
– Description of local patches• SIFT : Histograms of oriented gradients
![Page 5: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/5.jpg)
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx x
],,[ )2()2(12 dxx x
Kristen Grauman
![Page 6: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/6.jpg)
Goal: interest operator repeatability• We want to detect (at least some of) the
same points in both images.
• Yet we have to be able to run the detection procedure independently per image.
No chance to find true matches!
![Page 7: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/7.jpg)
Goal: descriptor distinctiveness• We want to be able to reliably determine
which point goes with which.
• Must provide some invariance to geometric and photometric differences between the two views.
?
![Page 8: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/8.jpg)
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
![Page 9: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/9.jpg)
yyyx
yxxx
IIIIIIII
yxwM ),(
xII x
yII y
yI
xIII yx
Recall: Corners as distinctive interest points
2 x 2 matrix of image derivatives (averaged in neighborhood of a point).
Notation:
![Page 10: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/10.jpg)
Since M is symmetric, we have TXXM
2
1
00
iii xMx
The eigenvalues of M reveal the amount of intensity change in the two principal orthogonal gradient directions in the window.
Recall: Corners as distinctive interest points
![Page 11: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/11.jpg)
“flat” region1 and 2 are small;
“edge”:1 >> 2
2 >> 1
“corner”:1 and 2 are large, 1 ~ 2;
One way to score the cornerness:
Recall: Corners as distinctive interest points
![Page 12: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/12.jpg)
Harris corner detector
1) Compute M matrix for image window surrounding each pixel to get its cornerness score.
2) Find points with large corner response (f > threshold)
3) Take the points of local maxima, i.e., perform non-maximum suppression
![Page 13: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/13.jpg)
Harris Detector: Steps
![Page 14: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/14.jpg)
Harris Detector: StepsCompute corner response f
![Page 15: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/15.jpg)
Harris Detector: StepsFind points with large corner response: f > threshold
![Page 16: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/16.jpg)
Harris Detector: StepsTake only the points of local maxima of f
![Page 17: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/17.jpg)
Harris Detector: Steps
![Page 18: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/18.jpg)
Properties of the Harris corner detectorRotation invariant?
Scale invariant?
TXXM
2
1
00
Yes
![Page 19: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/19.jpg)
Properties of the Harris corner detectorRotation invariant?
Scale invariant?
All points will be classified as edges
Corner !
Yes
No
![Page 20: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/20.jpg)
Scale invariant interest pointsHow can we independently select interest points in each image, such that the detections are repeatable across different scales?
![Page 21: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/21.jpg)
Automatic scale selectionIntuition: • Find scale that gives local maxima of some function f
in both position and scale.
f
region size
Image 1f
region size
Image 2
s1 s2
![Page 22: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/22.jpg)
What can be the “signature” function?
![Page 23: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/23.jpg)
Recall: Edge detection
gdxdf
f
gdxd
Source: S. Seitz
Edge
Derivativeof Gaussian
Edge = maximumof derivative
![Page 24: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/24.jpg)
gdxdf 2
2
f
gdxd
2
2
Edge
Second derivativeof Gaussian (Laplacian)
Edge = zero crossingof second derivative
Source: S. Seitz
Recall: Edge detection
![Page 25: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/25.jpg)
From edges to blobs• Edge = ripple• Blob = superposition of two ripples
Spatial selection: the magnitude of the Laplacianresponse will achieve a maximum at the center ofthe blob, provided the scale of the Laplacian is“matched” to the scale of the blob
maximum
Slide credit: Lana Lazebnik
![Page 26: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/26.jpg)
Blob detection in 2DLaplacian of Gaussian: Circularly symmetric
operator for blob detection in 2D
2
2
2
22
yg
xgg
![Page 27: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/27.jpg)
Blob detection in 2D: scale selection
Laplacian-of-Gaussian = “blob” detector 2
2
2
22
yg
xgg
fil
ter s
cale
s
img1 img2 img3Bastian Leibe
![Page 28: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/28.jpg)
Blob detection in 2DWe define the characteristic scale as the scale
that produces peak of Laplacian response
characteristic scale
Slide credit: Lana Lazebnik
![Page 29: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/29.jpg)
Example
Original image at ¾ the size
Kristen Grauman
![Page 30: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/30.jpg)
Original image at ¾ the size
Kristen Grauman
![Page 31: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/31.jpg)
Kristen Grauman
![Page 32: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/32.jpg)
Kristen Grauman
![Page 33: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/33.jpg)
Kristen Grauman
![Page 34: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/34.jpg)
Kristen Grauman
![Page 35: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/35.jpg)
Kristen Grauman
![Page 36: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/36.jpg)
)()( yyxx LL
1
2
3
4
5
List of (x, y, σ)
scale
Scale invariant interest pointsInterest points are local maxima in both position
and scale.
Squared filter response maps
![Page 37: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/37.jpg)
Scale-space blob detector: Example
Image credit: Lana Lazebnik
![Page 38: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/38.jpg)
We can approximate the Laplacian with a difference of Gaussians; more efficient to implement.
2 ( , , ) ( , , )xx yyL G x y G x y
( , , ) ( , , )DoG G x y k G x y
(Laplacian)
(Difference of Gaussians)
Technical detail
![Page 39: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/39.jpg)
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx x
],,[ )2()2(12 dxx x
![Page 40: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/40.jpg)
Geometric transformations
e.g. scale, translation, rotation
![Page 41: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/41.jpg)
Photometric transformations
Figure from T. Tuytelaars ECCV 2006 tutorial
![Page 42: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/42.jpg)
Raw patches as local descriptors
The simplest way to describe the neighborhood around an interest point is to write down the list of intensities to form a feature vector.
But this is very sensitive to even small shifts, rotations.
![Page 43: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/43.jpg)
SIFT descriptor [Lowe 2004] • Use histograms to bin pixels within sub-patches
according to their orientation.
0 2p
Why subpatches?Why does SIFT have some illumination invariance?
![Page 44: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/44.jpg)
CSE 576: Computer Vision
Making descriptor rotation invariant
Image from Matthew Brown
• Rotate patch according to its dominant gradient orientation
• This puts the patches into a canonical orientation.
![Page 45: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/45.jpg)
• Extraordinarily robust matching technique• Can handle changes in viewpoint
• Up to about 60 degree out of plane rotation• Can handle significant changes in illumination
• Sometimes even day vs. night (below)• Fast and efficient—can run in real time• Lots of code available
• http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT
Steve Seitz
SIFT descriptor [Lowe 2004]
![Page 46: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/46.jpg)
Example
NASA Mars Rover images
![Page 47: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/47.jpg)
NASA Mars Rover imageswith SIFT feature matchesFigure by Noah Snavely
Example
![Page 48: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/48.jpg)
SIFT properties• Invariant to
– Scale – Rotation
• Partially invariant to– Illumination changes– Camera viewpoint– Occlusion, clutter
![Page 49: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/49.jpg)
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
![Page 50: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/50.jpg)
Matching local features
Kristen Grauman
![Page 51: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/51.jpg)
Matching local features
?
To generate candidate matches, find patches that have the most similar appearance (e.g., lowest SSD)Simplest approach: compare them all, take the closest (or closest k, or within a thresholded distance)
Image 1 Image 2
Kristen Grauman
![Page 52: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/52.jpg)
Ambiguous matches
At what SSD value do we have a good match?To add robustness to matching, can consider ratio : distance to best match / distance to second best matchIf low, first match looks good.If high, could be ambiguous match.
Image 1 Image 2
? ? ? ?
Kristen Grauman
![Page 53: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/53.jpg)
Matching SIFT Descriptors• Nearest neighbor (Euclidean distance)• Threshold ratio of nearest to 2nd nearest descriptor
Lowe IJCV 2004
![Page 54: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/54.jpg)
Recap: robust feature-based alignment
Source: L. Lazebnik
![Page 55: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/55.jpg)
Recap: robust feature-based alignment
• Extract features
Source: L. Lazebnik
![Page 56: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/56.jpg)
Recap: robust feature-based alignment
• Extract features• Compute putative matches
Source: L. Lazebnik
![Page 57: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/57.jpg)
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
Source: L. Lazebnik
![Page 58: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/58.jpg)
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
• Verify transformation (search for other matches consistent with T)
Source: L. Lazebnik
![Page 59: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/59.jpg)
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
• Verify transformation (search for other matches consistent with T)
Source: L. Lazebnik
![Page 60: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/60.jpg)
Applications of local invariant features
• Wide baseline stereo• Motion tracking• Panoramas• Mobile robot navigation• 3D reconstruction• Recognition• …
![Page 61: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/61.jpg)
Automatic mosaicing
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
![Page 62: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/62.jpg)
Wide baseline stereo
[Image from T. Tuytelaars ECCV 2006 tutorial]
![Page 63: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/63.jpg)
Recognition of specific objects, scenes
Rothganger et al. 2003 Lowe 2002
Schmid and Mohr 1997 Sivic and Zisserman, 2003
Kristen Grauman
![Page 64: Local features: detection and description](https://reader036.vdocuments.site/reader036/viewer/2022062423/56816394550346895dd48c5a/html5/thumbnails/64.jpg)
Summary• Interest point detection
– Harris corner detector– Laplacian of Gaussian, automatic scale selection
• Invariant descriptors– Rotation according to dominant gradient direction– Histograms for robustness to small shifts and
translations (SIFT descriptor)