comparison of local feature descriptorsyang/courses/cs294-6/maji-presentation.pdfvarious feature...
TRANSCRIPT
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Comparison of Local Feature Descriptors
Subhransu Maji
Department of EECS,
University of California, Berkeley.
December 13, 2006
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
1 IntroductionLocal Features
2 BenchmarksMikolajczyk’s DatasetCaltech 101 Dataset
3 Experiments and ResultsEvaluation of Feature DetectorsEvaluation of Feature Descriptors
4 Future Work
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Local Features
Applications of Local Features
Multi Camera Scene reconstruction.
Robust to Backgrounds, Occlusions
Compact Representation of Objects for Matching, Recognitionand Tracking.
Lots of uses, Lots of options.
This work tries to address the issue of what features aresuitable for what task, which is currently a black art!!
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Local Features
Key properties of a good local feature
Must be highly distinctive, i.e. low probability of a mismatch.
Should be easy to extract.
Invariance, a good local feature should be tolerant to.
Image noiseChanges in illuminationUniform scalingRotationMinor changes in viewing direction
Question: How to construct the local feature to achieve
invariance to the above?
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Local Features
Various Feature Detectors
Harris detector find points at a fixed scale.
Harris Laplace detector uses the scale-adapted Harris function to localizepoints in scale-space. It then selects the points for which theLaplacian-of-Gaussian attains a maximum over scale.
Hessian Laplace localizes points in space at the local maxima of theHessian determinant and in scale at the local maxima of theLaplacian-of-Gaussian.
Harris/Hessian Affine detector does an affine adaptation of theHarris/Hessian Laplace using the second moment matrix.
Maximally Stable Exremal Regions detector finds regions such that pixelsinside the MSER have either higher (bright extremal regions) or lower(dark extremal regions) intensity than all the pixels on its outer boundary.
Uniform Detector(unif) - Select 500 points uniformly on the edge mapsby rejection sampling.
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Local Features
Various Feature Descriptors
Scale Invariant Feature Transformation A local image is path is dividedinto a grid (typically 4x4) and a orientation histogram is computed foreach of these cells.
Shape Contexts computes the ditance and orientaion histogram of otherpoints relative to the interst point.
Image Moments These compute the descriptors by taking various higherorder image moments.
Jet Decriptors These are essentially higher order derivatives of the imageat the interest point
Gradient Location and Orientaiton Histogram As the name suggests itconstructs a feature out of the image using the Histogram of location andOrientation in of points in a window around the interest point.
Geometric Blur These compute the average of the edge signal responseover small tranformations. Tunable parameters include the blurgradient(β = 1), base blur (α = 0.5) and scale multiplier (s = 9).
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Local Features
Example Detections
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Mikolajczyk’s DatasetCaltech 101 Dataset
Evaluation Criteria
We want the feature to be repeatable,repeatability = correct−matches
ground−truth−matches
Descriptor Performance:
recall vs 1-precision graphs.recall = #correct matches
#correspondances
correct matches found by neareast neignbour matching in thefeature space.correspondances obtained from ground truth matching.1 − precision = #falsematches
#false matches+#correct matces
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Mikolajczyk’s DatasetCaltech 101 Dataset
Mikolajczyk’s Dataset
8 Datasets, 6 Images per dataset.Ground Truth Homography available for these Images.
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Mikolajczyk’s DatasetCaltech 101 Dataset
Caltech 101 Dataset
101 Categories, man-made objects, motifs, animals and plants.
Foreground Mask is available. Obtain ground truth based on arough alignement of the contours.
Determine the scale, translation which maximizes area overlapof the contours.
Correspondance: Features of the images within a thresholddistance(10 Pixels) under the transformation.
Many clasification techniques use the structure of image forcomputing similarity. For e.g. SC based caracter recognitionusing TSP.
The performance of these algorithms is dependent ondetecting features on the right positions. Ideally we wouldwant the descriptor performance to be better on such a softernotion of matching.
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Mikolajczyk’s DatasetCaltech 101 Dataset
Best 8 and Worst 8
50 100 150 200 250 300
50
100
150
200
250
300100 150 200
50
100
150
200
250
300 350 400
50
100
150
200
250
50 100 150 200 250
40
60
80
100
120
140
160
180
200
220
0 50 100 150 200 250
0
50
100
150
200
250
300
40 60 80 100 120 140 160 180 200 220
80
90
100
110
120
130
50 100 150 200 250 300
0
50
100
150
200
250
30040 60 80 100 120 140 160 180 200 220 240
40
60
80
100
120
140
0.9781 0.9717 0.9642 0.9490 0.9486 0.9483 0.9405 0.9223
0 50 100 150 200 250 300
20
40
60
80
100
0 20 40 60 80 100 120 140 160 180
0
20
40
60
80
100
120
140
160
180
200
100 150 200 250
50
100
150
200
250
50 100 150 200 250
100
120
140
160
180
200
220100 120 140 160 180 200 220
60
80
100
120
140
160
180
200
220
50 100 150 200 250
50
100
150
200
250
50 100 150 200 250 300
20
40
60
80
100
120
140
160
180
20050 100 150 200 250 300
60
80
100
120
0.7097 0.6934 0.6919 0.6658 0.6614 0.6444 0.6426 0.3318
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Mikolajczyk’s DatasetCaltech 101 Dataset
Example Ground Truth Matches
100 200 300 400 500 600 700 800 900 1000
50
100
150
200
250
300
350 100 200 300 400 500 600
50
100
150
Faces car side
100 200 300 400 500 600
50
100
150
200
250
50 100 150 200 250 300 350 400 450 500
50
100
150
stop sign Motorbikes
Figure: Ground Truth matches. We use the harris Affine detector with adistance threshold of 5 pixelsSubhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Evaluation of Feature DetectorsEvaluation of Feature Descriptors
Repeatability Results on Benchmarks
Mikolajczyk Dataset: MSER was generally the best followedby Hessian Affine.
Hessian-Affine and Harris-Affine provide more regions than theother detectors, which is useful in matching scenes withocclusion and clutter.
Caltech 101 Dataset: Hessian Affine, Hessian Laplace, MSER,UNIF all perform equally well. Hessian Affine is slightly betterthan others in most cases.
Almost any detector is equally good as the matching is softer.
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Evaluation of Feature DetectorsEvaluation of Feature Descriptors
Desciptor Performance on Mikolajczyk’s Dataset
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1−precision
frac
of c
orre
ctEffect of scale − bikes
gbsiftscspinmomjla
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1−precision
frac
of c
orre
ct
Effect of scale − trees
gbsiftscspinmomjla
(1)bikes (2)trees
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
1−precision
frac
of c
orre
ct
Effect of scale − graf
gbsiftscspinmomjla
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
1−precision
frac
of c
orre
ct
Effect of scale − wall
gbsiftscspinmomjla
(3)graffiti (4)wall
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Evaluation of Feature DetectorsEvaluation of Feature Descriptors
Desciptor Performance on Mikolajczyk’s Dataset
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1−precision
frac
of c
orre
ct
Effect of scale − bark
gbsiftscspinmomjla
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1−precision
frac
of c
orre
ct
Effect of scale − boat
gbsiftscspinmomjla
(5)bark (6)boat
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1−precision
frac
of c
orre
ct
Effect of scale − leuven
gbsiftscspinmomjla
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
1−precision
frac
of c
orre
ct
Effect of scale − ubc
gbsiftscspinmomjla
(7)leuven (8)ubc
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Evaluation of Feature DetectorsEvaluation of Feature Descriptors
Desciptor Performance on Caltech 101
0.4 0.5 0.6 0.7 0.8 0.9 10
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
1−precision
frac
of c
orre
ctyin yang
siftscmomglohgbjet
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1−precision
frac
of c
orre
ct
Faces
siftscmomglohgbjet
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.05
0.1
0.15
0.2
0.25
1−precision
frac
of c
orre
ct
Faces Easy
siftscmomglohgbjet
0.95 0.96 0.97 0.98 0.99 10
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
1−precision
frac
of c
orre
ct
pizza
siftscmomglohgbjet
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Evaluation of Feature DetectorsEvaluation of Feature Descriptors
Desciptor Performance on Caltech 101
0.975 0.98 0.985 0.99 0.995 10
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1−precision
frac
of c
orre
ctbarrel
siftscmomglohgbjet
0.75 0.8 0.85 0.9 0.95 10
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
1−precision
frac
of c
orre
ct
car side
siftscmomglohgbjet
0.4 0.5 0.6 0.7 0.8 0.9 10
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
1−precision
frac
of c
orre
ct
stop sign
siftscmomglohgbjet
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.02
0.04
0.06
0.08
0.1
0.12
0.14
1−precision
frac
of c
orre
ct
Motorbikes
siftscmomglohgbjet
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Evaluation of Feature DetectorsEvaluation of Feature Descriptors
Results on Benchmarks
Mikolajczyk Dataset:1 SIFT and Shape Context do better on wall, bark datasets.2 Geometric Blur(GB) better on bikes, graf datasets3 Both are Comparable on ubc, leuven, boat, trees datasets
Caltech 101 Dataset: GB, Shape Context and SIFT do thebest in all cases.
GLOH which did the best in the Mikolajczyk’s Datasetperforms poorly.
In general the performance in Caltech 101 is much worse thanin Mikolajczyk’s dataset.
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Evaluation of Feature DetectorsEvaluation of Feature Descriptors
Some Observations
The performance difference in significant between SIFT andGB in both 1 and 2.
The performance of SIFT and SC are higly correlated.
The performance of SIFT and GB are higly negativelycorrelated.
Question: Do SIFT, GB carry complimentary information.
When is one more useful than the other?
SIFT does better when there is high texture. High Frequency
Information incorporated better? More experiments required...
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
Future Work
More flexible notion of Matching, rotations, non-rigidtransformations, etc to incorporate more classes
Extend the analysis to Different Datatsets like PASCAL
A systematic study of the Black Art!
Subhransu Maji Comparison of Local Feature Descriptors
OutlineIntroductionBenchmarks
Experiments and ResultsFuture Work
THANK YOU1
1beamer rocks!!Subhransu Maji Comparison of Local Feature Descriptors