Feature Extraction and Matching

Feature Tracking

Sudipta N SinhaSep 19, 2006

Outline• Feature Extraction and Matching (for Larger Motion)

– What are features ?– Tasks

• Detection: finding the feature locations • Representation: computing a compact descriptor• Matching: Finding distances in feature space.

– Algorithms: • Harris Corner Detector, SIFT.• More complex (wide-baseline correspondence)

• Tracking (for Small Motion)– Track geometric primitives (points, lines, patches, objects …)

from frame to frame in video.- High temporal coherence.- Typically required in a real-time system.

Matching comes up in all kinds of problems in computer vision

Panoramas, mosaics

Object recognition Structure from Motion ( F, T , … )

More: Detect object in clutter, Motion segmentation,

Image-based retrieval, Video mining .. (Check Papers in References)

The Correspondence Problem and Invariance

Invariance: Features need to be detected repeatedly at the same locations and the computed descriptors must be similar in-spite of the following type of changes observed in two images of the same scene.

Point Features (Interest Points)

Goal:• To detect the same

point in each imageindependently

Challenges:• Need repeatability in presence of Scale, Rotation, Affine

distortions and Illumination change

• Not all pixels are good candidates.– Texture-less regions, edges.

• Effect of noise on feature extraction.• Examples:

– Harris Corner Detector, SIFT

Harris Corner Detector

Idea: • Detect a patch which looks locally unique.• Shifting the patch in any direction will

give a large change in intensity.

Texture-less region: no change in all directions

Edge: no change along one direction.

Corner: large changes in all

direction.

A symmetric matrix represents an ellipse

Matrix is symmetric semi-definite

Harris Corner DetectorEigen-value analysisof the 2x2 matrix M:

Corners: Feature Descriptors and Matching.

• Simple Descriptor: convert a patch of n x n pixels centered at that pixel into a vector.

• Matching: SAD, SSD, ZMNCC• Invariance:

– Translation ? Yes – Rotation ? No. But the image patch could be re-sampled

using eigen-vector pair as the local coordinate frame.– Scale and Affine ? No – Brightness Change ? Yes, normalize image intensity

(ZMNCC)

Feature point in high dim feature space

Point Features: SIFT

First: Scale Invariant Feature Detection,Later: SIFT descriptors (rotational

invariance)

The SIFT Algorithm

Create Scale Space Stack : (Lowe IJCV’04)

• Intensity

• Gradient

• DoG

Images from SIFT Tutorial [Thomas F. El-Maraghi May 2004 ]

The SIFT Algorithm

• Find Local Extrema of DoG

in Scale Space.

Remove

• Low Contrast Point

• Points on Edges.

Images from SIFT Tutorial [Thomas F. El-Maraghi May 2004 ]

The SIFT Algorithm

• Descriptor represents Local Patch Appearance.

• Oriented Histograms built from Weighted Gradients.

Images from Lowe IJCV’04

SIFT: Results

Wide Baseline Matching:

Elliptical and Parallelogram features(Tuteylaar, Van Gool et. al. IJCV 2004)

Anchor point:

Traditional Corners

Wide Baseline Matching:

Elliptical and Parallelogram features(Tuteylaar, Van Gool et. al. IJCV 2004)

Anchor point:

local intensity maxima

Tracking Corners – The KLT algorithm

Main Idea: Assuming brightness constancy, try to find the new positions of some ‘salient’ image points in the second image (where the motion is small)

Steps:1. Detecting Salient Points to track (in current frame)2. Track those features in next frame

Could be done by Searching (Template matching) BUTKLT algorithm does this analytically, hence its faster !

KLT equations:

Assumption – Brightness Constancy

Find a displacement d, such that the error given by the following equation is minimized (over a tracking window )

KLT equations:

Assumption – Brightness Constancy

Find a displacement d, such that the error given by the following equation is minimized (over a tracking window )

KLT equations:

A symmetric form was later proposed by Tomasi, as follows

To estimate d, differentiate w.r.t d,

KLT equations:

Substituting Taylor Series Expansion for J(.) and I(.)

We get,

Setting derivative to zero at the minima, and re-arranging, we get

a linear system of equations for d

KLT equations

Multiscale and Iterative KLT Build Image PyramidCoarse to Fine Tracking Increases Effective Spatial Range within which features can be tracked.

View Dependent Effects : If surface patch is small, then large persective distortions can be approximated by an affine transformation

Brightness change = gain + offset (2 more parameters)

Affine KLT

Invariance to illumination

Acknowledgments

Slides/Figures were taken from –

• SIFT MATLAB tutorial - [Thomas F. El-Maraghi May 2004]

• Lecture Notes by Bill Freeman• Lecture Notes on Tracking: UWA Computer

Science, CITS 4240.• David Lowe’s SIFT papers• Stan Birchfield’s article on Symmetric

Version of KLT equations.

References and Papers• Stan Birchfield. KLT: An Implementation of the Kanade-Lucas-Tomasi Feature Tracker

• [2] Bruce D. Lucas and Takeo Kanade. An Iterative Image Registration Technique with an Application to Stereo Vision. International Joint Conference on Artificial Intelligence, pages 674-679, 1981.

• David Lowe, ‘Distinctive image features from scale-invariant keypoints’, Int. Journal of Computer Vision, 60(2):91–110, 2004.

• J. Matas, O. Chum, U. Martin, and T. Pajdla. Robust wide baseline stereo from maximally stable extremal regions. In Proc. British Machine Vision Conference, volume 1, pages 384–393, Sep 2002.

•  • K. Mikolajczyk and C. Schmid. Scale and affine invariant interest point detectors. Int. Journal of

Computer Vision, 1(60):63–86, 2004•  • T.Tuytelaars and L. Van Gool. Matching widely separated views based on affine invariant

regions. Int. Journal of Computer Vision, 1(59):61–85, 2004•  • K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and

L. Van Gool. A comparison of affine region detectors. Technical Report, accepted to IJCV, 2005

• KLT src code: http://www.ces.clemson.edu/~stb/klt/

• SIFT Matlab code: see Link at http://robots.stanford.edu/cs223b04/project9.html