An Introduction to Feature Tracking

N. D.

Outline

➔  Introduction: What is feature tracking?

➔  Features

➔  Feature Tracking Algorithm

➔  Alternatives

➔  Applications

➔  Q & A

Introduction: Motivation

How does object tracking work?

How do you determine the

similarity between images?

Introduction: Preview

You have two images. How does the first map onto the second?

Introduction: Feature Tracking

From Wikipedia: Motion estimation is...

“Process of determining motion vectors that describe the transformation

from one 2D image to another, usually from adjacent frames in a video

sequence. ”

Features: Definition

Official Definition:

“a n-tuple or vector whose components are functions of the initial

measurement pattern variables or some subset of the initial

measurement pattern variables.”

Human-Speak Definition:

A location on an image where you track change.

Features

(A location on an image where you track change

==

Any location) ?

Features

(A location on an image where you track change

==

Any location) ?

true

Features

What is a good feature?

Features: An Example

(1)

(3)

(5)

(2)

(4)

(6)

Features: Computed Answer

Features: Computed Answer

Features: The Good, the Bad, the Ugly

A good feature should be locally unique and invariant to geometric

transformations.

Good Bad Ugly

Features: Local Uniqueness

To determine local uniqueness, look at sub-parts of the feature.

Good

Bad

Ugly

. . .

. . .

. . .

Tracking: Algorithm Overview

Inputs: (a: Image, b: Image)

Step 1: vectora = Extract Features ( a ) vectorb = Extract Features ( b )

Step 2: matches = Compute Difference ( vectora , vectorb ) Step 3: result = Eliminate False Positives ( matches )

Tracking: Shoes Example

Inputs:

Tracking: Shoes Example

Output: Inputs:

Tracking: Shoes Example --- Step 1

Step 1: Feature Extraction using the Harris Operator

Tracking: Shoes Example --- Corner Detection

The Harris Operator is used in corner detection.

Tracking: Shoes Example --- Harris Points

Harris Operator output: bright points = locally unique candidates

Tracking: Shoes Example --- Choosing Points

Choose the brightest points that are at least delta distance apart.

Tracking: Shoes Example --- Choosing Points

Same result, on the colored images:

Tracking: Shoes Example --- Harris Equation

where ➔  W is a weights kernel (Gaussian kernel; smoothing) ➔  Ix is the horizontal Sobel operator (horizontal edge detection) ➔  Iy is the vertical Sobel operator (vertical edge detection)

Tracking: Shoes Example --- Sobel Operator

Sobelx and Sobely applied to Image a

[ Horizontal edge detection ] [ Vertical edge detection ]

Tracking: Shoes Example --- Step 2

Step 2: Feature Matching using the Sum of Squared Differences (alternative: Normalized Cross Correlation) to measure error and find

similarity between images.

Tracking: Shoes Example --- SSD

Low error measurements between features means a potential match.

Tracking: Shoes Example --- SSD Equation

where

➔  E(u, v) is the error measurement at (u, v) compared to (x, y) ➔  W(x, y) is a weights kernel (Gaussian kernel; smoothing) ➔  I(x, y) is the Sobel operator (edge detection)

Tracking: Shoes Example --- Step 3

Step 3: Eliminate False Positives using the Random Sample Consensus (RANSAC) method

Tracking: Shoes Example --- RANSAC

“An iterative method to estimate parameters of a mathematical model from a set of observed data which contains outliers. ”

Tracking: Flowers Example

Example #2:

Tracking: Museum Example

Example #3:

(A Few) Alternatives

➔  Pixel-by-pixel computation (horribly inefficient)

➔  Relational matching

◆  Bag-of-Words

◆  Both generative or discriminative models

➔  Particle Filter

➔  Multi-Scale Oriented Patches

➔  Eigenfaces / Space of Faces

Applications: Feature Extraction & Tracking

➔  Film ◆  Video tracking ◆  Replacement sets or characters (ex: Gollum) ◆  Motion tracking markers

➔  Biometrics

◆  Facial recognition (Eigenfaces) ◆  Xbox Kinect (?)

Questions ? Ask : Clap

(You may clap.)

Sources Gollum Motion Estimation - Wiki Ransac - Wiki Car with Tracking Markers Motion Tracking Markers Joffrey Clapping GIF Glossary of Computer Vision Terms Skeleton code for reading in/writing out images from ColorFront Flowers Pictures from ColorFront Computer Vision: Algorithms and Applications by Richard Szeliski