1

Object Tracking

Babol university of technology

ECE Dep.

Machine Vision

Prof: M. Ezoji

Presentation: Alireza Asvadi

Winter 2012

2

What is tracking?

Estimating the trajectory of an object over time by locating its position in every frame.

“Tracking by detection” Vs “Tracking with dynamics”

Tracking by Detection: we have a strong model of the object, we detect the object independently in each frame and can record its position over time.

Tracking with dynamics: we use object position estimated by measurement but also incorporate the position predicted by dynamics.

3

Time t Time t+1 Time t+n

1. detect the object independently in each frame

2. Link up the instances and we have a track

Tracking by detection:

Ref: Master Thesis, Alireza Asvadi, "Object Tracking from Video Sequence (using color and texture information and RBF Networks)"

4

Methods for detection of object :

Point detectors Background ModelingBackground Subtraction

Supervised ClassifiersContour evaluation….

Other Methods:

Template and density Based appearance models

Shape models

Ref: Master Thesis, Alireza Asvadi, "Object Tracking from Video Sequence (using color and texture information and RBF Networks)"

5

Point detectors:

Point detectors are used to find interest points in images which have an expressivetexture in their respective localities. The object is represented by points.commonly used interest point detectors include Moravec, Harris and SIFT detector.

Start

Take new image

Select Good Features

Track FeaturesIf lost

featuresReplace features

If images remaining

Store ImageReplace features

Stop

Yes

No

Yes

No

KLT control graph:

Ref: Ankit Gupta (1999183) Vikas Nair (1999219) Supervisor Prof M. Balakrishnan Electrical Engineering Department IIT Delhi

6Ref: A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Computing Surveys, Vol. 38, No. 4, pp. 1–45, December 2006.

Ref: M. Sonka,V. Hlavac, R.Boyle, “Image Processing, Analysis, and Machine Vision,” 3rd Edition, 2008.

7

x,y

Template Image

Input Image

I(x,y) O(x,y)

Output Image

x,yCorrelation

Template Matching:

Template matching is a brute force method of searching the image for a region similar to the object template.The position of the template in the current image is computed by a similarity measure, for example:

NCC=normalized cross-correlation SSD=Sum of square differences SAD=Sum of absolute differences

Ref: Ankit Gupta (1999183) Vikas Nair (1999219) Supervisor Prof M. Balakrishnan Electrical Engineering Department IIT Delhi

8

1

0

1

0

22

1

0)(

N

i

N

iii

i

N

i i

yyxx

yyxxcorx is the template gray level image

x is the average grey level in the templatey is the source image sectiony is the average grey level in the source imageN is the number of pixels in the section image (N= template image size = columns * rows)The value cor is between –1 and +1, with larger values representing a stronger relationship between the two images.

NCC:

Template Matching:

In Matlab: C = normxcorr2(template, A)

Affine Transformations could be use to Confirm a Match.

Ref: M. Sonka,V. Hlavac, R.Boyle, “Image Processing, Analysis, and Machine Vision,” 3rd Edition, 2008.

9

NCC

SAD=sum(abs(TEMP(:)- IMGBLK(:)))SSD=sum((TEMP(:)-IMGBLK(:)).^2)

Frame

Template

10

Shape Matching:

Shape matching can be performed similar to tracking based on template matching.

shape models are usually in the form of edge maps.

Ref: A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Computing Surveys, Vol. 38, No. 4, pp. 1–45, December 2006.

11

Background subtraction is often a good enough detector in applications where the background is known and all trackable objects look different from the background. The most important limitation of background subtraction is the requirement of stationary cameras.

Background Subtraction:

Ref: M. Sonka,V. Hlavac, R.Boyle, “Image Processing, Analysis, and Machine Vision,” 3rd Edition, 2008.

12

Supervised Classifiers:

During testing, the classifier gives a score to the test data indicating the degree of membership of the test data to the positive class.

Maximum classification score over image regions estimate the position of the object.

Object detection can be performed by learning different object views automatically from a set of examples by means of a supervised learning mechanism.

Ref: A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Computing Surveys, Vol. 38, No. 4, pp. 1–45, December 2006.

13

See Details inMean Shift Slides

14

Mean Shift algorithm:

15

Mean Shift Widely used, various enhancements (e.g. Robert Collins):

16

17

Problems with Tracking by detection Methods:

Occlusions:

Time

Similar Objects:

18

NCC

NCC map in First Frame

Problems with Tracking by detection Methods:

Or if the detector fails to detect object

(A template matching Example)

19

Tracking with dynamics:

Key idea: Given a model of expected motion, predict where objects will occur in next frame.

Observation (Detected object) + Dynamics

- Prediction vs. correction

- If the observation model is too strong, tracking is reduced to repeated detection

- If the dynamics model is too strong, will end up ignoring the data

20

Filtering Problem:

kzkx

kxa bc

Estimate of c based on

Prediction a and measurement b

In kalman Filter they are correspond with:

Kalman filter assumptions:

The Kalman filter model assumes that the state of a system at a time t evolved from the prior state at time t-1 according to:

Measurements of the system can be performed, according to the model:

AA

To have similarnotation

R. Faragher , “Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation,” IEEE Signal Processing Magazine, September 2012.

The Kalman filter:

Control information:(force)

state vector:(position & velocity)

The relationship between the forceDuring the time period Δt and position and velocity: In matrix form:

A

R. Faragher , “Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation,” IEEE Signal Processing Magazine, September 2012.

23

The Kalman filter:

The information from the predictions and measurements are combined to provide the best possible estimate of the location of the train.

THE PRODUCT OF TWO GAUSSIAN FUNCTIONS IS ANOTHER GAUSSIAN FUNCTION

R. Faragher , “Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation,” IEEE Signal Processing Magazine, September 2012.

24

The Kalman filter:

R. Faragher , “Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation,” IEEE Signal Processing Magazine, September 2012.

25

The Kalman filter:

R. Faragher , “Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation,” IEEE Signal Processing Magazine, September 2012.

26

Estimate(a posteriori)

Best Prediction prior to Zk(a priori)

Optimal Weighting(Kalman Gain)

Residual

Read details in References process noise

covariance

G. Welch, G. Bishop , “An Introduction to the Kalman Filter,” ,1995.

27

kkkkkk zHxHKkk

1|0

10 ˆlimlim

RR

1||00 ˆˆlim0lim kkkkk xxKkk PP

1 1ˆ ˆ ˆk | k k | k k k k k | kx x K z H x

Small prediction error:

the dynamics model is too strong, will end up ignoring the data

Small measurement error:

The observation model is too strong, tracking is reduced to repeated detection

The Kalman filter:

28

The Kalman filter:

kz

kx

G. Welch, G. Bishop , “An Introduction to the Kalman Filter,” ,1995.

29

The Kalman filter:

30

Problems: Yet it is not good enough

Kalman filter Estimate Is combination of Prediction and Measurement.

Result by applying The kalman filter

Till now we considered the entire measurements But what if we omit uninformative observations or highly unlikely measurements? It is Data Association.

NCC

NCC+Kalman

31

So far, we’ve assumed the entire measurement to be relevant to determining the state.

In reality, there may be uninformative measurements or measurements that are not necessarily resulted from the target of interest or may belong to different tracked objects.

Data association:

Data association: task of determining which measurements go with which tracks.

Simple strategy: only pay attention to the measurement that is “closest” to the prediction.

32

33

34

35

36

37

Tracking: Detection(observation)+ dynamics +Data association

Notice Predicted Values By Kalman Filter (Green)Applying Data association (Gating)

DA(Gating):Omit MeasurementsOutside the gate (say a circle with radius 50)

NCC

NCC+Kalman

NCC+Kalman+DA

38

Tracking by detectiondetect the object independently in each frame

tracking=Detection Detection Methods:

Tracking with dynamicsincorporate object dynamics to tracking Methods:tracking=Detection(observation)+dynamics

Applying Data associationEliminate highly unlikely measurementstracking=Detection(observation)+ dynamics +Data association

Summary:

Point detectorsTemplate matchingdensity Based appearance modelsShape modelsBackground Subtraction…

Filtering MethodsKalman filter…

Methods:Tracking MatchingGating…

39

Reference:

D. A. Forsyth, J. Ponce, “Computer Vision: A Modern Approach,” Prentice Hall,2nd Edition, 2012.

M. Sonka,V. Hlavac, R.Boyle, “Image Processing, Analysis, and Machine Vision,” 3rd Edition, 2008.

A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Computing Surveys, Vol. 38, No. 4, pp. 1–45, December 2006.

R. Faragher , “Understanding the Basis of the Kalman Filter Via a Simple and Intuitive Derivation,” IEEE Signal Processing Magazine, September 2012.

G. Welch, G. Bishop , “An Introduction to the Kalman Filter,” ,1995.