counting crowded moving objects vincent rabaud and serge belongie department of computer science and...

Counting Crowded Moving Objects

Vincent Rabaud and Serge BelongieDepartment of Computer Science and Engineering

University of California, San Diego{vrabaud,sjb}@cs.ucsd.edu

Presentation by: Yaron KoralIDC, Herzlia, ISRAEL

AGENDA

• Motivation• Challenges• Algorithm• Experimental Results

2

AGENDA


3

Motivation

• Counting crowds of people• Counting herds of animals• Counting migrating cells

• Everything goes as long as the crowdis homogeneous!!

4

AGENDA


5

Challenges

• The problem of occlusion– Inter-object– Self occlusion

• Large number of independent motions– Dozens of erratically moving objects– Require more than two successive

frames

6

Surveillance camera viewing a crowd from a distant viewpoint, but zoomed in, such that the effects of perspective are minimized.

AGENDA


7

Algorithm Highlights

• Feature Tracking with KLT• Increased Efficiency• Feature Re-Spawning• Trajectory Conditioning• Trajectory Clustering

8



9

Harris Corner Detector – What are Good Features? C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988

• We should easily recognize a corner by looking through a small window

• Shifting a window in any direction should give a large change in intensity

Harris Detector: Basic Idea

“flat” region:no change in all directions

“edge”:no change along

the edge direction

“corner”:significant

change in all directions

Harris Detector: Mathematics

2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

Change of intensity for shift in [u,v] direction:

IntensityShifted intensity

Window function

orWindow function w(x,y)=

Gaussian1 in window, 0 outside


2

,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

yx vIuIyxIvyuxI ,,

v

uMvu

v

u

III

IIIyxwvu

v

uyxIyxIyxwvuE

yyx

yxx

yxyx

2

2

,

2

),(

,,,,

For small [u,v]:

We have:


( , ) ,u

E u v u v Mv

For small shifts [u,v] we have a bilinear approximation:

2

2,

( , ) x x y

x y x y y

I I IM w x y

I I I

where M is a 22 matrix computed from image derivatives:

Harris Detector: MathematicsDenotes by ei the ith eigen-vactor of

M whose eigen-value is i:

Conclusions:

0 iiTi M ee

max1, ,maxarg e vuEvu

maxmax eE


( , ) ,u

E u v u v Mv

Intensity change in shifting window: eigenvalue analysis

1, 2 – eigenvalues of M

direction of the slowest change

direction of the fastest change

)max(-1/2

)min(-1/2

Ellipse E(u,v) = const


1

2

“Corner”1 and 2 are large,

1 ~ 2;

E increases in all directions

1 and 2 are small;

E is almost constant in all directions

“Edge” 1 >> 2

“Edge” 2 >> 1

“Flat” region

Classification of image points

using eigenvalues of M:

Sum of Squared Differences – Tracking FeaturesTracking Features

• SSD is optimal in the sense of ML when1. Constant brightness assumption2. i.i.d. additive Gaussian noise

Ayx

tyxtvyuxvu,

2,,1,,,SSD RI

Exhaustive Search

• Loop over all parameter space• No realistic in most cases

– Computationally expensive• E.g. to search 100X100 image in 1000X1000

image using only translation~1010 operations!

– Explodes with number of parameters– Precision limited to step size

The Problem

Find (u,v) that minimizes the SSD

over region A.

Assume that (u,v) are constant over all A

Ayx

yxvyuxvuSSD,

2,,, RI

Iterative Solution

• Lucas Kanade (1981)– Use Taylor expansion of I (the optical

flow equation)

– Find

Ayx

tyxvuvu

vuvuSSD,

2

,,min,min III

0 tyx vu III

Feature Tracking with KLT(We’re back to crowd counting…)• KLT is a feature tracking algorithm• Driving Principle:

– Determine the motion parameter oflocal window W from image I to consecutive image J

– The center of the window defines the tracked feature

23

Feature Tracking with KLT

• Given a window W– the affine motion parameters A and d

are chosen to minimize the dissimilarity

24

Feature Tracking with KLT

• It is assumed that only d matters between 2 frames. Therefore a variation of SSD is used

25

• A window is accepted as a candidate feature if in the center of the window, both eigenvalues exceed a predefined threshold t

min(min(λλ11,, λ λ22) > t) > t



33

Increased Efficiency #1• Associating only one window with

each feature– Giving a uniform weight function that

depends on 1/(window area |w|)– Determining quality by comparing:

• Computation of different Z matrices is accelerated by “integral image”[1]

34

|| w

z

[1] Viola & Jones 2004

Increased Efficiency #2

• Run on sample training frames first– Determine parameters that lead to the

optimal windows sizes– Reduces to less than 5% of the possible

parameter set– All objects are from the same class

35




36

Feature Re-Spawning

• Along time, KLT looses track:– Inter-object occlusion– Self occlusion– Exit from picture– Appearance change due to perspective

and articulation

• KLT recreates features all the time– Computationally intensive– Weak features are renewed

37

Feature Re-Spawning

• Re-Spawn features only at specific locations in space and time

• Propagate them forward and backward in time– Find the biggest “holes”– Re-spawn features

in frame with theweighted average oftimes

38



39

Trajectory Conditioning

• KLT tracker gives a set of trajectories with poor homogeneity– Don’t begin and end at the same times– Occlusions can result in trajectory

fragmentation– Feature can lose its strength resulting in

less precise tracks

• Solution: condition the data– Spatially and temporally

40

Trajectory Conditioning

• Each trajectory is influenced by its spatial neighbors

• Apply a box to each raw trajectory• Follow all neighbor trajectories from

the time the trajectory started

41



42

Trajectory Clustering

• Determine number of object at time tt by clustering trajectories

• Since at time tt objects may be close, focus attention on a time interval (half-width of 200 frames)

• Build connectivity graph– At each time step, the present features form the

nodes of a connectivity graph G – Edges indicate possible membership to a common

object.

43

Trajectory Clustering• Connectivity Graph

– Bounding Box: as small as possible, able to contain every possible instance of the object

– If two features do not stay in a certain box, they do not belong to the same object.

– The 3 parameters of this box are learned from training data.

44


Articulation factor

Trajectory Clustering• Rigid parts merging

– Features share similar movement during whole life span, belong to a rigid part of an objectrigid part of an object, and consequently to a common objectcommon object

– RANSAC is applied to sets of trajectories • Within time window • Connected in graph G

45

Trajectory Clustering

• Agglomerative Clustering– At each iteration, the two closest sets are

considered – If all features are linked to each other in the

connectivity graph, they are merged together.

– Otherwise, the next closest sets are considered

– Proceed until all possible pairs are analyzed

46

AGENDA


47

Experimental results

• Datasets– USC: elevated view of a crowd

consisting of zero to twelve persons– LIBRARY: elevated view of a crowd of

twenty to fifty persons– CELLS: red blood cell dataset consisting

of fifty to hundred blood cells

48


49


50

Estimated

Ground Truth


51

Conclusion

• A new way for segmenting motions generated by multiple objects in crowd

• Enhancements to KLT tracker• Conditioning and Clustering

techniques

52

Thank You!Thank You!

53

counting crowded moving objects vincent rabaud and serge belongie department of computer science and...

Documents

harris corner detector

edge detector

directionsharris detector

intensityharris detector

constharris detector

mathematicsfor small

100x100 image

image derivatives