Multi-Object Tracking through Occlusions by Local Tracklets Filtering andGlobal Tracklets Association with Detection Responses

Junliang Xing, Haizhou AiComputer Science and Technology Dept.

Tsinghua University, Beijing, Chinaahz@mail.tsinghua.edu.cn

Shihong LaoCore Technology Center

Omron Corporation, Kyoto, Japanlao@ari.ncl.omron.co.jp

AbstractThis paper presents an online detection-based two-stage

multi-object tracking method in dense visual surveillancesscenarios with a single camera. In the local stage, a par-ticle filter with observer selection that could deal with par-tial object occlusion is used to generate a set of reliabletracklets. In the global stage, the detection responses arecollected from a temporal sliding window to deal with am-biguity caused by full object occlusion to generate a set ofpotential tracklets. The reliable tracklets generated in thelocal stage and the potential tracklets generated within thetemporal sliding window are associated by Hungarian al-gorithm on a modified pairwise tracklets association costmatrix to get the global optimal association. This method isapplied to the pedestrian class and evaluated on two chal-lenging datasets. The experimental results prove the effec-tiveness of our method.

1. IntroductionMulti-object tracking is important for many computer vi-

sion applications. This is a relatively easy task when objectsare isolated and easily distinguished from each other andthe background. However, in complex and dense scenarios,many objects may have similar appearance, and occlusionshappen very frequently, such as object self-occlusion, oc-clusion between multiple objects, and occlusion by otherscene objects. In these situations, robust multi-object track-ing would be a very difficult problem and many traditionalobject tracking methods may fail. We propose a method thatcan robustly track multiple objects under such challengingconditions.

Recently with the significant progresses achieved in ob-ject detection researches, detection-based tracking methodsgain more and more attentions[11][15] since they are es-sentially more flexible and robust in complex environmentsno regard of whether camera moves or not and they arefully automatic which do not need outside initialization intracking which is of great importance in practical applica-

tions. However, the accuracy of state-of-the-art object de-tectors is still far from perfect. The detection performanceis usually a tradeoff between the detection rate and the falsealarm rate. Missed detections and false alarms and inac-curate responses happen frequently in the detection proce-dure which provides misleading information to tracking al-gorithms. Detection-based tracking method must overcomethese failures of the detector, and the difficulties caused byocclusions and similar appearance among multiple objects.

The aim of this work is to overcome the limitations ofcurrent object detectors to track multiple objects throughocclusions online in dense surveillance scenarios. Wepropose an online detection-based two-stage multi-objecttracking method. In the local stage, a particle filter with ob-server selection that could deal with partial object occlusionis used to generate a set of reliable tracklets. In the globalstage, the detection responses are collected from a temporalsliding window to deal with ambiguity caused by full objectocclusion to generate a set of potential tracklets. The reli-able tracklets generated in the local stage and the potentialtracklets generated within the temporal sliding window areassociated by Hungarian algorithm on a modified pairwisetracklets association cost matrix to get the global optimal as-sociation. The two stages incorporate with each other whichmake our algorithm seek both the local optimum trajectoryfor each object and the global optimum trajectories for allthe tracked objects.

The main contributions of this paper include: 1) a two-stage online tracking framework which seeks both the lo-cal optimum for each object and global optimum for all thetracked objects; 2) a human partition method concentrateson the upper human body in three different levels which ismore suitable for human tracking in common visual surveil-lance scenario; 3) a particle filter with observer selectionthat could deal with partial object occlusion for generatingreliable tracklets; 4) a modified pairwise tracklets associa-tion cost matrix for the Hungarian algorithm to solve dataassociation problem which could model tracklets associa-tion, initialization, termination and false alarms.

1200978-1-4244-3991-1/09/$25.00 ©2009 IEEE

The rest of the paper is structured as follows. Relatedwork is discussed in Section 2. The outline of our approachis described in Section 3. Problem formulation is given inSection 4. A detailed description of our approach is givenin Section 5. Experimental results are presented in Section6, and Section 7 concludes the paper.

2. Related workDetection based tracking algorithms obtain object hy-

potheses by applying an object detector to images. The de-tector is learned off-line from labeled training data. Givendetection responses generated by the detector, the trackingmethod needs to retrieve the real objects among those re-sponses and set ID for each of them in every frame. To dealwith this data association problem, there are usually twostrategies: one is to associate the responses locally (frameby frame) while the other associates them globally.

To associate the responses locally, Wu et al. [15] de-fined an affinity measure between detection responses basedon cues from position, size, and color and used a greedyalgorithm to associate object hypotheses and detection re-sponses. New object trajectories were initialized wheneverthe detection responses did not match with any existing tra-jectories for a certain number of frames; old trajectorieswere terminated when they were lost by the detector for acertain number of frames. This direct linking method de-pends a lot on the detector and is sensitive to temporal ob-servation missing. Li et al. [10] and Okuma et al. [12] usedparticle filter methods to associate the detection responsesof an unknown number of objects. The detection responseswere used to generate new particles and evaluate existingparticles. By particle filtering, the tracker could maintainmultiple hypotheses. However, increasing the number ofparticles requires more computational cost. To improve thecomputational efficiency, Li et al. [11] used multiple detec-tors (observers) to form a cascade particle filter. The orderin which the detectors were applied was determined basedon their computational costs: the faster the earlier. The lo-cal association methods used only the information in twoconsecutive frames which makes them incline to drift whenmultiple objects are close to each other.

To overcome the drifting problem of local associationmethod, one approach is to optimize multiple trajecto-ries simultaneously such as in Multi-Hypothesis Tracking(MHT) [13] or in Joint Probabilistic Data Association Fil-ters (JPDAF)[3]. Recently some approaches which try tosolve this problem globally have been developed, amongwhich Leibe et al. [9] used Quadratic Boolean Program-ming to couple the detection and estimation of trajectory hy-potheses, Andriluka et al. [2] used Viterbi algorithm to getoptimal object sequences, Zhang et al. [16] used a cost-flownetwork to model the MAP data association problem, andHuang et al. [6] did data association of detection responses

Detector

Output

# t-1Object State

Resample Select Update Estimate

Local Stage

Global Stage

# tObject State

Video Source

Input

Figure 1. Block diagram of our approach.

hierarchically. As the hypothese search space grows ex-pontially, the global association methods usually need lotsof computation which makes them unsuitable for real-timeprocessing.

To our best knowledge, there is not yet work to combinelocal filtering method and global association method for abetter tracking performance in detection based tracking lit-erature. Our work contributes in this direction.

3. Outline of Our ApproachAs illustrated in Figure 1, the main idea is a two-stage

framework that uses a particle filter algorithm to generatereliable tracklets and uses data association in a temporalsliding window for potential tracklets to optimize object tra-jectories globally.

In order to deal with partial occlusion problem, an ob-server selection process is introduced into the particle filterto choose a subset of all observations by the multi-view andmulti-part detector which corresponds to all visible objectparts.

Tracklets from the particle filter are associated with po-tential tracklets from a temporal sliding window around tra-jectory break points of detection responses by Hungarianalgorithm based on a modified pairwise tracklets associa-tion cost matrix for global trajectory optimization.

4. Problem FormulationGenerally object tracking can be formalized as a sequen-

tial Bayesian estimation problem. The trajectories of ob-jects in video are modeled as a sequence of states whereeach of them basically denotes its location and size. Letsit = (pi

t, sit) be the state of a particular object i at frame

t where pit is the position and si

t is the size, and si1:t =

{si1, . . . , s

it} be the trajectory of the object i up to frame

t. Denote St = {s1t , . . . , s

mt } as all the object states ap-

peared at frame t and S1:t = {s11:t, . . . , s

m1:t} as all object

trajectories up to frame t.Observations are generated by applying a detector on

each video frame. Similarly, let oit be the observation of

1201

the i-th object collected at frame t given the object state sit ,

and oi1:t = {oi

1, . . . ,oit} be all the observations of object i

up to frame t. Denote Ot = {o1t , . . . ,o

mt } as all the obser-

vations collected at frame t and O1:t = {o11:t, . . . ,o

m1:t} as

all the observations up to frame t.For a multi-object tracking system, the final aim is to

find the optimal trajectories for all the objects based on theobservation set. It is equivalent to maximize the posterioriprobability of S1:t giving the observations O1:t:

S∗1:t = arg max

S1:tp(S1:t|O1:t) (1)

Since the number of all possible enumerations of S1:t

given O1:t is huge, it prevents a brute force search to findthe global optimum. And in fact, global optimization istoo time-consuming for practical surveillance applicationsin which a long time delay is unaffordable. Therefore weturn to a progressive and recuperative strategy. Firstly, wedo optimization locally on each object which is equivalentto maximize the posteriori probability of si

t giving the ob-servations oi

1:t:s∗it = arg max

sit

p(sit|oi

1:t) (2)

This could be done by a particle filter. Denote thetracklet set generated as S

+1:t. Secondly, around trajectory

break points of potential tracklets detection responses froma temporal sliding window are generated and associatedby greedy linking method [15]. Denote this tracklet setgenerated as S

−1:t. Further, data association will be done

on{S+1:t, S

−1:t}. This above procedure is summarized as:

S∗1:t = arg max

S1:tp(S1:t|O1:t)

= arg maxS1:t

p(S1:t|S+1:t, S

−1:t)p(S+

1:t, S−1:t|O1:t)

= arg maxS1:t

p(S1:t|S+1:t, S

−1:t)p(S−

1:t|S+1:t, O1:t)p(S+

1:t|O1:t)

(3)5. Our approach

In common visual surveillance systems where a cameralooks down to the plane on a square or a passage wherepeople pass by, occlusion of different types often happensfrequently. Occlusion can be classified into three types ac-cording to the causation: object self-occlusion, inter-objectsocclusion and object occlusion by other scene objects. Ob-ject self-occlusion happens when an object changes its posewhich may make the detector fail. Inter-objects occlu-sion, which happens most frequently in dense environment,means that one object is occluded by other objects. Oc-clusion can also be classified into partial occlusion and fullocclusion. If an object is partially occluded, we can stilltry to get observations to track it. But if an object is fullyoccluded, no observation would be available. In this pa-per, we use a multi-view multi-part human detector to col-lect observations and a particle filter with observer selection

. . .Strong Classifier

. . .

. . .

. . .

FLR: Frontal/Left/Right

LNR1R

FNF1F

LNL1L

NFLR1NFLR �1FLR

Figure 2. The tree-structure multi-view human detector.

process to deal with partial object occlusion. For full objectocclusion, we tackle it by data association of the detectionresponses within a temporal sliding window.

5.1. Multi-View and Multi-Part Human Detectorfor Observation Collection

For full body human detection we use the algorithm pro-posed in [5] to train a tree-structure multi-view human de-tector as illustrated in Figure 2. To deal with partial occlu-sion of human body, part detectors are also trained to de-tect the partially occluded humans in which we partition thehuman body into three levels (head-shoulder (HS), head-torso (HT), full-body (FB)) (as shown in Figure 3) that isconcentrated on the upper human body which is differentfrom in [15] where head-shoulder, torso, leg, full-body wereused. This partition is preferred considering the followingthree factors: firstly, in common surveillance scenarios, theupper human body is most likely to be visible when occlu-sion happens; secondly, the upper human body undergoesless variations compared with other human body parts (suchas legs) which makes the upper body centered part detec-tors learned more robustly; last because of this special par-tition the three detectors can be efficiently co-trained by fea-ture sharing that guarantees a high computation efficiency intracking. This partition method describes a human body inthree levels of different representation power and trackabil-ity. The full-body part covering all the human body area hasthe highest representation power but the lowest trackabil-ity because it’s most likely to be occluded by other objectswhich makes it harder to be observed and tracked especiallyin crowded situation.

In our system, we use this multi-view multi-part humandetector to collect observations. A human hypothesis con-sists of three overlapped part areas of the human body: theFull Body, the Head-Shoulder, and the Head-Torso. Denotethe detection response as a 6-tuple rp = {l, p, s, t, v, a}where l is the label indicating the response type which couldbe FB, HS or HT, p is the position, s is the size, t is the timestamp (frame index), v is the visible score and a is the ap-pearance model. A combined response is the union of therepresentations of its parts, rc = {rpi|li = FB,HS,HT}.A human hypothesis H can be represented as {rc, u} where

1202

HS

24×24

24×32

24×58

HT FB

Figure 3. Hierarchy of upper body centered human partition.

Figure 4. Detection Responses generated by the detector (red forFB, yellow for HT, blue for HS, green is the human hypothesis).

u indicates the visible part. If u is none of the three responsetypes, it means the human is not visible.

In tracking, the detector is applied to each frame to geta set of detection responses. Under the assumption that hu-mans are on a plane to which the camera looks down, themethod in [15] is used to check the visibility of each de-tection response and to propose human hypotheses H . Thehuman hypotheses proposed in each frame are used both inthe local tracklets filter stage and the global tracklets asso-ciation stage later. Figure 4 shows the responses generatedby the detector on some sample frames.

5.2. Local Tracklets Filtering

In the local stage, a set of reliable tracklets S+1:t

(P (S+1:t|O1:t)) is to be generated. As formalized in sec-

tion 4, this is equivalent to sequentially estimate P (sit|oi

1:t),which stands for the distribution of the i-th object stategiven all the observations oi

1:t(hereafter, we suppress thesubscript t and superscript i for briefness when there is noambiguity).

This can be done by a particle filter or CONDENSA-TION [7] based on the following well-known two-step re-cursion procedure:

Predict : p(st|o1:t−1) =∫

p(st|st−1)p(st−1|o1:t−1)dst−1

Update : p(st|o1:t) ∝ p(ot|st)p(st|o1:t−1) (4)

The recursion requires a motion model p(st|st−1)and anobservation model p(ot|st). To handle complicated distri-butions which lead to analytical intractability, particle filterapproximates the two steps by a set of weighted samples{s(n)

t , π(n)t }N

n=1.When the system has a reliable observation model, the

state of the object can be well updated. This corresponds tothe situation when one object is isolated from other objectsand the detector can generate good observations. But in thedense environment where multiple objects are close to eachother and not all of the human bodies are fully visible, thedetector will fail to generate reliable observations or evengive wrong responses. In these situations, the predictionand update process of particle filter will give unpredictablefiltering results.

In order to deal with the unreliable observation prob-lems, we propose to select the best subset of correspondingobservations in the particle filter procedure.

Suppose we have l different observations for each states that is denoted as o = (o1, . . . , ol), where the k-th obser-vation is p(ok|s). Assuming observations are conditionallyindependent given the target state s, we have

p(o|s) = p(o1, . . . , ol|s) =l∏

i=1

p(oi|s) (5)

Instead of updating particle weights directly accordingto the above observation model, we dynamically select thebest subset of current observations which corresponds tovisible parts. For this purpose, the two-step recursion isextended to a three-step one as follows:

Predict : p(st|o1:t−1) =∫

p(st|st−1)p(st−1|o1:t−1)dst−1

Select : ot = arg maxot⊆ot

(p(ot|st)) (6)

Update : p(st|o1:t) ∝ p(ot|st)p(st|o1:t−1)

The Select procedure selects the best subset ot =(ol1t

, . . . , olctt

) of all the observations ot which maximizesthe observation probability based on the predicted objectstate st.

The initialization and termination of a reliable trackletare both guided by the global stage (which will be describedin subsection 5.3). Our particle filter with observer selectionprocess automatically selects the best subset of weightedparticle samples according to the object state in previousframe to update the object state in current frame to growthe tracklet. If the object state can not be observed (all theparts are invisible or lost by the observation model), the lo-cal tracklet filtering stage on this object stops and the track-let is buffered for the global stage.

1203

In implementation, a zero order motion model withGaussian diffusion was chosen for the dynamic model andthe confidence output by the detector was used as the obser-vation model [11].

5.3. Global Tracklets Association with DetectionResponses

In the global stage, a set of potential tracklets S−1:t

(P (S−1:t|S+

1:t, O1:t)) is to be generated from a temporal slid-ing window around trajectory break points using the de-tection responses and then they will be associated with thetracklet set S

+1:t from the particle filter with observer selec-

tion to optimize the final object trajectories S∗1:t (see equa-

tion 3 in section 4).

5.3.1 Detection Response Association for PotentialTracklets

Detection responses are associated within a temporal slid-ing window around trajectory break points (tails of reliabletracklets) to generate a set of potential tracklets. The size ofthe temporal sliding windows can be adjusted according tothe occlusion degree of the video data.

At each frame within the temporal sliding window, wefirst collect object hypotheses according to detection re-sponses. And then we add occluded object hypotheses ac-cording to the tracking results at the local stage. Theseobject hypotheses are then associated with the object hy-potheses in the previous frame based on the affinity of theresponses in position, size and appearance. The associationis guided by a greedy strategy as in Wu et al. [15]. If anobject hypothesis is associated in T (typically T = 5) con-secutive frames, then a new potential tracklet is generated.All the potential tracklets generated in this way will be fur-ther associated with the reliable tracklets as described in thefollowing subsection.

5.3.2 Pairwise Tracklet Association for Final ObjectTrajectories

The tracklet association approaches [14][8][6] model thejoint association likelihood P (T1, T2, . . . , Tn) of track-lets as the product of pairwise association likelihoodsP (Ti, Tj). The pairwise association likelihoods are thenrepresented in an association cost matrix C (with Cij =− log(P (Ti, Tj)) and the optimal association (maximizingthe joint likelihood) is computed using the Hungarian al-gorithm. In our implementation, we made some changemainly to the association cost matrix to meet the require-ment of associating two kinds of tracklets as illustrated be-low.

Supposing there are m tracklets in the Reliable Trackletsset S

+1:t and n tracklets in the Potential Tracklets set S

−1:t.

There may be four situations after tracklet association be-tween these two independent tracklet sets: TRi associateswith TPj ; TRi is a terminal object trajectory; TPj is aninitial object trajectory; TPj is a false trajectory. In orderto model all these four situations, we define tracklet associ-ation cost matrix as follows:

C =[Am×n Bm×m

Dn×n 0m×n

](7)

where A = {aij}m×n models the first situation, aij =− log P (TRi, TPj) is the pairwise association cost oftracklet TRi and TPj ; B = diag{b1, . . . , bm} mod-els the second situation, bi = − log P (TRi) is thecost of terminating the Reliable Tracklets TRi. D =diag{d1, . . . , dn} models the third and fourth situation,di = − 1

2 (log Pinit(TPj) + log Pfalse(TPj)) is the costof initializing the Potential Tracklet TPj as a new track ora false track.

In order to calculate the association likelihood betweentwo tracklets, we explore the appearance, shape and motionattributes of the tracklet. So we describe a tracklet Ti by a3-tuple {Ai,Si,Mi}, where Ai is the appearance model,Si is the shape model and Mi is the motion model.

Appearance Model Ai: We use color histograms as theappearance model of a tracklet. The color histograms arecalculated for each object part area and are updated over thedetection responses along the tracklet when the correspond-ing part is visible. A weighted Bhattacharya distance usingthe gaussian kernel between the two tracklets color modelsis calculated as the distance measure d(ai, aj) between thetracklets.

Pa(Ti, Tj) = exp(−d(ai, aj)) (8)

where d(ai, aj) = mean{d(aLi , aL

j )|L = FB,HT, HS}Size Model Si: Because the aspect ratio of an object is

a constant (determined by the aspect ratio of the trainingsamples), we just compute the estimated 3D object heightof each tracklet from the responses along the tracklet as theobject shape model. The expected 3D height averaged overthe entire tracklet is used as the 3D height estimation.

Ps(Ti, Tj) = exp(−|hi − hj |hi + hj

) (9)

Motion Model Mi: We calculate both the forward ve-locity and backward velocity of the tracklet as its MotionModel. The forward velocity is calculated from the refinedposition of the tail response of the tracklet while the back-ward velocity is calculated from the refined position of thehead response of the tracklet. Motion model in forward di-rection is represented by a gaussian {xF

i ,ΣFi }, and in back-

ward direction by a gaussian {xBi ,ΣB

i }.

Pm(Ti, Tj) =G(ptaili + vF

i Δt; pheadj ,ΣB

j )·G(phead

j + vBj Δt; ptail

i ,ΣFi )

(10)

1204

where pheadi is the position of the head response of tracklet

Ti and ptaili is the position of the tail response of tracklet Ti.

Assuming the independence among the three trackletmodels, the pairwise association likelihood of tracklet andcan be calculated as:

P (TRi, TPj) =Pa(TRi, TPj) · Ps(TRi, TPj)·Pm(TRi, TPj)

(11)

The likelihood of terminating the Reliable Tracklet TRi

is modeled as:Pterm(TRi) = Zt(1 − α)ω−b (12)

where Zt is a normalization factor, ω is the temporal slidingwindow size, b is the number of frames in which the objectis buffered because of occlusion.

The likelihood of the Potential Tracklet being a newtrack is modeled as:

Pinit(TPi) = Zt(1 − α)l (13)

where l is the length of the Potential Tracklet TPi, and be-ing a false track as:

Pfalse(TPi) = Zt(1 − α)2T−l (14)

So far we can apply the Hungarian algorithm on costmatrix C to get optimal association. Denoting M∗ =[m∗

ij ](m+n)(m+n) as the optimal assignment matrix ob-tained, for each m∗

ij = 1, do as follows:(1).If i ≤ m and j ≤ n, then associate TRi and TPj ;(2).If i ≤ m and j > m,then TRi is considered as a

terminated tracklet;(3).If i > m and j ≤ m, then TPj is considered as a

new track if Pinit(TPi) > Pfalse(TPi), or a false track ifPinit(TPi) < Pfalse(TPi);

Table 1 summarizes the overall algorithm of the twostage tracking framework.

6. ExperimentsWe evaluated our two-stage multi-object tracking algo-

rithm on two public datasets: the CAVIAR dataset [1] andthe ETH Mobile Scene (ETHMS) dataset [4]. The CAVIARdataset includes 26 video sequences of a walkway in a shop-ping center taken by a single camera with frame size of384 × 288 and frame rate of 25fps. The ETHMS datasetincludes 4 video sequences of street scenes taken by a mov-ing camera, with frame size of 640 × 480 and frame rateof 15fps. Both of the two datasets are very challenging be-cause of the heavy inter-person occlusions and poor imagecontrast between objects and background. We evaluatedour algorithm on its tracking performance, detection per-formance and speed, and compared our results with Wu etal.’s [15] method and Zhang et al.’s [16] work because theyhave reported the best results on the datasets.

Table 1. Algorithm of the two-stage tracking framework.Given: the tracked object set St−1 = {s1t−1, . . . , sm

t−1} with their ob-servations and other attributes at frame t− 1, the temporal sliding windowbuffer W.

Local Stage: For each tracked object:If si

t−1 can be observed:• Resample: for s = l1t−1, . . . , l

ct−1t−1 , simulate αn,os ∼

{πi,(n)t−1,os

}Nosn=1, and repace {si,(n)

t−1,os, π

i,(n)t−1,os

}Nosn=1 with

{si,(αn,os )t,os

, 1Nos

}Nosn=1

• Predict: simulate si,(n)n,os ∼ p(si

t|si,(n)t−1,os

)

• Select: oit = Select(oi

t) = [ol1t, . . . , ol

ctt

]

• Update:πi,(n)t−1,os

← p(oit|si,(n)

t )

• Estimate:sit =

∑s

∑n s

i,(n)t · πi,(n)

t,os

Else buffer the object for the global stage.

Global Stage:• Slide W tail to current frame;• Retrieve the reliable tracklets from the local stage;• Retrieve the potential tracklets larger than T within W and then

smooth the position and size using Kalman Filter;• Calculate the modified pairwise association cost matrix and apply

Hungarian algorithm;• Manage the object trajectories according to the assignment matrix;

Output:Estimate the objects state St = {s1t , . . . , smt }

Table 2. Comparison of the tracking results on CAVIAR dataset.Method GT MT PT ML FRMT IDS

Wu et al. [15] 140 106 25 9 35 17Zhang et al. [16] Alg.1 140 104 29 7 58 7Zhang et al. [16] Alg.2 140 120 15 5 20 15Our Local Stage 140 112 12 6 33 16Our two-stage 140 118 17 5 24 14

6.1. Tracking Performance

We adopt the same metrics as in [15] to evaluate thetracking performance:

• MT: Mostly Tracked trajectories, the number of trajec-tories that are successfully tracked for more than 80%;

• ML: Mostly Lost trajectories, the number of trajecto-ries that are tracked for less than 20%;

• PT: Partially Tracked trajectories, the number of tra-jectories that are tracked between 20% and 80%;

• FRMT: Fragmentation, the number of times a trajec-tory is interrupted;

• IDS: ID Switches, the number of times two trajectoriesswitch their IDs.

The tracking results on the CAVIAR dataset are showedin Table 2. As in [16], people that are too small in the im-ages (less than 24 pixels in width) are not counted in theevaluation. This dataset is fully independent from the train-ing dataset of our detector or tracker.

1205

Table 3. Comparison of the detection results.Dataset Method DR DA FAF

CAVIAR

Input[15][16] 72.8% N/A 0.270Wu et al. [15] 75.2% N/A 0.281Zhang et al. [16] Alg.1 75.2% N/A 0.081Zhang et al. [16] Alg.2 74.3% N/A 0.105Our Input 76.7% 0.657 0.262Our Local Stage 79.3% 0.739 0.078Our two-stage 81.8% 0728 0.136

ETHMS

Input[16] 64.3% N/A 1.51Zhang et al. [16] Alg.1 68.3% N/A 0.85Zhang et al. [16] Alg.2 70.4% N/A 0.97Our Input 65.1% 0.568 1.239Our Local Stage 71.8% 0.658 0.762Our two-stage 75.2% 0.643 0.939

From Table 2 we can see that the tracking results of ourlocal stage method outperform both the results of Algo-rithm.1 in [16] and the results in [15] in every aspect whichreflect the effectiveness of our local stage. Though we onlycollect global information within a temporal sliding win-dow in our two-stage tracking framework, we get compara-ble results to the Algorithm.2 in [16]. Figure 5 shows sometracking results from which we can find that our two-stagemethod can not only track partial occluded humans but alsocan recover trajectories from full occlusion.

6.2. Detection Performance

We use the detection rate (DR), detection accuracy (DA)and false alarm per frame (FAF) to evaluate detection per-formance and compare with direct detection result and pre-vious methods[15][16]. The detection accuracy is definedas:

DA =∑

i A(Di ∩ Gi)∑i A(Di ∪ Gi)

(15)

where Gi is the i-th ground truth (bounding box) while Di

is the corresponding detection result. A(·) calculates thearea of the input region.

The results are showed in Table 3 from which we can seethat our input observation set has a much higher DR and rel-ative lower FAR than the input observation set in [15][16].This proves the adaptability of our human body partitionmethod in common visual surveillance system. Comparedto our input observation set, both the local stage and two-stage tracking method make improvement on the detectionrate, detection accuracy and false alarm per frame.

6.3. Speed

The speed of our system is about 4fps on the CAVIARdataset and about 1.5fps on the ETHMS dataset which in-cludes frame by frame detection time. The test machineis an Intel Core 2 CPU with 2G RAM and the program iscoded in C++ without any parallel procedure. We analyzed

the execution time of our program and found that the detec-tion for collecting observations took up about 80% of thetotal execution time and the speed of frame by frame detec-tion is less than 5fps on the CAVIAR dataset. To improvethe detection efficiency, we make it scan only part of the im-age in each frame which results in a tracking that could runat 15fps without losing much accuracy. So given an inputvideo of size 384 × 288, the system can process it onlinesmoothly. The two-stage tracking system has a very hightracking efficiency which could be used for online visualsurveillance system.

7. ConclusionsIn this paper, we present an online detection-based two-

stage multi-object tracking framework which seeks both thelocal optimum trajectory for each object and the globaloptimum trajectories for all the tracked objects. It in-tegrates particle filter based tracker with data associationbased tracker efficiently to guarantee online tracking per-formance. It is not only different from the classical particlefilter which results in many breaks of object trajectories dueto occlusions but also different from the conventional asso-ciation based tracker which is time consuming and usuallyoffline. Experimental results on two challenging datasetsshow that the proposed method significantly improves thetracking performance both in tracking accuracy and track-ing efficiency in dense visual surveillance environment withdifferent types of occlusions.

8. AcknowledgementThis work is supported in part by National Basic Re-

search Program of China (2006CB303102), Beijing Edu-cational Committee Program (YB20081000303), and it isalso supported by a grant from Omron Corporation.

References[1] Caviar dataset. http://homepages.inf.ed.ac.uk/rbf/CAVIAR/.

6[2] M. Andriluka, S. Roth, and B. Schiele. People-tracking-by-

detection and people-detection-by-tracking. In CVPR, 2008.2

[3] I. J. Cox. A review of statistical data association for motioncorrespondence. IJCV, 10(1):53–66, 1993. 2

[4] A. Ess, B. Leibe, and L. V. Gool. Depth and appearance formobile scene analysis. In ICCV, 2007. 6

[5] C. Hou, H. Ai, and S. Lao. Multiview pedestrian detectionbased on vector boosting. In ACCV, 2007. 3

[6] C. Huang, B. Wu, and R. Nevatia. Robust object trackingby hierarchical association of detection responses. In ECCV,2008. 2, 5

[7] M. Isard and A. Blake. Condensation – conditional densitypropagation for visual tracking. IJCV, 28(1):5–28, 1998. 4

1206

Figure 5. Tracking result, the upper two rows are in CAVIAR, the bottom two rows are in ETHMS. (rectangle in red, yellow or blueindicates the largest part used for tracking, green rectangle is the local tracking result, gray rectangle is the global tracking result).

[8] R. Kaucic, A. G. A. Perera, G. Brooksby, J. Kaufhold, andA. Hoogs. A unified framework for tracking through occlu-sions and across sensor gaps. In CVPR, 2005. 5

[9] B. Leibe, K. Schindler, and L. V. Gool. Coupled detectionand trajectory estimation for multi-object tracking. In ICCV,2007. 2

[10] Y. Li, H. AI, C. Huang, and S. Lao. Robust head trackingbased on a multi-state particle filter. In FG, 2006. 2

[11] Y. Li, H. AI, T.Yamashita, S.Lao, and M. Kawade. Trackingin low frame rate video: A cascade particle filter with dis-criminative observers of different lifespans. In CVPR, 2007.1, 2, 5

[12] K. Okuma, A. Taleghani, D. Freitas, J. J. Little, and D. G.Lowe. A boosted particle filter: Multitarget detection andtracking. In ECCV, 2004. 2

[13] D. Reid. An algorithm for tracking multiple targets. IEEE

Transactions on Automatic Control, 24(6):843–854, Dec1979. 2

[14] C. Stauffer. Estimating tracking sources and sinks. In IEEEWorkshop on Event Mining in Video, 2003. 5

[15] B. Wu and R. Nevatia. Detection and tracking of multi-ple, partially occluded humans by bayesian combination ofedgelet based part detectors. IJCV, 75(2):247–266, 2007. 1,2, 3, 4, 5, 6, 7

[16] L. Zhang, Y. Li, and R. Nevatia. Global data association formulti-object tracking using network flows. In CVPR, 2008.2, 6, 7

1207