detecting and tracking people using an rgb-d camera via
TRANSCRIPT
Detecting and Tracking People using an RGB-D Camera via Multiple DetectorFusion
Wongun Choi ∗1 Caroline Pantofaru †2 Silvio Savarese ‡1
1Electrical and Computer Engineering, University of Michigan, Ann Arbor, USA2Willow Garage, Menlo Park, CA, USA
Abstract
The goal of personal robotics is to create machines thathelp us with the tasks of daily living, co-habiting with usin our homes and offices. These robots must interact withpeople on a daily basis, navigating with and around people,and approaching people to serve them. To enable this co-existence, personal robots must be able to detect and trackpeople in their environment. Excellent progress has beenmade in the vision community in detecting people outdoors,in surveillance scenarios, in Internet images, or in specificscenarios such as video game play in living rooms. The in-door robot perception problem differs, however, in that theplatform is moving, the subjects are frequently occluded ortruncated by the field-of-view, there is large scale variation,the subjects take on a wider range of poses than pedestri-ans, and computation must take place in near real time.
In this paper, we describe a system for detecting andtracking people from image and depth sensors on board amobile robot. To cope with the challenges of indoor mobileperception, our system combines an ensemble of detectorsin a unified framework, is efficient, and has the potential toincorporate multiple sensor inputs. The performance of ouralgorithm surpasses other approaches on two challengingdata sets, including a new robot-based data set.
1. Introduction
The primary mission of a personal robot is to help andco-habit with people. At any time, a robot might have tointeract with a person to perform a task such as delivering abeverage or receiving an object to put away. Other tasks arebest performed without bothering the human co-habitant, bynavigating around a home or an office smoothly, safely andunobtrusively. To function effectively in a human environ-
∗[email protected]†[email protected]‡[email protected]
Figure 1. Example tracking results generated by our algorithm.The left panel shows the detected targets in the RGB image. Theright panel shows a projection of the 3D location onto the groundplane.
ment, a robot must be able to detect and track people in itsvicinity. Perception in indoor human environments can beextremely challenging as the world that the robot “sees” iscomplex: all of its sensors are mounted on-board a movingplatform, the scene is full of clutter, and the dynamic rangeis very high. Moreover, the people it observes may take ona wide range of poses and motion patterns, be subject to oc-clusions and self-occlusions, and be truncated by the robot’sfield of view (Fig. 1). On top of all this, a robot must senseand react to people in near real time to enable interaction.
In this paper, we propose a new system for detection andmulti-target tracking of people as seen in RGB-depth im-agery acquired from an indoor mobile platform, as exem-plified in Fig. 1. To cope with the challenges of this task,we propose to integrate multiple complementary sensingmodalities and perceptual cues acquired using an ensembleof detectors. This ensemble includes image-based pedes-trian and upper body detectors, an image-based face detec-tor, a skin detector, as well as a depth-based shape detectorand motion detector. These detection algorithms are fusedinto a coherent framework using a sampling based method(Reversible Jump Markov Chain Monte Carlo particle fil-tering) constructed on a tracking-by-detection formulation.
Experimental results show that our approach outper-forms competing algorithms in a number of detection andlocalization tasks as seen in two challenging datasets. These
datasets include RGB-D sequences acquired from a mobileplatform, containing imagery of people performing activi-ties in a real office environment and under real-world varia-tion and constraints.
The main contribution of this paper is an algorithm forperson detection and tracking which is capable of handlingi) people in various poses, ii) occlusion, iii) a mobile plat-form, iv) scene clutter, and v) real-time performance. In ad-dition, we will contribute the code and data used to evaluateour algorithm.
2. Related WorkThere is a long history of approaches to person detec-
tion and tracking in the computer vision and robotics liter-ature. Many of these techniques (such as [11, 12, 9, 10])have been shown to be successful in outdoor scenarios orwhen scoped to the scenarios for which they were designed.They have been less successful, however, for indoor envi-ronments. In outdoor scenes, people are mostly observed inan up-right ‘pedestrian’ position, whereas in indoor scenespeople are often seen in more variable configurations (e.g.sitting on chairs, truncated by the image boundary, or oc-cluded.)
Methods have been proposed to track targets by learninga person specific appearance model given an initial posi-tion [4, 8]. These methods work relatively well when thebackground is not cluttered, but often suffer from the prob-lem of track drift [18] and require manual selection of ini-tial target positions. The improvement in people detectionalgorithms made it possible to design robust tracking-by-detection algorithms [26, 27, 5, 7, 6, 15]. Wu et al. [27]integrated an image based detection algorithm into a track-ing framework. Breitenstein et al. [5] proposed to use thedetector confidence value together with the detection out-put to generate a robust tracking algorithm. Whereas Khanet al. [15] proposed an MCMC particle filtering method totrack multiple interacting targets and employed it to analyzethe behavior of ants.
Similar to [15], Choi et al. [7] proposed an algorithmfor simultaneously tracking multiple targets as well as esti-mating camera parameters. To make tracking more robust,Wojek et al. [26] explicitly reasoned about the targets’ oc-clusions. Recently, [6] proposed a novel procedure based onmaximum weight independent sets to solve the correspon-dence problem among targets.
In order to improve robustness and accuracy, multi-ple approaches explore the idea of injecting knowledgeabout 3D structure of the scene into the tracking pro-cess [11, 20, 17]. [11, 20] proposed a system which com-bines depth information obtained from stereo cameras (orlaser scans) and detection responses obtained from an RGBcamera. In [3], a 2D lidar scanner is employed to detect andtrack people by identifying the legs’ cylinder shape.
3D Target to Image Projection fp
Multiple Cue Fusion
RJ-MCMC Particle Filtering
Targets
Target Hypothesis
Figure 2. Our algorithm combines RGB and depth image cues todetect and track multiple targets robustly. Given a hypothesis of avalid target location, an ensemble of detectors generates a confi-dence map (Sec.4) which is then used by Reversible Jump MarkovChain Monte Carlo particle filtering to identify and localize people(Sec.5).
The availability of an affordable RGB-D camera fromMicrosoft, the Kinect [19], made it possible for everyone toobtain useful depth information. Recently, [17] proposed apedestrian tracking algorithm using a combination of HOGand HOD features with multiple Kinect cameras. Basedon assumption that the person is detected and segmentedfrom the background, Shotton et al. [24] proposed a skele-ton tracking method based on Random Forest using Kinect.
The systems presented above, however, do not focus onthe indoor, mobile platform perception problem, and hencethey do not integrate all of the components necessary totackle such a task.
3. Model Representation
We tackle this problem within a sequential Bayesianframework. At each time stamp, we obtain the approximateposterior distribution of presence and location in 3D spacefor a set of targets using an RJ-MCMC algorithm (Sec. 5).Each hypothetical sample is then evaluated using an ensem-ble of detectors on the projection of the 3D location into theimage plane, as in Fig. 2. Below, we describe the Bayesianmodel observation likelihood and motion model. Sec. 4 de-scribes each of the detectors in the ensemble, and Sec. 5describes the RJ-MCMC sampling framework.
3.1. Bayesian ModelGiven a sequence of color and depth images I1∼t in time
(1, 2, ..., t), our goal is to detect and track people in 3Dspace. For consistency, we track the top position of eachperson’s head. This can be achieved by finding the maxi-mum a posteri (MAP) solution of the following probabilis-tic formulation. Let Zt = Z0
t , Z1t , ..., Z
kt be a set of tar-
gets at time stamp t and Zit be each target’s head location
parametrized as the 3D point (x, y, z). Following the Se-quential Bayesian formulation, the posterior probability of
targets P (Zt|I1∼t) can be written as:
P (Zt|I1∼t) ∝ P (It|Zt)︸ ︷︷ ︸(a)
∫P (Zt|Zt−1)︸ ︷︷ ︸
(b)
P (Zt−1|I1∼t−1)︸ ︷︷ ︸(c)
dZt−1
(1)where (a), (b) and (c) in Eq. 1 represent the observation
likelihood, the motion prior and the posterior at time t− 1,respectively.
Assuming independent motion between targets, we canfactorize the overall observation likelihood P (It|Zt) andmotion prior P (Zt|Zt−1) as
P (It|Zt) =
Mt∏i=0
P (It|Zit) , P (Zt|Zt−1) =
Mt∏i=0
P (Zit |Zi
t−1)
where Mt indicates the number of targets at time stamp t.
3.2. Observation Likelihood
Instead of using a single detection algorithm, we incor-porate multiple weak detectors to get a strong confidencevalue about the presence of a target(s) in the scene (seeSec.4 for more details). Assuming independence of the ob-servations, we can rewrite the observation likelihood of atarget i (P (It|Zi
t)) as follows:
P (It|Zit) =
N∏j=0
Pj(It|Zit) (2)
where j is the weak detector. The 3D location of Zit does
not directly correspond to a point in the color and depthimages It, so we project Zi
t into the image plane using thecamera parameters.
3.3. Motion Prior
The motion prior of each target encodes two character-istics: existence and smoothness. The former encodes theprior probability of the target’s presence at adjacent timestamps. The intuition is that if a target exists at time stampt − 1 then it is more likely for this target to exist at timestamp t, and vice versa. The latter enforces smoothness ofpeople’s motion in 3D space, e.g. people cannot jump todistant locations or heights in a short time. We model theexistence prior by two binomial probabilities Ps and Pe. Ps
is the probability of stay and Pe is the probability of en-trance. Ps encodes the likelihood that a target will exist intime t if it exists in time t − 1, and Pe encodes the proba-bility that a new target will appear in the scene. In practice,we use 0.9 for Ps and 0.1 for Pe. The motion smoothness ismodeled as a Gaussian distribution over (x, y, z) centeredon the location of the target at t− 1.
4. Observation CuesSince the color and depth images contain different in-
formation about the scene, combining them results in a
more robust detection algorithm. To that end, we combinefive different observation models: HOG, shape from depth,frontal face detection, skin and motion detection. In isola-tion, none of the detectors performs satisfactorily (e.g. theface detector misses people in profile or turned away fromthe camera, and the HOG detector yields many false posi-tives and missing detections), but combining all of them cangenerate much more reliable results. Note that our model isflexible enough to handle additional observation models asrequired to increase robustness.
In this section, we will adopt log likelihood l(It|Zit) in-
stead of the likelihood P (It|Zit) for simplicity. The entire
observation likelihood P (It|Zit) can be obtained by taking
the exponential of a weighted linear sum of each log likeli-hood. The following sections describe the detectors in de-tail.
P (It|Zit) ∝ exp(
∑j
wj lj(It|Zit)) (3)
4.1. Pedestrian and Upper Body Detector
The first cue originates from the distribution of gradi-ents in the image as encoded by the Histogram of OrientedGradient detector (HOG) [9]. To obtain a detection re-sponse, the HOG detector performs a dot product betweenthe model parameter w and the HOG feature h, and thresh-olds the value (above zero) to find person detections. In thiswork, we incorporate two HOG detection models, an upperbody detector and a full body detector, to cope with i) oc-clusion of the lower body, ii) different pose configurationsand iii) different resolutions of people in images.
Previous work [7] used a Gaussian model centered onpositive detections to obtain the observation model. How-ever, such approaches often fail when there are many misseddetections or false positives. Inspired by [5], we directly usethe detection response to model the observation likelihoodfrom the HOG detector.
lHOG(It|Zit) = w · h(fP (Zi
t)) (4)
where fP is an image projection function, h(fP (Zit)) repre-
sents the HOG feature extracted from the image projectionof Zi
t .
4.2. Depth-based Shape Detector
To model the likelihood of people in depth images, wedefine the log likelihood lShape as a distance between ashape template and the observed shape of a human. Thetemplate is defined for only the head and shoulder regionsince this is among the most stable body element across var-ious types of common human body configurations. Given adepth image and the location of a person, Zi
t , a W (width)by H (height) dimensional binary vector is constructed bythresholding the depth image around Zi
t . Then,
lShape(It|Zit) = τs − d(Stemp, S(fP (Z
it); It)) (5)
tZ i
l = τ - d shape sStemp S( f (Z ))p t
i( , )Figure 3. The shape vector is computed from the upper half ofthe depth image bounding box. The binary vector representingthe shape of the person’s head and shoulder area is compared to atemplate using the Hamming distance.
where τs is a threshold, Stemp is the template,S(fP (Zi
t); It) is the shape vector of Zit , and d(·, ·) is the
distance between template and shape vector of Zit . See
Fig.3 and Sec.6.1 for details.
4.3. Face Detector
The Viola-Jones face detector [25] as implemented inOpenCV [16, 1] detects people reliably if the image of theface is large enough (greater than 24 pixels) and the frontalside of the face is visible. We incorporate the face detectorresponse as another likelihood measure by calculating themaximum overlap ratio between detection output Xk
t of theface detector and image projection of hypothesis Zi
t acrossall K face detections at time t:
lFace = maxk
OR(Xkt , Tf (fP (Z
it))) (6)
where Tf is a face cropping transformation and OR(·, ·)is the standard overlap ratio between two rectangles (theintersection over the union of the two rectangles).
4.4. Skin Color Detector
The next cue used is skin color. If a person exists ina location Zi
t , then skin pixels corresponding to the faceregion are likely to be observed. To detect skin pixels, wethreshold each pixel in HSV color space and apply a medianfilter on the binary skin image ISkin. Given the output, thelikelihood is obtained by computing the ratio of skin pixelslying in the face region of a hypothesis:
lSkin =1
|Tf (fP (Zit))|
∑(x,y)∈Tf (fP (Zi
t))
ISkin(x, y) (7)
where | · | represents the area of a bounding box · and ISkin
is the filtered binary skin image.
4.5. Motion Detector
Finally, we use motion as an additional cue that implies(with high confidence) the presence of a person in the scene.In order to efficiently identify moving pixels in 3D, we ap-ply an octree-based change detection algorithm [14] to the
point clouds at two consecutive time stamps. A binary mo-tion image is obtained by projecting each of the movingpoints into the image. Similar to the skin detector, the like-lihood is obtained by calculating the ratio of moving pixelslying in the body region of a hypothesis.
lMotion =1
|fP (Zit)|
∑(x,y)∈fP (Zi
t)
IMotion(x, y) (8)
where IMotion is the binary motion image.Note that for many of these cues, such as face detection,
skin color detection and motion detection, a positive obser-vation increases the likelihood that a person is present, butthe lack of observation does not decrease the likelihood thata person is present.
5. Reversible Jump Markov Chain MonteCarlo Particle Filtering (RJ-MCMC)
Since people can enter and leave a scene at any timestamp, the vector Zt has variable dimension. To deal withthis variable dimensionality and efficiently find the MAPsolution over the posterior distribution P (Zt|I1∼t), we in-troduce a reversible-jump Markov Chain Monte Carlo (RJ-MCMC) algorithm [15]. The RJ-MCMC automatically de-tects new targets by using jump moves in different dimen-sions, e.g. adding new target or deleting existing target by arandom proposals.
Computing the posterior P (Zt−1|I1∼t−1) at t − 1 is in-tractable, so instead it is approximated using N unweightedsamples P (Zt−1|I1∼t−1) ≈ {Z(r)
t−1}Nr=1. Then the poste-rior P (Zt|I1∼t) at t can be approximated as follows:
P (Zt|I1∼t) ∝ P (It|Zt)∑r
P (Zt|Z(r)t−1) (9)
Performing RJ-MCMC sampling on the posteriorP (Zt|I1∼t) results in the posterior distribution for thenext time frame. In the following sections, we explainthe details of our proposal distribution and acceptanceratio computation for the Metropolis-Hasting algorithm.See [15] for further details of the RJ-MCMC algorithm.
For the remainder of this section, we assume that a weakdetection hypothesis Xt and the correspondences betweentargets and detections are available to guide the samplingprocedure. The detections are necessary to help initiate tar-gets and guide the sampling, but our algorithm is able tohandle missing detections and false positives.
5.1. Proposal Moves
For MCMC sampling to be successful, it is critical tohave a good proposal distribution which can explore the hy-pothesis space efficiently. Inspired by [15], we define fivedifferent reversible jump moves: Stay, Leave, Add, Delete,
and Update. Each move is designed as a reversible counter-part of another so as to guarantee the detailed balance of theMarkov Chain.
Stay: Even if a target does not exist in a current sam-ple Z(r)
t , the Stay move proposes to keep the target in thenew sample Z(r+1)
t . Among the targets in set S(r)t that are
not in a sample Z(r)t but existed in Zt−1, one target i is
randomly selected with a uniform probability 1
|S(r)t |
. Un-
like [15] (which samples from only the previous posteriorP (Zi
t |Zit−1)), we sample the new location of a target from
a mixture distribution of P (Zit |Xi
t) and P (Zit |Zi
t−1), whereXi
t is a corresponding detection. This makes the samplingprocess more robust when a target is moving. If no detec-tion is available for target i, the new proposal is sampledfrom the previous posterior distribution. The proposal canbe written as
QS(Z(r+1)t ;Z
(r)t ) =
{1
|S(r)t |
Qi(Zi(r+1)t ), if i ∈ S(r)
t
0, otherwise(10)
whereQi(Zi(r+1)t ) is equal to P (Zi
t |Zit−1) when there is no
corresponding detection, and 12 [P (Zi
t |Zit−1) + P (Zi
t |Xit)],
otherwise.Leave: If a target Stays in sample Z(r)
t , the Leave moveproposes to remove the target from the new sample Z(r)
t+1.This is the reverse jump move of Stay. Among the target setL(r)t that exist in Z(r)
t+1 and existed in Zt, one i is randomlyselected with a uniform probability 1
|L(r)t |
and removed. The
proposal can be represented as
QL(Z(r+1)t ;Z
(r)t ) =
{1
|L(r)t |
, if i ∈ L(r)t
0, otherwise(11)
Add: This proposal initiates a new target given the newdetections Xnew
t which do not correspond to any existingtargets. Among the detections A(r)
t = Xnewt \Z(r)
t that arenot in the current target set Z(r)
t , one i is randomly selectedwith a uniform distribution 1
|A(r)t |
and the new location of
target Zi(r)t is proposed from a distribution P (Z
i(r)t |Xi
t).The corresponding proposal distribution can be written as
QA(Z(r+1)t ;Z
(r)t ) =
{1
|A(r)t |
P (Zi(r)t |Xi
t), if i ∈ A(r)t
0, otherwise(12)
Delete: Similar to the Leave proposal, Delete is the re-verse jump move of Add. Among new detections D(r)
t =
Xnewt ∩ Z(r)
t that are already added in Z(r)t , one i is ran-
domly drawn with uniform probability 1
|D(r)t |
and removed
from Z(r+1)t .
QL(Z(r+1)t ;Z
(r)t ) =
{1
|D(r)t |
, if i ∈ D(r)t
0, otherwise(13)
Update: Update proposes a new location of a speci-fied target. Among all existing targets in a sample Z(r)
t ,a single target i is randomly selected and a new locationfor the sample is proposed from the proposal distributionQ(Z
i(r+1)t ;Z
i(r)t ) ∼ N (Z
i(r)t ,ΣU ). Note that one Update
move can be “reversed” by another Update move. The pro-posal can be expressed as follows:
QU (Z(r+1)t ;Z
(r)t ) =
{1
|Z(r)t |
Q(Zi(r+1)t ;Z
i(r)t ), if i ∈ L(r)
t
0, otherwise(14)
5.2. Acceptance Ratio
Following the Metropolis Hasting algorithm, we com-pute the acceptance ratio of the new sample Z(r+1)
t by theproduct of the three ratios:
a =P (It|Z(r+1)
t )
P (It|Z(r)t )
P (Z(r+1)t |I1∼t−1)
P (Z(r)t |I1∼t−1)
Q(Zrt ;Z
r+1t )
Q(Zr+1t ;Zr
t )(15)
The first term expresses the ratio between the image likeli-hoods, the second term represents the ratio between approx-imated predictions and the last encodes the ratio betweenproposal distributions. Since we change the state of onlyone target’s presence or location, most of the factors can becanceled out in the above computation. This characteris-tic makes the algorithm efficient and capable of processingvideos in real-time.
Algorithm 1 RJ-MCMC sampling
Require: Given Xt, It, {Z(r)t−1}Nr=1
Initialize Z(1)t
while r < N doSample Z ′t ∼ Q(Zt;Z
(r−1)t )
Compute a following Eq.15Z
(r+1)t ← Z ′t with probability a
end while
6. Experimental EvaluationIn this section, we discuss the experimental evaluation
of our algorithm, including the details of the actual imple-mentation. Testing was performed on both data collectedfrom on-board a mobile robot (the PR2) moving throughoutan office environment, as well as a stationary Kinect sensorplaced in a different office environment.
6.1. Implementation details
To begin with, an RGB-D sensor (in our case theKinect [19]) provides a pair of images, one RGB and onedepth. From these images, detection proposals are gener-ated using a combination of HOG detections, face detec-tions and 3D point cloud clusters [23].
The 3D clusters are generated by constructing a 3D pointcloud from the depth image and the camera parameters [23].Given the robot’s known 3D base location, 3D points on thefloor plane can be reliably removed. The remaining pointsare clustered using Euclidean clustering [22]. A single pro-posal for the location of a human head is given by the high-est point in a cluster. The proposals from all of the clustersare filtered by height (1.3∼2.3 m) as appropriate to our ap-plication and environment. The final remaining proposalsare associated with existing tracks using the Hungarian al-gorithm and 3D distance. This provides initialization forour algorithm and guides the MCMC sampling procedure.
To obtain HOG-based proposals and a confidence mapefficiently (sec. 4.1), we use the GPU implementation inOpenCV [1]. The detector can process 640x480 images in100∼200 milliseconds. The full body detections are ob-tained from the model trained on the INRIA dataset [9]and upper body detections from the CALVIN model [13].OpenCV also provides reliable face detections (sec. 4.3) us-ing the Haar feature-based face detection algorithm with afrontal face model [16]. Despite using OpenCV and theGPU, the HOG detector (full and upper body) and face de-tector are the slowest components in our system, so all threeare processed in parallel to the sampling algorithm.
To apply the depth-based upper body template, a ham-ming distance is computed between the template (fig.3) andthe shape in a depth-image window (sec. 4.2). The intro-duction of a more sophisticated distance such as a weighteddistance, or learning a depth template could provide a betterestimate, but we leave those as future work.
Skin pixels are found by thresholding the HSV between(2, 60, 40) and (15, 200, 200) (sec. 4.4). Finally, the octree-based motion detector is discretized to 3cm (sec. 4.5).
In the current implementation, the model weights foreach detector component are experimentally chosen usingvalidation data since no large dataset of RGB-D data froma moving, indoor platform is available. In future work, weintend to learn the weights.
Sampling is performed as follows. In each frame, wesample 2500 samples with 500 burn-in samples and 50samples for thinning to reduce correlation. The posteriordistribution at each time stamp is approximated by 40 un-weighted samples. This sampling takes about 100 millisec-onds to process one frame.
Since our algorithm is intended to run on a robot, ourimplementation is based on the ROS system [2]. ROS is amessage passing infrastructure for performing computationin a distributed fashion. ROS also provides standard func-tionality common to many robot platforms and applications,such as the ‘tf’ transform library which can be combinedwith the Kinect’s camera calibration to estimate the robotand camera’s location relative to a fixed point in the world.
Our implementation can process 5 frames per second on
average (using a GPU). The code for the entire system willbe posted on-line.
6.2. Dataset
We quantitatively evaluate our algorithm using two chal-lenging, new datasets. The first (static dataset) is acquiredin an office scenario with a fixed Kinect camera [19, 21].The dataset contains 17 videos each spanning 2 to 3 min-utes. The Kinect is mounted approximately 2 meters highand tilted downwards to improve the vertical field of view.Videos portray humans under different pose configurations(e.g. sitting on a chair, standing up), observed from dif-ferent view points (front, side, 3/4) and subject to variousdegrees of occlusion or self-occlusion.
The second dataset (the on-board dataset) was collectedwith a Kinect mounted on-board a robot (PR2). The robotdrives (tele-operated) around an office building, seeing peo-ple in offices, walking in corridors, in the cafeteria, and gen-erally going about their day. Both the robot and the peoplemay be moving. Subject distance from the camera, the num-ber of subjects in a frame, occlusion, lighting (includingdifficult skylights) and pose varies. This dataset includes 18segments.
For both datasets, we hand-annotated the people withbounding boxes around upper bodies, 3D locations inferredfrom the bounding boxes and depth images, and target ID.The annotation is provided for four images per second.
6.3. Results
We compared our algorithm against two baseline meth-ods: the Deformable Parts Model (DPM) [12] full body de-tector and upper body detector as trained by [13]. The DPMdetector is known to be more accurate than the HOG detec-tor, but the implementation of the HOG detector on a GPUis faster. We will integrate the DPM detector into our sys-tem in the future.
We report the log-average miss rate (LAMR) proposedby [26] as well as miss-rate vs. false-positive-per-image(FPPI) curve. As in [26], LAMR is computed by draw-ing equally spaced samples in log space of the FPPI. Weuse two evaluation protocols for identifying true positives.The first is based on the degree of overlap between detectedbounding boxes and ground truth bounding boxes(boundingbox overlap), and the second is a 3D distance threshold forlocalization evaluation.
As shown in Fig. 4(left), our algorithm significantly out-performs both baseline methods. On the static dataset, theimprovement is approximately 13% over both baselines. Onthe robot dataset, the improvement is 7% over the upperbody DPM detector and 20% over the full body detector.This improvement is seen despite using the weaker HOGdetector within our system. As hypothesized, the full bodydetector does not work well in our dataset due to occlusions,
tight field of view, and people who are in non-pedestrianposes such as sitting.
Next, we analyze the contribution of each detector mod-ule in our method, with results in Fig. 4(middle). As seen inthe plot, the depth shape detector plays the most significantrole, with the HOG detector providing the second largestcontribution.
When people do not show their face (as happens fre-quently in the static dataset), the face detector is not veryuseful. However, when people do show their faces, theface detector is a very strong contributor to overall detec-tion. This fact is somewhat diluted in the analysis in Fig. 4.Motion and skin indicators have similar issues, being verystrong in some instances and useless in others. The rea-sonable performance of the full algorithm as opposed to thevariability of the individual detectors is promising.
Finally, we evaluated our algorithm’s localization accu-racy. In Fig. 4(right), we show the LAMR measure overdifferent 3D distance thresholds. Our algorithm achievesmore accurate results when people are within approximately5 meters of the camera. This is to be expected as the Kinectprovides virtually no depth information past 5 meters dis-tance, and in fact the depth information past 3 meters isextremely noisy.
Overall, experiments show that our algorithm outper-forms state-of-the-art detectors. In addition, the fusion ofmultiple detection cues provides a more reliable final result.Our fusion method is capable of handling the variable per-formance of each individual detector.
7. ConclusionIn this paper, we have introduced an algorithm that can
detect and track people in indoor spaces from a mobile plat-form without instrumenting the environment. As shown inthe experimental evaluation, using an ensemble of detec-tion algorithms and fusing their results using RJ-MCMCparticle filtering results in robust and accurate person de-tection. Each detector module has different strengths andweaknesses, focusing on different body components or datacharacteristics, allowing the overall combination to handleocclusion, motion, truncation, and pose variation.
In future work, we intend to use data-driven training toimprove the algorithm’s parameters without hand tuning.For example, logistic regression can be used to map detec-tor confidence to better likelihood values. We also intendto learn person-specific detection models to improve dataassociation, especially when people leave the field of viewof the camera for short periods of time. Better matching oftracklets will improve overall algorithm robustness and alsoprovide more semantically meaningful track results fromwhich motion patterns can be learned. An extension of thisidea is to learn the interactions between people during dif-ferent activities, like walking together or standing in line.
Finally, this work is the first step in performing human posetracking from a mobile platform in a complex environment.Acknowledgement We would like to thank Yi-Hsuan Tsaifor his help on data collection and annotation. This work issupported by a grant from ONR (award #N000141110389)and NSF AEGER (award #1052762).
References[1] OpenCV, The Open Source Computer Vision library. http:
//opencv.willowgarage.com/wiki/. 4, 6[2] ROS, The Robot Operating System. http://www.ros.
org/. 6[3] K. Arras, S. Grzonka, M. Luber, and W. Burgard. Efficient
people tracking in laser range data using a multi-hypothesisleg-tracker with adaptive occlusion probabilities. In ICRA,2008. 2
[4] S. Avidan. Ensemble tracking. In PAMI, 2007. 2[5] M. D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier,
and L. V. Gool. Robust tracking-by-detection using a detec-tor confidence particle filter. In ICCV, 2009. 2, 3
[6] W. Brendel, M. Amer, and S. Todorovic. Multiobject track-ing as maximum weight independent set. In CVPR, 2011.2
[7] W. Choi and S. Savarese. Multiple target tracking in worldcoordinate with single, minimally calibrated camera. InECCV, September 2010. 2, 3
[8] D. Comaniciu and P. Meer. Mean shift : A robust approachtoward feature space analysis. In PAMI, 2002. 2
[9] N. Dalal and B. Triggs. Histograms of oriented gradients forhuman detection. In IEEE conference on Computer Visionand Pattern Recognition (CVPR), 2005. 2, 3, 6
[10] P. Dollar, S. Belongie, and P. Perona. The fastest pedestriandetector in the west. In BMVC, 2010. 2
[11] A. Ess, B. Leibe, K. Schindler, and L. van Gool. A mobilevision system for robust multi-person tracking. In CVPR,2008. 2
[12] P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ra-manan. Object detection with discriminatively trained partbased models. PAMI, 32(9), Sept. 2010. 2, 6
[13] V. Ferrari, M. Marin-Jimenez, and A. Zisserman. Progressivesearch space reduction for human pose estimation. In CVPR,2008. 6
[14] J. Kammerl. Octree point cloud compression inPCL. http://pointclouds.org/news/compressing-point-clouds.html, 2011. 4
[15] Z. Khan, T. Balch, and F. Dellaert. MCMC-based particlefiltering for tracking a variable number of interacting targets.PAMI, 2005. 2, 4, 5
[16] R. Lienhart and J. Maydt. An extended set of haar-like fea-tures for rapid object detection. In ICIP, 2002. 4, 6
[17] M. Luber, L. Spinello, and K. Arras. Learning to detect andtrack people in rgbd data. In RGB-D Workshop, RSS, 2011.2
[18] I. Matthews, T. Ishikawa, and S. Baker. The template updateproblem. PAMI, 26:810–815, 2004. 2
10−1 1000.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
false positives per image
mis
s ra
te
OURS (60.00%)DPM upper (73.41%)DPM full (73.91%)
10−1 1000.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
false positives per image
mis
s ra
te
Full (60.00%)
No Depth (84.47%)
No HOG (64.75%)
No Face (61.17%)
No Skin (62.60%)
No Motion (59.25%)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
distance threshold
log
aver
age
mis
s ra
te
Overall
Near(> 60 pixels)
Far(< 60 pixels)
10−1 1000.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
false positives per image
mis
s ra
te
OURS (60.01%)DPM upper (67.25%)DPM full (80.15%)
10−1 1000.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
false positives per image
mis
s ra
te
Full (60.01%)
No Depth (85.79%)
No HOG (69.48%)
No Face (60.80%)
No Skin (62.92%)
No Motion (63.69%)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
distance threshold
log
aver
age
mis
s ra
te
Overall
Near(> 60 pixels)
Far(< 60 pixels)
Figure 4. The top row shows results for the stationary Kinect dataset and the bottom row shows results for the on-board Kinect dataset.In both experiments our algorithm outperforms the baseline methods significantly (left panels). In the middle panel, we compare thecontribution of each detector. As expected, depth and the HOG detector contribute most to the tracking algorithm. The right panel showsLAMR measure over different distance thresholds. Detections within given threshold distance in the 2D ground plane and within 30 cm inheight from ground truth were considered as true positives. Targets whose upper body is larger than 60 pixels in image were considered tobe “near” (approximately within 5 meters).
Figure 5. Examples of tracking results. First row: results on the stationary camera dataset. Second row: results on the robot dataset.Detections are shown as bounding boxes in the images, and dots projected onto the ground plane in the top-down view. Each colorrepresents a different target. Notice that our algorithm can detect people in various poses, truncated by the image border, and despite thesevere occlusions between people which are common in indoor environments. Pedestrian detectors alone will generally fail in these cases.The last row shows difficult situations in which the people are beyond the Kinect’s depth range or under extreme lighting condition.
[19] Microsoft Corp. Kinect for XBOX. http://www.xbox.com/en-US/Kinect. 2, 5, 6
[20] L. Navarro-Serment, C. Mertz, and M. Hebert. Pedestriandetection and tracking using three-dimensional ladar data.IJRR, May 2010. 2
[21] PrimeSense. NITE natural interaction middleware. http://www.primesense.com/?p=515. 6
[22] R. B. Rusu. Semantic 3D Object Maps for Everyday Manipu-lation in Human Living Environments. PhD thesis, ComputerScience department, Technische Universitaet Muechen, Ger-many, October 2009. 6
[23] R. B. Rusu and S. Cousins. 3d is here: Point cloud library(pcl). In ICRA, Shanghai, China, May 9-13 2011. 5, 6
[24] J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio,R. Moore, A. Kipman, and A. Blake. Real-time human poserecognition in parts from a single depth image. In CVPR,June 2011. 2
[25] P. Viola and M. Jones. Robust real-time face detection. IJCV,57(2):137–154, 2003. 4
[26] C. Wojek, S. Walk, S. Roth, and B. Schiele. Monocular 3dscene understanding with explicit occlusion reasoning. InCVPR, 2011. 2, 6
[27] B. Wu and R. Nevatia. Detection and tracking of multi-ple, partially occluded humans by bayesian combination ofedgelet based part detectors. In IJCV, 2007. 2