multisensor-based human detection and tracking for smcb...

Download Multisensor-Based Human Detection and Tracking for SMCB  ·…

Post on 12-Jul-2018




0 download

Embed Size (px)



    Multisensor-Based Human Detection andTracking for Mobile Service Robots

    Nicola Bellotto, Student Member, IEEE, and Huosheng Hu, Senior Member, IEEE

    AbstractOne of fundamental issues for service robots ishumanrobot interaction. In order to perform such a task andprovide the desired services, these robots need to detect and trackpeople in the surroundings. In this paper, we propose a solution forhuman tracking with a mobile robot that implements multisensordata fusion techniques. The system utilizes a new algorithm forlaser-based leg detection using the onboard laser range finder(LRF). The approach is based on the recognition of typical legpatterns extracted from laser scans, which are shown to alsobe very discriminative in cluttered environments. These patternscan be used to localize both static and walking persons, evenwhen the robot moves. Furthermore, faces are detected using therobots camera, and the information is fused to the legs positionusing a sequential implementation of unscented Kalman filter.The proposed solution is feasible for service robots with a similardevice configuration and has been successfully implemented ontwo different mobile platforms. Several experiments illustrate theeffectiveness of our approach, showing that robust human track-ing can be performed within complex indoor environments.

    Index TermsLeg detection, people tracking, sensor fusion,service robotics, unscented Kalman filter (UKF).


    IN RECENT years, the social aspect of service robots hasmade it clear that these are not only expected to navigatewithin the environment they have been placed in but they alsohave to interact with people to provide useful services and showgood communication skills. The study of the so-called human-centered robotics and humanrobot interaction aims to achievesuch tasks and are currently some of the most fascinatingresearch fields in mobile robotics. In general, a service robothas to focus its attention on humans and be aware of theirpresence. It is necessary, therefore, to have a tracking systemthat returns the current position, with respect to the robot, ofthe adjacent persons. This is a very challenging task, as peoplesbehaviors are often completely unpredictable. Researchers havebeen using different methods to deal with this problem, in manycases, with solutions subjected to strong limitations, such astracking in rather simple situations with a static robot or usingsome additional distributed sensors in the environment.

    Manuscript received January 12, 2008; revised May 1, 2008 and July 17,2008. First published December 9, 2008; current version published January 15,2009. This paper was recommended by Associate Editor J. Su.

    This paper has supplementary downloadable material available at, provided by the authors. This includes five video filesshowing results of the implemented system. This material is 36.2 MB in size.

    The authors are with the Human Centred Robotics Group, Department ofComputing and Electronic Systems, University of Essex, CO4 3SQ Colchester,U.K. (e-mail:;

    Color versions of one or more of the figures in this paper are available onlineat

    Digital Object Identifier 10.1109/TSMCB.2008.2004050

    Service robotics has become one of the research areas at-tracting major attention for over a decade. The necessity of fastand reliable systems for tracking people with mobile robotsis evidenced in the literature by the growing number of real-world applications. Human tracking can help service robotsplan and adapt their movements according to the motion ofthe adjacent people or follow an instructor across differentareas of a building. For example, the tour-guide robot ofBurgard et al. [1] adopts laser-based people tracking both forinteracting with users and for mapping the environment, dis-carding human occlusions. Another field of application is auto-matic or remote surveillance with security robots, which canbe used to monitor wide areas of interest that are otherwisedifficult to cover with fixed sensors. An example is the systemimplemented by Liu et al. [2], where a mobile robot trackspossible intruders in a restricted area and signals their presenceto the security personnel.

    The most common devices used for people tracking are lasersensors and cameras. For instance, Lindstrm and Eklundh [3]propose a laser-based approach to track a walking person with amobile robot. The system detects only moving objects, keepingtrack of them with a heuristic algorithm, and needs the robot tobe static or move very slowly. Zajdel et al. [4] illustrate a vision-based tracking and identification system which makes use ofa dynamic Bayesian network for handling multiple targets.Even in this case, targets can be detected only when moving.Moreover, the working range is limited by the cameras angleof view; hence, it is difficult to track more than two subjectsat the same time. The robot described by Luo et al. [5] usesa tilting laser to extract body features, which are fused thenwith the face detected by a camera. The solution is useful forpursuing a person in front of the robot; however, the complexityof feature extraction limits its application to multiple peopletracking. Other implementations also use a combination offace detection and laser-based leg detection [6], [7]. In allthese cases, however, since they do not rely on any motionmodel, situations in which a person is temporarily occluded aredifficult to handle. Schulz et al. [8] describe a robot equippedwith two laser range sensors that can track several people,using a combination of particle filters and probabilistic dataassociation. Many other recent approaches make also use ofparticle filters, sometimes combining visual and laser data [9] totrack from a fixed position or using other kind of devices suchas thermal cameras [10].

    The solution presented in this paper adopts multisensor datafusion techniques for tracking people from a mobile robot usinga laser scanner and a monocular camera. A new detectionalgorithm has been implemented to find human legs by usinglaser scans, which works either in large empty environments

    1083-4419/$25.00 2008 IEEE

    Authorized licensed use limited to: IEEE Xplore. Downloaded on January 17, 2009 at 11:14 from IEEE Xplore. Restrictions apply.


    Fig. 1. Pioneer robot with laser and camera for legs and face detection.

    or small cluttered rooms. Different from other approaches, oursystem is also able to distinguish among different leg postures,improving the discrimination of false positives. Vision is thenused for face detection, and robust human tracking is performedwith a sequential unscented Kalman filter (UKF) fusing thetwo different sensor data. The solution is feasible for severalapplications of mobile service robots with a similar deviceconfiguration, although the computational efficiency makes itparticularly indicated for robots with limited processing power.The system has been tested in cluttered indoor environments,where human detection is made difficult by the presence offurniture or by the small size of a room, and proved to be robustenough to track people constantly even while the robot movesat a normal walking speed.

    This paper is organized as follows. Section II explains, indetail, the algorithm for leg detection and also introduces theface-detection module. The tracking system, including UKF,sensor fusion, and data association, is described in Section III.Then, Section IV presents several experiments and analyzes theresults. Finally, conclusions and future work are illustrated inSection V.


    The human-tracking algorithm adopts multisensor data fu-sion techniques to integrate the following two different sourcesof information: the first one is leg detection, based on thelaser scans of a Sick LRF, and the other one is face detection,which uses a monocular camera. The devices and their locationare shown in Fig. 1 for one of our mobile robots. Next, wedescribe, in detail, the principles underlying these two detectionalgorithms.

    A. Leg Detection

    In the literature, there are several systems using laser scans todetect human legs. However, most of them are simply based onthe search of local minima [6], [11], which, in general, workswell only for rather simple environments such as empty rooms

    Fig. 2. Leg patterns extracted from a laser scan. Most of the time,(left) LA and (center) FS patterns can be easily distinguished from otherobjects; however, (right) SL patterns can be very ambiguous in someenvironments.

    and corridors. In [12], it is reported that at least a single leg(SL) must always be well distinguishable, whereas attemptsof using more general solutions showed to be not very robustin cluttered environments [13]. Many other approaches canonly detect moving persons [3], [9], normally searching fordifferences between two or more scans. Aside from the problemof missing static humans, these methods are often unreliable formobile robots due to the difficulty of compensating for the egomotion.

    The algorithm for leg detection presented here extracts thenecessary features from a single laser scan, independently thenfrom the human or robot motion. However, in contrast with thelocal minima approach, we identify typical patterns (relative toparticular leg postures) that, in most of the cases, are well dis-tinguishable from the other objects in the environment. Thesepatterns are shown with an example in Fig. 2 and correspondto the following three typical situations: two legs apart (LA),forward straddle (FS), and two legs together or SL


View more >