IEEE International Conference on Multimedia & Expo, Toronto, July 2006 1

Coarse-to-Fine Pedestrian Localization and Silhouette

Extraction for the Gait Challenge Data Sets

Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos

The Edward S. Rogers Sr.Department of Electrical and Computer Engineering

University of Toronto

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 2

Motivation

The HumanID gait challenge data sets • Semi-automatic extraction

• Assuming paths are smooth to 2nd degreeBackground subtraction algorithms

• Pixel-based processing: not robust

• MRF-based processing: exploiting spatial & temporal dependencies but slow

Objective: fast & robust

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 3

Overview

Regional temporal difference

Gait video input

Pedestrian candidate detection

Detection position/size refinement

Localized background subtraction

Background model update

Detection post-processing

Pedestrian location and silhouettes

Fine detectionCoarse detection

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 4

Silhouette extraction difficulties

Heavy shadow Other subjects in the scene

Slow motion (eight successive frames)

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 5

Coarse detection

Coarse region Rc: centered at previous Bf

Gray map M1: maximum pixel difference

Foreground pixels F1: threshold Td

Spatial distribution examination:

foreground F2 and binary map M2

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 6

Coarse detection

Bounding box: centered at

Validation and variation control: Bc

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 7

Fine detection

BSMT on Rf (centered at Bc): Sr

Background update (GMM)

Fine detection Bf: projections of Sr and cluster analysis

Rfc : Rf centered horizontally

Final silhouette Sf: Sr within Rfc

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 8

Initial detection

No coarse detection

Whole frame as Rf

Confidence on localization: number of

foreground pixels in Bf exceeds 50

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 9

Experiments

285 sequences: gallery & probes B, D, H and K of USF gait challenge data sets

630 frames per sequence on average

Frame: color image of size 480x720

Parameters determined experimentally

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 10

Pedestrian localization results

50 frames to gain confidence on localization

0.07% of 165,749 frames in error (foreground pixel number & dislocation)

Errors: lower portion cut or missing & incomplete silhouettes

Useful for further processing: e.g. LDM

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 11

Silhouette extraction results

Evaluation: resemblance between extracted silhouettes & manual silhouettes (10005 frames available)

Metric: ratio of intersection to union

Consistently better than USF semi-auto extraction & MIT silhouette refinement

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 12

Performance comparison

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 13

Conclusions

Attack difficulties in gait recognition due to slow motion, heavy shadow, other moving subjects or objects

Coarse detection: locate subject Fine detection: robust and accurate

detection Future work: using models to help

silhouette extraction

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 14

Related work

Haiping Lu, K.N. Plataniotis and A.N. Venetsanopoulos, "A Layered Deformable Model for Gait Analysis", in Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition (FGR 2006), Southampton, UK, April 2006.

IEEE International Conference on Multimedia & Expo, Toronto, July 2006 15

Contact Information

Haiping Lu

Email: haiping@dsp.toronto.edu

Academic website:

http://www.dsp.toronto.edu/~haiping/