Multi-Group Tracking with Adaptive Target

Model

Loris Bazzani, PhD student (XXIV cycle)University of Verona

Department of Computer Science

PhD thesis proposal

2

ObjectivesCompare existing Multi-Target Tracking methods, studying the sampling technique

Propose a new tracking method: Multi-Group Tracking

Model robustly and adaptively the target

Integrate target model with Multi-Group tracking

3

Introduction

Multi-Target Tracking

Multi-Group Tracking

Target Modeling

Conclusions

Outline

State of the art

Open issues

Proposed ideas

Introduction (1)Tracking: spatial and temporal localization of a mobile object in an environment monitored by sensor(s)

Multi-target (MTT): keeping the identity of different targets

Reliable: insensible to noise and occlusions

Application to Automated Surveillance

5

Introduction (2)Multi-Group Tracking (MGT):

Spatial and temporal localization of groups of objects

Motivations:

Humans prefer to stay in group rather than alone

High-level representation of the relations among the targets

MGT is simpler than MTT in a crowded scenario

MGT can help MTT when occlusions occur

6

Introduction (3)Multi-Group Tracking (MGT): why is MGT a hard task?

7

Introduction (4)Target Model:

A general and representative example that summarizes any possible changing of the target

intrinsic variations: pose variation and shape deformation

extrinsic variations: illumination changes, camera movement, and occlusions

Not considering the above variation causes the failure of the tracking [Ross08]

Fit with the re-identification problem

8

Multi-Target Tracking (1)

Filtering

We observe the real world events as a state by the measurement process

Objective: estimating the state of the system at each instant given the measurements

Abstract Formulation [Arulampalam02]

State Space Approach for modeling discrete-time dynamic systems

State: abstract nature of the target

Measurement: “visible” dimensions of the state space

9

Multi-Target Tracking (2)Data Association [Bar-Shalom87]

The observer has at his disposal a huge amount of measurements

Finding the correct correspondences between measurements and states of the system

10

Multi-Target Tracking (3)

11

Multi-Target Tracking (4)Particle Filter (PF) [Isard01]

Multi-Target Tracking (5)State Space Conformation

(+) Efficient sampling(-) No interaction modeling

(+) Implicit interaction modeling(-) Curse of dimensionality

(+) Efficient sampling(+) Implicit interaction modeling

[Isard01]

[MacCormick00]

[Lanz06]

Multi-Target Tracking (6)- HJS vs. MHT [Bazzani09] -

MHT cons: 1) Multiple tracks cause proliferation in the number of tracks2) Occlusions generate new tracks3) Not robust to non-linear people motion

HJS pros: 1) One track is kept for each target2) Partial occlusions are handled; 3) Deal with non-linearity of people motion.

14

Multi-Target Tracking (7)

Open issues of PF-based MTT (and MGT):

Sampling Method

Dynamic Model

Linear-Gaussian model

Observation Model

State Estimation

Maximum-A-Posteriori or Weighted Mean

Multi-Target Tracking (8.1)

Sequential importance sampling/re-sampling [Arulampalam02]: classical PF + degeneracy problem avoiding by re-sampling

Regularized PF [Arulampalam02]: resamples applying a Kernel to the particles

MCMC [Andrieu03]: defines a Markov chain over the state space, such that the stationary distribution of the chain is equal to the sought posterior

Reversible-Jump MCMC [Khan05]: switches between variable dimensional state spaces

Rao-Blackwellizing PF [Schindler05]: analytically computes a portion of the distribution other the state space

Sampling

17

Multi-Target Tracking (8.2)

Likelihood: compare an observation z given a hypothesis of state of the system x

Usually defined in the Gibbs form: where d is a metric

x and z MUST be represented in the same feature space

ISSUES:

feature space (for x and z) and metric definitions

occlusion handling

Observation model

18

Multi-Target Tracking (9)

Occlusion Handling

Study and implement RJ-MCMC particle filter

Propose a set of jumps in order to cope with tracking a variable number of objects

Propose an observation model in RJ-MCMC framework

- Proposed Research -

19

Multi-Group Tracking (1)

defines a group as the moving regions, extracted from a foreground analysis

infers the MGT from the tracks estimation carried out by the MOT (e.g. tracks clustering)

uses the foreground information and MOT to detect groups, but then tracks them as different entities

(-) Social interactions cannot be caught by the foreground analysis(-) The inter-group dynamic yields a loss of appearance informations

(-) Direct dependence from MOT (-) The MOT estimation is not reliable when occlusions occur

(+) Cancel out the above problems(-) Model creation is a hard task

20

Multi-Group Tracking (2)Foreground-based MGT

Tracking at three levels of abstraction [McKenna00] :

Regions: connected component that have been tracked for T frames

People: one or more regions grouped together

Groups: one or more people grouped together, if they share a region

(+) Simplicity(-) heuristic FG

analysis

22

Multi-Group Tracking (3)MGT from MOT

MCMC PF for group tracking [Pang07]:

Track groups as ensemble of targets analyzing

: group variable

Treated as Bayesian estimation problem

Group structure model:

captures the relations among objects

23

Multi-Group Tracking (4)- Proposed Research -

Problems:

Definition of “Group”: an entity containing targets with similar characteristics (e.g. motion, interactions, ...)

Deterministic/formal definition as an ensemble of objects

Add non-deterministic component into the tracking method

Intra-group occlusions: if we know that the objects hasn’t left the group, we infer that it is still into the group

Inter-group occlusions: tracking of groups

24

Multi-Group Tracking (5)- Proposed Research -

25

Multi-Group Tracking (6)- Proposed Research -

Use a MOT method

Create a MGT method (track only groups)

Definition of group dynamics -> sociological studies

Definition of a group observation model

Define a collaborative probabilistic framework in order to share MGT and MOT informations

27

Target Modeling (1)

Train the model using the appearance data available before tracking begins

Adapt the model to account for its changes in appearance, using an on-line learning method

Open Issues:

Representation of the target: feature space

Leaning technique

28

Target Modeling (2)Feature

Color Histogram: which is the best color space for tracking? [Sebastian08]

(-) No spatial information

Color correlogram [Huang99], spatiogram [Birchfield05], multi-resolution histogram [Hadjidemetriou01]

(+) Add the spatial information

(-) Increase the computational burden

29

Target Modeling (3)Feature

Covariance descriptors [Porikli06]

Spatial and appearance attributes

(+) Natural way of fusing multiple features

(-) Computationally expensive -> integral images

30

Target Modeling (4)

Ensemble of localized features [Gray08]

Define a feature space and let machine learning approach find the best representation

AdaBoost extracts

the object representation: the most discriminative set of features, and

the similarity measures: the most discriminative set of likelihood ratio test

Used for re-identification problem

Fixed target models

31

Target Modeling (5)Adaptive target models

Incremental learning of Covariance-based descriptors [Porikli06]

Principal Component Analysis (PCA) incremental learning [Ross08]

Convex combination of models using a learning rate

Feature-based model

Parametric model

32

Target Modeling (6)Patch-based (local) updating [Kwon09]

Evolve photometric and geometric appearance

Local Patches can be added, deleted or moved to different position

Examining the patches by landscape analysisBad patches are modified on-line: background patches and patches in regions with high density of patch are deleted

Good patches are moved

Appearance model is updated with a convex combination

33

Target Modeling (7)- Proposed Research -

Part-based multiple-features:

Maximally Stable Color Regions

HS(V) histograms

Recurrent high-structured patches

Temporal updating:

Delete the non-stable features

Cover the Variability of the stable features

GlobalGlobal

LocalLocal

view point, partial view point, partial occlusionsocclusions

illumination, view illumination, view point, deformations, point, deformations,

partial occlusionspartial occlusions

partial occlusions, partial occlusions, illuminationillumination

Invariances:

34

ConclusionsCompare existing Multi-Target Tracking methods, studying the sampling technique

Propose a new tracking method: Multi-Group Tracking

Model robustly and adaptively the target

Integrate target model with Multi-Group tracking, using HJS and RJ-MCMC

35

The beginning

Thanks for attentionThanks for attention

Now, we “just” put the proposed ideas into practice

Questions?Questions?

36

References[Ross08] D.A. Ross, J. Lim, R.S. Lin, and M.H. Yang. Incremental learning for robust visual tracking. International Journal of Computer Vision, 77(1):125–141, 2008.

[Arulampalam02] M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Trans. on Signal Processing, 50(2):174–188, 2002.

[Bar-Shalom87] Y. Bar-Shalom. Tracking and data association. Academic Press Professional, Inc., San Diego, CA, USA, 1987.

[Isard01] M. Isard and J. MacCormick. Bramble: A bayesian multipleblob tracker. In IEEE Int. Conf. on Computer Vision, 2001.

[MacCormick00] John MacCormick and Andrew Blake. A probabilistic exclusion principle for tracking multiple objects. Int. J. Comput. Vision, 39(1):57–71, 2000.

[Lanz06] O. Lanz. Approximate bayesian multibody tracking. IEEE Trans. on Pattern Analysis and Machine Intelligence, 28(9):1436–1449, 2006.

[Andrieu03] Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. An introduction to mcmc for machine learning. Machine Learning, 50(1):5–43, January 2003.

37

References[Khan05] Z. Khan, T. Balch, and F. Dellaert. Mcmc-based particle filtering for tracking a variable number of interacting targets. IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(11):1805–1819, 2005.

[Schindler05] G. Schindler and F. Dellaert. A Rao-Blackwellized Parts-Constellation Tracker. In ICCV Workshop on Dynamical Vision; International Conference on Computer Vision, 2005. Springer

[Mckenna00] Stephen J. Mckenna, Sumer Jabri, Zoran Duric, Harry Wechsler, and Azriel Rosenfeld. Tracking groups of people. Computer Vision and Image Understanding, 2000.

[Pang07] Sze Kim Pang, Jack Li, and Simon Godsill. Models and algorithms for detection and tracking of coordinated groups. In Symposium of image and Signal Processing and Analisys, 2007.

[Sebastian08] Sebastian, P.; Yap Vooi Voon; Comley, R., "The effect of colour space on tracking robustness," Industrial Electronics and Applications, 2008. ICIEA 2008. 3rd IEEE Conference on , vol., no., pp.2512-2516, 3-5 June 2008

[Huang99] J. Huang, S. Ravi Kumar, M. Mitra, W.J. Zhu, and R. Zabih. Spatial color indexing and applications. International Journal of Computer Vision, 35(3):245–268, 1999.

[Birchfield05] ST Birchfield and S. Rangarajan. Spatiograms versus histograms for region-based tracking. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005.CVPR 2005, volume 2, 2005.

38

References

[Hadjidemetriou01] E. Hadjidemetriou, MD Grossberg, and SK Nayar. Spatial information in multiresolution histograms. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, volume 1, 2001.

[Porikli06] F. Porikli, O. Tuzel, and P. Meer. Covariance tracking using model update based on lie algebra. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, 2006.

[Gray08] D. Gray and H. Tao. Viewpoint Invariant Pedestrian Recognition with an Ensemble of Localized Features. In Proceedings of the 10th European Conference on Computer Vision: Part I, pages 262–275. Springer, 2008.

[Kwon09] J.S. Kwon and K.M. Lee. Tracking of a non-rigid object via patch-based dynamic appearance modeling and adaptive basin hopping monte carlo sampling. pages 1208–1215, 2009.