globally optimal solution to multi-object tracking with...

Globally Optimal Solution to Multi-Object Tracking with

Merged Measurements

João F. João F. João F. João F. HenriquesHenriquesHenriquesHenriques RuiRuiRuiRui CaseiroCaseiroCaseiroCaseiro Jorge BatistaJorge BatistaJorge BatistaJorge Batista

Institute of Systems and RoboticsUniversity of Coimbra

International Conference on Computer Vision, 2011

Motivation

• Tracking multiple people• Single, Single, Single, Single, uncalibrateduncalibrateduncalibrateduncalibrated camera• Realistic video surveillance scenes

Problem far from solved!Problem far from solved!Problem far from solved!Problem far from solved!

Proposed method:• Handles merged measurements• Global optima in polynomial time

•Tracking as global optimization1. Detect in all frames

2. Optimize all tracks simultaneously (minimize a cost function)

Solid theoretical treatment of groups

⇒

Contributions:Contributions:Contributions:Contributions:

Previous work

Multiple-matching

Target counts(group identification)

Recovering identities

⇒ ⇒

Previous work

Multiple-matching

Target counts(group identification)


⇒ ⇒

C. Huang et al.ECCV’08

Global solutionin

No merges/splitsNo merges/splitsNo merges/splitsNo merges/splitsNo groups Obtained trivially

A. Perera et al.CVPR’06

Local solutionin

(Global in )Limited to groups of 2

Limited togroups of 2

J. Sullivan et al.ECCV/CVPR’06

Local solutionin

No convergence guarantees

Global solutionin

J. F. J. F. J. F. J. F. HenriquesHenriquesHenriquesHenriqueset al., ICCV’11et al., ICCV’11et al., ICCV’11et al., ICCV’11

GlobalGlobalGlobalGlobal solutionsolutionsolutionsolutionin



Outline

1. Global optimization tracking review

2. Proposed method:

3. Conclusion and results

2.1. Multiple-matching

2.2. Target counts(group identification)

2.3. Recovering identities

⇒ ⇒

Outline


2. Proposed method:




⇒ ⇒2.1. Multiple-

matching

Global optimization tracking review

1. Detect and track with a simple method

• Stop trackers when confidence is low

• Output: trackletstrackletstrackletstracklets(short track segments)

2. Each tracklet is a nodenodenodenode in a graph


3. Connect each node to another if it occurs later in time

4. Calculate a costcostcostcost for each of these arcsarcsarcsarcs

Low cost ⇒ same target

(hopefully!)

Use distance, appearance similarity, size difference…

time

c1,2

c1,3node 1 node 2

node 3

• These are just hypothesesjust hypothesesjust hypothesesjust hypotheses.

• Optimization will choose the best set of arcs.


5. Termination nodes –hypothesis of ending a track

6. Initialization nodes –hypothesis of starting a track

7. Costs based on entry/exit locations model

c1init c1

term


• A typical graph:

• Possibly occluded targets –long-range arcs

• Optimization is free to choose between many hypotheses

(Some init./term. nodes omitted for clarity)

init.

init.

term.

term.

Note:

• Long occlusions ⇒ many arcs⇒ mistakes more likely

(merged measurements(merged measurements(merged measurements(merged measurementssolves this)solves this)solves this)solves this)


Optimal matching

• Find set of arcs X with a minimum total cost

• Subject to nononono----overlap restrictionoverlap restrictionoverlap restrictionoverlap restriction: only 1 ingoing and 1 outgoing arc in each node

∑=ji

ijijX

xcX,

* minarg

∀=∀=

∑∑

jx

ixts

i ij

j ij

,1

,1..

Hungarian AlgorithmHungarian AlgorithmHungarian AlgorithmHungarian Algorithm Solution in ( : #nodes)⇒

init.

init.

term.

term.

Merged measurements

• A single node single node single node single node may actually represent several targetsseveral targetsseveral targetsseveral targets.

• Merges and splits violate violate violate violate theno-overlap restriction.

⇓

Can’t use Hungarian Algorithmdirectly!

Somehow…Somehow…Somehow…Somehow…Change graph to support:

All the previous arcs+

Merges/splits hypotheses

(Cannot be solved with:• Min. cost flow• QP• QBP)

mergemergemergemerge splitsplitsplitsplit

Outline


2. Proposed method:





⇒ ⇒

Multiple-matching

node 1

node 2

node 3Don’t know a-priori if node 3 is:

• Same target as node 1• Same target as node 2• Both targets (merged)

Can calculate a cost for each hypothesis:

c13, c23, cmerge

????

Multiple-matching

node 1

node 2

node 3

cccc13131313

Multiple-matching

cccc23232323

node 1

node 2

node 3

Multiple-matching

cccc23232323

node 1

node 2

node 3

cccc13131313

Total cost

= cmerge

Problems:

• Total cost is not linearnot linearnot linearnot linear

• ViolatesViolatesViolatesViolates the no-overlap restriction

≠ c13 + c23

Multiple-matching

node 1

node 2

node 3

Conditions:

• Total cost of merge must be

⇒

Encode a merge as one normal Encode a merge as one normal Encode a merge as one normal Encode a merge as one normal link and one termination.link and one termination.link and one termination.link and one termination.

Key idea:

cccc23232323

cccc13131313

cccctermtermtermterm

????

Multiple-matching

node 1

node 2

node 3

Conditions:

• Total cost of merge must be

⇒

• If is active, must be active

⇒ Disable all other incoming arcs tonode 3

Encode a merge as one normal Encode a merge as one normal Encode a merge as one normal Encode a merge as one normal link and one termination.link and one termination.link and one termination.link and one termination.

Key idea:

cccc23232323

cccc13131313


????

Multiple-matching

node 1

node 2

node 3

cccc13131313

Multiple-matching

node 1

node 2

node 3

cccc23232323

Multiple-matching

node 1

node 2

node 3

cccc13131313


• Total cost= c13 + cterm = cmerge

as required.

• The proposed structure restricts the solutions to onlyonlyonlyonly these 3.

Multiple-matching

• For splits, reverse arc directions.

• The Hungarian Algorithmis free to choosefree to choosefree to choosefree to choose

• merges,• splits,• regular arcs

that better explain data.

• Same computational complexity Same computational complexity Same computational complexity Same computational complexity as simple matching.

Output

Outline


2. Proposed method:





⇒ ⇒

Groups

• Targets travelling alone can be tracked unambiguously.

But:

• Individual identities are lost in groups.

TargetTargetTargetTarget #1#1#1#1



TargetTargetTargetTarget ????????



1. Identify nodes that correspond to groups.

2. Match individual targets across the groups.

Strategy:

Target counts as flow circulation

• To robustly identify groups,count targets count targets count targets count targets in each node.

• Solve min-cost circulation:

Solved in polynomial time with the Edmonds-Karp algorithm.

Minimize number of targets

Flow conservation

Boundary conditions(at least 1 target per node)

Target counts as flow circulation

• Costs inversely proportional tobox size(larger boxes ⇒ larger groups)

• Counts obtained solely fromsolely fromsolely fromsolely fromthe graph topologythe graph topologythe graph topologythe graph topology.

• Equivalent to locating occluded targets.

Mostly as a tie-breaker

Outline


2. Proposed method:





⇒ ⇒


Groups subgraph

={arcs and nodes with flow > 1}

Entering a group

={arcs pointing to }

Exiting a group

={arcs pointing from }

To track across groups, match targets entering to targets exiting.

Simple matching problem⇒


• Match across groups:

1. Connect a node in entering to a node in exiting if it’s reachable through .

2. Calculate costs as usual.

3. Optimal matching with Hungarian Algorithm.


• Unlike other methods, wedon’t link across arbitrary gapsdon’t link across arbitrary gapsdon’t link across arbitrary gapsdon’t link across arbitrary gaps.

• Target’s trajectory is restrictedrestrictedrestrictedrestricted to group’s trajectory.

• Only admits physically plausiblephysically plausiblephysically plausiblephysically plausibletracks for occlusions.

Outline


2. Proposed method:




⇒ ⇒2.1. Multiple-

matching

Results

Detection method:foreground segmentation

• Simple to implement

• Frequent merges and splitsFrequent merges and splitsFrequent merges and splitsFrequent merges and splits⇒ Stress-test the framework

PETS 2006 PETS 2006 PETS 2006 PETS 2006 (single view)Good results, challenging videos.No other MOTA scores to compare.

PETS 2009 PETS 2009 PETS 2009 PETS 2009 (single view)Outperforms all 12 all 12 all 12 all 12 state-of-the-art systems at PETS 2010.

Results

Conclusion

Formalized group formationFormalized group formationFormalized group formationFormalized group formationas a global optimization problem.

PolynomialPolynomialPolynomialPolynomial----time algorithms time algorithms time algorithms time algorithms for:

• Multi-match problem.• Counting problem.• Identity match across groups.

Improves state-of-the-artwith simple detection and low-level stages.

Questions?

globally optimal solution to multi-object tracking with...

Documents