simultaneous segmentation and 3d pose estimation of humans philip h.s. torr pawan kumar, pushmeet...

Simultaneous Segmentation and 3D Pose Estimation of Humans

Philip H.S. TorrPawan Kumar, Pushmeet Kohli, Matt Bray

Oxford Brookes University

Arasanathan Thayananthan, Bjorn Stenger, Roberto CipollaCambridge University

Algebra

Unifying Conjecture

Tracking = Detection = Recognition Detection = Segmentation

• therefore Tracking (pose estimation)=Segmentation?

Objective

Image Segmentation Pose Estimate??

Aim to get a clean segmentation of a human…

Developments

ICCV 2003, pose estimation as fast nearest neighbour plus dynamics (inspired by Gavrilla and Toyoma & Blake)

BMVC 2004, parts based chamfer to make space of templates more flexible (a la pictorial structures of Huttenlocher)

CVPR 2005, ObjCut combining segmentation and detection.

ICCV 2005 Dynamic Graph Cuts ECCV 2006, interpolation of poses using the MVRVM

(Agarwal and Triggs) ECCV 2006 combination of pose estimation and

segmentation using graph cuts.

Tracking as Detection (Stenger et al ICCV 2003)

Detection has become very efficient,e.g. real-time face detection, pedestrian detection

Example: Pedestrian detection [Gavrila & Philomin, 1999]: Find match among large number of exemplar templates

Issues: Number of templates needed Efficient search Robust cost function

Cascaded Classifiers

First filter : 19.8 % patches remaining

1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7

Filter 10 : 0.74 % patches remaining

1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7

Filter 20 : 0.06 % patches remaining

1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7

Filter 30 : 0.01 % patches remaining

1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7

Filter 70 : 0.007 % patches remaining

1280x1024 image, 11 subsampling levels, 80sAverage number of filter per patch : 6.7

Hierarchical Detection Efficient template matching (Huttenlocher & Olson,

Gavrila) Idea: When matching similar objects, speed-up by

forming template hierarchy found by clustering Match prototypes first, sub-tree only if cost below

threshold

Trees

These search trees are the same as used for efficient nearest neighbour.

Add dynamic model and • Detection = Tracking = Recognition

Evaluation at Multiple Resolutions

One traversal of tree per time step

Evaluation at Multiple Resolutions

Tree: 9000 templates of hand pointing, rigid

Templates at Level 1

Templates at Level 2

Templates at Level 3

Comparison with Particle Filters

This method is grid based,• No need to render the model on line• Like efficient search• Can always use this as a proposal process for

a particle filter if need be.

Interpolation, MVRVM, ECCV 2006

Code available.

Energy being Optimized, link to graph cuts

Combination of• Edge term (quickly evaluated using chamfer)• Interior term (quickly evaluated using integral

images)

Note that possible templates are a bit like cuts that we put down, one could think of this whole process as a constrained search for the best graph cut.

Likelihood : Edges

Edge Detection Projected Contours

Robust EdgeMatching

Input Image 3D Model

Chamfer MatchingInput image Canny edges

Distance transform Projected Contours

Likelihood : Colour

Skin Colour ModelProjected Silhouette

Input Image 3D Model

Template Matching

Template Matching =

Template Matching = constrained search for a cut/segmentation?

Detection = Segmentation?

Objective

Image Segmentation Pose Estimate??

Aim to get a clean segmentation of a human…

MRF for Interactive Image Segmentation, Boykov and Jolly [ICCV 2001]

EnergyMR

F

Pair-wise Terms MAP SolutionUnary likelihoodData (D)

Unary likelihood Contrast Term Uniform Prior(Potts Model)

Maximum-a-posteriori (MAP) solution x* = arg min E(x)x

=

However…

This energy formulation rarely provides realistic (target-

like) results.

ObjCut (yesterday)

Unary potential

Pairwise potential

Pose parameters

Labels

Pixels

Prior Potts model

Pose-specific MRF

Layer 2

Layer 1

Transformations

Θ1

P(Θ1) = 0.9

Cow Instance

Do we really need accurate models?

Do we really need accurate models?

Segmentation boundary can be extracted from edges

Rough 3D Shape-prior enough for region disambiguation

Energy of the Pose-specific MRFEnergy to be

minimizedUnary term

Shape prior

Pairwise potential

Potts model

But what should be the value of θ?

The different terms of the MRF

Original image

Likelihood of being foreground given a

foreground histogram

Grimson-Stauffer

segmentation

Shape prior model

Shape prior (distance transform)

Likelihood of being foreground

given all the terms

Resulting Graph-Cuts

segmentation

Can segment multiple views simultaneously

Solve via gradient descent

Comparable to level set methods

Could use other approaches (e.g. Objcut)

Need a graph cut per function evaluation

Formulating the Pose Inference Problem

But…But…

… to compute the MAP of E(x) w.r.t the pose, it means that the unary terms will be changed at EACHEACH iteration and the maxflow recomputed!

However…However… Kohli and Torr showed how dynamic graph cuts can

be used to efficiently find MAP solutions for MRFs that change minimally from one time instant to the next: Dynamic Graph Cuts (ICCV05).

Dynamic Graph Cuts

PB SB

cheaperoperation

computationally

expensive operation

Simplerproblem

PB*

differencesbetweenA and B

A and Bsimilar

PA SA

solve

Dynamic Image Segmentation

Image

Flows in n-edges Segmentation Obtained

First segmentation problem MAP solution

Ga

Our Algorithm

Gb

second segmentation problem

Maximum flow

residual graph (Gr)

G`

differencebetween

Ga and Gbupdated residual

graph

Energy Minimization using Graph cuts

EMRF(a1,a2)

Sink (1)

Source (0)

a1 a2

Graph Construction for Binary Random Variables

Energy Minimization using Graph cuts

Sink (1)

Source (0)

a1 a2

EMRF(a1,a2) = 2a1

2t-edges

(unary terms)

Energy Minimization using Graph cuts

EMRF(a1,a2) = 2a1 + 5ā1

Sink (1)

Source (0)

a1 a2

2

5

Energy Minimization using Graph cuts

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2

Sink (1)

Source (0)

a1 a2

2

5

9

4

Energy Minimization using Graph cuts

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2

Sink (1)

Source (0)

a1 a2

2

5

9

4

2

n-edges(pair-wise term)

Energy Minimization using Graph cuts

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Sink (1)

Source (0)

a1 a2

2

5

9

4

2

1

Energy Minimization using Graph cuts

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Sink (1)

Source (0)

a1 a2

2

5

9

4

2

1

Energy Minimization using Graph cuts

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Sink (1)

Source (0)

a1 a2

2

5

9

4

2

1

a1 = 1 a2 = 1

EMRF(1,1) = 11

Cost of st-cut = 11

Energy Minimization using Graph cuts

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Sink (1)

Source (0)

a1 a2

2

5

9

4

2

1

a1 = 1 a2 = 0

EMRF(1,0) = 8

Cost of st-cut = 8

Energy Minimization using Graph cuts

• Most probable (MAP) configuration ≡ minimum cost st-cut.

• st-mincut is in general a NP-hard problem - negative edge weights

• Solvable in polynomial time- non-negative edge weights- corresponds to sub-modular (regular) energy functions

• The Max-flow Problem- Edge capacity and flow balance constraints

Computing the st-mincut from Max-flow algorithms

• Notation- Residual capacity (edge capacity – current flow)- Augmenting path

• Simple Augmenting Path based Algorithms- Repeatedly find augmenting paths and push flow.- Saturated edges constitute the st-mincut. [Ford-Fulkerson Theorem]

Sink (1)

Source (0)

a1 a2

2

5

9

42

1

9 + α

4 + α

Adding a constant to both thet-edges of a node does not change the edges constituting the st-mincut.

Key Observation

Sink (1)

Source (0)

a1 a2

2

5

2

1

E (a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

E*(a1,a2 ) = E(a1,a2) + α(a2+ā2)

= E(a1,a2) + α [a2+ā2 =1]

Reparametrization

9 + α

4

All reparametrizations of the graph are sums of these two types.

Other type of reparametrization

Sink (1)

Source (0)

a1 a2

2

5 + α

2 + α

1 - α

Reparametrization, second type

Both maintain the solution and add a constant α to the energy.

s

Gt

original graph

0/9

0/7

0/5

0/2 0/4

0/1

xi xj

flow/residual capacity

Graph Re-parameterization

s

Gt

original graph

0/9

0/7

0/5

0/2 0/4

0/1

xi xj

flow/residual capacity

Graph Re-parameterization

t residual graph

xi xj0/12

5/2

3/2

1/0

2/0 4/0st-mincut

ComputeMaxflow

Gr

Edges cut

Update t-edge Capacities

s

Gr

t residual graph

xi xj0/12

5/2

3/2

1/0

2/0 4/0

Update t-edge Capacities

s

Gr

t residual graph

xi xj0/12

5/2

3/2

1/0

2/0 4/0

capacitychanges from

7 to 4

Update t-edge Capacities

s

G`t

updated residual graph

xi xj0/12

5/-1

3/2

1/0

2/0 4/0

capacitychanges from

7 to 4

edge capacityconstraint violated!(flow > capacity)

= 5 – 4 = 1

excess flow (e) = flow – new capacity

add e to both t-edges connected to node i

Update t-edge Capacities

s

G`t

updated residual graph

xi xj0/12

3/2

1/0

2/0 4/0

capacitychanges from

7 to 4

edge capacityconstraint violated!(flow > capacity)

= 5 – 4 = 1

excess flow (e) = flow – new capacity

5/-1

Update t-edge Capacities

s

G`t

updated residual graph

xi xj0/12

3/2

1/0

4/0

capacitychanges from

7 to 4

excess flow (e) = flow – new capacity

add e to both t-edges connected to node i

= 5 – 4 = 1

5/0

2/1

edge capacityconstraint violated!(flow > capacity)

Update n-edge Capacities

s

Gr

t

residual graph

xi xj0/12

5/2

3/2

1/0

2/0 4/0

• Capacity changes from 5 to 2

Update n-edge Capacities

s

t

Updated residual graph

xi xj0/12

5/2

3/-1

1/0

2/0 4/0

G`

• Capacity changes from 5 to 2- edge capacity constraint violated!

Update n-edge Capacities

s

t

Updated residual graph

xi xj0/12

5/2

3/-1

1/0

2/0 4/0

G`

• Capacity changes from 5 to 2- edge capacity constraint violated!

• Reduce flow to satisfy constraint

Update n-edge Capacities

s

t

Updated residual graph

xi xj0/11

5/2

2/0

1/0

2/0 4/0

excess

deficiency

G`

• Capacity changes from 5 to 2- edge capacity constraint violated!

• Reduce flow to satisfy constraint- causes flow imbalance!

Update n-edge Capacities

s

t

Updated residual graph

xi xj0/11

5/2

2/0

1/0

2/0 4/0

deficiency

excess

G`

• Capacity changes from 5 to 2- edge capacity constraint violated!

• Reduce flow to satisfy constraint- causes flow imbalance!

• Push excess flow to/from the terminals

• Create capacity by adding α = excess to both t-edges.

Update n-edge Capacities

Updated residual graph

• Capacity changes from 5 to 2- edge capacity constraint violated!

• Reduce flow to satisfy constraint- causes flow imbalance!

• Push excess flow to the terminals

• Create capacity by adding α = excess to both t-edges.

G`

xi xj0/11

5/3

2/0

2/0

3/0 4/1

s

t

Update n-edge Capacities

Updated residual graph

• Capacity changes from 5 to 2- edge capacity constraint violated!

• Reduce flow to satisfy constraint- causes flow imbalance!

• Push excess flow to the terminals

• Create capacity by adding α = excess to both t-edges.

G`

xi xj0/11

5/3

2/0

2/0

3/0 4/1

s

t

Complexity analysis of MRF Update Operations

MRF Energy Operation

Graph Operation Complexity

modifying a unary term

modifying a pair-wise term

adding a latent variable

delete a latent variable

Updating a t-edge capacity

Updating a n-edge capacity

adding a node

set the capacities of all edges of a node zero

O(1)

O(1)

O(1)

O(k)*

*requires k edge update operations where k is degree of the node

• Finding augmenting paths is time consuming.

• Dual-tree maxflow algorithm [Boykov & Kolmogorov PAMI 2004]- Reuses search trees after each augmentation.- Empirically shown to be substantially faster.

• Our Idea • Reuse search trees from previous graph cut computation• Saves us search tree creation tree time [O(#edges)]• Search trees have to be modified to make them consistent with

new graphs• Constrain the search of augmenting paths

– New paths must contain at least one updated edge

Improving the Algorithm

Reusing Search Trees

c’ = measure of change in the energy

• Running time– Dynamic algorithm (c’ + re-create search tree )– Improved dynamic algorithm (c’)– Video Segmentation Example

- Duplicate image frames (No time is needed)

Dynamic Graph Cut vs Active Cuts

Our method flow recycling

AC cut recycling

Both methods: Tree recycling

• Compared results with the best static algorithm.- Dual-tree algorithm [Boykov & Kolmogorov PAMI 2004]

• Applications- Interactive Image Segmentation- Image Segmentation in Videos

Experimental Analysis

Experimental Analysis

additional segmentation

cues

user segmentation cuesstatic: 175 msecdynamic : 80 msecdynamic (optimized): 15 msec

static : 175 msec

Interactive Image segmentation (update unary terms)

EnergyMRF =

Experimental Analysis

Image resolution: 720x576 static: 220 msec dynamic (optimized): 50 msec

Image segmentation in videos (unary & pairwise terms)

Graph CutsDynamic Graph Cuts

EnergyMRF =

Experimental Analysis

Image resolution: 720x576 static: 177 msec dynamic (optimized): 60 msec

Image segmentation in videos (unary & pairwise terms)

Graph CutsDynamic Graph Cuts

EnergyMRF =

Experimental Analysis

MRF consisting of 2x105 latent variables connected in a 4-neighborhood.

Running time of the dynamic algorithm

Other uses

Can be used to compute uncertainty in graph cuts via max marginals.

Max marginals can be used for parameter learning in MRF’s.

Inference in Graphical Models

Graphical Model

Topology

Tree Graph with cycles

Belief Propagation and variants

Exact solution

True Marginals/ min-marginals

Approximate solution

Approximate Marginals/ min-marginals

Graph Cuts

No Marginals/

Min-Marginals

Class 1: Max-flow Computation, Exact

Class 2: Alpha expansions, Approximate Solution with bounds

Class 3: Local Minima (with no bounds)

Inference in Graphical Models

Min-Marginals Energies(ψ)- Minimize joint energy over all other variables.

- Related to max-marginals as:

- Can be used to compute confidence as:

σj = µj / Σa µa = exp(-ψi) / Σa exp(-ψa)

µj = (1/z)*exp(-ψj)

Energy Projections and Graph Construction

EMRF(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2 + Kā2

a1

a2

2

5

9

4

2

1

Sink (0)

Source (1)

∞

A high unary term (t-edge) can be used to constrain the solution of the energy to be the solution of the energy projection.

Alternative Construction

K

Segmentation Comparison

Gri

mson

-G

rim

son

-S

tau

ffer

Sta

uff

er

Bath

ia0

Bath

ia0

44O

ur

Ou

r m

eth

od

meth

od

Face Detector and ObjCut

Segmentation

Segmentation

Conclusion

Combining pose inference and segmentation worth investigating.

Tracking = Detection Detection = Segmentation Tracking = Segmentation. Segmentation = SFM ??