enforcing constraints for human body tracking david demirdjian artificial intelligence laboratory,...

Enforcing Constraints for Human Body Tracking

David DemirdjianArtificial Intelligence Laboratory, MIT

WOMOT 2003

Goal Real-time articulated body tracking from

stereo accounting for constraints on pose

WOMOT 2003

Approach

Differential tracking: assuming the articulated body pose t-1 is known, estimate the pose t (or

equivalently the set of limb rigid motions i=(tii) between

poses t-1 and t) that minimizes the distance between the articulated model and the observed 3D data

tracking as a constrained optimization problem

WOMOT 2003

Approach

Differential tracking: assuming the articulated body pose t-1 is known, estimate the pose t (or

equivalently the set of limb rigid motions i=(tii) between

poses t-1 and t) that minimizes the distance between the articulated model and the observed 3D data

tracking as a constrained optimization problem– Solve unconstrained optimization problem– Project solution on constraint surface

WOMOT 2003

Projection-based approach

unconstrained optimum)

human motion manifold

WOMOT 2003

Approach

Estimate limb motions i=(tii) independently using standard multi-object tracking algorithm

Projection: find the closest body motion =(i’) to =(i) that satisfies human body constraints: – articulated constraints – other constraints: joint limit, …

WOMOT 2003

Previous work

Particle sampling: Sidenbladh & al. ECCV’00

Sminchisescu & Triggs CVPR’01

Differential tracking: Plankers & Fua ICCV’99

Jojic & al. ICCV’99

Delamarre & Faugeras ICCV’99

WOMOT 2003

Plan

Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion

WOMOT 2003

Multi-object tracking

Assuming the articulated body pose t-1 is known, estimate the set of limb rigid motions i=(tii) minimizes the distance between the (moved) limb and the observed 3D data

Consists in estimating limb motions i=(tii) independently:

– Estimate visible 3D mesh of each limb– Current implementation uses the ICP algorithm to

register each limb w.r.t 3D data

WOMOT 2003

Iterative Closest Point

3D registration– find the rigid transformation that maps shape St (limb model) to

shape Sr (3D data)

SrSt

),(minarg1

2*

n

i

rt SSGd

WOMOT 2003

Iterative Closest Point

Matching points• For all points in St, we search for the closest point in Sr by

computing the distance and keep the closest

SrSt

WOMOT 2003

Iterative Closest Point

Energy function minimization• Estimate the rigid transformation that minimizes the sum of

squared distances between corresponding matched points

SrSt

WOMOT 2003

Iterative Closest Point

Energy function minimization• Estimate the rigid transformation that minimizes the sum of

squared distances between corresponding matched points

SrSt

WOMOT 2003

Iterative Closest Point

Optimal rigid transformation (and uncertainty ) found by combining all the elementary displacements

WOMOT 2003

Plan

Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion

WOMOT 2003

ProjectionThe unconstrained optimal body motion is given by

=(1, 2 … N)With uncertainty

=(1, 2 … N)

)()( 12 TE

with =(1’, 2’ … N’) satisfying articulated constraints

Articulated constraints enforcement: find that minimizes the Mahalanobis distance:

WOMOT 2003

Articulated motion estimation

If Mij is a joint between objects i and j:

 Mij joint

(Ri,ti)

(Rj,tj)obj. i

obj. j

Motion of point Mij

on limb i

Motion of point Mij

on limb j=

)(')(' ijjiji MM

WOMOT 2003

Articulated motion estimation

If Mij is a joint between objects i and j:

 Mij joint

(Ri,ti)

(Rj,tj)obj. i

obj. j

Motion of point Mij

on limb i

Motion of point Mij

on limb j=

'''' jijjiiji tMRtMR

)(')(' ijjiji MM

WOMOT 2003

Articulated motion estimation

If Mij is a joint between objects i and j:

 Mij joint

(Ri,ti)

(Rj,tj)obj. i

obj. j

Motion of point Mij

on limb i

Motion of point Mij

on limb j=

0'')''(][

0'']''[

')]'[(')]'[(

''''

jijiij

jiijji

jijjiiji

jijjiiji

ttM

ttM

tMItMI

tMRtMR

[.]x denotes skew-symmetric matrix

)(')(' ijjiji MM

WOMOT 2003

Articulated motion estimation

If Mij is a joint between objects i and j:

 Mij joint

(Ri,ti)

(Rj,tj)obj. i

obj. j

Motion of point Mij

on limb i

Motion of point Mij

on limb j=

0'')''(][ jijiij ttM

[.]x denotes skew-symmetric matrix

)(')(' ijjiji MM

WOMOT 2003

Articulated motion estimation

If Mij is a joint between objects i and j:

 Mij joint

(Ri,ti)

(Rj,tj)obj. i

obj. j

Motion of point Mij

on limb i

Motion of point Mij

on limb j=

0 ijS

)(')(' ijjiji MM

0'')''(][ jijiij ttM

=(1’, 2’ … N’)

WOMOT 2003

Articulated motion estimation

If Mij is a joint between objects i and j:

 Mij joint

(Ri,ti)

(Rj,tj)obj. i

obj. j

Motion of point Mij

on limb i

Motion of point Mij

on limb j=

(Stack for all joints)0

)(')(' ijjiji MM

0 ijS

0'')''(][ jijiij ttM

=(1’, 2’ … N’)

WOMOT 2003

Articulated motion estimation

All the joint constraints can be written as a linear constraint:

0

is a linear combination of vectors in the nullspace of Therefore there exists a matrix V such that:

V

V can be estimated by SVD of

WOMOT 2003

Articulated motion estimation

)()( 12 VVE T

111 )( TT VVVVP

)()( 12 TE

unconstrainedmotion

articulatedmotion

Find minimum of E2 in nullspace of

P

…

(linear projection)

WOMOT 2003

Plan

Unconstrained problem Articulated constraints enforcing Other constraints Tracking results Application (Multimodal interface) Conclusion

WOMOT 2003

Other constraints

Constraints:– Static: Joint angle bounds, gravity law, …– Dynamic: Maximum velocity, …

Motivation:– Using more constraints to reduce local minima

and therefore increase tracking robustness– Application context can reduce tremendously

the dimension of the pose space

WOMOT 2003

Other constraints

)()( *1*2 TE

Pose constraints modeled by a (user-defined) function f, such that valid poses correspond to f()>0

ex: f()=min(g1(), g2(), … gN()) with g1() = angle(l-arm, l-forearm) – min_angle

g2() = max_angle - angle(l-arm, l-forearm)….

Constraints enforcement: find * that minimizes the Mahalanobis distance:

with * satisfying Ft-1(*)=f( *(t-1))>0

WOMOT 2003

Other constraints

)()( *1*2 TE

articulated motionarticulated constrained

motion

** V

)()(

)()(*1*2

*1*2

VVE

VVVVETT

T

with (local parameterization)

WOMOT 2003

Constrained optimization algorithm

Alternate between binary and stochastic searches

WOMOT 2003

Constrained optimization algorithm

Alternate between binary and stochastic searches

WOMOT 2003

Constrained optimization algorithm

Alternate between binary and stochastic searches

E2 = E0

WOMOT 2003

Constrained optimization algorithm

Alternate between binary and stochastic searches

E2 = E1

WOMOT 2003

Constrained optimization algorithm

Alternate between binary and stochastic searches

WOMOT 2003

TRACKING SEQUENCE

WOMOT 2003

Future work

Quantitative measurement(comparing results with tethered motion capture system)

Appearance/Shape information(learning color distribution + shape of limbs)

Motion/gesture(including dynamic constraints)

Learning human motion constraints (instead of giving them explicitly.. [ICCV’03])

WOMOT 2003

Applications Multimodal Human-Computer Interaction

(gesture + speech)

WOMOT 2003

WOMOT 2003

Conclusion

Projection-based approach for articulated body tracking– articulated constraints enforced by (linearly)

projecting unconstrained limb motion on articulated motion manifold

– other constraints enforced using a stochastic constrained optimization algorithm