Learning a correlated model of identity and pose-dependent body shape variation

for real-time synthesis

Brett Allen1,2, Brian Curless1, Zoran Popović1, and Aaron Hertzmann3

1 University of Washington2 Industrial Light & Magic

3 University of Toronto

Motivation

movies games

telepresence design

Goal

• We would like to be able to generate body models of any individual in any pose.

Identity +

pose shape

- want to synthesize models in real-time

- model should be learnable from real data

Data

• CAESAR data set: 44 subjects in 2 poses• Multi-pose data set: 5 subjects in 16 poses• Dense-pose data set: 1 subject in 69 poses

[Anguelov et al. 2005]

Related work

Anatomical methods

Aubel 2002

Chadwick et al. 1989Turner and Thalmann 1993Scheepers et al. 1997Wilhelms and Van Gelder 1997…

Example-based methods

7000

7000

7000

0

0

0

z

y

x

z

y

x

v

),( Φpv f

v = shape vectorp = example parameters = function parameters

Given: n examples v0…vn and n sets of parameters p0…pn (optional)find .

Scattered data interpolation

Allen et al. 2002

Lewis et al. 2000Sloan et al. 2001Kry et al. 2002

i

iif φΦp ),(columns of = key shapesi = reconstruction weights from applying k-nearest neighbors or RBFs on p

1

2

43

1

3

2

4

An aside on enveloping

Enveloping + scattered data interpolation= “corrective enveloping”

Latent variable modelsvWxvWx }),{,(f

x = latent variable (component weights)W = components in columnsv = average shape

Blanz & Vetter 1999

Allen et al 2003Seo et al 2003Anguelov et al 2005…

Pose variation vs body variation

Sloan et al. 2001

Pose variation vs body variation

Anguelov et al. 2005

KωvWxv

Our approach

iii

k

φv

b

φ

φ

c

cWxc

1

c = “character vector”: all information needed to put a character in any posev = shape in a particular pose

Intrinsic skeletonparameters: bonelengths and carrying angles

Two Problems

1. Scans might not be at “key” poses

2. Scans are not complete

Maximize: p(c | {e()})

…actually, we don’t know the pose or skinning weights either:

Maximize: p(c, s, q{} | {e()})

Maximum a posteriori estimation

})({log)(log)(log

2

1)2log(5.1log

})({)()(),,|(

}){|}{,,(

1 1

2)()(2

2

1 1

)(

)(

qsc

ve

qscqsc

eqsc

v

v

vvv

ppp

nnP

pppepP

pP

n n

iii

n n

ii

Results

Going to multiple characters

• One possibility: Learn several character vectors separately, then run PCA.

• Two problems:– the character vector contains values that have

different scales (rest positions, offsets, bone lengths)

– we don’t have enough data!

Identity variation

),0();,(),,( 2),(

NNg ii I0xqsce

cWxc

~ ~

c is the character vector of the th example persong(c,s,q) applies skinning and pose space deformations

Alternating optimization

• We initialize the {x} with the weights from running PCA on estimated skeleton parameters.

• We then optimize W, c, s, q.

• Then we optimize for {x}.

• Repeat…

Results

Results

(video)

Conclusions

We present a flexible approach for learning body shape variation between individuals and between poses, including the interrelationship between the two.+ very general: can handle irregular and incomplete sampling in regard to both the poses/identities scanned, and in the surfaces themselves+ the learned model can generate body shapes very quickly (over 75 fps)

Limitations

• You need a lot of data! Our data set was too sparse in some areas.

• Some poses are hard to capture.• It’s very hard to compensate for the

skinning artifacts.• The shape matching could be improved

(high-frequency details are lost if the matching is poor).

Acknowledgements

• UW Animation Research Labs• Washington Research Foundation• National Science Foundation, NSERC, CFI• Microsoft Research, Electronic Arts, Sony, Pixar• Kathleen Robinette and the AFRL lab• Dragomir Anguelov• Domi Pitturo