character animation and control using human motion data jehee lee carnegie mellon university jehee

Character Animation and Controlusing Human Motion Data

Jehee LeeCarnegie Mellon University

http://www.cs.cmu.edu/~jehee

Character Animation

Final Fantasy Movie Characters

from www.finalfantasy.com

Final Fantasy X

NBA Courtside 2002

NFL 2k2 WWF Raw

All game characters from www.gamespot.com

Motion Capture

• Record movements of live performers– Realistic, highly detailed data can be obtained

Motion capture lab at CMU

Animation from Motion Capture

MotionDatabase

Preprocess

On-lineController

Motion EditingToolbox

MotionSensor

Data

ConvincingAnimation

ControllableResponsiveCharacters

High-Level UserInterfaces

The Art ofAnimation

Animation from Motion Capture

MotionDatabase

Preprocess

On-lineController

Motion EditingToolbox

MotionSensor

Data

ConvincingAnimation

ControllableResponsiveCharacters

MappingLive

Performance

High-Level UserInterfaces

The Art ofAnimation

ComputerPuppetry

Interactive 3D Avatar Control

• How to organize data ?– Large collection of motion data

• How to control ?– User interfaces

MotionDatabase

PreprocessOn-line

Controller

MotionSensor

Data

ControllableResponsiveCharacters

High-Level UserInterfaces

Related Work (Motion Control)

Rule-based Control system

[Bruderlin & Calvert 96]

[Perlin & Goldberg 96]

[Chi 2000]

[Cassell et at 2001]

[Hodgins et al 95]

[Wooten and Hodgins 96]

[Laszlo et al 96]

[Faloutsos et al 2001]

Example-based Statistical Models

[Popovic & Witkin 95]

[Bruderlin & Willams 95]

[Unuma et al 95]

[Lamouret & van de Panne 96]

[Rose et al 97]

[Wiley & Hahn 97]

[Gleicher 97, 98, 01]

[Sun & Mataxas 2001]

[Bradley & Stuart 97]

[Pullen & Bregler 2000]

[Tanco & Hilton 2000]

[Brand & Hertzmann 2000]

[Galata et al 2001]

[Lee et al 02]

Related Work (User Interfaces)

Graphical User Interfaces

Performance

(Motion capture devices)

Performance

(Vision-based)

[Bruderlin & Calvert 96]

[Laszlo et al 96]

[Rose et al 97]

[Chi 2000]

[Badler et al 93]

[Semwal et al 98]

[Blumberg 98]

[Molet et al 99]

“Mocap Boxing” (Konami)

[Blumberg & Galyean 95]

[Brand 1999]

[Rosales et al 2001]

[Ben-Arie et al 2001]

Motion Database

• In computer games– Many short, carefully planned, labeled motion clips– Manual processing

Walk Cycle StopStart

Left Turn

Right Turn

Motion Database

• Our approach– Extended, unlabeled sequences of motion– Automatic processing

Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted.

Sketch Interface

Motion Data for Rough Terrain

Motion Data for Rough Terrain

Unstructured Input Data

Connecting Transitions

Local Search for Path Following

Comparison to Real Motion

Comparison to Real Motion

User Interfaces

Choice-based Interface

• What is available in database ?– Provided with several options– Select from among available behaviors

Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted.

Jehee Lee, Jinxiang Chai, Paul Reitsma, Jessica Hodgins, and Nancy Pollard, Interactive Control of Avatars Animated with Human Motion Data, submitted.

How to Create Choices ?

Clustering

Find Reachable Clusters

A

B

C

D E

F

G

Most Probable Paths

Cluster Forest

B

C

DE

F

G

BD

E

F

G

Performance Interface

MotionDatabase

SearchEngine

AnimateAvatars

Vision-basedInterface

Silhouette extraction and matching implemented by Jinxiang Chai

Database Search3 sec

Animation from Motion Capture

MotionDatabase

Preprocess

On-lineController

Motion EditingToolbox

MotionSensor

Data

ConvincingAnimation

ControllableResponsiveCharacters

MappingLive

Performance

High-Level UserInterfaces

The Art ofAnimation

ComputerPuppetry

The Art of Animation

• Animators need good tools– Modify, vary, blend, transition, filter, …

MotionDatabase

Motion EditingToolbox

ConvincingAnimation

The Art ofAnimation

Challenges in Motion Editing

• Reusability and flexibility– Motion data is acquired

• For a specific performer• Within a specific environment• In a specific style/mood

• High dimensionality

• Inherent non-linearity of orientation data

Related Work

Physically-based

Signal processing/

Interpolation

Optimization + Interpolation

Stochastic

Modify [Popovic& Witkin 99]

[Unuma et al 95]

[Bruderlin &

Williams 95]

[Sun&Metaxas 01]

[Lee & Shin 01, 02]

[Gleicher 97, 98, 01]

[Lee & Shin 99]

[Perlin 95] [Bradley&Stuart 97]

[Pullen&Bregler 00]

Transition/

Blend[Rose et al 96]

[Lamouret & van de Panne 96]

[Rose et al 97]

[Sun&Metaxas 01]

[Lee & Shin 01, 02]

[Tanco&Hilton 00]

[Brand &

Hertzmann 00]

[Galata et al 01]

Basic Techniques

• Multiresolution Analysis– Signal processing approach– Transition, blend, modify style/mood, and

resequence

• Hierarchical displacement mapping– Constraint-based approach– Interactive editing– adaptation to different characters/environments.

Multiresolution Analysis

• Represent signals at multiple resolutions– give hierarchy of successively smoother signals– facilitate a variety of signal processing tasks

)0(m)3(m )1(m)2(m

Decomposition

• Reduction: upsampling followed by smoothing• Expansion: smoothing followed by downsampling

)(nm

)1( nm

)1( nd

Reduction Expansion

)(nm )1( nm )2( nm )0(m

)1( nd )2( nd )0(d

)(nm )1( nm )2( nm )0(m

)1( nd )2( nd )0(d

Decomposition

Reconstruction

Enhance / Attenuate

Jehee Lee and Sung Yong Shin, General Construction of Time-Domain Filters for Orientation Data, IEEE Transactions on Visualization and Computer Graphics, to appear.

Jehee Lee and Sung Yong Shin, A Coordinate-Invariant Approach to Multiresolution Motion Analysis and Synthesis, Graphical Models (formerly GMIP), 2001.

Enhance / Attenuate

Walk

Limp

Turn

?Turn with a Limp

Walk

Limp

Turn

Turn with a Limp

Analogy

• Low frequency (Content)

Result = Limp + (Turn – Walk)

• High frequency (Style)

Result = Turn + (Limp – Walk)

Walk Turn

LimpTurn with

A limp

Walk

Strut

Run

Stub toes Limp

Stitched

Re-sequence

Reconstruction

)0(m

)0(d

)1(d

)2(d

)0(m

Reconstruction

)(E 0m

)0(m

)0(d

)1(d

)2(d

Reconstruction

)(E 0)0()1( dmm

)0(m

)0(d

)1(d

)2(d

Orientation Representation

• Inherent non-linearity of orientation space

12222 zyxw 3S

Filtering Orientation Data

kikikiki aaa qqqq 0)(F

• How to generalize convolution filters ?

3S

Related Work• Re-normalization

• Azuma and Bishop (94)

• Global linearization• Johnstone and Williams (95)

• Local linearization• Welch and Bishop (97)• Fang et al. (98)• Lee and Shin (2002)

• Multi-linear• Shoemake (85)

• Optimization• Lee and Shin (96)• Hsieh et al. (98)• Buss and Fillmore (2001)

Re-normalization

iq

1iq

2iq

1iq

2iq

5

1,

5

1,

5

1,

5

1,

5

1

Linearization

3S 3R

Exponential and Logarithm3SqT

q

Exponential and Logarithm

1q

qq 1

Exponential and Logarithm33 RSI T

),,,( 0001I

),,,( zyx0

Exponential and Logarithm33 RSI T

log exp

I

Global Linearization

ii pq log I

Angular Displacement

ip

ii pp 1

1ip

1iq

)( iiii pppp 11

iq3S3R

Angular Displacement

ip

ii pp 1

1ip1

1

ii qq

)( iiii pppp 11

ii qqI 1

Angular Displacement

ip

ii pp 1

1ip1

1

ii qq

)( iiii pppp 11

ii

iii

exp1

q

qq

ii qqI 1 11

i log ii qq

Local Linearization

3S 3R

iii pqq

11log

The Drifting problem

iii pqq

11log

Our Approach

3S 3R

iiii ppqq

111log

Filtering Orientation Data

ip

1ip 2ip1ip2ip

i

1i

1i

2i

Filter Properties

• Coordinate invariant

• Time invariant

• Symmetric

ii qqRH,RRH where

niinnn

qqSH,SSH where

3where Sbabqabaq ,,)H()H( ii

Coordinate Invariance

m

)0(

)0(

)1(

)(

mdd

d

n

)0(

)0(

)1(

)(

mdd

d

T

n

mTDecomposition Reconstruction

T

T

Coordinate Invariance

• Independent to the choice of coordinate systems

Basic Techniques

• Multiresolution Analysis– Signal processing approach– Transition, blend, modify style/mood, smoothen,

resequence

• Hierarchical displacement mapping– Constraint-based approach– Interactive editing and adaptation

Motion Editing through Optimization

• Constraints[Witkin & Kass 88] [Cohen 92] [Gleicher 98]

– Features to be retained– New features to be accomplished

• Find a new motion– Satisfy given constraints– Preserve original characteristics

Jehee Lee and Sung Yong Shin, A Hierarchical Approach to Interactive Motion Editing for Human-Like Figures, Siggraph 99

Motion Representation

• Motion of articulated characters– Bundle of motion signals– Each signal describe positions / orientations / joint angles

Basic Idea

• Inter-frame relationship– Enforce constraints– By inverse kinematics

• Inter-frame relationship– Avoid jerkiness– By curve fitting

Displacement Mapping

Displacement Map

Original Motion

Target Motion

Hierarchical Displacement Mapping

• Representation of displacement maps– An array of spline curves– Over a common knot sequence

• Flexibility in representation– Hard to determine knot density– Adaptive refinement is needed

Adaptive Refinement

• Multi-level or hierarchical B-splines[Lee, Wolberg, and Shin 97] [Forsey and Bartel 95]

– Sum of uniform B-spline functions– Coarse-to-fine hierarchy of knot sequences

Multi-Level B-spline Fitting

0f

1f 2f

10 ff 210 fff

Adaptation to Rough Terrain

Jehee Lee and Sung Yong Shin, A Hierarchical Approach to Interactive Motion Editing for Human-Like Figures, Siggraph 99

Adaptation to New Characters

Character Morphing

Animation from Motion Capture

MotionDatabase

Preprocess

On-lineController

Motion EditingToolbox

MotionSensor

Data

ConvincingAnimation

ControllableResponsiveCharacters

MappingLive

Performance

High-Level UserInterfaces

The Art ofAnimation

ComputerPuppetry

Hyun Joon Shin, Jehee Lee, Michael Gleicher, and Sung Yong Shin, Computer Puppetry: An Importance-based Approach, ACM Transactions on Graphics, 2001.

The videos were made by Hyun Joon Shin, Tae Hoon Kim, Hye-Won Pyun, Seung-Hyup Shin, Jehee Lee, Sung Yong Shin, and many others at the Korea Broadcasting System.

Summary

• Motion data processing– Multiresolution analysis– Hierarchical displacement mapping

• Interactive control– Motion databases– User interfaces: Choice, sketch, performance

Future Work

• Autonomous virtual humans– Convincing appearance, movements– Reasonable level of intelligence

• Collect real world data– Motions, pictures, videos, voices, facial

expressions, and physical properties

Computer Puppetry

• Immediate mapping from a performer to an animated character

MotionSensor

DataMapping

LivePerformance

ComputerPuppetry

Time Invariance

• Independent to the position in the signal

Time

Statistical Model

Statistical Model

Motion Representation

StatisticalModel

MarkovProcess

User Control

UpdateAvatar Pose

Markov Process

• Raw data– Extended– Unstructured

• Processed data– Connected– Flexible

Cluster Forest

Cluster Forest