ABSTRACT

Storyboard is a classical animation tool to help the cre-

ators to organize scenes. In this paper, we propose a newmethod for producing animation from storyboard. The 3Dposition and behavior of the characters are estimated from

2D views using the constraints optimization and example-based interpolation method. We also show animation ex-amples produced from the storyboard.

1. INTRODUCTION

Rapid making of CG animation is useful for various fields

such as entertainment and educations. Recent developmentof low-cost PC enables users to make animations for theirown uses. However, making animation is still a difficult taskespecially for novice users. The users have to learn many

technical terms and skills such as bones, skeleton, inversekinematics, and so on.

More intuitive method for describing the scenes is mak-

ing a storyboard. Storyboard is a classical animation tool tohelp the creators to organize scenes [1,2,3,4]. In the worldof animation, Walt Disney and his staff developed a

storyboard system in 1928. Managing the thousands of draw-ings and the progress of a project was nearly impossible, soDisney had his artists pin up their drawings on the studio

walls. This way, progress could be checked, and scenes addedand discarded with ease.

Starboard may be a good user interface for novice users

to organize idea. However, the users still need many stepsto create animation from storyboards. If we can automati-cally generate animation from storyboards, it would be a

useful tool for create animation.

In this paper, we present an intelligent storyboard forprototyping CG animation. We address the two typical prob-lem in generating animation from 2D storyboards:

INTELLIGENT STORYBOARD FORPROTOTYPING ANIMATION

1) Estimation of 3D location of characters from 2D view

2) Generating continuous character motion from the sparsepose specified by the storyboards

Storyboards generally represent a coarse description of

the character motion. We need to interpolate the charactermotion between each storyboards. However, the conventionalkeyframe interpolation is not enough because the pose of

the characters can be very different between the two adja-cent frames. We use the example-based interpolation tosupplement missing character motion.

2. SYSTEM OVERVIEW

Fig.2 shows the concept of the intelligent storyboard. Theinput to the system is storyboards written by the user. The

Fig.2 Concept of the intelligent storyboard

Junichi Hoshino Yumi Hoshino

University of Tsukuba / PRESTO, JST

Fig.1 Example of the storyboards

Estimating 3D location

Example-based interpolation

Storyboards

Animation

Object andMotionDatabase

DigitalBankSystem

problem of making animations from storyboards is that there

is not enough information to generate 3D character motions.We supplement the missing information by using the geo-metric constraint and example-based learning method. 3D

model of the scene and the characters are stored in the cur-rent system.

3. ESTIMATION OF 3D CHARACTERLOCATIONS FROM 2D VIEW

The first step of generating animation is to estimate 3Dcharacter position from 2D view. We estimate the 3D char-acter position from 2D line segments on the storyboards. Fig.

4 shows the relationship of 2D center line on the image planeand a 3D body part. The position and orientation of a bodyparts can be represented as x=[r,t]T where t is the translation

and r is the rotation along the xyz axis.

The error of a 2D center line and a body part can becalculated as the minimum distance between 3D line seg-

ment P=(p1,p

2) and plane M.

h( x, l)= ( Rp1+ t)⋅ n( Rp2+ t)⋅ n (1)

R is a3x3 matrix derived from r. n is a unit normal vectorof plane M.

n= q1′ × q2

′

q1′ × q2

′ (2)

h(x,l) can be linearized around l = l̂i and x = x̂i−1 usingthe Taylor expansion.

h x l h x lh x l

xx x

h x l

ll li i i

i i ii

i i ii( , ) ( ˆ , ˆ )

( ˆ , ˆ )( ˆ )

( ˆ , ˆ )( ˆ )≈ + − + −−

−−

−1

11

1∂∂

∂∂

(3)

where ∂∂h

x ,

∂∂h

l are partial derivatives.

∂∂

∂∂

∂∂

h

x

h

r

h

t=

, , ∂∂h

r

n R p I

n R p I

T

T=

××

[ ]

[ ]1

2 ,

∂∂h

t

n

n

T

T=

(4),(5),(6)

∂∂h

l can be represented as follows.

∂∂

∂∂

∂∂

∂∂

∂∂

h

l

h

p

h

p

h

q

h

q=

1 2 1 2

, , ,' ' (7)

∂∂

h

p

n RT

1 0=

,

∂∂

h

p n RT2

0=

(8),(9)

∂∂

n

q

q qRp t q I nn q I

q qRp t q I nn q I

T T

T T1

1 2

1 2 2

1 2

2 2 2

1

1'

' '

' '

' '

' '

[ ] [ [ ] [ ]]

[ ] [ [ ] [ ]]=

×+ − × + ×

×+ − × + ×

(10)

∂∂

n

q

q qRp t q I nn q I

q qRp t q I nn q I

T T

T T2

1 2

1 1 1

1 2

2 1 1

1

1'

' '

' '

' '

' '

[ ] [ [ ] [ ]]

[ ] [ [ ] [ ]]=

×+ − × + ×

×+ − × + ×

(11)

these coefficients are used for kalman filter.

We minimize the errors between 2D center line and 3Dbody parts. We use Extended Kalman Filter (EKF) for opti-mization. The measurement equation is

z H x vi i i= + (12)

Fig.4 Fitting 2D line segments and 3D body center

Fig.5 Action trajectory in eigen space. The complex changesof each joints can be represented as a simple trajectory ineigenspace.

storyboard

O

3D character

h(x,l) M

PQ

Q'

xk

PQ Q'

x'k

xk

O O

storyboard storyboard

Fig.3 A relationship of the storyboard and 2D-3D line seg-ments

e1

e2

0

where

zh x l

xx h x li

i ii i i= −−− −

∂∂

( ˆ , ˆ )ˆ ( ˆ , ˆ )1

1 1 (13)

Hh x l

xii i= −∂

∂( ˆ , ˆ )1 (14)

vh x l

ll li

i ii i= −−∂

∂( ˆ , ˆ )

( ˆ )1 (15)

The optimization can be done using the Kalman Filter

equations [5,6].

ˆ ˆ [ ˆ ]x x K z H xi i i i i i= + −− −1 1 (16)

K H H H Ri i i i iT

i= +− −

−σ σ1 11( ) (17)

σ σ σi i i i iK H= −− −1 1 (18)

4. EXAMPLE-BASED INTERPOLATIONOF THE CHARACTER MOTION

Storyboards generally represent a coarse description of

the character motion. We need to interpolate the charactermotion between each storyboards. However, the conventionalkeyframe interpolation is not enough because the pose of

the characters can be very different between the two adja-cent frames. We use the example-based interpolation tosupplement missing character motion.

We represent character’s pose parameters in eigen spaceto reduce the computational cost for searching. Fig.5 showsthe example of the walking action in eigenspace. We can see

that the complex change of each joints can be representedas a simple curve in eigenspace.

Let the 3D pose parameters xp of the person p be

xt = [xt1, xt

2,....xtm]T (11)

where i is the number of frames of training gesture and m isthe total number of the parameters (n=30 for the example inFig.2). The average of the pose parameters can be estimatedas

c = 1N

xtΣt = 1

N

(12)

where N is the number of the whole sample action data. Thedifference D

p can be estimated as

D p = xp – c (13)

The covariance matrix of the pose parameters Q is

Q = XXT (14)

where. X = [D1,D2,...D p]. Eigen vector and eigen value canbe obtained from the following equation.

Qui = λ iui (15)

The k-dimensional eigenspace is constructed by the keigenvectors, which correspond to the largest k eigenvalues,

as basis vectors. A set of the pose parameters xi is projected

onto the eigenspace by

gt =[e1, e2,....ek]T(xt – c) (16)

Assuming the pose displacement between two successiveframes to be small, the two projections onto the eigenspaceare close to each other. So the actions are represented as a

continuous trajectory in the eigenspace. These action trajec-tories are used to interpolate character motion.

The personal characteristics of motion can be also repre-

sented in eigenspace. Fig.6 shows the samples from the threepersons doing the same action. (a) shows the difference be-tween the person. By estimating the variance of motion be-

tween different persons, we can synthesize many other mo-tion with different personal characters. (b) shows the aver-age action and reconstructed motion with different personal

characters.

Fig.7 Sample dance sequences in eigenspace

-3

-2

-1

0

1

2

3

-4 -3 -2 -1 0 1

e 2

e1

Fig.6 Analysis and synthesis of personal characters

(a) sample of 3 persondoing the same action

(b) average action and recon-structed action

0

0.5

1

1.5

-1 0 1 2e

2

1e

0

0.5

1

1.5

-1 0 1 2

e2

average action

generated action

5. ANIMATION EXAMPLES

Fig. 7 shows the example of the dance sequences ineigenspace. Sample 3D motion is extracted from video us-

ing our computer vision technique [7]. Fig.8 (a) is the col-lection of storyboards for input. Fig.8 (b) is the generatedanimation sequences.

Fig.9 shows the example of generating walking anima-tion. (a) is the input storyboards and (b) is the generated ani-mation sequences.

6. CONCLUSIONS

In this paper, we propose an intelligent storyboard forprototyping CG animation. The 3D position of the charac-

ters are estimated from 2D views. We also describe the mo-tion synthesis method using the example-based interpola-tion. The future work includes more sophisticated interpre-

tation of line drawing to create characters. Higher level rec-ognition of the scenes and simple language instruction maybe also useful.

7. REFERENCES

[1] John Hart:”The Art of Storyboard: Storyboarding for Film, TV,and Animation”, Focal Press, 1999

[2] Mark Simon:”Storyboards: Motion in Art”, Focal ress, 2000

[3] Amy Spaulding:”The Page as a Stage Set: Storyboard PictureBooks”, Scarecrow, 1995

[4] “A Visit With Disney's the Little Mermaid and FriendsStoryboard”, Mouse Works, 1993

[5] A. Gelb et al.:"Applied Optimal Estimation", Cambridge, MA,MIT Press.

[6] H.W. Sorenson:"Kalman Filtering: Theory and Application",New York, IEEE Press, 1985

[7] J. Hoshino, H. Saito, M. Yamamoto:”Automatic Registrationof Virtual Objects onto Human Image Sequences”, IAPR Int.Conf. on Pattern Recognition, ICPR 2000, pp. 175-177, 2000

(a) input storyboards

(b) Generated animation sequence

Fig. 9 Example of walking animation produced fromstoryboards

Fig.8 Example of generating dance sequences. Colors showsthe correspondance of input storyboards and generated ani-mation sequences.

(a) input storyboards

(b) generated animation sequence