learning to synthesize arm motion to music by example · 2020. 7. 2. · various work...

Learning to Synthesize Arm Motion to Music By Example

Sageev OoreSaint Mary’s University, Halifax,

[email protected]

Yasushi AkiyamaSaint Mary’s University,

Halifax, [email protected]

ABSTRACT

We present a system that generates arm dancing motion to new music tracks, based on sample motion captured data of dancingto other pieces of music. Rather than adapting existing motion, as most music-animation systems do, our system is novel in thatit analyzes both the supplied motion and music data for certain characteristics (eg. melodic contour, loudness, etc), and learnsrelationships between the two. When new music is provided, the characteristics are analyzed as before, and used to predictcharacteristics of the motion. A generative process then creates motion curves according to these constraints. A demonstrationis presented showing the results on both slow and fast tunes, including an automatically-generated trio of backup dancers for ajazz standard. The system could be extended naturally to other movements beyond arm dancing.

Keywords: animation, learning, motion/music synchronization.

1 INTRODUCTIONThere are a myriad ways to create character motion tomusic. Certain characters may have a particular stylesof dancing, based both on their ways of moving, andalso on how they perceive the music, or based on thedirections of a choreographer. How does the nature ofsome of these dance motions— especially in the “back-ground” contexts that one might want to automate—relate to musical characteristics such as loudness, pitch,and note density? There is no constant relationship, ofcourse, but for a given context or character, there maylikely be a particular connection or association. Our pa-per presents an initial approach towards answering thisquestion by analyzing the characteristics of music andmotion data, and then by providing a prototype tool foranimators to specify some of these relationships implic-itly by example.

Various work [ES03, Lyt90, LL05, CBBR02, KPS03,WLXS02,GBBM96,GM95,PN03] has been focused onthe problem of adapting existing motions to music. Inmany of these systems, the primary mappings betweenthe features of music and motion are specified explic-itly by the user. In others, motions are synthesized froman existing database. This research proposes a systemfor learning the characteristics of desired motions as-sociated with musical phrases by user-provided exam-ples for some musical phrases, in order to produce ananimated figure that responds not only to the musicalphrases that are used to train it, but will also be able

Permission to make digital or hard copies of all or part of thiswork for personal or classroom use is granted without fee providedthat copies are not made or distributed for profit or commercialadvantage and that copies bear this notice and the full citation on thefirst page. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee.

WSCG 2006 conference proceedings, ISBN 80-86943-03-8WSCG’2006, January 30 – February 3, 2006Plzen, Czech Republic.Copyright UNION Agency – Science Press

to handle new musical ideas. Such a tool would beintended for use by animators to automatically chore-ograph characteristics of dance for a musical perfor-mance, so the final motions can be created automati-cally. For example, ultimately the dance style in a cer-tain nightclub or a set of backup singers/dancers for amusical act, or recurring dancers in a video game, couldbe done in this way.

Since the motion-music relationships are inferredfrom the data, it follows that different data sets willlead to different dance styles. For example, if all of thesupplied training data were to depend exclusively onthe pitch, and not be affected in any way by changesin loudness (e.g. the ‘dynamics of the phrasing’,in musical terms), then this should be visible in theresulting animation. The system is not intended tolearn a single definitive relationship between motionand music based on a large comprehensive motiondatabase, but rather to allow certain relationships– asneeded for a particular set of characters or situations–to be shown by example. In Section 9 we demonstratethis with automatically generated arm motions for a setof backup dancers for a jazz standard.

2 RELATED WORKThe issue of synchronizing animation to music hasbeen addressed by a number of researchers. Kim etal. [KPS03] and Alankus et al. [ABB04] created sys-tems using motion graphs [KGP02] and beat analysis tosynthesize a motion from existing sample motions syn-chronized to background music. The systems by Cardleet al. [CBBR02] and Lee and Lee [LL05] synchronizemotion curves by locally modifying motions using per-ceptual cues obtained from the music. Lytle’s [Lyt90]system creates animation of musical instruments fromorchestrated MIDI music. Goto and Muraoka [GM95]have multiple musicians, each assigned to control spe-cific features of a single animated character. Penasse

1

WSCG2006 Full Papers proceedings 201 ISBN 80-86943-03-8

and Nakamura [PN03] used the analog sound signal andadjusted key frames so that they fall on the beats of mu-sic. In these systems, using explicit mappings of fea-tures between music and animation seems like a stan-dard approach. ElKoura and Singh [ES03] use inputmusic data in an augmented tablature notation for gui-tar. Their system solves the problem of multiple con-current reaching tasks by finding the likely hand con-figurations in the realistic hand configurations obtainedfrom example motions, and minimizing a cost functionfor hand motions. Wang et al [WLXS02] predict thejoint angle trajectories to create conducting motions byusing kernel-based Hidden Markov Models. Learningfrom the music with the time signature that has the sameor similar musical accents and relying on beat informa-tion provided in the MIDI file may limit its flexibilityof creating motions responding to music that is playedmore freely.

3 SYSTEM OVERVIEW AND OUT-LINE

The goal of our system is to accept examples of syn-chronized motion and music, in order to learn a rela-tionship between the two, and subsequently use this in-formation to generate animation for new music input.For demonstrative purposes, the present system focuseson arm motions recorded to melodic lines in MIDI for-mat; scalability is discussed in Section 10.

We now give an outline of this paper, along with anoverview of the two phases, in which our system oper-ates.1) Training Phase: Fig 1A set of examples is provided consisting of hand-motion recorded in synchrony with music in MIDIformat (described in Section 4). Based on ourmotion model, the motion data is analyzed for cer-tain characteristics: distance-from-body, amplitude,centres-of-motion (described in Section 5). Likewise,the music data is analyzed for characteristics such asloudness and note density (Section 6). A relation-ship between the input and output characteristics islearned so as to be able to predict a distribution overlikely output motion characteristics given music inputcharacteristics (Section 7).2) Generative Phase: Fig 2New music input is provided. The system analyzes itas before, and uses the learned model to stochasticallygenerate a set of motion characteristics. A generativeprocess is applied to create motion curves for the finalanimation (Section 8).

4 DATA COLLECTION

Music Data .Music is input in a standard MIDI file format.

Each note event contains information pertaining to

Figure 1: Training Phase: Synchronized examples ofmotion and music are provided to the system.

Figure 2: Generative Phase: New music is now pro-vided, and using the learned model, motion is now theoutput rather than input (as indicated by the reversedarrows).

pitch, loudness, time, and a NOTE ON and NOTEOFF signal, indicating whether the note is starting orstopping. The MIDI format we use is Type 1, whichallows a file to have multiple tracks. This is convenientas it lets us focus directly on processing individual lines(eg. melody, bass) and avoid the task of separating agiven score into parts [SNK01].

Motion Capture Two 3D motion capture devices areused to collect the sample motion data. The dancer fa-miliarizes herself with the music prior to the motionrecording session. We acquire 3 degrees of freedom(DOF) from each sensor, measuring hand position. Asthe dancer is aware that the motion will be used for thearm motion of a character who is standing in one spotduring recording, the dancer also remains in one spot toensure that the motion examples are representative.

5 DANCE MOTION ANALYSIS

Our motion model is developed based on a few keyproperties observed in the captured sequences. In thefollowing discussion, letB1,B2, . . . ,BN representN dif-ferent recorded motions, each of which was danced insync to the same music trackM.


5.1 Spatial Characteristics

An important visible feature of the arm motion isextent:the distance of the hands from the body [Lab88,NF03].

We first computeDk(t), the Euclidian distance of thehands from the root node at timet, for motionBk, wherethe root can simply be approximated by a point near themiddle the torso for this purpose. We then estimate theglobal motion centreby computing the mean positionof the entire hand motion. Finally, we computedistk(t),the Euclidian distance of the hands from the global mo-tion centre. This is shown in Figure 3. Figure 4 showsdistk(t) of four sample captured dance sequences to thesame piece of musicM.

Figure 3: Root position, Dk(t), the global motion cen-tre, and distk(t)

The motions tend to be comprised of oscillating seg-ments, corresponding to back-and-forth (or side-to-sideetc) movement of the hands. We model such oscilla-tions by estimating, for each time frame, an approxi-mate centre of extent about which the current oscilla-

Figure 4: This illustrates 4 different sets of dist(t) fora single music track. Each colour corresponds to onerecorded motion. While each curve is different, theyshare certain characteristics, and those characteristicsare what we would like to learn. E.g. the two largestpeaks happen around the same time in all the motionsamples, and the ranges of the oscillation amplitudesare very similar within a certain period of time. Also,they share similar oscillatory nature.

tion is taking place, as well as its approximate ampli-tude. To do this we define a windowV(ti) around timeti , corresponding to a set ofS+ 1 frames of recordeddance motion from(ti − S/2) to (ti + S/2). This isshown in Figure 5.

Figure 5: For each frame, mk(ti) and vk(ti) are com-puted in the moving window V(ti) .

For each windowV(ti) we compute the mean handdistance (or “center of oscillation”):

mk(ti) =1

S+1 ∑ti∈V(ti)

(distk(ti)) (1)

Similary, we compute the variance for windowV(ti):

Figure 6: The mean hand distance mk(t). The samecolors correspond to the same motions in the previousand next figures.

vk(ti) =1

S+1 ∑ti∈V(ti)

(distk(i)−mk(ti))2 (2)

The square-root of this value,√

vk(ti), relates to theamplitude of the oscillations, as it tells us how farthe hand moves from the local centre of oscillation.Figures 6 and 7 showmk(t) andvk(t), respectively, forclips from a set of motions all corresponding to onepiece of music. Looking at these graphs, we see that, fora given piece of music, there are sections where thereis considerable variance in themk, and other sections


Figure 7: The variance vk(t)

wherein themk are relatively tight. Similarly, lookingat the estimated amplitudes over the same motion set,more coherence is evident in some areas than in oth-ers. This suggests that, if we are to predictm(t) andv(t) for some new piece of music, and if we want to beable to create multiple animations sharing “characteris-tics”, it is not enough to simply predict a single valuefor m(t), but rather adistributionwhich we will specifyby a meanµm(t) and varianceσm(t), from which wewill later sample values (Section 8). Similarly wewill predict µv(t) andσv(t). In order to learn to predictthese values for a new piece of music, we first estimatethem for the given data:

µm(ti) =1N

N

∑k=1

(mk(ti)) (3)

σm(ti) =1

N−1

N

∑k=1

(mk(ti)−µm(ti))2 (4)

and

µv(ti) =1N

N

∑k=1

(vk(ti)) (5)

σv(ti) =1

N−1

N

∑k=1

(vk(ti)−µv(ti))2. (6)

In Section 6 we will describe some of the music char-acteristics that may drivem(t) andv(t), but first we con-sider some spatial characteristics of the motion.

5.2 Temporal Characteristics

While the frequency is of course not constant, thereis likely a relationship between the motion and therhythm of the music, and once again, we will be try-ing to capture certain characteristics of this relation-ship. One type of visually salient event is thepeakofan oscillation— a hand stopping and going in anotherdirection. This event corresponds to zero-crossingsof derivative, and we would like to model thesepeakpointsboth in time and in space. The amplitude and os-cillation centre predictors(µv,σv,µm,σm) will give us,based on the music characteristics, a distribution over

where the peak point should be in space; another pre-dictor will be used to stochastically choose the frame atwhich that peak point should occur so that it is in syn-chrony with musical characteristics. To provide train-ing data for the learning algorithm, we create, for eachmotionBk, a binary variable

peakk(t)={

1 if a local extremum occurs int±∆t,0 otherwise

where∆t is a small constant≈ 0.1secto allow for slighttemporal discrepancies between the data and the music.Combining peakk(t) for eachBk, wherek = 1, . . . ,N,we obtain an estimate, for a given piece of music, ofthe independent probability of a peak event occuringat each time slice. We repeat this for each of thegiven music tracks and corresponding sets of anima-tions. This gives us a set of music tracks, each with acorresponding functionpeak(t) marking possible mo-ments at which such events may occur.

6 MUSIC ANALYSISBefore we can apply a machine learning tool to predictthe motion characteristics described above, we need toconsider the ‘input’ variables to the system. That is, wenow extract certain characteristics of a given musicaltrack that may influence the resulting animation. Thethree key concepts in which we are interested are (1)melodic contour, (2) loudness and (3) note density. Torealize this, we need to compute various quantities asdescribed below.

6.1 Dealing with Multiple ScalesSuppose that the current note is somewhat higher (orlouder) than the previous note. What is the signifi-cance of this, with the view of choreographing a danceto the music? This would depend not only on the pre-vious note, but also on where this note fits within thelarger context of the pitch (or loudness) contour. Thus,we consider musical progression at several time-scales,by computing characteristics for three distinct time-windows,W1 = [w0,w1],W2 = [w1,w2],W3 = [w2,w3],and refer to the collection of these subwindows asW =[w0,w3], with w j < w j+1 and w3 being the currentframe. This is illustrated in Fig 8.

Note that the windows used here for the music con-text are different from the windows used for the motiondata analysis, in that the music windows are to cap-ture the music contour that leads up to the current noteevent, while the motion windows are used to capturethe behaviouraroundthe current event. This is why themusic window ends on the current event, but the motionwindow is centred on the current event.

6.2 Melodic Contour InformationMelodic contour refers to the rise and fall of the pitches.If a passage is played slowly and staccato, then most of


Figure 8: The windows W1,W2 and W3 for multiplescales.

the time there is no "active" note, yet a contour is stillimplied, and animated motion may correspond to thiscontour. This can be solved by simply keeping track ofthe previous note. However, suppose a note is embel-lished with a trill, tremolo or turn (e.g. notes alternatingquickly back and forth). The contour could be consid-ered static in this case as opposed to varying along withthe trill. Furthermore, in a fast passage, notes may lo-cally be going up and down, but the overall shape mightbe a gradually rising progression, so that the musicalcontour is rising. This motivates a weighted moving-average filter, which also allows the contour value tobe expressed as a function of frame number. That is, acontour value is expressed even when there is no notebeing played. The resolution of the interpolation isdetermined according to the sampling rate of the mo-tion capture data, so that each motion frame has a cor-responding note event. We can then compute the fol-lowing characteristics:

1. Pitch samples. Once the pitches have been pro-cessed as described above, processed pitch valuesp(ti) can be sampled as needed fromW. This inputparameter describes the general contour of the musicin windowW.

2. Mean and variance of pitches. Statistics such asmeanµp and varianceσp of the pitch contourp(t)can be calculated for each of the windowsWj . µp

can be considered as a local phrase centre (i.e.,around which register the notes occur) whileσp

gives us the idea of how quickly the notes go up (ordown) in the given window.

3. Pitch-peaks. Just as peaks are salient features of mo-tion, they are salient features of melodic contour,and therefore a pitch-peak variable is used to flagwhether the current event is (close to) a local maxi-mum (1), or a local minimum (-1), and 0 is assignedotherwise.

4. Duration to next/previous closest peaks. A peak maybe perceived to be more significant if it is more iso-lated from any other nearby peaks. By includingthese parameters, one can perceive whether or notthe peak is isolated from others or just one of manypeaks happening in a short period of time.

5. Phrase endings and duration of phrase. Althoughalgorithms to detect phrase endings are widely avail-able, a very simple procedure of finding two consec-utive notes with the inter-onset time larger than athreshold was used in this study. This method is in-spired by Pachet’sContinuator[Pac02]. The thresh-old is typically based on the average inter-onset timeof all the notes.

6.3 Loudness CharacteristicsLoudness information is processed differently from thepitch information. Firstly, we assume that when anote is played, it decays continuously while the key ispressed, and then as soon as the key is released, the notestops. This phenomenon of decaying sound is observedin many of acoustic instruments such as piano and gui-tar. For non-decaying instruments, we have found it tobe sufficient to simply keep loudness constant while thekey is being pressed.

For each time frame, we compute a sum of the loud-ness of all active notes. The loudnessl j(t) of a jth noteat timeti is given by:

l j(ti) = l j(t j 0)κ(ti−t j 0) (7)

wherel j(t j 0) andt j 0 are the initial loudness and NOTEON time of the jth note, respectively, andκ thedecayconstant (κ < 1). Thiscumulativeloudness is rele-vant because it captures articulation of notes as well asevents that are important for realization of the loudness.For example, in the events such as trills and tremolos,the overall loudness is kept near constant as if a sin-gle note ringers without decaying even though thereare many notes played one after another. In anothercase when a group of notes are played simultaneously(e.g. chords), the music is usually louder than one-linemelodies. Figure 9 shows the result of the cumulativeloudness.

Once we have obtained the cumulative loudness ofthe music, we then extract the information in a mannersimilar to the process of pitch information. (i.e. loud-ness samples, mean and variance, peaks, and durationto neighbouring peaks.)

6.4 DensityAt each timeti , thedensityρ(ti) of notes is computedfor each of the three windows leading up to that time.Let the number of notes in a windowWk(ti) and the timeduration ofWk(ti) be, Nk(ti) and∆Tk(ti), respectively,ρk(ti) is computed as:


Figure 9: Cumulative loudness processed from MIDIdata

ρk(ti) =Nk(ti)∆Tk(ti)

(8)

This information describes how many notes areplayed in a given window, hence, it gives us a generalidea of how fast the music is played. Because of thefiltering of the pitch information and the cumulationof the loudness, the information of the note density issomewhat lost without explicitly providing it.

7 LEARNING CHARACTERISTIC RE-LATIONSHIPS

Once the various characteristics have been selected andextracted, a standard neural network model, with tanhfunctions in the hidden layer, is used to learn relation-ships based on the provided example data. Other non-linear models could be also be used.

Our system predicts two types of characteristics ofthe output motion: temporal and spatial characteristics.The input data is based on a series of music tracks, eachwith a set of corresponding recorded motions. The in-put data is processed as described above, so that eachtime slice of the music contributes one training exam-ple. For each frame at timeti , let u(ti) represent theset of musical characteristics computed as described inSection 6. One model is trained to predict the probabili-ties peak(ti) as described in Section 5 given inputu(ti).The other model learns to estimate the spatial charac-teristics of the distribution, also described in Section 5.

The temporal model has 22 input parameters (Table1) and 1 output parameter (Table 3), which is the prob-ability peak(t) indicating the likelihood of the frame attime t being a peak in the motion output. The spatialmodel has 41 input parameters (Table 2) and 4 outputparameters,µm(t), σm(t), µv(t), andσv(t) (Table 3),which are used to form distribution functions. We nextexplain how these output parameters are used to gener-ate motion curves.

8 GENERATING MOTION CURVES

The generative procedure begins by analyzing a newmusical input track, and using thepeak(t) estimator(predicted by a sigmoidal unit, which can be interpretedas a binary probability value) to stochastically select

# Parameter1-10 Time distances to the last 10 pitch peaks11 Time distance to the next pitch peak

12 - 21 Time distances to the last 10 loudness peaks22 Time distance to the next loudness peak

Table 1: Input parameters for the temporal predictor

# Parameter1 - 20 Pitch/velocity samples21 - 23 Note density in each window24 - 29 Mean/variance of pitch in each window30 - 35 Mean/variance of loudness in each window36 - 37 Flags to indicate peaks in pitch/loudness38 - 39 Time duration of peaks

40 Flags to indicate phrase endings41 Time duration of phrases

Table 2: Input parameters for the spatial predictor

# Parameter1 µm(t)2 σm(t)3 µv(t)4 σv(t)5 peak(t)

Table 3: Output parameters

certain times as moments where motion peaks can oc-cur. The system then goes through all the candidatepeak points to make sure that the consecutive peaks arenot too close in time. Whether or not they are too closeis determined based on the human physical ability. Themaximum values of velocity and acceleration are ob-tained from the example motions. If candidate motionpeaks are detected to create movements with higher val-ues than these minimum values, one of them is excludedfrom the list of peaks. A common previous approach tocreate animation to music is to assume the beat informa-tion is embedded in the MIDI file [KPS03, WLXS02],which also requires an unwavering tempo throughoutthe piece, or by using beat extraction algorithms suchas [Meu02]. However, restricting dance motions syn-chronizing only to beat is merely one aspect of chore-ography, and the movements seen in dance are not nec-essarily accented only by the beat factor. Furthermore,consider music playedrubato(freedom of tempo) as of-ten found in classical and jazz music, or music in tempobut that has tuplets (e.g. triplets, quintuplets). In thesecases, synchronizing the motion with the beat is onlyone of the possible options; having motion that can besynchronized with articulation of music itself is equallyimportant.

Once the peaks have been selected, then for eachpeak point at timeti , the system computes an exact loca-tion of the hand as follows. Letµm(ti),σm(ti),µv(i), and


σv(i) be predicted by the model according to the musiccharacteristicsu(ti) at that time. A local motion cen-tre and amplitude fordist(ti) are chosen by samplinga gaussian distribution with these parameters. Notethat thedist(t) function specifies distance of the arm,but does not indicate the direction in which the distanceshould be chosen. While this is an additional character-istic that could be predicted by the system (discussed inSection 10), we currently randomly set several primaryaxes of motionθ1,θ2, . . . ,θr for each character’s handsasmeandirections, and then for any given peak atti ,we randomly choose one of them,k and sample froma normal distribution centered atθk. Together, the axisand distance specifies a point relative to the character.

A quartic polynomial is used to interpolate betweenthe generated peak points. First derivatives are con-strained to be 0 at the peaks, and second-derivative con-tinuity is enforced at the join points.

The orientation angles at the characters’ links arecomputed using an IK algorithm.

9 RESULTSThe model was trained on a data set consisting of sev-eral pieces of music, each one a few minutes long, to-gether with multiple corresponding samples of recordedhand motion for each piece. Specifically, the averagetraining music sample had 3000-4000 frames, of whichwe sampled every 10 frames, and used 8 recorded mo-tions per piece, giving roughly 2400 data samples perpiece. New pieces of music were then given to the sys-tem, which were automatically analyzed, and distribu-tion parameters were predicted for the correspondingmotions. Sampling from these parameters and interpo-lating, as described in above, allowed the creation ofmultiple motions, sharing certain characteristics, whilestill differing in the details due to the stochastic natureof the generative process (see the accompanying videos,with sample frames shown in Figure 10). The musicalpieces can range from being highly rhythmic, to hav-ing rhythmic patterns that shift from slow and rubato tofast, demonstrating that the characters respond appro-priately to these complex changes. Testing the systemwith various pieces shows that the motion can relate tothe music even when there is no steady tempo, while atother times it can correspond to certain clear accents inthe music. Note that, for each test, the motions of onlyone dancer were used, so the results corresponded to thepatterns specific to that set. This is, indeed, the goal ofthe system: to learn a mapping based on the motions ofa particular choreographer; the system is not intendedto learn a general “all-purpose” mapping.

10 CONCLUSION AND FUTUREWORK

We have demonstrated a novel approach to creating an-imated motion to new musical scores for virtual ac-

Figure 10: Screenshots of hand motions createdstochastically for a new piece of music (see accom-panying video).

tors. We have proposed a set of key features of musicand motions, and shown how to extract these features.Rather than adapting existing recorded dance motion,we use recorded motions synchronized to music sound-tracks, and automatically analyze salientcharacteris-tics of both the motion and the music. A non-linearparametric model is then used to learn the relationshipbetween these characteristics by example, rather thanby requiring an animator to specify mappings explicitly.Once the training data has been specified, then the sub-sequent musical analysis and stochastic generative pro-cess are automated procedures, and thus can be appliedin any context where an autonomous animated agent isacting.

Finally, while our present work focuses on animat-ing motion of hand and arm, the same ideas could beextended to include a range of body motion, such asleg and torso motion. In this case, unlike the some-what free movements of hands, additional constraintswould need to be incorporated (e.g. ground contact) inorder to maintain physically plausible postures. We ex-pect that further extensions of this sort will contributesignificantly to the naturalness of the final synthesizedmotion.

ACKNOWLEDGEMENTS

We would like to thank Dirk Arnold and the anonymousreviewers.

REFERENCES[ABB04] Gazihan Alankus, A. Alphan Bayazit, and O. Burchan

Bayazit. Automated motion synthesis for virtual chore-ography. Technical Report WUCSE-2004-66, Depart-ment of Computer Science and Engineering, Washing-ton University in St. Louis, St. Louis, MO, 2004.

[CBBR02] Marc Cardle, Loic Barthe, Stephen Brooks, and PeterRobinson. Music-driven motion editing: Local mo-tion transformations guided by music analysis. InEu-rographics UK Conference (EGUK), pages 38–44, Le-icester, UK, June 2002.


[ES03] George ElKoura and Karan Singh. Handrix: Animat-ing the human hand. InACM SIGGRAPH / Eurograph-ics Symposium on Computer Animation, San Diego, CA,2003.

[GBBM96] Christian Greuel, Mark T. Bolas, Niko Bolas, and Ian E.McDowall. Sculpting 3d worlds with music; advancedtexuring techniques. InIST/SPIE Symposium on Elec-tronic Imaging: Science & Technology, The enginerringReality of Virtual Reality III, 1996.

[GM95] Masataka Goto and Youichi Muraoka. Interactive per-formance of a music-danced cg dancer. InWorkshop onInteractive Systems and Software (WISS) ’95, 1995.

[KGP02] Lucas Kovar, Michael Gleicher, and Frdric Pighin. Mo-tion graphs. InSIGGRAPH ’02: Proceedings of the 29thannual conference on Computer graphics and interac-tive techniques, pages 473–482, New York, NY, USA,2002. ACM Press.

[KPS03] Tae-Hoon Kim, Sang Ill Park, and Sung Yong Shin.Rhythmic-motion synthesis based on motion-beat anal-ysis. ACM Transactions on Graphics, 22(3):392–401,2003.

[Lab88] Rudolf Laban. The Mastery of Movement. NorthcoteHouse, London, 4 edition, 1988. Revised by Lisa Ull-man.

[LL05] Hyun-Chul Lee and In-Kwon Lee. Automatic syn-chronization of background music and motion in com-puter animation. InEUROGRAPHICS 2005, volume 24,September 2005.

[Lyt90] Wayne T. Lytle. Driving computer graphics animationfrom a musical score. InScientific Excellence in Super-computing, The IBM 1990 Contest Prize Papers, vol-ume 2, page 644, Ithaca, NY, 1990.

[Meu02] Benoit Meudic. A causal algorithm for beat-tracking. In2nd Conference on Understanding and Creating Music,Caserta, Italy, November 2002.

[NF03] Michael Neff and Eugene Fiume. Aesthetic edits forcharacter animation. InACM SIGGRAPH/EurographicsSymposium on Computer Animation, 2003.

[Pac02] François Pachet. Interacting with a musical learning sys-tem: The continuator. InICMAI ’02:Proceedings of theSecond International Conference on Music and Artifi-cial Intelligence, pages 119–132, London, UK, 2002.Springer-Verlag.

[PN03] Vincent Penasse and Yoshihiko Nakamura. Dance mo-tion synthesis with music synchronization. InRoboticsSociety of Japan 2003, volume 21, 2003.

[SNK01] H.-H. Shih, S. S. Narayanan, and C.-C. J. Kuo. Au-tomatic main melody extraction from midi files with amodified lempel-ziv algorithm. InInternational Sympo-sium on Intelligent Multimedia, Video Speech Process-ing, 2001.

[WLXS02] Tianshu Wang, Yan Li, Ying-Qing Xu, and Heung-Yeung Shum. Learning kernel-based hmms for dynamicsequence synthesis. In10th Pacific Conference on Com-puter Graphics and Applications (PG’02), 2002.


learning to synthesize arm motion to music by example · 2020. 7. 2. · various work...

Documents