New Algorithm

Using the results above the following algorithm for the estimation of the sources,

delays and mixing matrix can be derived:

0ij

n

jjiji sx

1

222 )(||||)(||

ix

Solutions can be obtained by different methods like positive PCA , nonnegative

matrix factorisation or positive ICA .

2. Initialization of the delays:

js

2a) Numerical solution of

ijj

n

jjijii ssxx

)arg(||||)arg(||

1

222

for the term )arg( js

; integration to obtain

2b) Exploiting the knowledge of the sources, the mixing and

delay matrix are updated by optimization of the cost function:

2

,

||)(||minarg, jjj

A

jj SAxA

jj

kijkikj tsS ,))((:)(

kjkj aA )(:

2c) Update and go back to 2a), until convergence. ij

Iteration of the following two steps:

H. Hicheur1, L. Omlor2, A. Barliya3, M. Ousov-Fridin3, U. Maoz3, C. Roether2, J. Grezes1, L. Yahia-

Cherif1 ; B. de Gelder4,, M. Giese2, A. Berthoz1, T. Flash3

(1 LPPA, CNRS-Collège de France, Paris, France; 2ARL, Hertie Institute for Clinical Brain Sciences,

Tübingen, Germany; 3Dept. of Computer Science, Weizmann Institute, Rehovot, Israel; 4Dept. of

Psychology, Tilburg, Netherlands)

Facial expressions of emotions have received great attention in research over the past century. Fundamental components of

emotional facial expressions have been characterized in the physiological and psychological literature, and have been used

as basis for the synthesis of facial expressions in computer graphics and robotics. By contrast, the bodily expression of

emotions has been much less systematically studied.

In the current consortium, we initiated a multidisplinary approach (combining classical methods from kinematic analysis,

machine learning approaches -blind source separation and sparse feature learning- and approaches from differential

geometry) where we aim to provide an integrative view on the bodily expression of emotions during human gait. The

expected results would likely be useful for the realization of convincing emotional body movements in computer graphics

and robotics.

Building of a kinematic data-basisWe have studied the motor behavior for the expression of emotions by full-body movements. Body movements of

experienced actors expressing different emotions (Anger AN, Sadness SA, Joy JO, Fear FE and Neutral NE) were recorded.

Figure 1: Example of the markers-set used in part of the experiments in the motion capture room

(right). From the 3D coordinates of these markers, the body segments were computed

(stickdiagrams on the left).

A total of 13 actors were asked to express 5 types of emotional gaits and the movements of

around 40 body markers were recorded at a 120 Hz sampling frequency using a Vicon motion

capture system equipped with 24 infrared cams. Part of this database was used in order to

analyze, at both the behavioral and computational levels, how a specific emotion affect the

locomotor behaviour.

Similarities in the perception and the

production of emotional gaitsHere, we quantitatively examined the effects of a particular emotion on the locomotor behaviour. This was realized through

measurements of step parameters changes (step length, duration, width…) as well as by analyzing the spatial specificity

(which particular body segment) and the temporal specificity (when during the step cycle) of a particular emotion. In

parallel, we used recorded movies from actors in order to assess how well naïve subjects recognize an emotion from a

particular movie. We designed a simple psychophysical experiment where subjects had to press one of 5 screen buttons

(corresponding to the number of tested emotions) when he/she perceived a particular emotion from the displayed movie.

As illustrated in figure 2, we observed a pairing between the emotions Joy and Anger on one hand, and between the emotions Sadness and Fear on

the other hand, compared to the Neutral NE condition. This pairing was evident both from general gait parameters (Fig. 2, left) and from the angular

motion properties of the lower body (Fig. 2, right). This kinematic similarity between the two groups of emotions was associated with perceptual

ambiguities (Fig 2., middle): subjects tend to misperceive one particular emotion of a given group (for example FE) for another emotion of the same

group (f.ex. SA).

Figure 2: Left) Step parameters Middle) Recognition performance of naive subjects Right)

angular motion properties of some body segment, across the different emotions

Computational analysis of the motor patterns of

human bodily expression of emotions

Static Body Expression of Emotion

To develop a computational model of body expression of emotion (BEE) based on static postures and to investigate the nature of BEE.

We use a heuristic approach to model BEE, assuming that BEE can be considered as a complex behavior composed of a set of simple

building blocks called primitives. The composition follows combinatorial syntax rules in order to form a complete representation.

Four basic emotions were considered: Joy, Sadness, Anger, and Fear.

The input material consisted of still mages derived from video films in which professional actors and ordinary subjects freely portrayed

body postures expressing emotions. To extract basic features, the photographs were subjected to image processing procedures (body

segmentation from the background; building a multi-joint human body model; labeling body parts; estimating head position and

gesture).Algorithm and Training

Using the different extracted features (f) , e.g., head and hand postures, configurations of different body parts, a

computational algorithm to calculate the maximum mutual information (I(f;C)) between the different features and classes of

emotions (C) was developed. Features with the highest mutual information scores were defined as primitives. Then,

combination synergies were defined as consisting of sets of primitives where additional mutual information of the sets

with the different emotions was found. Following the training, sets were sorted based on these mutual information scores

and those (one or many) with the highest mutual information were defined as emotion-specific over-all body

configurations.

Locomotion and emotion: inter segmental coordinationHuman locomotion is a well coordinated activity in which the elevation angles of leg segments: thigh, shank and foot are

constrained by the intersegmental law of covariation. The elevation angles do not evolve independently during the gait

cycle but when plotted with respect to each other, they are constrained to a plane and form the shape of a “tear-drop”.

Thus, reducing the actual degrees of freedom of the leg to two (Borghese et al. 1996).

ModelWe have developed a mathematical model that describes the

properties of the plane, accurately predicts its orientation and

rotation with a change of gait speed. The model relies on the

ability to well describe each elevation angle with one sinusoid:

1 1( ) sini ii i iAt a t The model then predicts that the orientation of the plane would be

defined by the expression:

2 3 2 3

1 3 1 3

1 2 1 2

sin( )

sin( )

sin( )

A A

n A A

A A

Elevation vs. Anatomical Angles•The planar constraint holds only for the elevation angles.

• It is insufficient to describe the anatomical angles with only one

sinusoid and higher harmonics have to be introduced in order to

obtain good results.

• The different elevation angles of the leg are characterized by equal

natural frequencies of the sinusoids which immediately implies

planar covariation according to the model. In the case of anatomical

angles it is not as simple.

• In case of anatomical angles, it is commonly observed that higher

harmonics appear with higher energy than lower ones.

•There is no simple transformation between the two angular

representations.

Emotion

Elevation Angles’ Profiles

Angry

Happy

Neutral

Sad

Fear

The plane of intersegmental covariation holds for all emotions. Changes appear in the description of the harmonics.

Sinusoidality Measures

Conclusions

The model predicts the properties of the plane taking into account only the amplitudes and phases of the segments‟ elevation angles. The

emotions differ in the amplitudes, and the ratios between the amplitudes of the first 2 harmonics. The time spent during the swing and

stance phases and the orientation of the plane are also distinguishable. These parameters are directly related to energy expenditure. This

topic will be further investigated.

Goal

Results -Joy (Joint Angles)

MIMax-Min

(1) Flash T., Hochner B. (2005) Current Op. in Neurobiology 15:660-666

(2) Maoz, U. Portugaly, E, Flash, T. and Weiss Y. (2005) NIPS.

(3) Barliya A, Rother C.L., Giese M.A., and Flash T. (2006) CMCW II, BGU, Israel.

(4) Flash T, Handzel A.A. (2007) Biological Cybernetics 96(6):557-601

(5) Ousov-Fridin M., De Gelder B., and Flash T. (2007) CMCW III, BGU, Israel.

We gained insight into the nature of BEE, by developing a system that can recognize

emotion from various sets of extracted features. The recognition rates were as low as

46% looking only at the body silhouette, and as high as up to 70%, when more

informative primitives were added. We will further investigate whether these

primitives are also informative in human BEE perception and recognition processes.

Conclusions

1. First Primitive

2. K-primitive:

max additional information 1 1argmax ( ; | )k kF I C F B

1 argmax ( ; )F

F I C F

The converging evidence obtained with these methods suggests the existence of defined emotion-specific components in

motor behavior. Psychophysical experiments suggest also a relevance of these components for the visual perception of

emotions from body movements. The obtained results are likely useful for the realization of convincing emotional body

movements in computer graphics and robotics. Ongoing work studies the physiological basis of such components by means of

electro-myographical recordings.

Supported by a HFSP grant « The bodily expression of emotions »

General Conclusion

These experimental observations are now completed by a theoretical work devoted to the extraction of emotion-specific

locomotor primitives.

Published work (in relation with the HFSP program):

Hicheur, H., Terekhov A.V., Berthoz A., Intersegmental coordination during human locomotion: does planar covariation of elevation angles reflect

central constraints?, Journal of Neurophysiology, (2006), 96:1406-1419.

Computational approach

Blind Source Separation

Phase shifts (unknown)

Observed Data

Linear weights (unknown)

Sources (unknown)

)()(1

ij

n

jjiji tstx

• New algorithm for estimating over-determined anechoic mixtures.

• Better performance than other existing methods for tested synthetic and natural data sets.

• Accurate reconstruction of gait trajectories with very few sources.

• Extracted motor components seem to be relevant for visual perception of emotions from gait

Motivation•Kinematic redundancy: large number of degrees of freedom (DOF) makes the inverse kinematics

problem ill-posed.

•Bernstein [2] proposed that control might be simplified by coupling of DOFs; small number of

functional control units Motor Synergies!

•EMG Patterns can be approximated by linear combinations of few underling source signals [1,3,4].

•Motor primitives as approach to solve the DOF problem in robots and humans [7].

Comparison with Psychophysical Results on Emotion Recognition

Estimated mixing weights for emotional walking were decomposed into a component that characterizes neutral

walking aneutral , and a specific part Daj for emotion j. Emotion-specific parts estimated by sparse linear regression:

Critical emotion-specific kinematic features, extracted from the

(motor) trajectories, match the features that are relevant

for visual recognition.

Features extracted with other demixing methods (e.g. PCA)

fail to match important features for visual recognition.

-*-*

Mixing weights Daij: emotional - neutral

New Algorithm

Important features for the visual perception of emotional gaits have

been identified in psychophysical experiments [5,6,8]. Comparison

with emotion-specific weight components shows an almost perfect

match. (±Symbols indicate increased (+) or decreased (-) angle

amplitude compared to neutral walking, if feature was critical for

emotional expressiveness in visual recognition; -*: feature found in

own psychophysical experiments.)

aj aneutral + Daj = aneutral + Cj

•Temporal coordination between joints suggest a new unsupervised learning model including time delays.

Over-determined Anechoic-Mixture Model:

Conclusions

(1) D„Avella A, Bizzi E (2005). PNAS 102(8): 3076-3081

(2) Bernstein, N. A. (1967). The Co-ordination and Regulation of

Movements, Pergamon Press,Oxford.

(3) Flash T., Hochner B. (2005) Current Op. in Neurobiology 15:660-666

(4) Ivanenko,Y.P. et al. (2005) J. of Neuroscience 25(31):7238-7253

(5) Montepare, et al Nonverb. Behav. Vol 11 (1).

(7) Schaal S., Schweighofer S. (2005). Curr. Op. in Neurobio. 15:1-8

(8) Wallbott, et al (1998). Europ. J. of Social Psych.,Vol 28:879-89

Wilhelm Busch

Schmied und

Teufel

1. Computation of the Fourier transforms of the data and solution of

the factorization equation:

Published work (in relation with the HFSP program): Omlor L, Giese M A (2006) Unsupervised learning of spatio-temporal primitives of emotional gait. In: Perception and

Interactive Technologies 2006, Lecture Notes in Artificial Intelligence 4021, Springer, New York, pp. 188-192.

Omlor L, Giese M A (2006) Extraction of spatio-temporal primitives of emotional body expressions. Neurocomputing, in press.