modeling expressivity in ecas

1

Modeling Expressivity in ECAs

Catherine Pelachaud,Maurizio Mancini

LINC - University of Paris 8

Behavior Behavior is related to the (Wallbott, 1998):

quality of the mental state (e.g. emotion) it refers to

quantity (somehow linked to the intensity factor of the mental state)

Behaviors encode: content information (the ‘What is

communicating’) expressive information (the ‘How it is

communicating’)

Behavior expressivity refers to the manner of execution of the behavior

Behavior Representation

Behavior= signal shape, movement, expressivity

Gesticon (Gesture Lexicon): Dictionary of behavior description (B. Krenn, H. Pirker OFAI)

Modalities: face hand and arm gesture body movement and posture gaze

Behavior Representation Face:

facial expression duration: onset, apex, offset

Gesture: phases: preparation, pre-hold-stroke, stroke, post-hold-

stroke, retraction gesture shape and movement for each phase

Head: head direction head movement

Gaze eye direction

Expressivity Dimensions

Expressivity dimensions: Spatial: amplitude of movement Temporal: duration of movement Power: dynamic property of movement Fluidity: smoothness and continuity of movement Repetitiveness: tendency to rhythmic repeats Overall Activation: quantity of movement across

modalities

Implemented for gesture and facial expression

Overall Activitation

• Threshold filter on atomic behaviors during APML tag matching

• Determines the number of nonverbal signals to be executed.

Spatial Parameter

• Amplitude of movement controlled through asymmetric scaling of the reach

• Space that is used to find IK goal positions

• Expand or condense the entire space in front of agent

Temporal parameter

Stroke shift / velocity control of a beat gesture

Y p

osit

ion

of w

rist

w.r

.t. s

houl

der

[cm

]

Frame #

• Determine the speed of the arm movement of a gesture's meaning-carrying stroke phase

• Modify speed of stroke

Fluidity

• Continuity control of TCB interpolation splines and gesture-to- gesture• Continuity of arms’ trajectory paths• Control the velocity profiles of an action

coarticulation

X p

osit

ion

of w

rist

w.r

.t. s

houl

der

[cm

]

Frame #

Power

• Tension and Bias control of TCB splines;• Overshoot reduction• Acceleration and deceleration of limbs

Hand shape control for gestures that do not need hand configuration to convey their meaning (beats).

Repetitivity

• Technique of stroke expansion: Consecutive emphases are realized gesturally by repeating the stroke of the first gesture.

Expressivity

Expressivity values may act: over the whole animation: EmoTV, analysis-synthesis every instant of the movement: Greta Music on every gesture: GEMEP corpus on a particular phase of the gesture: Attract attention

study

Exploratory studies based on various data types: acted data real data 2D cartoon literature

Research Issue

Behavior representation what to encode at which levels of representation dynamism

Implementation refinement

Expressivity over the WHOLE animation

One set of values are set extracted manually from annotation of

real data video corpus

extracted automatically from video corpus of acted data using image analysis technique

From annotations to animation Jean-Claude Martin, Laurence Devillers, LIMSI-CNRS; Maurizio Mancini, Paris8

Consider what is visible: annotate signals, how there are displayed, how they are

perceived Model what is visible:

represent signals, animate them with expressivity Two-steps approach

Elaborate rules by analysis (video corpus) Animate by “copy synthesis”

No model of the processes underlying the display of the signals

Annotation extraction animation


EmoTV: 51 clips French TV Interviews Annotation

Emotion labels: single and blend of emotions Multimodal behavior Expressivity dimensions

AnnotationAnnotation

StepsSteps

ExtractionExtraction GenerationGenerationAnimationAnimation

EmoTVEmoTVclipclip

GRETAGRETAanimationanimation


Expressivity parameter Anger DespairAngerDespair

from annotations

Temporal Extent 1 -1 1

Fluidity -1 1 0.58

Power 1 -0.5 0.11

Repetition 1 -1 1

values obtained from literature

values obtained from annotation

Expressivity of gestures in mixed emotion


Real and Virtual World SensingA. Raouzaiou, G. Caridakis, K. Karpouzis ICCS; C. Peters, E. Bevacqua, M. Mancini, Paris 8

Real World Sensing

Virtual Sensing

SensoryStorage

Perception

Generation

Attention Planning

Interpretation

Personality

SceneOntology

Goals


Expressivity over

the WHOLE

animation

ApplicationScenario

Interpretation: gesture specified by symbolic name facial expression:

emotion label: if the facial expression corresponds to one of the prototypical facial expression of emotions

otherwise, FAPs values

Planning: modulate expressivity parameters module emotional expressions


Generation Input to ECA system:

a symbolic description of a gesture emotion label or set of FAPs value expressivity parameters value

Output: facial and gesture animation


Expressivity on Every Frames

Greta Music Roberto Bresin, KTH - Maurizio Mancini,

Paris8

One set of expressivity paramters values are extracted automatically from acoustic data in real-time and fet to the ECA system.

Design a tool for real-time visual feedback to expressive performance


From music expression to facial expression

From acoustic cues to emotion : extraction of acoustic cues:

Tempo, Sound Level, Articulation (staccato/legato), Attack Velocity, Spectrum, Vibrato rate, Vibrato Extent, Pitch

From acoustic cues to facial expression: mapping of acoustic cues:

music emotion facial expression music volume spatial and power music tempo temporal and overall activation music articulation fluidity


Music version of Greta

Input:Expressivity Parameters

Output:FAP values (animated head)

This version of Greta allowsonly the following actions:

• head moving• eyes blinking• emotional expression• skin colouring


Video


Attraction of attention Corpus: videos from traditional animation that

illustrate different types of conversational interaction

Analysis: the modulations of gesture expressivity over time play a role in managing communication, thus serving as a pragmatic tool

France Telecom

Expressivity on Gesture Phases

Attraction of attention Irregularities

the principle of anticipation: it enhances the visibility of a gesture it enhances our propensity to gaze at this gesture.

Discontinuities create a contrast between successive

gestures function to isolate a particular gesture from a

sequence of gestures France Telecom


Irregularity


Irregularity – slow motion


Discontinuity – slow motion


Application: ECA as web presenter

Discontinuity – spatial parameter


35

Annotation of multimodal behavior signals on 3 modalitiesmodalities

arm gesturehead movementbody movement

phases of each signaleach phase: physical shape + timing

expressivity of each signal

Expressivity on Each Gesture

36

Annotation of multimodal behavior


37

Animation format

we have defined a file format for the specification of behavior for our animation engine (Greta)

we translate from the XML annotation file (ANVIL) to the engine animation file

ANVILANVILannotationannotation

animationfile


38


Animation format

41

Demo

demo!

modeling expressivity in ecas

Documents