ron artstein dusan jan steve solomon david traum filedata analysis cuny, georgetown, u pitt, cmu...

File Identifier R_pos [red, positive]O_pos [orange, positive]Y_pos [yellow, positive]B_pos [black, positive]W_rev [white, reverse]

Example: tns_logo_right_W_rev.eps

9

On photographic background, The NewSchool university logo can be used in color,black and white versions. The demonstra-tions at left show which colors look best ondark backgrounds and which work better onlight backgrounds. Because every image isdifferent, be sure to choose a color with astrong contrast to the background. Place thelogo in an area of the image that is not busyin order to enhance legibility. (See page 10for “don’ts”.)

WhiteCMYK 0/0/0/0RGB 255/255/255

The New School RedPantone® 1795 CCMYK 0/94/100/0RGB 255/19/0

The New School OrangePantone® 158 CCMYK 0/61/97/0RGB 255/99/9

The New School YellowPantone® 143 CCMYK 0/35/85/0RGB 254/166/32

BlackPantone® Black 4 C*CMYK 0/22/100/89RGB 28/22/1

University logo

Color: on photography

Whiteon dark background

The New School Redon dark background

The New School Orangeon dark background

The New School Yellowon dark background

Blackon light background

The New School Redon light background

The New School Orangeon light background

The colors shown throughout thismanual have not been evaluated byPantone, Inc. for accuracy and maynot match the PANTONE ColorStandards. PANTONE® is a registeredtrademark of Pantone, Inc.

*See pages 47–49 for a completeexplanation of the file naming structure.

*100% Process Black may be substi-tuted in restrictive printing instances.

Dialogue modeling and analysis for virtual humans

Ron Artstein Dusan Jan Steve Solomon David Traum

Institute for Creative Technologies, University of Southern California



9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Theory Formation

Identify Cultural Factors CUNY, Georgetown, CMU

Computational Models CMU, USC

Virtual Humans USC

Implementation CMU

RESEARCH PRODUCTS

Surveys & Interviews CUNY, CMU, U Mich, Georgetown

Cross-Cultural Interactions

U Pitt, CMU

Data Analysis CUNY, Georgetown,

U Pitt, CMU

validation

validation

validation Validated Theories

Models

Modeling Tools

Briefing Materials

Scenarios

Training Simulations

Common task

Subgroup task

MURI 14 Review — September 10, 2009



9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












1 Culture in Virtual Humans

2 Modeling and parameterizing culture

3 Dialogue analysis




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Virtual Humans

9

ICT Conversational Virtual Humans

Transitioned Training

Technology Advanced Research

Dr. Perez SASO-ST, SASO-EN

Elder-Al-Hassan

SASO-EN

Group Conversation Characters

C3IT Cultural training

Justin Virtual Patient

Hassan Tactical Questioning

Natural Language Dialogue

Processing

Sgt. Star

Sgt. Blackwell

Justina Virtual Patient

Clinical Pysch

Diagnosis Be a Reporter

Sgt, Medic Mission Rehearsal

Exercise Ahmad Wani Sudan

Information Activeworlds

Amani & Mohammed

Tactical Questioning

Question- Answering Characters




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Virtual Humans in the study of culture

$'? ?

?

? ?

$�

Computational theory Ethnographic studies

Behavioral models

implementation

VIRTUAL HUMANS

Perception and interactionstudies

validation andrefinement

Transitioned technologytraining, entertainment, . . .




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Working definitions of culture

Culture = set of rulesNormative rules: how people in particular roles shouldbehave

Communicative rules: what behavior signifies whichmeanings in which contexts

Inferential rules: what new knowledge can be derivedfrom existing knowledge

Rules used in multiple waysNorms identify compliance, identity, group membership

Communicative rules produce meaningful behavior,interpret behavior of others




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Aspects of cultural differences

Varying dimensions: Appearance, behaviors, meanings,contexts, frequencies, mapping from behavior and contextto meanings

Culture-specific communicationVerbal communication: Phonemes, morphemes, words,sentences.

Non-verbal behavior: Proxemics, gaze, facial expressions,body postures, hand gestures, prosody and intonation.

Social interaction: Turn-taking, greetings & closings,sequential interaction, grounding, boosting & downplaying,ritualized behavior.




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Culture for Virtual HumansVirtual Humans as members of a culture

Cultural knowledge and goalsRoles in the cultureBehave according to cultural rules

Virtual Human culture modelsExternal model: impression of being from a culture

AppearanceBehavior — performance, frequency, and context

Internal model: interprets the world along a cultural modelIdentity & roleInterpretation of behaviors

Perception not necessarily on the intended dimensionCulturally appropriate behavior perceived from outside asaggressive/subdued personality




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo














3 Dialogue analysis




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Group conversational behaviors (ICT & UTEP)

PhenomenaProxemics

Ara

bA

mer

ican

Gaze in conversational rolesSpeaker, addressee, listener . . .

Silence/Overlap at turn boundariesTime offset between speakers

Cultures(Levantine) Arab

Mexican

Anglo-American




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Proxemics model

Social force model for positioning and movementFspeaker : attractive force towards speaker

Fnoise : repelling force from outside noise

Fconvex : force towards convex hull of all participants

Fproximity : repelling force from characters that are too close

Culture Social Zone (meters)

Anglo-American 1.2–2.7(Hall 1968)Mexican 1.0–2.0Arab 0.7–1.5




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Perception of group behavior

ExperimentIdentify group behavior that models a culture

20 subject from each culture6 movies, 2-minutes eachRate for realism relative to the culture in which they grew up

ResultsArab subjects reliably identify Arab proxemics modelMexican, Anglo-American do not reliably distinguishbetween groupsNo reliable discrimination based on gaze and turn taking




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Culturally Affected Behavior

Explicit, modular cultural component

Overlaid modelsTask: Police captain / doctor

Socio-cultural: German / Iraqi / Japanese

EvaluationEvaluating a Framework for Representing CulturalNorms for Human Behavior ModelsBest paper award, BRIMS 2009




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Sample cultural model (Iraqi Sunni)

3

agent’s-view-that-agent-is-familiar-with-participant intrinsic utility:150, current utility: 0

Sample of Culture Model for Iraqi-Sunni Culture

agent’s-view-of participant’s-view-that-agent-is-respectful-of-modesty

intrinsic utility: 165, current utility:165

invite-agent-to-play-poker

show-picture-of-wife agent’s-view-that-agent-is-respectful-of-modesty


agent’s-view-of-participant’s-view-that-agent-is-observant-of-Islam


agent’s-view-that-agent-is-observant-of-Islam intrinsic utility: 100, current utility:100

agent’s-view-that-agent’s-face-is-saved intrinsic utility:150, current utility: 0

talk-about-cars-and-driving

give-alcohol

+0.1

+0.1

+0.05

+0.1

-0.5

-0.5 -0.95

-0.95

-0.1

-0.6 -0.6




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Evaluation: learning through interaction

Cultural model based on Appalachia Scots-Irish

Method

Phase Experimental group(interactive)

Control group(paper, no responses)

1. Scripted Perform scripted actions Read scripted actions2. Guided Choose & perform actions Choose & mark actions3. Rating Rate all possible actions for cultural appropriateness

ResultsOnly experimental group canidentify culturally appropriateactions. 1

23

45

Condition

Mea

n ra

ting

●

●

Control Experimental

Inappropriate

Appropriate




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Tactical questioning characters

Training for human intelligence collectors

Interview / question-answering

Complex behaviorCharacter requests assurancesWithholds sensitive informationMay lie

Modular authoringDomainDialogue actsBehavior networks




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Parameterizing the characters

Pre-conditions for divulging informationReveal identity only if offered safety

Dialogue behavior (networks)Time spent in social chit-chatProclivity for lying

Alternative responses

Assertive

{ High I will only tell you this information if you can provideme with some kind of assurance of safety.

Low I’m very sorry but if you wouldn’t mind, I hope youwill be able to offer me protection before I give youthis information.




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Perception of parametric differencesVarying Personality in Spoken Dialogue with a Virtual Human, IVA 2009

General personality traits2 personalities consistent across several dimensions

Assertiveness, modesty, honesty, trust . . .Differences detected along some dimensions

Significant: trust, conscientiousness, complianceTrends: modesty, honesty, positive emotion

Facets of assertivenessAsseriveness manipulated across several behaviors

Initiative of getting to businessPoliteness of asking for offersClarification vs. attack for misunderstandings

Difference detected through interactionIPIP test – International Personality Item Pool




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo














3 Dialogue analysis




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Data sources

Role-plays of negotiation scenarioCollected at the Marshall School of Business (USC)Student population, English languageTranscribed at ICT

Group dialogues (4 speakers per group)Collected at University of Texas El PasoArabic, Spanish, and English speaking groupsVarious tasks (personal story, discuss a movie, name a toy)4 Arabic dialogues (2 groups) transcribed and translated atICT

Negotiation dialogues in the Middle EastTo be collected by UMD in the fall




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Analysis

Language and dialogue patterns used in negotiationCore speech acts

suggest, counter-propose, accept, reject, explain, attack

Modes of topic progressionInteractional goals (e.g. becoming closer)Politeness; social status and power; independence ordominance; downplaying and boosting

Strategies for negotiationQ&A (Questions and Answers): getting to partners desiresand motivesS&O (Substantiation and Offers): infer underlying motivesthrough exchange of offers




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Negotiation: Questions and AnswersCaptain . . . Um, the one important thing for us is we feel that it is not in a

good locationElder the clinic?Captain The clinic. As it currently is. Um, we don’t feel it is safe. . . .Doctor I appreciate your consideration about our clinic but I’d rather be stay

closer to the people . . .Elder So, so your your biggest issue is you does not want to move, right?Doctor Yes.Elder And, and that you have issues with money . . . and medical supplies?Doctor Medical supply equipment and like . . . and ambulance and also

medicines and . . . and also lack of staff....

Elder My prerogative is that it stays within the boundaries of the town . . .Captain Sure.Elder Um, let’s ok, how about let’s back up a second and talk about money.




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Negotiation: Substantiation and Offers

Captain Well, uh, our proposal is to relocate the clinic to the place, uh, near toour army bases, army base so the uh we can provide more securityto the clinic. . . .

...Doctor . . . I would rather keep it where it is and operate as I am . . .Captain So do you have any preference about the relocation place?Elder Um, you know, I am still a little confused. The reason why you want to

move the base, is is because how is that affecting your operations?Captain . . . the primary purpose of our proposal to relocate the clinic is purely

about the security of the clinic. . . .Elder So why moving to the base rather moving to let’s say downtown?

...Elder So I am not exactly, so it seems like you could go, you can move it to

the US base and you can also move it within the town center, right?




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Arabic toy-naming: making suggestions

2. This is a beetle, name it . . . beetle �é�ñ

®

J

k . . AîDÒ� ,

�é�ñ

®

J

k ø

Yë

...4. A name that the child will accept éÊJ.

�®�J�� É

®¢Ë@ Õæ� @

3. Will accept and not difficult, and so áKYªK. ð , I. ª��

�Óð éÊJ.�®�J��

2. Yeah Fola, Fola, Fola C¯ , C

¯ , C

¯ é

�<Ë @ð

1. Yes Fola is somewhat popular with us ø

ñ�

� PñîD��Ó A

KY

J« A

Jk@ C

¯ Ñª

K

2. We have the name Fola, Fola is well , ú

æªK

¬ðQªÓ C

¯ . . ø

Aë C

¯ A

KY

J«

known, but for this the problem is �éÊ¾

�� È

�ð@ @Yë áºË

4. But this is not Fola @Yë C¯

��Ó @Yë ��.

1. this is not Fola @Yë C¯

��Ó




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Arabic toy-naming: arguing

3. . . . male or female? ?�

I�K

ñÓ Bð Q�»

YÓ . .

4. Female, female, Zeina, Zeina, female, ,�

I�K

ñÓ ,�éJK

P ,�éJK

P ,�

I�K

ñÓ ,�

I�K

ñÓ

Zeina, Zeina �éJK

P ,�éJK

P

3. My opinion is like this ø

Aë É�JÓ ú

G@P A

K @

4. Naèoul, Naèoul Èñm�' , Èñm�

'

2. Naèoul is for a man, a man’s name Ég. P Õæ�@ , is for man Èñm�'

3. Naèoul won’t work Èñm�' ¡J.

��K. AÓ ��.

4. Zeina, Zeina �éJK

P ,�éJK

P

2. Zeina, Zeina, Zeina �éJK

P ,�éJK

P ,�éJK

P

3. A nice name would be Zeina and Naèoul " Èñm�'ð

�éJK

P\

àñºK ñÊg Õæ�@

1. This is a bee [naèla] ø

X�éÊm�

'

3. This is not a bee �éÊm�

' �

�Ó ø

Aë

2. No, this is a beetle, a beetle �é�ñ

®

J

k , ø

X

�é�ñ

®

J

k , B




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Arabic toy-naming: reaching an agreement

3. Maybe something like this, Bobby, , úG.ñK. , ½Jë ø

P

�éÊ

ª

�� áºÜØ

as when Mahmoud said at the beginning XñÒm× È�ðBAK. ÈA

�¯ A�ÜÏ ø

P ú

æªK

Khanfous [beetle] is ok, enough �C

g , úæ��Öß é

�QK. �ñ

®

J

k

2. Khanfous �ñ®

J

k

3. Khanfous, something “Khonfo”, yes, because é�KB , è @ "ñ

®

J

k\

�éÊ

ª

�� , �ñ

®

J

k

we have this thing like there are bugs �èQå

��k éJ

¯ ø

P ø

Aë A

KY

J« éJ

¯

1. He gave it a nickname “Khonfo” �éKñ

�� AêªËYKð "ñ

®

J

k\ AêªËX ñë

4. Gave it a nickname, yes è @ AêªËX

toge

ther

3. He’s giving it some nickname �éKñ

�� AêªËYJK.

1. He’s giving it a nickname . . . on their own ÑîE. A�k úÎ« . . AêªËYJK.




9


WhiteCMYK 0/0/0/0RGB 255/255/255





University logo












Plans for next year

Continued dialogue analysisBoth existing and newly collected data

Collaboration on computational models

Initial implemetation of cultural Virtual Humans


ron artstein dusan jan steve solomon david traum filedata analysis cuny, georgetown, u pitt, cmu...

Documents