new vr19 virtual humans - tu-freiberg.de · 2007. 1. 30. · virtuelle realität 1 part 19: virtual...

Virtuelle Realität

1

Part 19:

Virtual Humans

Virtuelle RealitätWintersemester 2006/07

Prof. Bernhard Jung

Prof. B. Jung Virtuelle Realität, WS 2006/07

Overview

Interactive Virtual HumansApplicationsModeling Facial Animation

lip-synchronized speechemotions

Body AnimationEmbodied Conversational Agents

Further ReadingNadia Magnenat-Thalmann und Daniel Thalmann. Handbook of

Virtual Humans. Wiley & Sons. 2004.

Virtuelle Realität

2


Why Virtual Humans?

Virtual Environmentsstatic and dynamic 3D objectsreal-time navigation & explorationhuman-like interaction & communication? With whom?

embodiment of inhabitantse.g. virtual presenters, virtual guides, virtual actors, …

Ultimate Goal: Simulation of Virtual Worlds inhabited by Virtual Human Society

Virtual humans' abilities should be similar to human abilitiesperception, language understanding & generation, emotions, goal-driven behavior, memory, inference, personality, social skills


Virtual Humans – Application Areas

Virtual characters for computer games Virtual presenters for TV and the WebVirtual representations of participants in conferencesVirtual mannequins for the clothing industries Virtual ancient people in cultural heritage applicationsVirtual inhabitants in architectural simulationsVirtual soldiersVirtual patients for surgery, orthopedicsVirtual people for simulation-based learning and training, team coordination, decision makingVirtual users for ergonomic analysis in work environments…

Virtual Try-On, www.human-solutions.com

Ananova, ananova.com

SimPatient, www.rti.org

Virtuelle Realität

3


Virtual Humans for Prototyping:Ergonomic Analyses

Visibility Analyses

www.ergomax.de



Reach Analyses

www.ergomax.de

Virtuelle Realität

4



Posture (Comfort) Analyses

www.ergomax.de



Climate-control simulation e.g. TIM – "thermophysiological occupant model"

computer model that simulates most of the human body in a total of 14 areas, also taking into account the blood circulation and heat generation DaimlerChrysler (e.g. development of 2008 C-Class)

http://www.worldcarfans.com/news.cfm/newsID/2070110.003/page/3/country/gcf/lang/eng/mercedes/new-mercedes-c-class-comfort-developmenthttp://www.emercedesbenz.com/Jan07/09_The_New_Mercedes_C_Class_Superlative_Ride_Comfort_Based_On_Digital_Prototkypes_1.html http://www.puz.de/_shared/php/file_get.php?w=Y36B2MIO862V3HQVL5E5

Virtuelle Realität

5


Virtual Humans for Prototyping:Ergonomic Analyses - Demos

http://www.ugs.com/products/efactory/jack/movies.shtml

maintenance tasks

welding gun operation sorting the mail(posture anaylses)

reach analysis


Virtual Humans for Social Interaction Training

… for police officersJust-Talk, http://www.rti.org

… for doctorsAffective Virtual Patient, http://www.isnm.de

Virtuelle Realität

6


Modeling Virtual Humans

Segmented Modelsarticulated figureindividual segments not deformablee.g. VRML

Skinned Modelsbones + skin (mesh)skeletal animation

continuous deformation of body mesh

Multi-Layer Modelsbones, muscles, skinmain anatomical layers of the body are explicitly represented and deformed


Skeletal Animation Example

source: Milkshape3D help

Assigning mesh verticesto the right knee

bad- some vertices unassigned

good

Virtuelle Realität

7


Modeling Virtual Humans A Body Modeling Production Pipeline for Virtual Reality

PoserHigh-level tool for 3D modeling of virtual people

3D Studio MaxSkeleton based on character Studio or simple bones H-Anim compliance?

VR environmentuses cal3d – an open source, skeletal based 3d character animation library

character skinning in 3DS Max

Poser

skinned character in 3DS Max


3DS Max Character StudioBiped Hierarchy

Virtuelle Realität

8


H-ANIM (MPEG-4)Skeleton

open standard

www.h-anim.org


Anthropometric Databases

statistical data on weight, stature, grip reach, …

E.g. NASA datasethttp://msis.jsc.nasa.gov/sections/section03.htm

E.g. AnthroKids - Anthropometric Data of Childrenhttp://www.itl.nist.gov/iaui/ovrt/projects/anthrokids/anthrokids.html

http://msis.jsc.nasa.gov/sections/section03.htm

Virtuelle Realität

9


Modeling Virtual Humans Facial Modeling & Animation

Real-time animation of 3D "talking heads"expensive hardware no longer required

Specialized 3D modeling tools availablee.g. Haptek People Putty (www.haptek.com)

Lip-synchronized speech animationprerecorded audio or text-to-speech (still limited in emotional expression)

Capable of highly emotional facial expressionMPEG4 facial animation parameters

High-level animationusually scripted; cognitive architectures part of AI research


Talking Heads

Synchronize lip animation with speech

e.g. Baldi, Ananova, Pophead

Visemes can be considered as the visual speech counterparts of phonemes

They are associated to the 3D deformations of a neutral faceAnimation is achieved by concatenating visemes

demo: visemes inHaptek FigureMaker

demo: lip-sync text-to-speech

http://www.speech.kth.se/multimodal/

viseme 'o'

Virtuelle Realität

10


Talking Heads

McGurk effect

http://www.media.uio.no/personer/arntm/McGurk_english.htmlWhat is this person saying?Play the clip several times, alternating between looking at the talking head while listening, and listening with your eyes shut. Most adults (98%) think they are hearing "DA" - a so called "fused response" - where the "D" is a result of an audio-visual illusion.In reality you are hearing the sound "BA", while you are seing the lip movements "GA". The "McGurk effect" was first described by Harry McGurk and John MacDonald in "Hearing lips and seeing voices", Nature 264, 746-748 (1976)


Talking Heads"Basil of the Mirror"www.themeaddicts.com

Virtuelle Realität

11


Facial Animation Examples


Facial Animation by Motion Capture

http://www.softimage.com/products/face_robotSoftimage Face Robot

Virtuelle Realität

12


Facial Animation in MPEG – 4

Set of featurepoints


Facial Expression of Emotions 1/2

[Ekman&Friesen, 1975]laughter

submissive: broad grin, apprehension around the eyes. smile: relaxed, teeth together but lips are barley parted. laughter: teeth often parted, partially covered by the lips.

surprisesudden opening of the eyes, raising the eyebrowes, opening the mouth, wrinkles on the brow and forehead

fearsimilar to surprised (mouth and eyes open) but muscles at the corner of the mouth and brow are pulled back producing a somewhat sqareish mouth and attenuating the raised eyebrows.

Virtuelle Realität

13


Facial Expression of Emotions 2/2

[Ekman&Friesen, 1975]Interest

eyes wide open (object is close) or squint in (great distance), fixed on the object. Mouth open slightly or closed.

Angerbrows are lowered, staring expression; mouth either partially open showing clenched teeth or firmly closed; eyebrows drawn down and together.

Contemptsimilar to anger and reaction of dislike but insignificant object; canine tooth on the side of the face, producing a sneer; wrinkles under the eyes; upper eyelids may be partially closed.


Emotion theory Ortony, Clore and Collins (OCC)

Extremely widely used for agentsOrtony, A; Clore, G. & Collins, A. 1988 The cognitive structure of emotions. Cambridge University Press

AppraisalsAssessments of events, actions, objects

ValenceWhether emotion is positive or negative

ArousalDegree of physiological response

Generating appraisalsDomain-specific rulesProbability of impact on agent’s goals

Virtuelle Realität

14


OCC Model of EmotionsOrtony, A., Clore, G., & Collins, A. (1988). The Cognitive Structure of Emotions. Cambridge: Cambridge University Press.


Animating Virtual Humans- Motion Capture

source: http://www.ptiphoenix.com/

source: http://vrlab.epfl.ch/HandbookVHumans

Virtuelle Realität

15


"One thing you will see is much more sophisticated artificial intelligence," said David Zucker, the CEO of Chicago-based Midway Games. "The artificial intelligence, it's the way in which the world works. When you walk up to an individual in the world, he reacts one way if you punch him. If you say hello to him, he does something else. It's how objects work when acted upon."

The Xbox 360 makes these possibilities available en masse.

In Microsoft's new fantasy game Kameo, players can ride a horse into an army of 3,500 ogres. Amazingly, each ogre has its own intelligence and reacts to the player independent of the thousands of ogres around it.

Animating Virtual Humans

News report on upcoming Xbox 360, May 2005


Animating Virtual Humans - Cognitive Animation

Animation architecture: several layers includingbehavior (reactive) layer: sensorimotor procedurescognitive (deliberative) layer: planning, language, …

Steve architecture (Rickel) Max architecture (Kopp et al.)

Funge, 1999

Virtuelle Realität

16


Embodied Conversational Agents (ECA)

Artificial Intelligence meets Virtual Humans

narrower focus: simulate verbal and nonverbal communicative behavior of humans

speech, gesture, emotional facial expressiontiming, synchronization of multimodal behavior

wider context: simulate wide range of cognitive and social abilities in integrated fashion

communication, planning, emotional states, …agent teams

Note: not all ECAs possess complete anthropomorphic bodies


Taxonomy of Nonverbal Behavior

Virtuelle Realität

17


REA – Reale Estate Agent


Rea - ArchitectureJustine Cassell, MIT

Virtuelle Realität

18


REA: Communicative Functions

Propositional informationcontent of conversationspeech, gesture, intonation

Interactional informationkeeps communication channel openregulatory speech ("huh", "go on"non-verbal behavior (quick head nods, turning to listener, …)


Animating Virtual Humans - Cognitive Animation

MRE – Mission Rehearsal Environment (Gratch et al. 2003)

Steve (Rickel & Johnson 1999)

Max (Kopp, Jung, Lessmann & Wachsmuth 2003)

Examples of Embodied Conversational Agents

new vr19 virtual humans - tu-freiberg.de · 2007. 1. 30. · virtuelle realität 1 part 19: virtual...

Documents