Virtuelle Realität
1
Part 19:
Virtual Humans
Virtuelle RealitätWintersemester 2006/07
Prof. Bernhard Jung
Prof. B. Jung Virtuelle Realität, WS 2006/07
Overview
Interactive Virtual HumansApplicationsModeling Facial Animation
lip-synchronized speechemotions
Body AnimationEmbodied Conversational Agents
Further ReadingNadia Magnenat-Thalmann und Daniel Thalmann. Handbook of
Virtual Humans. Wiley & Sons. 2004.
Virtuelle Realität
2
Prof. B. Jung Virtuelle Realität, WS 2006/07
Why Virtual Humans?
Virtual Environmentsstatic and dynamic 3D objectsreal-time navigation & explorationhuman-like interaction & communication? With whom?
embodiment of inhabitantse.g. virtual presenters, virtual guides, virtual actors, …
Ultimate Goal: Simulation of Virtual Worlds inhabited by Virtual Human Society
Virtual humans' abilities should be similar to human abilitiesperception, language understanding & generation, emotions, goal-driven behavior, memory, inference, personality, social skills
Prof. B. Jung Virtuelle Realität, WS 2006/07
Virtual Humans – Application Areas
Virtual characters for computer games Virtual presenters for TV and the WebVirtual representations of participants in conferencesVirtual mannequins for the clothing industries Virtual ancient people in cultural heritage applicationsVirtual inhabitants in architectural simulationsVirtual soldiersVirtual patients for surgery, orthopedicsVirtual people for simulation-based learning and training, team coordination, decision makingVirtual users for ergonomic analysis in work environments…
Virtual Try-On, www.human-solutions.com
Ananova, ananova.com
SimPatient, www.rti.org
Virtuelle Realität
3
Prof. B. Jung Virtuelle Realität, WS 2006/07
Virtual Humans for Prototyping:Ergonomic Analyses
Visibility Analyses
www.ergomax.de
Prof. B. Jung Virtuelle Realität, WS 2006/07
Virtual Humans for Prototyping:Ergonomic Analyses
Reach Analyses
www.ergomax.de
Virtuelle Realität
4
Prof. B. Jung Virtuelle Realität, WS 2006/07
Virtual Humans for Prototyping:Ergonomic Analyses
Posture (Comfort) Analyses
www.ergomax.de
Prof. B. Jung Virtuelle Realität, WS 2006/07
Virtual Humans for Prototyping:Ergonomic Analyses
Climate-control simulation e.g. TIM – "thermophysiological occupant model"
computer model that simulates most of the human body in a total of 14 areas, also taking into account the blood circulation and heat generation DaimlerChrysler (e.g. development of 2008 C-Class)
http://www.worldcarfans.com/news.cfm/newsID/2070110.003/page/3/country/gcf/lang/eng/mercedes/new-mercedes-c-class-comfort-developmenthttp://www.emercedesbenz.com/Jan07/09_The_New_Mercedes_C_Class_Superlative_Ride_Comfort_Based_On_Digital_Prototkypes_1.html http://www.puz.de/_shared/php/file_get.php?w=Y36B2MIO862V3HQVL5E5
Virtuelle Realität
5
Prof. B. Jung Virtuelle Realität, WS 2006/07
Virtual Humans for Prototyping:Ergonomic Analyses - Demos
http://www.ugs.com/products/efactory/jack/movies.shtml
maintenance tasks
welding gun operation sorting the mail(posture anaylses)
reach analysis
Prof. B. Jung Virtuelle Realität, WS 2006/07
Virtual Humans for Social Interaction Training
… for police officersJust-Talk, http://www.rti.org
… for doctorsAffective Virtual Patient, http://www.isnm.de
Virtuelle Realität
6
Prof. B. Jung Virtuelle Realität, WS 2006/07
Modeling Virtual Humans
Segmented Modelsarticulated figureindividual segments not deformablee.g. VRML
Skinned Modelsbones + skin (mesh)skeletal animation
continuous deformation of body mesh
Multi-Layer Modelsbones, muscles, skinmain anatomical layers of the body are explicitly represented and deformed
Prof. B. Jung Virtuelle Realität, WS 2006/07
Skeletal Animation Example
source: Milkshape3D help
Assigning mesh verticesto the right knee
bad- some vertices unassigned
good
Virtuelle Realität
7
Prof. B. Jung Virtuelle Realität, WS 2006/07
Modeling Virtual Humans A Body Modeling Production Pipeline for Virtual Reality
PoserHigh-level tool for 3D modeling of virtual people
3D Studio MaxSkeleton based on character Studio or simple bones H-Anim compliance?
VR environmentuses cal3d – an open source, skeletal based 3d character animation library
character skinning in 3DS Max
Poser
skinned character in 3DS Max
Prof. B. Jung Virtuelle Realität, WS 2006/07
3DS Max Character StudioBiped Hierarchy
Virtuelle Realität
8
Prof. B. Jung Virtuelle Realität, WS 2006/07
H-ANIM (MPEG-4)Skeleton
open standard
www.h-anim.org
Prof. B. Jung Virtuelle Realität, WS 2006/07
Anthropometric Databases
statistical data on weight, stature, grip reach, …
E.g. NASA datasethttp://msis.jsc.nasa.gov/sections/section03.htm
E.g. AnthroKids - Anthropometric Data of Childrenhttp://www.itl.nist.gov/iaui/ovrt/projects/anthrokids/anthrokids.html
http://msis.jsc.nasa.gov/sections/section03.htm
Virtuelle Realität
9
Prof. B. Jung Virtuelle Realität, WS 2006/07
Modeling Virtual Humans Facial Modeling & Animation
Real-time animation of 3D "talking heads"expensive hardware no longer required
Specialized 3D modeling tools availablee.g. Haptek People Putty (www.haptek.com)
Lip-synchronized speech animationprerecorded audio or text-to-speech (still limited in emotional expression)
Capable of highly emotional facial expressionMPEG4 facial animation parameters
High-level animationusually scripted; cognitive architectures part of AI research
Prof. B. Jung Virtuelle Realität, WS 2006/07
Talking Heads
Synchronize lip animation with speech
e.g. Baldi, Ananova, Pophead
Visemes can be considered as the visual speech counterparts of phonemes
They are associated to the 3D deformations of a neutral faceAnimation is achieved by concatenating visemes
demo: visemes inHaptek FigureMaker
demo: lip-sync text-to-speech
http://www.speech.kth.se/multimodal/
viseme 'o'
Virtuelle Realität
10
Prof. B. Jung Virtuelle Realität, WS 2006/07
Talking Heads
McGurk effect
http://www.media.uio.no/personer/arntm/McGurk_english.htmlWhat is this person saying?Play the clip several times, alternating between looking at the talking head while listening, and listening with your eyes shut. Most adults (98%) think they are hearing "DA" - a so called "fused response" - where the "D" is a result of an audio-visual illusion.In reality you are hearing the sound "BA", while you are seing the lip movements "GA". The "McGurk effect" was first described by Harry McGurk and John MacDonald in "Hearing lips and seeing voices", Nature 264, 746-748 (1976)
Prof. B. Jung Virtuelle Realität, WS 2006/07
Talking Heads"Basil of the Mirror"www.themeaddicts.com
Virtuelle Realität
11
Prof. B. Jung Virtuelle Realität, WS 2006/07
Facial Animation Examples
Prof. B. Jung Virtuelle Realität, WS 2006/07
Facial Animation by Motion Capture
http://www.softimage.com/products/face_robotSoftimage Face Robot
Virtuelle Realität
12
Prof. B. Jung Virtuelle Realität, WS 2006/07
Facial Animation in MPEG – 4
Set of featurepoints
Prof. B. Jung Virtuelle Realität, WS 2006/07
Facial Expression of Emotions 1/2
[Ekman&Friesen, 1975]laughter
submissive: broad grin, apprehension around the eyes. smile: relaxed, teeth together but lips are barley parted. laughter: teeth often parted, partially covered by the lips.
surprisesudden opening of the eyes, raising the eyebrowes, opening the mouth, wrinkles on the brow and forehead
fearsimilar to surprised (mouth and eyes open) but muscles at the corner of the mouth and brow are pulled back producing a somewhat sqareish mouth and attenuating the raised eyebrows.
Virtuelle Realität
13
Prof. B. Jung Virtuelle Realität, WS 2006/07
Facial Expression of Emotions 2/2
[Ekman&Friesen, 1975]Interest
eyes wide open (object is close) or squint in (great distance), fixed on the object. Mouth open slightly or closed.
Angerbrows are lowered, staring expression; mouth either partially open showing clenched teeth or firmly closed; eyebrows drawn down and together.
Contemptsimilar to anger and reaction of dislike but insignificant object; canine tooth on the side of the face, producing a sneer; wrinkles under the eyes; upper eyelids may be partially closed.
Prof. B. Jung Virtuelle Realität, WS 2006/07
Emotion theory Ortony, Clore and Collins (OCC)
Extremely widely used for agentsOrtony, A; Clore, G. & Collins, A. 1988 The cognitive structure of emotions. Cambridge University Press
AppraisalsAssessments of events, actions, objects
ValenceWhether emotion is positive or negative
ArousalDegree of physiological response
Generating appraisalsDomain-specific rulesProbability of impact on agent’s goals
Virtuelle Realität
14
Prof. B. Jung Virtuelle Realität, WS 2006/07
OCC Model of EmotionsOrtony, A., Clore, G., & Collins, A. (1988). The Cognitive Structure of Emotions. Cambridge: Cambridge University Press.
Prof. B. Jung Virtuelle Realität, WS 2006/07
Animating Virtual Humans- Motion Capture
source: http://www.ptiphoenix.com/
source: http://vrlab.epfl.ch/HandbookVHumans
Virtuelle Realität
15
Prof. B. Jung Virtuelle Realität, WS 2006/07
"One thing you will see is much more sophisticated artificial intelligence," said David Zucker, the CEO of Chicago-based Midway Games. "The artificial intelligence, it's the way in which the world works. When you walk up to an individual in the world, he reacts one way if you punch him. If you say hello to him, he does something else. It's how objects work when acted upon."
The Xbox 360 makes these possibilities available en masse.
In Microsoft's new fantasy game Kameo, players can ride a horse into an army of 3,500 ogres. Amazingly, each ogre has its own intelligence and reacts to the player independent of the thousands of ogres around it.
Animating Virtual Humans
News report on upcoming Xbox 360, May 2005
Prof. B. Jung Virtuelle Realität, WS 2006/07
Animating Virtual Humans - Cognitive Animation
Animation architecture: several layers includingbehavior (reactive) layer: sensorimotor procedurescognitive (deliberative) layer: planning, language, …
Steve architecture (Rickel) Max architecture (Kopp et al.)
Funge, 1999
Virtuelle Realität
16
Prof. B. Jung Virtuelle Realität, WS 2006/07
Embodied Conversational Agents (ECA)
Artificial Intelligence meets Virtual Humans
narrower focus: simulate verbal and nonverbal communicative behavior of humans
speech, gesture, emotional facial expressiontiming, synchronization of multimodal behavior
wider context: simulate wide range of cognitive and social abilities in integrated fashion
communication, planning, emotional states, …agent teams
Note: not all ECAs possess complete anthropomorphic bodies
Prof. B. Jung Virtuelle Realität, WS 2006/07
Taxonomy of Nonverbal Behavior
Virtuelle Realität
17
Prof. B. Jung Virtuelle Realität, WS 2006/07
REA – Reale Estate Agent
Prof. B. Jung Virtuelle Realität, WS 2006/07
Rea - ArchitectureJustine Cassell, MIT
Virtuelle Realität
18
Prof. B. Jung Virtuelle Realität, WS 2006/07
REA: Communicative Functions
Propositional informationcontent of conversationspeech, gesture, intonation
Interactional informationkeeps communication channel openregulatory speech ("huh", "go on"non-verbal behavior (quick head nods, turning to listener, …)
Prof. B. Jung Virtuelle Realität, WS 2006/07
Animating Virtual Humans - Cognitive Animation
MRE – Mission Rehearsal Environment (Gratch et al. 2003)
Steve (Rickel & Johnson 1999)
Max (Kopp, Jung, Lessmann & Wachsmuth 2003)
Examples of Embodied Conversational Agents