july 24, 20001smart animated agents -- siggraph course #24 smart animated agents norman i. badler --...

July 24, 2000 1Smart Animated Agents -- SIGGRAPH Course #24

SMART ANIMATED AGENTS

Norman I. Badler -- Course #24 Co-Organizer Norman I. Badler -- Course #24 Co-Organizer (with John Funge)(with John Funge)

Center for Human Modeling and SimulationCenter for Human Modeling and Simulation

University of PennsylvaniaUniversity of Pennsylvania

Philadelphia, PA 19104-6389Philadelphia, PA 19104-6389

http://www.cis.upenn.edu/~badlerhttp://www.cis.upenn.edu/~badler

Norman I. Badler -- Course #24 Co-Organizer Norman I. Badler -- Course #24 Co-Organizer (with John Funge)(with John Funge)

Center for Human Modeling and SimulationCenter for Human Modeling and Simulation

University of PennsylvaniaUniversity of Pennsylvania

Philadelphia, PA 19104-6389Philadelphia, PA 19104-6389

http://www.cis.upenn.edu/~badlerhttp://www.cis.upenn.edu/~badler


Course Speakers

• Norman Badler (U. Pennsylvania)Norman Badler (U. Pennsylvania)

• Justine Cassell (MIT Media Lab)Justine Cassell (MIT Media Lab)

• John Funge (Sony Computer Entertainment John Funge (Sony Computer Entertainment America)America)

• Jeff Rickel (ISI / U. So. California)Jeff Rickel (ISI / U. So. California)

• Bruce Blumberg (MIT Media Lab)Bruce Blumberg (MIT Media Lab)

• Norman Badler (U. Pennsylvania)Norman Badler (U. Pennsylvania)

• Justine Cassell (MIT Media Lab)Justine Cassell (MIT Media Lab)

• John Funge (Sony Computer Entertainment John Funge (Sony Computer Entertainment America)America)

• Jeff Rickel (ISI / U. So. California)Jeff Rickel (ISI / U. So. California)

• Bruce Blumberg (MIT Media Lab)Bruce Blumberg (MIT Media Lab)


Course Schedule (a.m.)

8:30-8:35 Badler (Introduction)8:30-8:35 Badler (Introduction)

8:35-10:00 Badler8:35-10:00 Badler

10:00-10:15 (break)10:00-10:15 (break)

10:15-11:45 Cassell10:15-11:45 Cassell

11:45-12:00 Questions and Issues11:45-12:00 Questions and Issues

12:00-1:30 (lunch)12:00-1:30 (lunch)

8:30-8:35 Badler (Introduction)8:30-8:35 Badler (Introduction)

8:35-10:00 Badler8:35-10:00 Badler

10:00-10:15 (break)10:00-10:15 (break)

10:15-11:45 Cassell10:15-11:45 Cassell


12:00-1:30 (lunch)12:00-1:30 (lunch)


Course Schedule (p.m.)

1:30-2:30 Funge1:30-2:30 Funge

2:30-3:00 Rickel2:30-3:00 Rickel

3:00-3:15 (break)3:00-3:15 (break)

3:15-3:45 Rickel3:15-3:45 Rickel

3:45-4:45 Blumberg3:45-4:45 Blumberg


1:30-2:30 Funge1:30-2:30 Funge

2:30-3:00 Rickel2:30-3:00 Rickel

3:00-3:15 (break)3:00-3:15 (break)

3:15-3:45 Rickel3:15-3:45 Rickel

3:45-4:45 Blumberg3:45-4:45 Blumberg



Course #24 Topics

Action Primitives and Action RepresentationAction Primitives and Action Representation

Natural Language Interfaces Natural Language Interfaces

Conversational and Communicative AgentsConversational and Communicative Agents

Cognitive ModelingCognitive Modeling

Pedagogical AgentsPedagogical Agents

Task-Oriented CollaborationTask-Oriented Collaboration

LearningLearning

Action Primitives and Action RepresentationAction Primitives and Action Representation

Natural Language Interfaces Natural Language Interfaces

Conversational and Communicative AgentsConversational and Communicative Agents

Cognitive ModelingCognitive Modeling

Pedagogical AgentsPedagogical Agents

Task-Oriented CollaborationTask-Oriented Collaboration

LearningLearning


Building Smart Agents (Badler)

• Introduction and ApplicationsIntroduction and Applications

• Smart AvatarsSmart Avatars

• Parameterized Action Representation (PAR)Parameterized Action Representation (PAR)

• Natural Language Instructions Natural Language Instructions

• Automating AttentionAutomating Attention

• Agent MannerAgent Manner

• Building PARsBuilding PARs









Building Smart Agents
















Introduction to and Applications for Embodied Agents:

• Engineering Ergonomics.Engineering Ergonomics.

• Design and Maintenance Assessment.Design and Maintenance Assessment.

• Games/Special Effects.Games/Special Effects.

• Military Simulations.Military Simulations.

• Job Education/Training.Job Education/Training.

• Medical Simulations.Medical Simulations.

• Engineering Ergonomics.Engineering Ergonomics.

• Design and Maintenance Assessment.Design and Maintenance Assessment.

• Games/Special Effects.Games/Special Effects.

• Military Simulations.Military Simulations.

• Job Education/Training.Job Education/Training.

• Medical Simulations.Medical Simulations.


Virtual Human “Dimensions”

• AppearanceAppearance

• FunctionFunction

• TimeTime

• AutonomyAutonomy

• IndividualityIndividuality

• AppearanceAppearance

• FunctionFunction

• TimeTime

• AutonomyAutonomy

• IndividualityIndividuality


Appearance:

2D drawings 2D drawings >> 3D wireframe 3D wireframe >>

3D polyhedra 3D polyhedra >> curved surfaces curved surfaces >> freeform deformations freeform deformations >>

accurate surfaces accurate surfaces >> muscles, fat muscles, fat >> biomechanics biomechanics >> clothing, equipment clothing, equipment >> physiological effects (perspiration, physiological effects (perspiration, irritation, injury)irritation, injury)

2D drawings 2D drawings >> 3D wireframe 3D wireframe >>

3D polyhedra 3D polyhedra >> curved surfaces curved surfaces >> freeform deformations freeform deformations >>

accurate surfaces accurate surfaces >> muscles, fat muscles, fat >> biomechanics biomechanics >> clothing, equipment clothing, equipment >> physiological effects (perspiration, physiological effects (perspiration, irritation, injury)irritation, injury)


Function:

cartoon cartoon >> jointed skeleton jointed skeleton >>

joint limits joint limits >> strength limits strength limits >>

fatigue fatigue >> hazards hazards >> injury injury >> skills skills >> effects of loads and stressors effects of loads and stressors >> psychological models psychological models >>

cognitive models cognitive models >> roles roles >> teaming teaming

cartoon cartoon >> jointed skeleton jointed skeleton >>

joint limits joint limits >> strength limits strength limits >>

fatigue fatigue >> hazards hazards >> injury injury >> skills skills >> effects of loads and stressors effects of loads and stressors >> psychological models psychological models >>

cognitive models cognitive models >> roles roles >> teaming teaming


Time (to create / move each):

off-line animation off-line animation >>

interactive manipulation interactive manipulation >>

real-time motion playback real-time motion playback >> parameterized motion synthesis parameterized motion synthesis >> multiple agents multiple agents >>

crowds crowds >> coordinated teams coordinated teams

off-line animation off-line animation >>

interactive manipulation interactive manipulation >>

real-time motion playback real-time motion playback >> parameterized motion synthesis parameterized motion synthesis >> multiple agents multiple agents >>

crowds crowds >> coordinated teams coordinated teams


Autonomy:

drawing drawing >> scripting scripting >>

interacting interacting >> reacting reacting >>

making decisions making decisions >>

communicating communicating >> intending intending >>

taking initiative taking initiative >> leading leading

drawing drawing >> scripting scripting >>

interacting interacting >> reacting reacting >>

making decisions making decisions >>

communicating communicating >> intending intending >>

taking initiative taking initiative >> leading leading


Individuality:

generic character generic character >>

hand-crafted character hand-crafted character > >

cultural distinctions cultural distinctions >>

sex and age sex and age >> personality personality >>

psychological-physiological profiles psychological-physiological profiles >>

specific individualspecific individual

generic character generic character >>

hand-crafted character hand-crafted character > >

cultural distinctions cultural distinctions >>

sex and age sex and age >> personality personality >>

psychological-physiological profiles psychological-physiological profiles >>

specific individualspecific individual


Comparative Graphical AgentsComparative Graphical Agents

Application Appear. Function Time Autonomy Individ.Application Appear. Function Time Autonomy Individ.

CartoonsCartoons highhigh low low high high lowlow high high

Sp. EffectsSp. Effects highhigh low low high high lowlow med med

MedicalMedical highhigh high high medmed medmed med med

ErgonomicsErgonomics medmed high high medmed medmed low low

GamesGames highhigh low low low low med/high medmed/high med

MilitaryMilitary medmed med med lowlow med/high lowmed/high low

EducationEducation medmed low low low low med/high medmed/high med

TrainingTraining medmed low low low low highhigh med med

Application Appear. Function Time Autonomy Individ.Application Appear. Function Time Autonomy Individ.

CartoonsCartoons highhigh low low high high lowlow high high

Sp. EffectsSp. Effects highhigh low low high high lowlow med med

MedicalMedical highhigh high high medmed medmed med med

ErgonomicsErgonomics medmed high high medmed medmed low low

GamesGames highhigh low low low low med/high medmed/high med

MilitaryMilitary medmed med med lowlow med/high lowmed/high low

EducationEducation medmed low low low low med/high medmed/high med

TrainingTraining medmed low low low low highhigh med med





• Parameterized Action RepresentationParameterized Action Representation













Why Smart Avatars? For Motion Control

• Point-and-click (menu or direct 2D Point-and-click (menu or direct 2D manipulation).manipulation).

• Directly sensed (3D motion capture).Directly sensed (3D motion capture).

• Language commands (text or speech).Language commands (text or speech).

Use instructions -- as if the agent were Use instructions -- as if the agent were oneself or another oneself or another realreal person: person:

A Smart AvatarA Smart Avatar

• Point-and-click (menu or direct 2D Point-and-click (menu or direct 2D manipulation).manipulation).

• Directly sensed (3D motion capture).Directly sensed (3D motion capture).

• Language commands (text or speech).Language commands (text or speech).

Use instructions -- as if the agent were Use instructions -- as if the agent were oneself or another oneself or another realreal person: person:

A Smart AvatarA Smart Avatar


Smart Agent Requirements

Actions to Execute:Actions to Execute:• Action Representation - Action Representation - What it can doWhat it can do..

Behavior Model:Behavior Model:• The agent’s decision-making, “thought,” and reaction The agent’s decision-making, “thought,” and reaction

processes - processes - What it should do or wants to do.What it should do or wants to do.

Inputs to Effect Behavior:Inputs to Effect Behavior:• Incoming knowledge about the outside world - Incoming knowledge about the outside world - What it What it

needs to knowneeds to know..

Actions to Execute:Actions to Execute:• Action Representation - Action Representation - What it can doWhat it can do..




needs to knowneeds to know..


“Classic” AI: Agent Action Cycle

Sense

Act

Control

Agent modelAgent model

• Messages• Sensors• Situation

World modelWorld model



Actions to execute:Actions to execute:• Action Representation - Action Representation - What it can do.What it can do.


processes.processes.

Inputs to Effect Behavior:Inputs to Effect Behavior:• Incoming knowledge about the outside world.Incoming knowledge about the outside world.

Actions to execute:Actions to execute:• Action Representation - Action Representation - What it can do.What it can do.





4 Levels of Action Representation

0:0: Basic Motion Generators Basic Motion Generators

1:1: Parallel -Transition Networks Parallel -Transition Networks

2:2: Parameterized Actions Parameterized Actions

3:3: Natural Language Instructions Natural Language Instructions

0:0: Basic Motion Generators Basic Motion Generators

1:1: Parallel -Transition Networks Parallel -Transition Networks

2:2: Parameterized Actions Parameterized Actions

3:3: Natural Language Instructions Natural Language Instructions


Level 0: Basic Human Movement Capabilities

• Gesture / Reach / Grasp.Gesture / Reach / Grasp.

• Walk / Turn / Climb.Walk / Turn / Climb.

• Posture Transitions (Sit / Stand)Posture Transitions (Sit / Stand)

• Visual Attention / Search.Visual Attention / Search.

• Pull / Lift / Carry.Pull / Lift / Carry.

• Motion playback (captured or scripted).Motion playback (captured or scripted).

• ‘ ‘Noise’ or secondary movements.Noise’ or secondary movements.

• Gesture / Reach / Grasp.Gesture / Reach / Grasp.

• Walk / Turn / Climb.Walk / Turn / Climb.

• Posture Transitions (Sit / Stand)Posture Transitions (Sit / Stand)

• Visual Attention / Search.Visual Attention / Search.

• Pull / Lift / Carry.Pull / Lift / Carry.

• Motion playback (captured or scripted).Motion playback (captured or scripted).

• ‘ ‘Noise’ or secondary movements.Noise’ or secondary movements.


Synthesized Motions -- Leverage Economy of Expression

Few parameters controlling many:Few parameters controlling many:

• Inverse kinematics for arms, legs, spine.Inverse kinematics for arms, legs, spine.

• Paths or footsteps driving locomotion.Paths or footsteps driving locomotion.

• Balance constraint on whole body.Balance constraint on whole body.

• Dynamics control from forces and torques.Dynamics control from forces and torques.

• Facial expressionsFacial expressions

Few parameters controlling many:Few parameters controlling many:

• Inverse kinematics for arms, legs, spine.Inverse kinematics for arms, legs, spine.

• Paths or footsteps driving locomotion.Paths or footsteps driving locomotion.

• Balance constraint on whole body.Balance constraint on whole body.

• Dynamics control from forces and torques.Dynamics control from forces and torques.

• Facial expressionsFacial expressions



Actions to execute:Actions to execute:• Action Representation.Action Representation.









Raise Level of Behavioral Control from Level 0

AnimNL project (~1988-1994):AnimNL project (~1988-1994):• “ “Go into the kitchen and get me the coffee urn” Go into the kitchen and get me the coffee urn”

(manual scripting of actions)(manual scripting of actions)

• SodaJack (action planner + object specific SodaJack (action planner + object specific reasoner)reasoner)

Needed a better underlying paradigm upon Needed a better underlying paradigm upon which to build smarter agents.which to build smarter agents.

AnimNL project (~1988-1994):AnimNL project (~1988-1994):• “ “Go into the kitchen and get me the coffee urn” Go into the kitchen and get me the coffee urn”

(manual scripting of actions)(manual scripting of actions)

• SodaJack (action planner + object specific SodaJack (action planner + object specific reasoner)reasoner)

Needed a better underlying paradigm upon Needed a better underlying paradigm upon which to build smarter agents.which to build smarter agents.


Level 1: Parallel Transition Networks (PaT-Nets)

A Virtual Parallel Execution Engine for agent A Virtual Parallel Execution Engine for agent actions (a.k.a. Finite State Machines):actions (a.k.a. Finite State Machines):

• Processes are nodes.Processes are nodes.

• Instantaneous (conditional or probabilistic) Instantaneous (conditional or probabilistic) transitions are edges.transitions are edges.

• Hierarchic.Hierarchic.

• Message passing and synchronization.Message passing and synchronization.

Emerging common paradigm for agent control.Emerging common paradigm for agent control.

A Virtual Parallel Execution Engine for agent A Virtual Parallel Execution Engine for agent actions (a.k.a. Finite State Machines):actions (a.k.a. Finite State Machines):

• Processes are nodes.Processes are nodes.

• Instantaneous (conditional or probabilistic) Instantaneous (conditional or probabilistic) transitions are edges.transitions are edges.

• Hierarchic.Hierarchic.

• Message passing and synchronization.Message passing and synchronization.

Emerging common paradigm for agent control.Emerging common paradigm for agent control.


PaT-Net Applications

• Conversational agents. (SIGGRAPH ‘94)Conversational agents. (SIGGRAPH ‘94)

• Hide and seek. (VRAIS ‘96)Hide and seek. (VRAIS ‘96)

• MediSim: Physiological models. (Presence ‘96)MediSim: Physiological models. (Presence ‘96)

• Jack Presenter. Jack Presenter. (AAAI-97 Workshop/IEEE CG&A)(AAAI-97 Workshop/IEEE CG&A)

• Delsarte Presenter. Delsarte Presenter. (Pacific Graphics ‘98)(Pacific Graphics ‘98)

• JackMOO.JackMOO. (WebSim ‘98, VR ‘99) (WebSim ‘98, VR ‘99)

• AVA (Attention). (Autonomous Agents ‘99)AVA (Attention). (Autonomous Agents ‘99)

• Conversational agents. (SIGGRAPH ‘94)Conversational agents. (SIGGRAPH ‘94)

• Hide and seek. (VRAIS ‘96)Hide and seek. (VRAIS ‘96)

• MediSim: Physiological models. (Presence ‘96)MediSim: Physiological models. (Presence ‘96)

• Jack Presenter. Jack Presenter. (AAAI-97 Workshop/IEEE CG&A)(AAAI-97 Workshop/IEEE CG&A)

• Delsarte Presenter. Delsarte Presenter. (Pacific Graphics ‘98)(Pacific Graphics ‘98)

• JackMOO.JackMOO. (WebSim ‘98, VR ‘99) (WebSim ‘98, VR ‘99)

• AVA (Attention). (Autonomous Agents ‘99)AVA (Attention). (Autonomous Agents ‘99)


What’s Missing?

• PaT-Nets are effective but hand-coded.PaT-Nets are effective but hand-coded.

• No matter what artificial language we No matter what artificial language we introduce it is not the way people introduce it is not the way people conceptualize the situation. (Badler/Webber)conceptualize the situation. (Badler/Webber)

• Connect language and animation through Connect language and animation through an intermediate level ---an intermediate level ---

• PaT-Nets are effective but hand-coded.PaT-Nets are effective but hand-coded.

• No matter what artificial language we No matter what artificial language we introduce it is not the way people introduce it is not the way people conceptualize the situation. (Badler/Webber)conceptualize the situation. (Badler/Webber)

• Connect language and animation through Connect language and animation through an intermediate level ---an intermediate level ---


Level 2: Parameterized Action Representation (PAR)

• Derived from Derived from BOTHBOTH Natural Language Natural Language analyses and animation requirements:analyses and animation requirements:

– Agent, Objects, Sub-Actions.Agent, Objects, Sub-Actions.

– Preparatory Specifications, Postconditions.Preparatory Specifications, Postconditions.

– Applicability and Termination Conditions.Applicability and Termination Conditions.

– Purpose (Achieve, Generate, Enable).Purpose (Achieve, Generate, Enable).

– Path, Duration, Motion, Force.Path, Duration, Motion, Force.

– Agent Manner.Agent Manner.

• Derived from Derived from BOTHBOTH Natural Language Natural Language analyses and animation requirements:analyses and animation requirements:








Level 3: Natural Language Instructions

• Instructions say what to do.Instructions say what to do.

• Instructions depend on underlying action Instructions depend on underlying action skills.skills.

• Instructions build agent behaviors (future Instructions build agent behaviors (future actions or standing orders).actions or standing orders).

• Instructions say what to do.Instructions say what to do.

• Instructions depend on underlying action Instructions depend on underlying action skills.skills.



Integrated Approach: The JackMOO Testbed

• JackMOO goal: Create a multi-user, shared, JackMOO goal: Create a multi-user, shared, 3D virtual environment with full body avatars 3D virtual environment with full body avatars and autonomous human agents, language-and autonomous human agents, language-based commands, and low network based commands, and low network bandwidth.bandwidth.

• Based on lambdaMOO engine with Jack 3D Based on lambdaMOO engine with Jack 3D environment and simple imperative environment and simple imperative commands.commands.

• JackMOO goal: Create a multi-user, shared, JackMOO goal: Create a multi-user, shared, 3D virtual environment with full body avatars 3D virtual environment with full body avatars and autonomous human agents, language-and autonomous human agents, language-based commands, and low network based commands, and low network bandwidth.bandwidth.

• Based on lambdaMOO engine with Jack 3D Based on lambdaMOO engine with Jack 3D environment and simple imperative environment and simple imperative commands.commands.


JackMOO Smart Avatar Experiments

• Greetings (Gender and culture specific)Greetings (Gender and culture specific)

• Go to … (Sit in chair; Go to bed; Leave)Go to … (Sit in chair; Go to bed; Leave) — Do unspecified but necessary Do unspecified but necessary preparatory actionspreparatory actions..

• Relationships (Follow me)Relationships (Follow me)— Mutual agreement.Mutual agreement.

• Autonomous Agents (Waiter)Autonomous Agents (Waiter)— Reacts to environment & states of other agents.Reacts to environment & states of other agents.

• Greetings (Gender and culture specific)Greetings (Gender and culture specific)

• Go to … (Sit in chair; Go to bed; Leave)Go to … (Sit in chair; Go to bed; Leave) — Do unspecified but necessary Do unspecified but necessary preparatory actionspreparatory actions..

• Relationships (Follow me)Relationships (Follow me)— Mutual agreement.Mutual agreement.

• Autonomous Agents (Waiter)Autonomous Agents (Waiter)— Reacts to environment & states of other agents.Reacts to environment & states of other agents.


Expanding the Agent Model

• Create individuals or specific people.Create individuals or specific people.

• Link perceptions of context to action. Link perceptions of context to action.

• Embed action planning capabilities.Embed action planning capabilities.

• Add emotional planner.Add emotional planner.

Agent = Agent =

Intentions X Personality X Emotions X Intentions X Personality X Emotions X Context X History X Capabilities X …Context X History X Capabilities X …

• Create individuals or specific people.Create individuals or specific people.

• Link perceptions of context to action. Link perceptions of context to action.

• Embed action planning capabilities.Embed action planning capabilities.

• Add emotional planner.Add emotional planner.

Agent = Agent =

Intentions X Personality X Emotions X Intentions X Personality X Emotions X Context X History X Capabilities X …Context X History X Capabilities X …


Adapt the OCC Model of Agent Emotional Response

• Consequences of events:Consequences of events:– Consequences for selfConsequences for self

– Consequences for othersConsequences for others

• Actions of agents:Actions of agents:– SelfSelf

– OthersOthers

• Actions of objects.Actions of objects.

• Consequences of events:Consequences of events:– Consequences for selfConsequences for self

– Consequences for othersConsequences for others

• Actions of agents:Actions of agents:– SelfSelf

– OthersOthers

• Actions of objects.Actions of objects.


Valenced Reaction To

Consequences of Events Actions of Agents Aspects of Objects

pleased, displeased approving, disapproving liking, disliking

FOCUSING ON FOCUSING ON

Consequences for Others

Consequences for Self

DesirableUndesirable

Self Agent

OtherAgent

Happy-for GloatingResentment Pity

FORTUNES OF OTHERS

ProspectsRelevant

ProspectsIrrelevant

HopeFear

Confirmed Disconfirmed

Satisfaction ReliefFears-confirmed Disappointment

PROSPECT-BASED

JoyDistress

WELL-BEING

Pride AdmirationShame Reproach

ATTRIBUTION

LoveHate

ATTRACTION

Gratification GratitudeRemorse Anger

WELL-BEING/ATTRIBUTIONCOMPOUNDS

OCC Model of Emotions







needs to know.needs to know.





needs to know.needs to know.


Response Requires Input

• Sensing the state of events:Sensing the state of events:– Self (action postconditions)Self (action postconditions)

– Others (messages; observations)Others (messages; observations)

• Sensing the actions of agents:Sensing the actions of agents:– Self knowledge (what am I doing)Self knowledge (what am I doing)


• Sensing the actions of objects.Sensing the actions of objects.– Smart objectsSmart objects

• Sensing the state of events:Sensing the state of events:– Self (action postconditions)Self (action postconditions)


• Sensing the actions of agents:Sensing the actions of agents:– Self knowledge (what am I doing)Self knowledge (what am I doing)


• Sensing the actions of objects.Sensing the actions of objects.– Smart objectsSmart objects


Input Sensing

• Message passing.Message passing.– Explicit transfer or direct knowledge of state Explicit transfer or direct knowledge of state

information between agents.information between agents.

• Artificial perception.Artificial perception.– Visual/auditory/haptic [collision detection] sensing Visual/auditory/haptic [collision detection] sensing

to attend to and observe local context.to attend to and observe local context.

• Situation awareness.Situation awareness.– Recognizing complex relationships.Recognizing complex relationships.

• Message passing.Message passing.– Explicit transfer or direct knowledge of state Explicit transfer or direct knowledge of state

information between agents.information between agents.

• Artificial perception.Artificial perception.– Visual/auditory/haptic [collision detection] sensing Visual/auditory/haptic [collision detection] sensing

to attend to and observe local context.to attend to and observe local context.

• Situation awareness.Situation awareness.– Recognizing complex relationships.Recognizing complex relationships.


Training the Agent Model

Sense

Act

Control

Agent modelAgent model

• Hand-coded procedures.• Rule-based systems.• Natural Language

instructions.• By example

(demonstration).


Recall: Parameterized Action Representation (PAR)

• Representation derived from Representation derived from BOTHBOTH NL NL analyses and animation requirements:analyses and animation requirements:







• Representation derived from Representation derived from BOTHBOTH NL NL analyses and animation requirements:analyses and animation requirements:








Examples of PAR Action Fragments

• Preparatory Specifications:Preparatory Specifications:– If not at proper location to execute action, get there If not at proper location to execute action, get there

and get into correct pose to continue.and get into correct pose to continue.

• Applicability Conditions:Applicability Conditions:– In order to use a gun, the agent must have one; he In order to use a gun, the agent must have one; he

does does notnot have to go find one. have to go find one.

• Termination Conditions:Termination Conditions:– ““Draw gun” terminates when the gun is no longer in Draw gun” terminates when the gun is no longer in

the holster.the holster.

• Preparatory Specifications:Preparatory Specifications:– If not at proper location to execute action, get there If not at proper location to execute action, get there

and get into correct pose to continue.and get into correct pose to continue.

• Applicability Conditions:Applicability Conditions:– In order to use a gun, the agent must have one; he In order to use a gun, the agent must have one; he

does does notnot have to go find one. have to go find one.

• Termination Conditions:Termination Conditions:– ““Draw gun” terminates when the gun is no longer in Draw gun” terminates when the gun is no longer in

the holster.the holster.


Examples of PAR Action Fragments

• Path parameters:Path parameters:– Walk to a given location.Walk to a given location.– Reach to a given place.Reach to a given place.

• Agent manner:Agent manner:– Walk style.Walk style.– Expressive content (EMOTE parameters).Expressive content (EMOTE parameters).

• Postconditions:Postconditions:– After “receive object” action terminates, agent has After “receive object” action terminates, agent has

object.object.

• Path parameters:Path parameters:– Walk to a given location.Walk to a given location.– Reach to a given place.Reach to a given place.

• Agent manner:Agent manner:– Walk style.Walk style.– Expressive content (EMOTE parameters).Expressive content (EMOTE parameters).

• Postconditions:Postconditions:– After “receive object” action terminates, agent has After “receive object” action terminates, agent has

object.object.


Case Study: The Virtual Reality Checkpoint Trainer

• Joint ONR Project between UPenn, UHouston, Joint ONR Project between UPenn, UHouston, and EAI.and EAI.

• Multi-agent and/or avatar situation.Multi-agent and/or avatar situation.

• Process simulator for traffic.Process simulator for traffic.

• Autonomous agents.Autonomous agents.

• Real-time behaviors and reactions.Real-time behaviors and reactions.

• Natural Language input for “standing orders”. Natural Language input for “standing orders”.

• (Next step: Live trainees in VR.)(Next step: Live trainees in VR.)

• Joint ONR Project between UPenn, UHouston, Joint ONR Project between UPenn, UHouston, and EAI.and EAI.

• Multi-agent and/or avatar situation.Multi-agent and/or avatar situation.

• Process simulator for traffic.Process simulator for traffic.

• Autonomous agents.Autonomous agents.

• Real-time behaviors and reactions.Real-time behaviors and reactions.

• Natural Language input for “standing orders”. Natural Language input for “standing orders”.

• (Next step: Live trainees in VR.)(Next step: Live trainees in VR.)


Virtual Environment


The Checkpoint Scene


Components of the Checkpoint Scenario

• The PAR system architecture.The PAR system architecture.

• Agents and behavior rules.Agents and behavior rules.

• The Actionary.The Actionary.

• Actions currently represented.Actions currently represented.

• Python implementation.Python implementation.

• Natural Language inputs.Natural Language inputs.

• But first, the video.But first, the video.

• The PAR system architecture.The PAR system architecture.

• Agents and behavior rules.Agents and behavior rules.

• The Actionary.The Actionary.

• Actions currently represented.Actions currently represented.

• Python implementation.Python implementation.

• Natural Language inputs.Natural Language inputs.

• But first, the video.But first, the video.


NL2PANL2PARRNL2PANL2PARR

ExecutioExecutionnEngineEngine

PAR SYSTEM PAR SYSTEM ARCHITECTUREARCHITECTURE

VisualizVisualizerer

Jack Jack ToolkitToolkit

Motion Motion GeneratoGeneratorsrs

Rule Rule ManagManagerer

ActionaryActionary

Actions

Objects

Agent Proc 1Agent Proc 1QueueManager

ProcessManager


ProcessManager

Agent Proc NAgent Proc NQueueManager

ProcessManager


Execution Engine

• Main system control loop.Main system control loop.

• Maintains global timer for synchronization.Maintains global timer for synchronization.

• Inputs NL instructions.Inputs NL instructions.

• Outputs scene updates.Outputs scene updates.

• Main system control loop.Main system control loop.

• Maintains global timer for synchronization.Maintains global timer for synchronization.

• Inputs NL instructions.Inputs NL instructions.

• Outputs scene updates.Outputs scene updates.


PAR Representations

UPARUPAR (Uninstantiated PAR):(Uninstantiated PAR): Contains default Contains default applicability conditions, preparatory applicability conditions, preparatory specifications, execution steps; stored in the specifications, execution steps; stored in the ActionaryActionaryTMTM..

IPARIPAR (Instantiated PAR): (Instantiated PAR): UPAR instantiated UPAR instantiated with specific information on agent, physical with specific information on agent, physical objects, manner, termination conditions, etc. objects, manner, termination conditions, etc.

UPARUPAR (Uninstantiated PAR):(Uninstantiated PAR): Contains default Contains default applicability conditions, preparatory applicability conditions, preparatory specifications, execution steps; stored in the specifications, execution steps; stored in the ActionaryActionaryTMTM..

IPARIPAR (Instantiated PAR): (Instantiated PAR): UPAR instantiated UPAR instantiated with specific information on agent, physical with specific information on agent, physical objects, manner, termination conditions, etc. objects, manner, termination conditions, etc.









ActionaryActionary

Actions

Objects


ProcessManager


ProcessManager


ProcessManager


Agent Process

• QueueQueue ManagerManager maintains a priority-based, maintains a priority-based, multi-layered, preemptive queue of all the multi-layered, preemptive queue of all the IPARs to be executed by the agent.IPARs to be executed by the agent.

• ProcessProcess ManagerManager processes each IPAR, processes each IPAR, checks terminations, applicability, checks terminations, applicability, preparatory specs, and execution steps.preparatory specs, and execution steps.

• Trigger actions based on emotion and Trigger actions based on emotion and context.context.

• QueueQueue ManagerManager maintains a priority-based, maintains a priority-based, multi-layered, preemptive queue of all the multi-layered, preemptive queue of all the IPARs to be executed by the agent.IPARs to be executed by the agent.

• ProcessProcess ManagerManager processes each IPAR, processes each IPAR, checks terminations, applicability, checks terminations, applicability, preparatory specs, and execution steps.preparatory specs, and execution steps.

• Trigger actions based on emotion and Trigger actions based on emotion and context.context.


Agent Rule Manager

• Relays IPARs generated, for “immediate Relays IPARs generated, for “immediate instructions,” by the NL2PAR module to the instructions,” by the NL2PAR module to the correct Agent Process.correct Agent Process.

• Stores translated “standing orders” as Stores translated “standing orders” as complex rules in a rule table.complex rules in a rule table.

• Evaluates the rules at each frame of the Evaluates the rules at each frame of the simulation and sends the generated IPARs, if simulation and sends the generated IPARs, if any, to the appropriate Agent Process.any, to the appropriate Agent Process.

• Relays IPARs generated, for “immediate Relays IPARs generated, for “immediate instructions,” by the NL2PAR module to the instructions,” by the NL2PAR module to the correct Agent Process.correct Agent Process.

• Stores translated “standing orders” as Stores translated “standing orders” as complex rules in a rule table.complex rules in a rule table.

• Evaluates the rules at each frame of the Evaluates the rules at each frame of the simulation and sends the generated IPARs, if simulation and sends the generated IPARs, if any, to the appropriate Agent Process.any, to the appropriate Agent Process.









ActionaryActionary

Actions

Objects


ProcessManager


ProcessManager


ProcessManager


The Actionary™

• Links natural language and actions.Links natural language and actions.

• Holds persistent definitions (database) of Holds persistent definitions (database) of actions as UPARs.actions as UPARs.

• Constructed through GUI or (eventually) Constructed through GUI or (eventually) natural language input.natural language input.

• Goes beyond motion capture libraries.Goes beyond motion capture libraries.

• Makes use of PaT-Nets and all lower level Makes use of PaT-Nets and all lower level motion generation tools during execution.motion generation tools during execution.

• Links natural language and actions.Links natural language and actions.

• Holds persistent definitions (database) of Holds persistent definitions (database) of actions as UPARs.actions as UPARs.

• Constructed through GUI or (eventually) Constructed through GUI or (eventually) natural language input.natural language input.

• Goes beyond motion capture libraries.Goes beyond motion capture libraries.

• Makes use of PaT-Nets and all lower level Makes use of PaT-Nets and all lower level motion generation tools during execution.motion generation tools during execution.


PAR Tree for Checkpoint Actions

Tree showing current Actionary of PARs for Tree showing current Actionary of PARs for checkpoint scene. checkpoint scene. Tree showing current Actionary of PARs for Tree showing current Actionary of PARs for checkpoint scene. checkpoint scene.


Actions in Detail


• Actionary of defined PARs and objects is Actionary of defined PARs and objects is coded in coded in PythonPython..

• The execution steps of a PAR defined in a The execution steps of a PAR defined in a Python script are interpreted and Python script are interpreted and dynamically expanded into C++ PaT-Nets.dynamically expanded into C++ PaT-Nets.

• A PAR can be completely created from the A PAR can be completely created from the GUI and dynamically added to the working GUI and dynamically added to the working memory.memory.

• Actionary of defined PARs and objects is Actionary of defined PARs and objects is coded in coded in PythonPython..

• The execution steps of a PAR defined in a The execution steps of a PAR defined in a Python script are interpreted and Python script are interpreted and dynamically expanded into C++ PaT-Nets.dynamically expanded into C++ PaT-Nets.

• A PAR can be completely created from the A PAR can be completely created from the GUI and dynamically added to the working GUI and dynamically added to the working memory.memory.

Python Integration in PAR (1)


Python Integration in PAR (2)

• Actionary database loaded into working Actionary database loaded into working memory during each application run.memory during each application run.

• Specifications of the conditional properties Specifications of the conditional properties (termination, applicability, preparatory) of a (termination, applicability, preparatory) of a PAR are stored as Python scripts and can be PAR are stored as Python scripts and can be easily altered on the fly.easily altered on the fly.

• Actionary database loaded into working Actionary database loaded into working memory during each application run.memory during each application run.

• Specifications of the conditional properties Specifications of the conditional properties (termination, applicability, preparatory) of a (termination, applicability, preparatory) of a PAR are stored as Python scripts and can be PAR are stored as Python scripts and can be easily altered on the fly.easily altered on the fly.


Why a Natural Language Interface?

Give Give complexcomplex commands (when a menu just commands (when a menu just won’t do!)won’t do!)

• Instructions with conjunctions and relative Instructions with conjunctions and relative clauses:clauses:

– ““Go to the closet Susan opened and get a Go to the closet Susan opened and get a flashlight.”flashlight.”

Give Give complexcomplex commands (when a menu just commands (when a menu just won’t do!)won’t do!)

• Instructions with conjunctions and relative Instructions with conjunctions and relative clauses:clauses:

– ““Go to the closet Susan opened and get a Go to the closet Susan opened and get a flashlight.”flashlight.”


Provide Information

• Answer questions about the virtual Answer questions about the virtual environment, other agents, or the tasks:environment, other agents, or the tasks:

– “ “What is now in the tool box?”What is now in the tool box?”

– “ “Where is Sam going?”Where is Sam going?”

– “ “Can Lucy see Charlie?”Can Lucy see Charlie?”

– “ “What’s the next step in this procedure?”What’s the next step in this procedure?”

– “ “When do I stop doing this?”When do I stop doing this?”

• Answer questions about the virtual Answer questions about the virtual environment, other agents, or the tasks:environment, other agents, or the tasks:

– “ “What is now in the tool box?”What is now in the tool box?”

– “ “Where is Sam going?”Where is Sam going?”

– “ “Can Lucy see Charlie?”Can Lucy see Charlie?”

– “ “What’s the next step in this procedure?”What’s the next step in this procedure?”

– “ “When do I stop doing this?”When do I stop doing this?”


Give “Standing Orders”

• Provide persistent instructions that depend Provide persistent instructions that depend on trigger conditions:on trigger conditions:

– “ “When the door opens, go inside.”When the door opens, go inside.”

– “ “If someone’s glass is empty, fill it.”If someone’s glass is empty, fill it.”

– “ “Drink only from your own glass.”Drink only from your own glass.”

– “ “If the driver has a gun, run for cover.”If the driver has a gun, run for cover.”

• More examples in the video.More examples in the video.

• Provide persistent instructions that depend Provide persistent instructions that depend on trigger conditions:on trigger conditions:

– “ “When the door opens, go inside.”When the door opens, go inside.”

– “ “If someone’s glass is empty, fill it.”If someone’s glass is empty, fill it.”

– “ “Drink only from your own glass.”Drink only from your own glass.”

– “ “If the driver has a gun, run for cover.”If the driver has a gun, run for cover.”

• More examples in the video.More examples in the video.


Natural Language Instructions

• Uses Uses XTAG Tree Adjoining GrammarXTAG Tree Adjoining Grammar: parser : parser w/ broad coverage English grammar.w/ broad coverage English grammar.

• XTAG translates parse trees into PARs.XTAG translates parse trees into PARs.

• Uses modeled environment to choose Uses modeled environment to choose correct lexical semantics (sense correct lexical semantics (sense disambiguation and reference binding).disambiguation and reference binding).


• Uses Uses XTAG Tree Adjoining GrammarXTAG Tree Adjoining Grammar: parser : parser w/ broad coverage English grammar.w/ broad coverage English grammar.

• XTAG translates parse trees into PARs.XTAG translates parse trees into PARs.

• Uses modeled environment to choose Uses modeled environment to choose correct lexical semantics (sense correct lexical semantics (sense disambiguation and reference binding).disambiguation and reference binding).



Working Toward Human-ness:Two Hypotheses

• Better understanding of real human Better understanding of real human movement ought to increase the movement ought to increase the naturalnessnaturalness of embodied agent behaviors.of embodied agent behaviors.

• Human body motor systems work in Human body motor systems work in integratedintegrated ways, controlled by high level ways, controlled by high level goals and intentions and only weakly goals and intentions and only weakly accessible to deliberate intervention.accessible to deliberate intervention.

““We can’t help how we act.”We can’t help how we act.”

• Better understanding of real human Better understanding of real human movement ought to increase the movement ought to increase the naturalnessnaturalness of embodied agent behaviors.of embodied agent behaviors.

• Human body motor systems work in Human body motor systems work in integratedintegrated ways, controlled by high level ways, controlled by high level goals and intentions and only weakly goals and intentions and only weakly accessible to deliberate intervention.accessible to deliberate intervention.

““We can’t help how we act.”We can’t help how we act.”


How to we Realize these Hypotheses?

• Use Cognitive Science models and data if Use Cognitive Science models and data if possible.possible.

• Use experience and models from human Use experience and models from human motion experts to develop appropriate motion experts to develop appropriate character motions.character motions.

• Drive character from within, not just by Drive character from within, not just by reaction.reaction.

• Build more integrated motion controllers.Build more integrated motion controllers.

• Use Cognitive Science models and data if Use Cognitive Science models and data if possible.possible.

• Use experience and models from human Use experience and models from human motion experts to develop appropriate motion experts to develop appropriate character motions.character motions.

• Drive character from within, not just by Drive character from within, not just by reaction.reaction.

• Build more integrated motion controllers.Build more integrated motion controllers.


AVA: Automated Visual Attending(Sonu Chopra) Input:Input:

• List of cognitive and motor tasks (e.g., walk to table, search for target, monitor List of cognitive and motor tasks (e.g., walk to table, search for target, monitor

object).object).

Output:Output:

• Animation of character’s head, eye, and body movements. Attending behavior Animation of character’s head, eye, and body movements. Attending behavior

emerges as a competition between deliberate, involuntary and spontaneous emerges as a competition between deliberate, involuntary and spontaneous

attention.attention.


Purpose

Visual AttentionVisual Attention is an important characteristic of is an important characteristic of human activity:human activity:• Fills in unspecified “behavioral detail”.Fills in unspecified “behavioral detail”.

• Models competing events, increasing cognitive load and Models competing events, increasing cognitive load and visual idling.visual idling.

• Modifies motor activity based on visual inputs.Modifies motor activity based on visual inputs.

• Interleaves eye and motor behaviors.Interleaves eye and motor behaviors.

Visual AttentionVisual Attention is an important characteristic of is an important characteristic of human activity:human activity:• Fills in unspecified “behavioral detail”.Fills in unspecified “behavioral detail”.

• Models competing events, increasing cognitive load and Models competing events, increasing cognitive load and visual idling.visual idling.

• Modifies motor activity based on visual inputs.Modifies motor activity based on visual inputs.

• Interleaves eye and motor behaviors.Interleaves eye and motor behaviors.


Psychologically Motivated

Model structure and inputs from:Model structure and inputs from:

• Cognitive psychologyCognitive psychology

• Biologically inspired models of computer Biologically inspired models of computer vision vision

• Human ergonomicsHuman ergonomics

Implement as a PaT-Net: GazeNet.Implement as a PaT-Net: GazeNet.

Model structure and inputs from:Model structure and inputs from:

• Cognitive psychologyCognitive psychology

• Biologically inspired models of computer Biologically inspired models of computer vision vision

• Human ergonomicsHuman ergonomics

Implement as a PaT-Net: GazeNet.Implement as a PaT-Net: GazeNet.


Cognitive Psychology --Types of Attending Behavior

Deliberate (voluntary or endogenous) attention

Involuntary (exogenous and spontaneous) attention

These compete for the attention resource.These compete for the attention resource.


Types of Eye Behaviors

• Visual SearchVisual Search

• MonitoringMonitoring– LocomotionLocomotion– Visual TrackingVisual Tracking– Limit ConditionsLimit Conditions

• Reach and GraspReach and Grasp

• Attention Capture by Peripheral MotionAttention Capture by Peripheral Motion

• Spontaneous Looking (Idling)Spontaneous Looking (Idling)

• Visual SearchVisual Search

• MonitoringMonitoring– LocomotionLocomotion– Visual TrackingVisual Tracking– Limit ConditionsLimit Conditions

• Reach and GraspReach and Grasp

• Attention Capture by Peripheral MotionAttention Capture by Peripheral Motion

• Spontaneous Looking (Idling)Spontaneous Looking (Idling)


Major Components of AVA GazeNet

• Queue of sites (or objects) Queue of sites (or objects) related to related to deliberate tasks that are currently vying for deliberate tasks that are currently vying for attention -- attention -- IntentionListIntentionList

• Queue of objects Queue of objects in an agent’s periphery in an agent’s periphery that are moving -- that are moving -- PlistPlist

• Spontaneous Looking Spontaneous Looking -- look at locations -- look at locations with highest local feature contrast.with highest local feature contrast.


AArrcchhititeeccttuurree

Implement as a PaT-Net: GazeNet.


Motor Activity Modified byVisual Input

• If attention captured by peripheral motion, If attention captured by peripheral motion, continue to “deliberately track” object if on continue to “deliberately track” object if on collision course.collision course.

• Slow down motion (walk or reach) in case Slow down motion (walk or reach) in case of increasing cognitive load.of increasing cognitive load.

• Increase response time to task targets with Increase response time to task targets with increasing (deliberate) load or in the increasing (deliberate) load or in the presence of peripheral motion.presence of peripheral motion.

• If attention captured by peripheral motion, If attention captured by peripheral motion, continue to “deliberately track” object if on continue to “deliberately track” object if on collision course.collision course.

• Slow down motion (walk or reach) in case Slow down motion (walk or reach) in case of increasing cognitive load.of increasing cognitive load.

• Increase response time to task targets with Increase response time to task targets with increasing (deliberate) load or in the increasing (deliberate) load or in the presence of peripheral motion.presence of peripheral motion.


Peripheral Motion Sensor

• Sample for motion (by querying scene Sample for motion (by querying scene graph) those objects that fall in an agent’s graph) those objects that fall in an agent’s peripheral field of view (between 10º and 90º peripheral field of view (between 10º and 90º horizontal and 10º and 65º vertical).horizontal and 10º and 65º vertical).

• Add objects that move to Add objects that move to PlistPlist..


Peripheral Motion Sensor

• If moving object “captures” attention, agent If moving object “captures” attention, agent estimates collision likelihood based on estimates collision likelihood based on velocity and heading. If likely, deliberate velocity and heading. If likely, deliberate tracking is performed.tracking is performed.

• Presence increases response time to Presence increases response time to deliberate targets even if agent doesn’t deliberate targets even if agent doesn’t overtly orient.overtly orient.


Interleaving and theTaskQ Manager

Distinction between eye behavior and Distinction between eye behavior and underlying agent motion allows for underlying agent motion allows for attention attention interleavinginterleaving. If the agent is expert, eye . If the agent is expert, eye behavior for the next task assigned to agent behavior for the next task assigned to agent is initiated before prior motor activity is is initiated before prior motor activity is complete: e.g., looking at goal of reach complete: e.g., looking at goal of reach before walk to destination complete.before walk to destination complete.


Potential Applications

• Human data (eye tracking studies) can be Human data (eye tracking studies) can be input to AVA and visualized in an embodied input to AVA and visualized in an embodied agent.agent.

• AVA can be extended to model situation AVA can be extended to model situation awareness (when do critical events remain awareness (when do critical events remain unattended) for:unattended) for:

– gamesgames

– real-time simulationsreal-time simulations

• Human data (eye tracking studies) can be Human data (eye tracking studies) can be input to AVA and visualized in an embodied input to AVA and visualized in an embodied agent.agent.

• AVA can be extended to model situation AVA can be extended to model situation awareness (when do critical events remain awareness (when do critical events remain unattended) for:unattended) for:

– gamesgames

– real-time simulationsreal-time simulations


Animating Agent Mannerfrom PAR (Chi, Costa, Zhao)

• Motion control paradigm for a Motion control paradigm for a parameterized range of natural-looking parameterized range of natural-looking movements.movements.

• Based on Effort component of Rudolf Based on Effort component of Rudolf Laban’s movement theory (LMA).Laban’s movement theory (LMA).

• Proceduralizes qualitative aspects of Proceduralizes qualitative aspects of movement while providing textual movement while providing textual descriptors along just four dimensions.descriptors along just four dimensions.

• Motion control paradigm for a Motion control paradigm for a parameterized range of natural-looking parameterized range of natural-looking movements.movements.

• Based on Effort component of Rudolf Based on Effort component of Rudolf Laban’s movement theory (LMA).Laban’s movement theory (LMA).

• Proceduralizes qualitative aspects of Proceduralizes qualitative aspects of movement while providing textual movement while providing textual descriptors along just four dimensions.descriptors along just four dimensions.


Effort Motion Factors

Four factors range from an Four factors range from an

indulgingindulging extreme to a extreme to a fightingfighting extreme: extreme:

SpaceSpace: Indirect ------------------ Direct: Indirect ------------------ Direct

WeightWeight: Light --------------------- Strong: Light --------------------- Strong

TimeTime: Sustained ------------- Sudden: Sustained ------------- Sudden

FlowFlow: Free --------------------- Bound: Free --------------------- Bound

Four factors range from an Four factors range from an

indulgingindulging extreme to a extreme to a fightingfighting extreme: extreme:

SpaceSpace: Indirect ------------------ Direct: Indirect ------------------ Direct

WeightWeight: Light --------------------- Strong: Light --------------------- Strong

TimeTime: Sustained ------------- Sudden: Sustained ------------- Sudden

FlowFlow: Free --------------------- Bound: Free --------------------- Bound


Shape Motion Factors

Four factors:Four factors:

SagittalSagittal: Advancing ----------- Retreating: Advancing ----------- Retreating

VerticalVertical: Rising ----------------- Sinking: Rising ----------------- Sinking

HorizontalHorizontal: Spreading ----------- Enclosing: Spreading ----------- Enclosing

FlowFlow: Growing -------------- Shrinking: Growing -------------- Shrinking

Four factors:Four factors:

SagittalSagittal: Advancing ----------- Retreating: Advancing ----------- Retreating

VerticalVertical: Rising ----------------- Sinking: Rising ----------------- Sinking

HorizontalHorizontal: Spreading ----------- Enclosing: Spreading ----------- Enclosing

FlowFlow: Growing -------------- Shrinking: Growing -------------- Shrinking


Kinematic Model of Effort and Shape

path curvature path curvature

interpolation spaceinterpolation space

# of frames between # of frames between keypointskeypoints

velocity curvevelocity curveparametersparameters

path curvature path curvature

interpolation spaceinterpolation space

# of frames between # of frames between keypointskeypoints

velocity curvevelocity curveparametersparameters

anticipationanticipation

overshootovershoot

squash & stretchsquash & stretch

breathbreath

wrist bendwrist bend

arm twistarm twist

limb volumelimb volume

anticipationanticipation

overshootovershoot

squash & stretchsquash & stretch

breathbreath

wrist bendwrist bend

arm twistarm twist

limb volumelimb volume

• Designed with LMA expert guidance.Designed with LMA expert guidance.• Parameters include:Parameters include:


EMOTE (Expressive MOTion Engine) Implementation

• 3D animation control module using LMA 3D animation control module using LMA Effort and Shape motion control scheme.Effort and Shape motion control scheme.

• Spatial description as series of end-effector Spatial description as series of end-effector positions.positions.

• Qualitative description using Effort and Qualitative description using Effort and Shape sliders (parameters).Shape sliders (parameters).

• Works with Works with inverse kinematicsinverse kinematics to generate to generate real-time motion.real-time motion.

• 3D animation control module using LMA 3D animation control module using LMA Effort and Shape motion control scheme.Effort and Shape motion control scheme.

• Spatial description as series of end-effector Spatial description as series of end-effector positions.positions.

• Qualitative description using Effort and Qualitative description using Effort and Shape sliders (parameters).Shape sliders (parameters).

• Works with Works with inverse kinematicsinverse kinematics to generate to generate real-time motion.real-time motion.


EMOTE Interface:


EMOTE Interface -- Effort and Shape Sliders


EffortPhrasing


ShapePhrasing


EMOTE Sampler

VideoVideo VideoVideo


Define Links between Gesture Selection and Agent Model

Gesture performance cues agent state.Gesture performance cues agent state.

• Model individuals (specific people).Model individuals (specific people).

• Normal people show a variety of EMOTE Normal people show a variety of EMOTE parameters during gestures.parameters during gestures.

• Emotional states and some pathologies Emotional states and some pathologies indicated by reduced spectrum of EMOTE indicated by reduced spectrum of EMOTE parameters.parameters.

• Model such parameter distributions. Model such parameter distributions.

Gesture performance cues agent state.Gesture performance cues agent state.

• Model individuals (specific people).Model individuals (specific people).

• Normal people show a variety of EMOTE Normal people show a variety of EMOTE parameters during gestures.parameters during gestures.

• Emotional states and some pathologies Emotional states and some pathologies indicated by reduced spectrum of EMOTE indicated by reduced spectrum of EMOTE parameters.parameters.

• Model such parameter distributions. Model such parameter distributions.


Towards Interactively Building Parameterized Actions

• Motion abstraction and mapping with Motion abstraction and mapping with spatial constraints (SIGGRAPH ‘99 Sketch: spatial constraints (SIGGRAPH ‘99 Sketch: Rama Bindiganavale)Rama Bindiganavale)

• Automatically abstract semantically Automatically abstract semantically significant points in an agent’s action into significant points in an agent’s action into spatial and visual constraints which are then spatial and visual constraints which are then used to construct a PAR for that motion.used to construct a PAR for that motion.

• Populate the Actionary from real examples.Populate the Actionary from real examples.

• Motion abstraction and mapping with Motion abstraction and mapping with spatial constraints (SIGGRAPH ‘99 Sketch: spatial constraints (SIGGRAPH ‘99 Sketch: Rama Bindiganavale)Rama Bindiganavale)

• Automatically abstract semantically Automatically abstract semantically significant points in an agent’s action into significant points in an agent’s action into spatial and visual constraints which are then spatial and visual constraints which are then used to construct a PAR for that motion.used to construct a PAR for that motion.

• Populate the Actionary from real examples.Populate the Actionary from real examples.


Objectives

• Map PARs to agents of different Map PARs to agents of different anthropometric sizes while maintaining the anthropometric sizes while maintaining the spatial and visual constraints and similar spatial and visual constraints and similar motionmotion style style..

• Consider actions that involve external Consider actions that involve external objects and the agent’s own body.objects and the agent’s own body.

• Focus on goal achievement rather than Focus on goal achievement rather than trajectories.trajectories.

• Map PARs to agents of different Map PARs to agents of different anthropometric sizes while maintaining the anthropometric sizes while maintaining the spatial and visual constraints and similar spatial and visual constraints and similar motionmotion style style..

• Consider actions that involve external Consider actions that involve external objects and the agent’s own body.objects and the agent’s own body.

• Focus on goal achievement rather than Focus on goal achievement rather than trajectories.trajectories.


Related Animation Techniques

• Motion re-targetingMotion re-targeting

• Motion signal processingMotion signal processing

• But we use concepts from computer vision But we use concepts from computer vision analysis:analysis:• Constraint recognitionConstraint recognition

– for end effectors and visual attention.for end effectors and visual attention.

• Spatial proximity of tagged featuresSpatial proximity of tagged features

– zero-crossings of acceleration.zero-crossings of acceleration.

• Motion re-targetingMotion re-targeting

• Motion signal processingMotion signal processing

• But we use concepts from computer vision But we use concepts from computer vision analysis:analysis:• Constraint recognitionConstraint recognition

– for end effectors and visual attention.for end effectors and visual attention.

• Spatial proximity of tagged featuresSpatial proximity of tagged features

– zero-crossings of acceleration.zero-crossings of acceleration.


Method

• Capture motion from primary agent.Capture motion from primary agent.

• Find acceleration zero-crossings.Find acceleration zero-crossings.

• Co-occurrence of zero-crossings and Co-occurrence of zero-crossings and spatial proximity of end-effectors with spatial proximity of end-effectors with objects automatically indicate start/end of objects automatically indicate start/end of constraints.constraints.

• Objects can be self / fixed / mobile.Objects can be self / fixed / mobile.

• Capture motion from primary agent.Capture motion from primary agent.

• Find acceleration zero-crossings.Find acceleration zero-crossings.

• Co-occurrence of zero-crossings and Co-occurrence of zero-crossings and spatial proximity of end-effectors with spatial proximity of end-effectors with objects automatically indicate start/end of objects automatically indicate start/end of constraints.constraints.

• Objects can be self / fixed / mobile.Objects can be self / fixed / mobile.


“Drink from a mug”


Motions Stored in PAR

• Constraints become part of PAR Constraints become part of PAR description of movement.description of movement.

• Motions may be replayed on different-sized Motions may be replayed on different-sized agents.agents.

• Movement style and significant semantic Movement style and significant semantic features are preserved.features are preserved.

– contactscontacts

– visual attentionvisual attention

• Constraints become part of PAR Constraints become part of PAR description of movement.description of movement.

• Motions may be replayed on different-sized Motions may be replayed on different-sized agents.agents.

• Movement style and significant semantic Movement style and significant semantic features are preserved.features are preserved.

– contactscontacts

– visual attentionvisual attention


Motion Abstraction

VideoVideo VideoVideo


Conclusions

• Instruction execution must be context-, Instruction execution must be context-, perception-, and agent-sensitive.perception-, and agent-sensitive.

• Cognitive Science and Movement Analysis Cognitive Science and Movement Analysis help model human behaviors.help model human behaviors.

• Language interfaces (through PAR) expand Language interfaces (through PAR) expand usability and agent building.usability and agent building.

• A new concept in dictionaries: The A new concept in dictionaries: The Actionary™ Actionary™ translates text into action.translates text into action.

• Instruction execution must be context-, Instruction execution must be context-, perception-, and agent-sensitive.perception-, and agent-sensitive.

• Cognitive Science and Movement Analysis Cognitive Science and Movement Analysis help model human behaviors.help model human behaviors.

• Language interfaces (through PAR) expand Language interfaces (through PAR) expand usability and agent building.usability and agent building.

• A new concept in dictionaries: The A new concept in dictionaries: The Actionary™ Actionary™ translates text into action.translates text into action.


Acknowledgments

ColleaguesColleagues: Martha Palmer, Aravind Joshi, Jan : Martha Palmer, Aravind Joshi, Jan Allbeck, Aaron Bloomfield, MeeRan Byun, Diane Allbeck, Aaron Bloomfield, MeeRan Byun, Diane Chi, Sonu Chopra, Monica Costa, Rama Chi, Sonu Chopra, Monica Costa, Rama Bindiganavale, Charles Erignac, Ambarish Bindiganavale, Charles Erignac, Ambarish Goswami, Karin Kipper, Seung-Joo Lee, Sooha Goswami, Karin Kipper, Seung-Joo Lee, Sooha Lee, Jianping Shi, Hogeun Shin, William Schuler, Lee, Jianping Shi, Hogeun Shin, William Schuler, and Liwei Zhao.and Liwei Zhao.

Sponsors: Sponsors: NSF, ONR, DARPA, NASA, ARO NSF, ONR, DARPA, NASA, ARO THANK YOU!THANK YOU!

ColleaguesColleagues: Martha Palmer, Aravind Joshi, Jan : Martha Palmer, Aravind Joshi, Jan Allbeck, Aaron Bloomfield, MeeRan Byun, Diane Allbeck, Aaron Bloomfield, MeeRan Byun, Diane Chi, Sonu Chopra, Monica Costa, Rama Chi, Sonu Chopra, Monica Costa, Rama Bindiganavale, Charles Erignac, Ambarish Bindiganavale, Charles Erignac, Ambarish Goswami, Karin Kipper, Seung-Joo Lee, Sooha Goswami, Karin Kipper, Seung-Joo Lee, Sooha Lee, Jianping Shi, Hogeun Shin, William Schuler, Lee, Jianping Shi, Hogeun Shin, William Schuler, and Liwei Zhao.and Liwei Zhao.

Sponsors: Sponsors: NSF, ONR, DARPA, NASA, ARO NSF, ONR, DARPA, NASA, ARO THANK YOU!THANK YOU!

july 24, 20001smart animated agents -- siggraph course #24 smart animated agents norman i. badler --...

Documents

siggraph course

teamssmart animated

injurysmart animated

issuessmart animated

lunchsmart animated

course schedule

embodied agents

smart animated agentsnorman