The Entertainment Industry Meets Simulation:

The Challenge of the Holodeck

Bill Swartout

Institute for Creative TechnologiesUniversity of Southern California

The ICT is intended to serve as a nexus between the simulation

and entertainment communities

BizTechShow

Institute For Creative Technologies

How can we best simulate reality?

• Two approaches– Military simulation– Entertainment industry

Simulating Reality:Two Approaches

Military Simulation• Literal: try to include as much reality in

simulation as possible• Homogeneous Entertainment Industry• Goal-directed: create effect of real experience

by selectively focusing on elements (real and unreal)

• Hybrid Approach: combine disparate elements

Movie clip

To the Holodeck and beyond…

Mission Rehearsal Exercise Project: New Operations

Mission Rehearsal Exercise Project: ICT VR Theater

What we don’t want…

Movie clip

New Research Areas• Speech Recognition• Natural Language Understanding• Dialogue Management• Natural Language Generation• Speech Synthesis• Gesture Generation• Perception• Personality Variation• Sound Spatialization• Interactive Story Structure

Movie clip

Integration Raises New Challenges:

Natural LanguageProject Leaders: Ed Hovy (ISI), Kevin Knight (ISI),

David Traum (ISI), Shri Narayanan (IMSC)• Speech recognition: Noisy

environment• Multimodal: Face To Face

(speech+gesture), Radio• Multiple Interactors (Sgt, medic,

squads,…)– Messages tailored for multiple

addressees/overhearers• Emotion

• Most speech synthesis sounds unnatural• Even the best have limited

expression (AT&T NextGen)– No command voice

• Use hybrid approach:– Multiple synthesizers with

different “voices”– Command voice

Text toSpeech

Synthesis“Squad Leaders listen up!”

Challenge: Speech SynthesisProject Leaders: Lewis Johnson (ISI), Shri Narayanan

(IMSC)

• Use pre-recorded library for highly emotional speech

Challenge: Modeling EmotionsProject Leaders: Jon Gratch (ICT), Stacy Marsella

(ISI)

Challenge: Modeling EmotionsProject Leaders: Jon Gratch (ICT), Stacy Marsella

(ISI)

Movie clip

Challenge: Modeling EmotionsProject Leaders: Jon Gratch (ICT), Stacy Marsella

(ISI)• Problem addressed:

– Virtual humans need emotions to be believable & realistic

• Approach:– Characters have beliefs, goals, utilities– Construct plans– Events evaluated against plans– If goals thwarted (or enabled) characters

react

Emotional Reaction

Movie clip

Modeling Emotions: Issues

• Integrating gesture, natural language & voice to project realistic behavior

• Managing unintended interpretations

• Hollywood vs. Psychology• Movie emotions quite different from natural

emotions

• Recognizing human emotions

Challenge:Story is Critical

• More than an event list• Successful stories:

– Juxtapose contrasting elements– Surprise (plot twists)– Dilemmas– Things go wrong– Build to climax (and ebb)

Structuring Stories for Intelligent Agents: Issues

• Traditional story telling tricks require linear structure

• Immersed participant needs freewill

• Need “bumpers”– Possibility: director agent controls

environment & characters to limit choice

Reasons for Optimism

• Hybrid approach– Use all the tools in the toolbox

• Entertainment industry as source of knowledge and techniques

• Story context constrains possibilities• Synergy from graphics, sound &

emotional engagement– The mind is a willing collaborator