Using a Goal/Action Architectureto Integrate Modularity and Long-Term Memory into AI Behaviors

Marc AtkinJohn Abercrombie

Why Do We Want Better Why Do We Want Better AI?AI?• Bad AI is always noticedBad AI is always noticed• AI often inconsistent AI often inconsistent

between games – why can’t between games – why can’t we build on previous work?we build on previous work?

• The AI is a big part of what The AI is a big part of what makes a game world makes a game world immersive and funimmersive and fun

What’s Stopping Us?What’s Stopping Us?

• What makes AI complex?What makes AI complex?– AI has many possible reactions AI has many possible reactions

to many situationsto many situations– Many behaviors are operating Many behaviors are operating

on the AI at any given timeon the AI at any given time– Hard to account for all the Hard to account for all the

possible interactionspossible interactions

The Basic Idea: Use The Basic Idea: Use ModularityModularity• It’s not just for coding It’s not just for coding

anymore!anymore!• Produces AI that is easier to Produces AI that is easier to

understand and debugunderstand and debug• Produces AI that it reusable Produces AI that it reusable

within and between projectswithin and between projects• The key pointThe key point: Design the AI : Design the AI

architecture so that it architecture so that it facilitates modularityfacilitates modularity

Part 1: TyrionPart 1: Tyrion

TyrionTyrion

• An architecture for An architecture for specifying and executing AI specifying and executing AI behaviorbehavior

• Implements a Implements a languagelanguage in in which to write behaviorwhich to write behavior

• Encourages hierarchical, Encourages hierarchical, modular designmodular design

Resources, Goals, and Resources, Goals, and ActionsActions• GoalGoal: A description of a: A description of a

desired world statedesired world state• ActionAction (or (or behaviorbehavior):):

A method for achieving a goalA method for achieving a goal• ResourceResource::

An entity An entity required torequired toperform an actionperform an action

Resource HierarchyResource HierarchyTeam

Squads

Vehicles

Characters

Driver/Gunners

Legs/Arms/Head

Resource hierarchyResource hierarchyTeam Strategic level

Tactical levelSquads

Vehicles

Characters

Driver/Gunners

Legs/Arms/Head

IndividualUnits

Motor Control

Action/Goal HierarchyAction/Goal Hierarchy

Attack From All Sides

NormalAttack FlankingAttack

MoveTo

Squad

Character

Legs /Arms

Attack Goal

Fire Weapon

MoveTo GoalFireAt Goal

Attack Goal

Attack Goal

Action StructureAction Structure

• Goal satisfied by actionGoal satisfied by action• Selection heuristicSelection heuristic

– Evaluates appropriateness of actionEvaluates appropriateness of actionin a given situationin a given situation

• Message callback functions for child Message callback functions for child actionsactions

• Message callback functions for Message callback functions for sensorssensors

• Body (Unreal script latent code)Body (Unreal script latent code)– Typically executes over a number of Typically executes over a number of

game ticksgame ticks

Action IdiomsAction Idioms

• Simplify writing action Simplify writing action bodiesbodies

• Encapsulate commonly Encapsulate commonly performed tasksperformed tasks

• Examples:Examples:– WaitForAllGoalsWaitForAllGoals– WaitForAnyGoalWaitForAnyGoal

Execution ModelExecution Model

• 10 times a second:10 times a second:– Iterate over every resource:Iterate over every resource:

• Order unmatched goals by priorityOrder unmatched goals by priority• Find actions that achieve themFind actions that achieve them• Tick every running actionTick every running action• Tick every active periodic sensor;Tick every active periodic sensor;

send messagessend messages• If an action completes, mark its If an action completes, mark its

goal as achieved/failed; send goal as achieved/failed; send messagesmessages

Part 2: Building Modular Part 2: Building Modular ActionsActions

SWAT4SWAT4

• What is SWAT4?What is SWAT4?– First person tactical shooter First person tactical shooter – Player is team leaderPlayer is team leader– Single player – player Single player – player

commands four officer AI’s commands four officer AI’s – Object is to kill or subdue all Object is to kill or subdue all

enemies, and rescue the enemies, and rescue the hostageshostages

What Makes Up the What Makes Up the SWAT4 AI?SWAT4 AI?• Hierarchical resources for

organizing behaviors

• Layer behaviors to form Layer behaviors to form more complex behaviorsmore complex behaviors

Tactical

Individual

Motor Control

CharacterCharacter

SquadSquad

MovementMovement WeaponWeapon

Modularity in SWAT4Modularity in SWAT4

• Start from the ground up – Motor Start from the ground up – Motor ControlControl– The Legs - Movement ResourceThe Legs - Movement Resource

• Basic Movement – move to a position, move to Basic Movement – move to a position, move to a particular object, follow another AIa particular object, follow another AI

• Lower body rotationLower body rotation

– The Arms – Weapon ResourceThe Arms – Weapon Resource• Aiming – aim at a position, to a rotation, at Aiming – aim at a position, to a rotation, at

somethingsomething• Item usage – (i.e. guns, grenades, toolkit, Item usage – (i.e. guns, grenades, toolkit,

explosives)explosives)

Modularity in SWAT4Modularity in SWAT4

• The CharacterThe Character– Controls arms and legsControls arms and legs– Plays special full body Plays special full body

animationsanimations

• The SquadThe Squad– Controls individual characters, Controls individual characters,

tells them what to do so they tells them what to do so they can work togethercan work together

ReuseReuse

• Tyrion framework encourages Tyrion framework encourages tasks to be broken apart into sub-tasks to be broken apart into sub-taskstasks– Higher level behaviors use lower level Higher level behaviors use lower level

behaviors to accomplish complicated behaviors to accomplish complicated taskstasks

– Lower level behaviors can be used Lower level behaviors can be used over and over again by different over and over again by different higher level behaviors higher level behaviors • i.e. Movement, aiming, weapon usagei.e. Movement, aiming, weapon usage

LayeringLayering

• Each behavior on a resource is Each behavior on a resource is responsible for controlling responsible for controlling everything below iteverything below it– i.e. Officers move in a formation i.e. Officers move in a formation

while aiming at points of interest in while aiming at points of interest in the world (Fall in Command)the world (Fall in Command)

• Using multiple characters in a Using multiple characters in a squad allows group to work squad allows group to work together to accomplish moretogether to accomplish more

Organizing BehaviorsOrganizing Behaviors

• Use of action idiomsUse of action idioms– waitForGoal, waitForAllGoals, waitForGoal, waitForAllGoals, waitForAnyGoals, waitForAnyGoals, waitForResourcesAvailablewaitForResourcesAvailable

• Parent action posts child goals to Parent action posts child goals to accomplish tasksaccomplish tasks– on the same resource or one on the same resource or one

lower in the hierarchylower in the hierarchy

Organizing BehaviorsOrganizing Behaviors

• In SWAT4 the Squad posts child In SWAT4 the Squad posts child goals on characters (the officers)goals on characters (the officers)

• Character behaviors usually post Character behaviors usually post child goals on movement and/or child goals on movement and/or weapon resourcesweapon resources

• Each action can wait for child Each action can wait for child goals to complete using action goals to complete using action idiomsidioms

Example: Move Squad to a Example: Move Squad to a PointPoint• Squad behavior (MoveSquadTo):Squad behavior (MoveSquadTo):

– Posts MoveTo Goal on each officerPosts MoveTo Goal on each officer– WaitForAllGoals(MoveToGoals)WaitForAllGoals(MoveToGoals)

• Character behavior (MoveTo):Character behavior (MoveTo):– Posts Aim Around Goal on weapon Posts Aim Around Goal on weapon resourceresource

– Posts MoveToLocation Goal on Posts MoveToLocation Goal on movement resourcemovement resource

– WaitForGoal(MoveToLocation)WaitForGoal(MoveToLocation)

Stack Up (on a door)Stack Up (on a door)

• User points at door, gives User points at door, gives commandcommand– Squad behavior determines points Squad behavior determines points

for Officers to stack up atfor Officers to stack up at– Each officer locomotes to point Each officer locomotes to point

given - while lookinggiven - while lookingaround or aimingaround or aimingat the door ifat the door ifthey are ablethey are able 1

21

2

How Stack Up WorksHow Stack Up Works

• Squad Behavior (Squad Stack Up)Squad Behavior (Squad Stack Up)– Post Stack Up goal on each officerPost Stack Up goal on each officer

• Pass Stack Up Point as destination Pass Stack Up Point as destination parameterparameter

– WaitForAllGoals(StackUpGoals)WaitForAllGoals(StackUpGoals)• Character Behavior (Stack Up)Character Behavior (Stack Up)

– Post Aim At Door goalPost Aim At Door goal– Post MoveTo(StackUpPoint) goalPost MoveTo(StackUpPoint) goal– WaitForGoal(MoveToGoal)WaitForGoal(MoveToGoal)

• Result: Movement & Weapon behaviors Result: Movement & Weapon behaviors act independently of each otheract independently of each other

Move & ClearMove & Clear

• Inherits from the Stack Up behavior Inherits from the Stack Up behavior – Re-use functionality of getting AI’s stacked Re-use functionality of getting AI’s stacked

up on the doorup on the door• Adds new behaviors:Adds new behaviors:

– After finishing stacking up (but not trying After finishing stacking up (but not trying the door): the door): • Officer near the door opens itOfficer near the door opens it• Another Officer told to start moving as Another Officer told to start moving as

soon as the door is openedsoon as the door is opened– Team then moves through the door into next Team then moves through the door into next

room to points specified by a designerroom to points specified by a designer

Breach, Bang & ClearBreach, Bang & Clear

• Inherits from the Move & Clear behaviorInherits from the Move & Clear behavior– Open Door behavior replaced: first officer Open Door behavior replaced: first officer

blows door open using C2 explosives or a blows door open using C2 explosives or a breaching shotgunbreaching shotgun

– Second officer throws flashbang grenade Second officer throws flashbang grenade into room before enteringinto room before entering

• Squad organizes charactersSquad organizes characters• Some basic character behaviors are reused Some basic character behaviors are reused

across many commandsacross many commands– i.e. deploy grenade, deploy shotgun, or i.e. deploy grenade, deploy shotgun, or

deploy C2deploy C2

Movement & W

eapon Behaviors

Squad StackUpSquad

StackUp

Squad BreachBang & Clear

Squad BreachBang & Clear

Squad Move

& Clear

Squad Move

& Clear

Stack UpStack Up

Move & Clear

Move & Clear

Open Door

Open Door

Throw GrenadeThrow

Grenade

Deploy C2

Deploy C2Deploy

Shotgun

DeployShotgu

n

Squad Behaviors

Character Behaviors

Aim AtAim AtMove

ToMove

To

UseGrenad

e

UseGrenad

e

Aim Around

Aim Around

Move To

Move To

Move To

Move To

Move To

Move To

Move To

Move To Inherits From

Child Behavior

Summary and ResultsSummary and Results

• Framework is key to organizing use of systemFramework is key to organizing use of system• Time to prototype and develop AI behaviors Time to prototype and develop AI behaviors

reducedreduced• Interesting SWAT4 Stats:Interesting SWAT4 Stats:

– Number of Squad Behaviors: 33Number of Squad Behaviors: 33– Number of Character Behaviors: 65Number of Character Behaviors: 65– Number of Weapon Behaviors: 10Number of Weapon Behaviors: 10– Number of Movement Behaviors: 11Number of Movement Behaviors: 11– Number of Months Programming: 16 Number of Months Programming: 16

(May 2003 – September 2004)(May 2003 – September 2004)

Part 3: Long-term Part 3: Long-term MemoryMemory

The Problem of The Problem of Forgetful AIForgetful AI• ““Oh hello – have weOh hello – have we

met before?”met before?”• State machines don’tState machines don’t

remember past events remember past events • What’s a natural way to What’s a natural way to

represent task-related represent task-related memory?memory?

Two Types of MemoryTwo Types of Memory

• Task memory:Task memory:– What needs to be doneWhat needs to be done– The context of the current actionThe context of the current action– Represented by the set of goalsRepresented by the set of goals

• Cognitive memory:Cognitive memory:– Sensor data that has Sensor data that has

accumulated during the AI’s life accumulated during the AI’s life timetime

– Represented inside sensors and Represented inside sensors and actionsactions

Tribes: VengeanceTribes: Vengeance

• First-person shooterFirst-person shooter• Emphasizes speed and Emphasizes speed and

freedom of movement freedom of movement (jetpacks!)(jetpacks!)

• 33rdrd in the series in the series

A Day in the Life of a A Day in the Life of a GruntGrunt• A Grunt AI with three goals:A Grunt AI with three goals:

– PatrolGoal (priority 40)PatrolGoal (priority 40)– AttackGoal (priority 50, dormant)AttackGoal (priority 50, dormant)– DodgeGoal (priority 90, dormant)DodgeGoal (priority 90, dormant)

• AttackGoal will activate when AttackGoal will activate when an enemy is sightedan enemy is sighted

• DodgeGoal will activate when DodgeGoal will activate when a visible projectile will hit the a visible projectile will hit the AIAI

Patrol is executing 4040

PatrolGoal

AttackGoal

DodgeGoal

legslegs

armsarms

Patrol is executing 4040

PatrolGoal

Enemy spotted! Sensor wakes up AttackGoal

Attack executes; posts subgoalsfor legs & arms

4040 5050

AttackGoal

DodgeGoal

legslegs

legslegs

armsarms

armsarms

5050X

Patrol is executing 4040

PatrolGoal

Enemy spotted! Sensor wakes up AttackGoal

Attack executes; posts subgoalsfor legs & arms

4040 5050

AttackGoal

DodgeGoal

legslegs

legslegs

armsarms

Projectile spotted! Sensor wakes up DodgeGoal

Dodge executes; needs legs;arms continue executing Attack!

90904040

5050

armsarms

5050

legslegs

armsarms

X

X 5050X

Patrol is executing 4040

PatrolGoal

Enemy spotted! Sensor wakes up AttackGoal

Attack executes; posts subgoalsfor legs & arms

4040 5050

AttackGoal

DodgeGoal

legslegs

legslegs

armsarms

Projectile spotted! Sensor wakes up DodgeGoal

Dodge executes; needs legs;arms continue executing Attack!

90904040

5050

Dodge finishes: legs resumeAttack subgoal

4040 5050

armsarms

5050

legslegs

armsarms

legslegs

armsarms 5050

X

X

X

5050X

Seeing the Big PictureSeeing the Big Picture

• Task memoryTask memory: An AI’s goal set : An AI’s goal set constitutes the “big picture” of constitutes the “big picture” of tasks to be accomplishedtasks to be accomplished– Failed or interrupted goals hang aroundFailed or interrupted goals hang around

• Cognitive memoryCognitive memory: actions : actions potentially contain lots of stored potentially contain lots of stored datadata– High level actions have long life span; High level actions have long life span;

their data sticks around for that time, their data sticks around for that time, tootoo

Seeing the Big PictureSeeing the Big Picture

• Explicit goal representation:Explicit goal representation:AI can reason about AI can reason about remaining tasksremaining tasks

• Behaviors can be chosen that Behaviors can be chosen that achieve multiple goalsachieve multiple goals

• Separate rules decide when Separate rules decide when to remove unachieved goalsto remove unachieved goals

Part 4: CollaborationPart 4: Collaboration

Why Collaborate?Why Collaborate?

• Increases productivityIncreases productivity• Both working on FPS’s starting at about Both working on FPS’s starting at about

the same time using the same engine the same time using the same engine (Unreal)(Unreal)

• Development teams based in Boston Development teams based in Boston (SWAT4) and Canberra, Australia (Tribes) (SWAT4) and Canberra, Australia (Tribes)

• Teams shared code for rendering, Teams shared code for rendering, networking, and morenetworking, and more

• Tyrion was an already tested systemTyrion was an already tested system

How to Work Together?How to Work Together?

• CommunicationCommunication• Shared CodebaseShared Codebase• PerforcePerforce• Programming (of course!)Programming (of course!)

CommunicationCommunication

• Initial technical design document was Initial technical design document was written in Australia, then discussed written in Australia, then discussed via e-mail between offices for weeksvia e-mail between offices for weeks

• E-mail response time was usually a E-mail response time was usually a day (due to time difference)day (due to time difference)

• More complicated work could be More complicated work could be accomplished over the phoneaccomplished over the phone

• An in-person visit helped SWAT4 make An in-person visit helped SWAT4 make greater use of Tyrion’s featuresgreater use of Tyrion’s features

Communication: Lessons Communication: Lessons LearnedLearned• We underestimated the amount of time We underestimated the amount of time

required for effective communication required for effective communication – During design phase, both of us spent During design phase, both of us spent

about 1/3about 1/3rdrd of our time communicating – less of our time communicating – less later onlater on

• We underestimated the amount of time We underestimated the amount of time required for “tech support”required for “tech support”

• Collaboration allowed both teams to Collaboration allowed both teams to identify weaknesses in Tyrion that we identify weaknesses in Tyrion that we would discuss and addresswould discuss and address

Shared CodebaseShared Codebase

• Shared code between offices Shared code between offices to speed development of to speed development of both Tribes & SWAT4both Tribes & SWAT4

• One shared codebase, which One shared codebase, which both projects could integrate both projects could integrate code from or tocode from or to

PerforcePerforce

• Very useful program, allowed Very useful program, allowed for easy collaboration for easy collaboration between officesbetween offices

• We didn't have to worry We didn't have to worry about sharing the code, we about sharing the code, we got to worry about the code got to worry about the code itself!itself!

ProgrammingProgramming

• Marc did most of the implementationMarc did most of the implementation• John's prior experience in Unreal helped John's prior experience in Unreal helped

optimize and streamline the codeoptimize and streamline the code• We both felt comfortable editing the code We both felt comfortable editing the code

in the shared codebase for our own needsin the shared codebase for our own needs• Code today is clean and well commented, Code today is clean and well commented,

although it does contain a fewalthough it does contain a few#if IG_SWAT#if IG_SWAT and and #if IG_TRIBES3#if IG_TRIBES3 preprocessor definitions!preprocessor definitions!

Programming: Lessons Programming: Lessons LearnedLearned• Would have been nice to have shared Would have been nice to have shared

a common AI base classa common AI base class

• Both games exposed weaknesses Both games exposed weaknesses that were addressedthat were addressed– Early in SWAT, the idle goal needed to be Early in SWAT, the idle goal needed to be

satisfied by two subclasses of the base satisfied by two subclasses of the base Idle Action classIdle Action class• Change was made to system to allow Change was made to system to allow

subclasses of actions to satisfy goalssubclasses of actions to satisfy goals

– Claiming leaf (motor control) resources Claiming leaf (motor control) resources issueissue

Programming: Lessons Programming: Lessons LearnedLearned• Inheritance vs Building BlocksInheritance vs Building Blocks

– Building blocksBuilding blocks: new behaviors are : new behaviors are implemented by combining simpler onesimplemented by combining simpler ones

– InheritanceInheritance: Change an existing behavior by : Change an existing behavior by sub-classing sub-classing Tyrion didn’t intend for thisTyrion didn’t intend for this

– Not everyone will use your engine the way Not everyone will use your engine the way you intendedyou intended• Having multiple ways of accomplishing tasks Having multiple ways of accomplishing tasks

allows clients to create and innovateallows clients to create and innovate• Engine’s strength determined by client’s effective Engine’s strength determined by client’s effective

useuse

Collaboration ResultsCollaboration Results

• The shared development of using The shared development of using Tyrion worked wellTyrion worked well

• We feel both games had challenging We feel both games had challenging and fun AI and fun AI

• Its use has made prototyping AI Its use has made prototyping AI behaviors easy and quick!behaviors easy and quick!

• Marc got to wake up at 7 a.m. in the Marc got to wake up at 7 a.m. in the morning for a fire alarm while morning for a fire alarm while staying at John's housestaying at John's house

The FutureThe Future

• We move on to new projects, and We move on to new projects, and will continue to use Tyrionwill continue to use Tyrion

• Still finding improvements and Still finding improvements and new ways of using Tyrionnew ways of using Tyrion

• Engine ImprovementsEngine Improvements– Better goal/sensor integrationBetter goal/sensor integration– Make AI more “designer-friendly”Make AI more “designer-friendly”– Shared AI subclassShared AI subclass

Questions?Questions?