constructing complex npc behavior via multi-objective neuroevolution

Constructing Constructing Complex NPC Complex NPC

Behavior via Multi-Behavior via Multi-Objective Objective

NeuroevolutionNeuroevolutionJacob Schrum – Jacob Schrum – schrum2@cs.utexas.eduschrum2@cs.utexas.edu

Risto Miikkulainen – Risto Miikkulainen – risto@cs.utexas.eduristo@cs.utexas.edu

University of Texas at AustinUniversity of Texas at AustinDepartment of Computer SciencesDepartment of Computer Sciences

IntroductionIntroduction Goal: discover NPC behavior automaticallyGoal: discover NPC behavior automatically BenefitsBenefits

Save production time/effortSave production time/effort Learn counterintuitive behaviorsLearn counterintuitive behaviors Find weaknesses in static scriptsFind weaknesses in static scripts Tailor behavior to human playersTailor behavior to human players

ChallengesChallenges Games are complexGames are complex Multi-modal behavior is requiredMulti-modal behavior is required Success often depends on multiple objectivesSuccess often depends on multiple objectives

RL & Evolution popular approaches RL & Evolution popular approaches Current methods do not explicitly consider multiple Current methods do not explicitly consider multiple

objectives!objectives!

Multi-Objective Multi-Objective OptimizationOptimization

Exceptional performance in any objective is Exceptional performance in any objective is importantimportant

Imagine game with two Imagine game with two objectives: objectives: Damage DealtDamage Dealt Health RemainingHealth Remaining

AA dominates dominates BB iff iff AA is is strictly better in one strictly better in one objective and at least objective and at least as good in as good in othersothers

Population of points Population of points not dominated are best: not dominated are best: Pareto Front Pareto Front

High health but did not deal much damage

Dealt lot of damage,but lost lots of health

Tradeoff between objectives

NSGA-IINSGA-II Evolution: natural approach for finding optimal Evolution: natural approach for finding optimal

population population Non-Dominated Sorting Genetic Algorithm II*Non-Dominated Sorting Genetic Algorithm II*

Population P with size N; Evaluate PPopulation P with size N; Evaluate P Use mutation to get P´ size N; Evaluate P´Use mutation to get P´ size N; Evaluate P´ Calculate non-dominated fronts of {P Calculate non-dominated fronts of {P P´} size 2NP´} size 2N New population size N from highest fronts of {P New population size N from highest fronts of {P P´}P´}

*K. Deb et al. 2000

NeuroevolutionNeuroevolution Genetic Algorithms + Artificial Neural NetworksGenetic Algorithms + Artificial Neural Networks NNs good at generating behaviorNNs good at generating behavior GA creates new nets, evaluates themGA creates new nets, evaluates them Three types of mutations (no crossover used)Three types of mutations (no crossover used)

Perturb Weight Add Connection Add Node

Battle DomainBattle Domain

15 NPCs attack player

PlayerBatInfinite Plane

NPC Fitness ObjectivesNPC Fitness Objectives NPCs have several goalsNPCs have several goals

Inflict damage as a group → Attack Assist BonusInflict damage as a group → Attack Assist Bonus Avoid damage individually → Damage ReceivedAvoid damage individually → Damage Received Stay alive individually → Time AliveStay alive individually → Time Alive

Objectives are contradictory and distinctObjectives are contradictory and distinct

NPCs take damage from bat Player is knocked back by NPC

ExperimentExperiment Multi-objective vs. Single-objectiveMulti-objective vs. Single-objective

Single-objective uses weighted sum of z-Single-objective uses weighted sum of z-scoresscores

Evolve each method against Evolve each method against static static player strategiesplayer strategies Spinning strategySpinning strategy Alternating strategyAlternating strategy Chasing strategyChasing strategy

NPCs must beat NPCs must beat previous player previous player strategy to progress strategy to progress

ResultsResults

Differences for Alternating and Chasing significant with p < .05

Evolved NPC BehaviorsEvolved NPC Behaviors Single-ObjectiveSingle-Objective

Cannot find balance Cannot find balance between attack and defensebetween attack and defense

Behavior is ineffectualBehavior is ineffectual

Multi-ObjectiveMulti-Objective Baiting BehaviorBaiting Behavior

One agent acts as baitOne agent acts as bait Others attack player on blind Others attack player on blind

sideside Charging BehaviorCharging Behavior

Try to dodge bat while Try to dodge bat while rushing inrushing in

Maximizes damage dealtMaximizes damage dealt

DiscussionDiscussion BenefitsBenefits

Multi-modal behaviorMulti-modal behavior Exploits dynamics of Exploits dynamics of

environmentenvironment Individuals can have Individuals can have

different behaviorsdifferent behaviors Future WorkFuture Work

Evolve against humansEvolve against humans Learn many behaviors Learn many behaviors

in advancein advance More complex games: More complex games:

Unreal TournamentUnreal Tournament

ConclusionConclusion Multi-objective encourages multi-modal Multi-objective encourages multi-modal

behaviorbehavior Multi-objective better than single-objectiveMulti-objective better than single-objective

Deals with contradictory objectivesDeals with contradictory objectives Leads to several different but effective solutionsLeads to several different but effective solutions

Lead to multi-modal behavior in future gamesLead to multi-modal behavior in future games

Questions?Questions?See movies at: See movies at:

http://nn.cs.utexas.edu/?multiNPCsE-mail: E-mail: schrum2@cs.utexas.eduschrum2@cs.utexas.edu