adaptive intelligent agent in real-time strategy games
DESCRIPTION
Adaptive Intelligent agent in real-time strategy games. A Hybrid Online Case-Based Planning &Reinforcement Learning Approach. Project Members. Omar Enayet. Abdelrahman Al- Ogail. Ahmed Atta. Amr Saqr. Dr. Mostafa Aref. Dr. Ibrahim Fathy. Agenda. Project Research Area & Domain - PowerPoint PPT PresentationTRANSCRIPT
ADAPTIVE INTELLIGENT AGENT IN REAL-TIME STRATEGY GAMES
A Hybrid Online Case-Based Planning &Reinforcement Learning Approach
Omar Enayet
Amr Saqr
Ahmed Atta
Abdelrahman Al-Ogail
Dr. Mostafa
Aref
Dr. Ibrahim Fathy
PROJECT MEMBERS
Project Research Area & Domain Motivations. Problem Definition Objectives Related Work Our Methodology.
Offline Stage. Online Stage.
Testing and Results. Conclusion and Future Work. Demo. References.
AGENDA
AI Learning
Make the machine learn.
AI Planning
Plan then re-plan according to new
givens.
Knowledge
SharingLet everyone know instantly what you
knew through experience.
PROJECT RESEARCH AREA
RTS GamesReal-Time Strategy Games.
Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions
PROJECT DOMAIN
RoboticsFor interest for
military which uses battle simulations in training programs.
War Simulation
For the corporation of robots.
Experimental Relevance
They constitute well-defined environments
to conduct experiments.
MOTIVATIONS
Experience Loss Static ScriptsComputer AI relies on
static scripting techniques.
The Absence of sharing experience costs a lot
PROBLEM DEFINITION
Predictability Non-Adaptability
Computer Opponent doesn’t adapt to
changes in human actions.
Computer Opponent actions easily
predicted.
PROBLEM CHALLENGES
Adaptive A.I.Making the Computer
Opponent adapt to changes like human do.
Mobile Experience
Import/Export your experience !
OBJECTIVES
•Eric Kok introduced : Adaptive Reinforcement Learning Agents in RTS Games, which merged BDI Agents technology with Reinforcement Learning, 2009
•Santi Ontanon introduced Darmok 2 which is an Online Case-Based Planning system designed to play Wargus in 2010.
•M.Johansen devised a CBR/RL system for learning micromanagement in real-time strategy games, 2009
RELATED WORK
ARCHITECTURE OVERVIEW
CASE REPRESENTATIONCase
3
0.6Performance
Behavior
BuildBase1TrainForce(TinyLandForce)TrainForce(TinyAirForce)Attack(NearWoodPeon)
Snippet
UnitExist(Peon)
Preconditions
PlayerBuildings > 0
Alive conditions
Rush-AttackStrategy
WinWargusGoal
NumberOfPlayerPeons = 10NumberOfPlayerCanonTower = 3
PlayerHasFortress = 1..
Shallow Features
PathExists = 1DistanceToEmeny = 40
.
.
Deep Features
0.8Prior Confidence
BeginningSituation
Eligibility
PERCEPTION
GAME STATE ANALYZER
Offline Stage:Learning from
human demonstration before shipping
game
CASE ABSTRACTION
CASE ABSTRACTION (CONT’D) Simplify case complexity. Increase the flexibility.
Abstractor
Point(10, 137) Unguarded Region
CASE ACQUISITION
Generates Cases from Human’s game play.
AcquisitionAbstract Trace Casebase
CASE ACQUISITION (CONT’D)
Online Stage:Learning what’s the
best to do while playing
CASE RETRIEVAL
CASE RETRIEVAL – CONT’D
RetrieverCase Base
Goal
Case
CASE ADAPTATION
Adapts Behaviors according to current game state. Removal of unnecessary actions. Adaptation for unsatisfied preconditions.
AdaptationBehavior Adapted Behavior
CASE ADAPTATION(CONT’D)
PlanBuild(Barracks)
PlanBuild(Great-Hall)
Train(Peon)Build(Barracks)
ONLINE PLAN EXPANSION & EXECUTION
ONLINE PLAN EXPANSION & EXECUTION (CONT’D)
ACTION CONTROLLER
CASE CONCRETIZATION
CASE CONCRETIZATION (CONT’D) Adapt the abstract actions to suit current
situation.
Concreter Point(10, 137)Unguarded Region
CASE REVISION
CASE REVISION (CONT’D)
Uses reinforcement learning, TD-learning SARSA( )λ
ReviserCase
Used Case Evaluation
TESTING AND RESULTS
Offline Le
arning
Adap
tation
Retriev
al
Abstr
action
Concre
tizati
on
Build
ing Pl
acemen
t
Respon
ds to
attack
s
Attack
s
Resourc
e Gath
ering
Traini
ng Fo
rces
0102030405060708090
100
Success Percentage
CONCLUSION A Hybrid Architecture of case based reasoning and
reinforcement learning was introduced to play strategy games.
The architecture merged online case based planning with Sarsa(λ) with eligibility traces. The system showed promising simulation of human behavior; however it still needs a lot extra effort and testing to become industrially capable.
Also, the concept of an abstract case base was introduced which opens the door for generic AI engines for games which is never implemented till the date of writing of this document.
Demo!
FUTURE WORK1) Cooperative AI Agents.
2) Opponent Modelling.
3) Strategy visualization tool.
4) Generic situation assessment.
5) Learn weights of Game State through neural network.
6) Online I-Strategizers.
7) Generic Abstraction/ Concretization.
WEB RESOURCES To get introduced for the whole project journey, evolution,
technical summaries, presentations, discussions, meeting minutes and others visit project blog:
http://rtsairesearch.wordpress.com/
For full materials of papers, technical summaries, documentations, articles, external links and running version of WARGUS (our test best) use the repository link:
svn checkout http://rtsairesearch.googlecode.com/svn/trunk/ rtsairesearch-read-only
For downloading the latest source code for the I-Strategizer Project, please use the following:
svn checkout http://istrategizer.googlecode.com/svn/trunk/ istrategizer-read-only
REFERENCES [1] Martin Johansen Gunnerud. A CBR/RL system for learning
micromanagement in real-time strategy games. In Norwegian University of Science and Technology, 2009
[2] Santi Ontañón, Neha Sugandh, Kinshuk Mishra, Ashwin Ram. On-Line Case-Based Planning. In Computational Intelligence, 26(1):84-119, 2010.
[3] Brain Schwab. AI Game Engine Programming. Charles River Media, 2009.
[4] Santi Ontañón and Kane Bonnette and Prafulla Mahindrakar and Marco A. G´omez-Mart´ın and Katie Long and Jainarayan Radhakrishnan and Rushabh Shah and Ashwin Ram. Learning from Human Demonstrations for Real-Time Case-Based Planning. In AAAI 2008
[5] Ralph Bergmann and Wolfgang Wilke, On the role of abstraction in case-based reasoning
REFERENCES – CONT. [6] Kinshuk Mishra, Santiago Santi Ontañón, and Ashwin Ram.
Situation Assessment for Plan Retrieval in Real-Time Strategy Games. In 9th European Conference on Case-Based Reasoning (ECCBR 2009), Trier, Germany.
[7] Neha Sugandh and Santiago Santi Ontañón and Ashwin Ram . On-Line Case-Based Plan Adaptation for Real-Time Strategy Games. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008)
[8] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, An Introduction. MIT press, 2005.
[9] Wikipedia, the free encyclopedia. http://www.wikipedia.com
[10] Michael Buro, Call for Research in RTS AI, AAAI 2004