scrum alliance 2013-mmog
DESCRIPTION
TRANSCRIPT
Saturday, 27th July, 2013
Introduction and Background
Sajit Vasudevan, Technical Architect, Cognizant
Technology Solutions
Nirmalya Sengupta, Software Craftsman and Coach,
CeeZone Consulting
Is to show that
Complex projects can be executed by
following Scrum
Adaptability to emerging architectural
needs is achievable
Adoption of Scrum mindset helps in
succeeding
Is to give some concrete examples
Objective
Project Background
Dutch company – Online lottery giant, decided to enter Online Poker market
Scope: Building a complete offering of online betting Poker game environment (first version in 12 months)
Technology choices: Java, C++, GWT,Test Scripting, Linux, MySQL, RabbitMQ, XUnit, Cruisecontrol, Artifactory, Maven, RedDwarf
Application: a bird’s eye view
Player Portal
Game Server
Lobby Server
Database
RabbitMQ
Tournament
Server
Poker Lobby – showing all table types, tournament types etc
Poker lobby – Tables list
Game table
Servers must be ...
Robust
Network ready
Support for ‘Statefulness’
Easy to interface with external systems
Highly threaded
Minimum supportable concurrent player base: ~20000
Logged in players don't always play but add load
Speed and Order of message exchange between Client
and Server are of essence
Crucial Challenges
Inherently asynchronous
Arbitrarily fired Timers affect software's behaviour
Players may decide to sit out and re-join anytime
Network connection with player can drop anytime: game
must continue
Randomness must be satisfactorily built in and proven
(legal requirement)
Game History must be retained (legal requirement)
Multiple simultaneous tables, Multiple simultaneous
tournaments
Example of asynchronous behaviour
P1
P2 P4
P3
step1. P2 plays
step2. P3 sits-out /
gets disconnected
NEW HAND!
step3. Hand over!
ASYNCHRONOUS
MESSAGE!
Now P3 is
informed of
new Hand
allowed to re-
join
Tricky to test software which..
is inherently asynchronous
is supposed to support many users, while
maintaining performance
is asking testers to be good/bad Poker players
is affected indeterminately by Timers (arbitrariness)
requires preservation of history of every move
is expected to display minimum latency always
`
Infrastructure - revisited
Player Portal
Game Server
Lobby Server
Database
RabbitMQ
Tournament
Server
.
.
.
.
.
.
.
.
. . .
.
BOTs
Human Players
JMX Console
Some initial Architectural decisions Game logic implemented as a handcrafted FSM
Game Server separate from Tournament Server
All components inside a Game/Tournament Server
communicate through messages
{ Game | Tournament | Lobby } server exchange
information through Database
Clients ‘pull’ information from Lobby Server
Poker Lobby – showing all table types, tournament types etc
Poker lobby – Tables list
Game table
Problems popped up: Pig headed or headless chicken
Understanding of tournament server was clearer over
time
Performance bottleneck:
Identified earlier that lobby would be heavily loaded
Clients ‘pulling’ information from Lobby Server was a
bottleneck
Exchanging information through Database was
difficult
Plugging in 3rd party authentication mechanism was not
straightforward
Architectural adaptations led to
Replacing Game Server with Tournament Server
We realised that every game played can be considered to be a
part of ‘default’ tournament with no competition
Game/Tournament and Lobby Servers decoupling
through a Queue
Lobby Server keeps its own database
Lobby Server gets latest update from queue
Previous updates can be forgotten – saves memory and
processing
Building functionality
Significant focus on unit testing
Regular and frequent builds
By not having adequate automated tests, we paid the
price in
Incorrectly calculating how to divide money at
stake, amongst players
Incorrectly recording the HandHistory of games
(both of the above, legal requirements and hence,
extremely important)
Debugging effort and cost was huge
Our Scrum experience reminds us that
Collaboration between product management,
developers, testers and architect is crucial
Changes from the product management could be
accommodated at any point of time
Architectural alterations can be made to the initial
architecture
Brickbats