game on: automating sports video game testing
DESCRIPTION
Sports video games are generally on a short cycle time—tied to the start of a particular sport’s season. Like all video games, the pressure is always on to add more features to sell more games, and the list of “cool” features is endless. Getting buy-in to implement automated testing in this environment can be a challenge. And once you get that buy-in, your next challenge is to ensure it provides significant value to the game team. Fazeel Gareeboo shares the lessons they learned at EA Sports—lessons you can take back to your project. Fazeel describes the sports video game development environment and discusses how you can overcome the reluctance of product owners to spend precious resources on implementing automated testing. Fazeel addresses how to get buy-in from the team and finally how to make your product—game or not—more successful using automated testing.TRANSCRIPT
T24
Special Topics
5/8/2014 3:00:00 PM
Game On: Automating Sports
Video Game Testing
Presented by:
Fazeel Gareeboo
EA Sports
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Fazeel Gareeboo
EA Sports
A software development director at EA Sports Tiburon, Fazeel Gareeboo manages a team that provides automated testing for the studio. Before moving into management, he developed computer aided design (CAD) software and wrote device drivers for Windows and MicroStation, including the first 2048x2048 Windows device driver-for Bill Gates’ house. Fazeel has a special interest in automating any repetitive work and in creating great teams for the kinds of work that computers cannot do yet. He grew up on the island of Mauritius and worked in Europe before settling in the United States. Reach Fazeel at [email protected].
4/26/2014
1
Fazeel Gareeboo,
EA Sports
� Sports Video games
� Game development
� Game stability during development
� Automated smoke testing
� Technology
� Q&A
4/26/2014
2
� Key differences from other software:◦ Requirements
� Usually not ‘given’
� Internally generated by game designers
◦ ‘Metric’ for success – Sales – hard to predict
◦ Platforms
� Consoles are closed
� Need licensing
� Fixed release date – tied to sports’ season
� Game subject to licensing authorities’ approval
4/26/2014
3
� Fixed release date
� Requirements TBD
� Multiple external approvals (consoles, licensors)
� Pick a set of features that would make a great game
� Develop and polish the features in time for the release date
4/26/2014
4
Alpha
• Feature
Complete
Beta
• 0 bugs
Final
• Ship it !
� ‘unknown’ number of defects + Fixed release date
� Hard to add more resources to project
�“Crunching”
4/26/2014
5
� Make sure the game is stable all the way during development
� “Stop the line” – Toyota way manufacturing◦ Freeze the depot
� Automated testingAutomated testingAutomated testingAutomated testing is key because manual testing cannot keep up with the pace of changes
4/26/2014
6
� Given the existing challenges of developing the game, investing in automated testing does not rank high on the list �
� Case for a stable game throughout development is received more favorably by those underwriting the game – company executives/publishers, …◦ CostCostCostCost� Invest a known amount in setting up Automated testing
◦ Benefits� Lower development costs:
� Team not blocked during development by game instability
� No need to crunch during Alpha
� Lower risk to ship date:� Game stable throughout development
4/26/2014
7
� Central team for Automated Smoke Testing
� Central technology created to support automated testing on consoles
� Game teams supported by central team, and we run set of smoke tests on every build of the game and provide results to the game teams
New Game
Built
Copy to
game
console
Run test
scripts on
game
Report
results to
Dashboard
4/26/2014
8
� No, not really �� Having test results is not sufficient.� Need effective policing of breaks…◦ Individuals� You checked something in that broke the game
� Did you not test this ?
� From now on, we need you to run all the tests before you check in (we have a dashboard to verify)
◦ Team� Create and publish a stability policy
� “If we don’t have a full set of passes in 2 days, we freeze the depot”
4/26/2014
9
� Major pieces:◦ Automation infrastructure� Runtime component that is integrated into the game� Server component ◦ Test scripts� Easily edited� Stored in game depot ◦ UI automation layer� ‘Blind navigation’ leads to brittle scripts
� E.g. move to x,y then press ‘A’� Need to provide a way for the test script to talk to the UI layer� What UI components are on screen ?� Select component Button | ‘Ok’� Is UI ready to accept input ?
� ‘industry standards’ / ‘open source’ – not readily available, because consoles are closed systems
� Applies to :◦ Automated testing technology
◦ UI automation layer
4/26/2014
10
� Automated testing of automated testing tech highly recommended
� Making the case for Automated smoke testing may be easier higher up the food-chain
� Having the test results is not enough. You have to drive through the culture of stability at the individual and team levels
4/26/2014
11