automated acceptance testing of high capacity network gateway
DESCRIPTION
Experience paper presentation from XP2010. Paper itself can be found from: http://gosei.fi/material/xp2010-final-public-v1.pdfTRANSCRIPT
RANNICON NSN
Automated Acceptance Testing of High Capacity Network Gateway
Ran Nyman, Ismo Aro, Ronald Wagner
RANNICON NSN
Content
Background
Initial Phase
Growing
Scaling Now
Agile Acceptance Testing
CI
Challenges with ATDD
Conclusions
RANNICON NSN
Who are We
Ran NymanAgile Coach
Consultant
Developer
Ismo AroAgile Coach
Scrum Master
Tester
Developer
RANNICON NSN
Background
Product development started in 2007
The product High capacity GGSN and SAE
Gateway
Two teams and cultures Waterfall Agile
RANNICON NSN
The Starting Phase
Two teams
Mixing teams
No legacy code
First steps: PBL Initial architecture and Development environment
RANNICON NSN
Growing First Wave
Two more teams from traditional background
C team grasped new development ideas fast
D team had difficulties in changing
RANNICON NSN
Growing Second Wave
More local teams
Subcontractor on different continent Same rules apply CI challenges Customs problems Lot's of traveling
And it slowly started to work
RANNICON NSN
Current Setup
More than 20 teams
Located in four countries
Development staff is still growing More teams on existing sites
RANNICON NSN
How Does it Work
Release at least every other sprint
“As a result of our agility in developing products, we have been able to demonstrate the product capabilities at an early stage of the development cycle itself, helping us to win new customers cases.”
Head of the product line
RANNICON NSN
Agile Acceptance Testing
RANNICON NSN
Agile Acceptance Testing
Acceptance test are decided in beginning of sprint
Tests are automated during development
All acceptance test are automated
Test scripts are only documentation that is written from tests
RANNICON NSN
Tools
CI servers: Bamboo, BuildBot
Automation Framework: Robot Framework
Protocol tester: Catapult
RANNICON NSN
Continuous Integration
CI Build (compilation and Unit Tests) Sanity Nightly regression
RANNICON NSN
CI Initial
RANNICON NSN
CI Current
RANNICON NSN
Test Growth
RANNICON NSN
RANNICON NSN
Challenges with ATDD
Limited e2e testing:
No visibility what is ready for shipping
Hard to troubleshoot
Changing functionality leads to unpredictable failures in tests
ATDD is done on to low level
RANNICON NSN
E2E Tests in Future
Test cases are created before code and they work as a tool of communication.
Test Cases are going to be written using BDD. eg.
Given activate customer license
When customized packet is send to A interface
Then packet is delivered from A interface to B interface.
And no counters are increased.
RANNICON NSN
Conclusions
Automated acceptance testing pays off
100% test automation is the target
Real life is more complex than you think
Do not forget the CI challenges
Splitting requirements and tracking progress
RANNICON NSN
Questions
RANNICON NSN
Contact Us
Ran Nyman
[email protected] www: http://rannicon.com
Ismo Aro
[email protected] Twitter: ismoaro www: http://www.graceland.fi
Roland Wagner