-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
1/27
Agile Test Automation
a Behaviour Driven Approach
Ray Hua
2
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
2/27
1. A journey of test automation
2. Some lessons learned
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
3/27
2 years ago
No automation
NowBDD
Skilled testers
Manual
testers
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
4/27
A commercial tool
Some contractors
A project delivery approach
Half a million dollars
Some automation
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
5/27
A mix of open-source & commercial tools
In-house R&D
One automation engineer
BAU delivery
Some automation
that didnt work
An in house solution
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
6/27
Start redesigning test framework
Start training testers
Some automation
that didnt work
In house solution QA team on board
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
7/27
7 projects
90% functional coverage
Migrated to Selenium/Grid 2, more VMs
Code migration completed by testers
Some automation
that didnt work
In house solution QA team on board Grow regression
suite and upgrade
technology
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
8/27
Executable requirements
BDD engine for .net SpecFlow
BA/PM on board
BDD automation, scripted by testers
Some automation
that didnt work
In house solution QA team on board Grow regression
suite and upgrade
technology
Behaviour
Driven
Development
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
9/27
C#
Visual Studio
TFS
Nunit
Resharper
Selenium/Grid
Team City
Vmware
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
10/27
Behaviour Driven?
A specialized version of TDD
Name of test methods should besentences
Readable when test fails
A ubiquitous language for analysisExecutable acceptance criteria
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
11/27
Tools
Java JBehave.Net SpecFlow
Ruby Cucumber/RSpec
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
12/27
A BDD scenario:
Given the account is overdrawn
And the card is validWhen the customer requests cash
Then ensure a rejection message is displayed
And ensure cash is not dispensed
And ensure the card is returned.
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
13/27
BDD engine generates stubs (SpecFlow):
[Given(TheAccountIsOverdrawn)]
Public void TheAccountIsOverdrawn (){}
[And(TheCardIsValid)]
Public void TheCardIsValid (){}
[When(TheCustomerRequestsCash)]
Public void TheCustomerRequestsCash (){}
[Then(EnsureARejectionMessageIsDisplayed)]
Public void EnsureARejectionMessageIsDisplayed (){}
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
14/27
[Given
(TheAccountIsOverdrawn)]Public void TheAccountIsOverdrawn (){
setupAccount(overdrawn);}
[And(TheCardIsValid)]
Public void TheCardIsValid (){
setupCard(valid);}
[When(TheCustomerRequestsCash)]
Public void TheCustomerRequestsCash (){aCustomer.Setup(standard);
customer.requestCash();}
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
15/27
The team lovesReadable test report for PM with colours
indicating pass, fails and in-progress
IntelliSense for BA to write stories
Tool generates method stubs to help
Testers focusing on what to test next.
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
16/27
Challenges
Unskilled testers
Test framework
Unstable tests
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
17/27
Unskilled testersStart training with motivated testers
Set examplesLearn by following, then creating
Give enough time
Always have helps available
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
18/27
Test framework
Use established patternsLeverage application code
Make it earlier to understand
Naming and conventions
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
19/27
Unstable tests
Decouple tests from build pipeline
Categorize tests
Leverage application codeWait and retry and verify
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
20/27
Catches
An additional layer = more code
More code = more maintenance
Given/When/Then format can be
too verbose
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
21/27
TipsCategorize tests
Stabilize tests
Speed up tests
Design testable architecture
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
22/27
Categorize tests
Warm up tests < 1 minute
Smoke tests < 5 minutes
Regression tests < 15 minutes
Daily regression tests < 1 hourUnstable tests
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
23/27
Stabilize tests
Ensure atomicityControl the environment (system, application
and data)
Run tests in random orderWait, retry and verify
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
24/27
Speed up tests
Run tests in parallel [Selenium Grid] [SauceLabs]
One code base, use application code where
necessary
Smaller tests
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
25/27
Design testable architectureBuilt-in testability
Automation hooksE.g. Client side MVC/API/server side MVC
Lots of unit tests
Less integration testsEven less end-to-end tests
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
26/27
Keep tests
Green(its the heartbeat)
and
fix them when they fail
Most Importantly
-
7/30/2019 Iqnite 2012 Agile Test Automation - A Behaviour Driven Approach
27/27