Test Automation PyramidT. Alexander Lystad, Chief QA Architect, VESD

Test AutomationWhy?

Allows you to test more with less effortAllows you to deploy/release more quickly and more frequentlyHigher quality software

How?As a natural part of the development processA story is not done until appropriate automated tests are passing

Test Automation PyramidShows different technical levels on which

you can have automated testsA modelGoing up

Scope increasesExecution time increasesEffort increases

Guideline for relative distribution of your testing effort

Unit TestsTesting smallest possible parts of the system

in isolationUse of test doubles to replace dependencies

Dummies, stubs, spies, fakes, mocksSmallest scope

Easier to locate and understand errorsBest performance

Execute early and oftenWon’t catch all defects

Component Integration Tests●Testing that multiple components work together●Test doubles are not used for the integration(s)

you want to test●Varying scope●Good performance●Won’t catch all defects

API TestsTesting that your APIs behave as expected

RESTSOAP...

Big scopeInvolves many components

Decent performanceWon’t catch all defects

UI TestsEnd-to-end testingMaximum scope

Can catch many problemsWorst performance

Don't use UI for setup and teardownParallelize

Can be brittle, flaky and a lot of work to maintainTest critical user workflows

Won’t catch all defects

Exploratory TestingManual testingNot regression testingExperience + creativityUsed to learn about the system, discover defects and

to improve automated testingCan be based on a mission/test charter or persona

Test for accurate and helpful error messages when registering an expense incorrectly

Test for unexpected results when configuring company details in two different tabs

Impatient manager new to the system

Where is…?●Acceptance testing●BDD●Smoke testing●Security testing●Performance testing●System testing●System integration testing●…

What should I aim for?Depends, discuss in your teamAs an example, Google suggests

70% unit tests20% integration tests10% end-to-end tests

Tools

Selenium WebDriver (VAFT)JAutomate, Sikuli

SoapUI, RestSharp

JIRA Capture

Unit Test Frameworks

BDD Frameworks

Hour-glassIce cream cone

Anti-patterns

Other Test Automation Pyramids

Testing Week TaskConfluence: Test Automation Pyramid - Testing Week 2015How does your team's test automation effort fit into the Test Automation Pyramid,

how does your pyramid look today, and how can it improve going forward?

Example workflowIdea → Production

Example workflowDesign process

Acceptance criteria drafted

Example workflowRefinement process

3 amigos: PO/BA + dev + QA/tester

Acceptance criteria finalizedAcceptance tests finalized

Risks and mitigationWhat should be tested, on

which level

Example workflowImplementation

Developers have a clear picture of what is expected

Decided automated tests are implemented as part of story development

Example workflow: Test executionBalance between quick feedback

and discovering problems earlyDuring implementationAfter implementationAt the beginning of your delivery

pipeline (CI server)After deployments

Internal TestUser AcceptanceProduction

Exploratory testing

SummaryThe test automation pyramid is a useful thinking tool and discussion referenceMake conscious decisions about what to test on which level, and also where you

run which testsTry to distribute your tests optimally

High qualityHigh velocityMedium effort

Discuss with your teamDo the Testing Week task

Questions or comments?