Quality Initiatives in the DevOps Tool Chain

Jan Sahlström, Betsson Technologies@testsson_se

A QA’s Role in a DevOps World

Nordic Testing Days 2018Tallinn, Estonia

Photo: Jan Sahlström, 2017

Source: Wikipedia

“A term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment, where building, testing, and releasing software can happen rapidly, frequently, and more reliably.”

DevOps“A term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment, where building, testing, and releasing software can happen rapidly, frequently, and more reliably.”

DevOps is a culture, not (only) a role

• There is no clear definition• Developed by developers and ops, no testers involved• ”Everything can be automated”

Are there any problems with DevOps?

DevOps ToolchainTest

here!

DevOps Toolchain Test here!

DevOps Toolchain

Test here!

Test here!

Test here!

Test here!

Test here! Test

here!

Test here!

Inspired by: @DanAshby04

DevOps Toolchain

1. Understand the activity in each stage

CreateCreate is composed of the building, coding, and configuring of the software development . The process specific activities are:• Design of the software and configuration• Coding including code quality and performance• Software build and build performance• Release candidate

a.k.a: Coding

DevOps Toolchain

1. Understand the activity in each stage

2. Identify quality initiatives

Create | QA initiativesWhat Comment How

TDD Help drive the testing and understanding of the feature Sapient

Unit Test Verify the functionality of a specific section of code Machine

Integration Test Verify the interfaces between components Machine

Component interfaceTest

Check the handling of data passed between various units Machine

Code Quality Configured and measured using tools Machine

Code Review Any form of four eye principle Sapient

Code Coverage Where are the weak spots? Machine

DevOps Toolchain

RRepeat for each section

R

R R

R

R

R

Continuous Delivery Pipeline

Define where the variousinitiatives fit and what shouldhappen if a verification step fails

Some initiatives

Adapting to the increasing speed of deliveries and ways of working

Everyone have a test environment.Some are lucky enough to have another environment for their production code.

Environments

LOCAL TEST QA PRODUCTION

Unit Test Smoke Test System Test Load Test(!)

Integration Test RST UAT

Acceptance Test Regression Test Performance Test

4 Eye Principle System Test

Dev PC

MONITORING

PreProd

MONITORINGMONITORING

PROD

Frequent -Guarded

Highly Controlled Deployment

Controlled Deployment

LOCAL TEST QA PROD

Environments

LOCAL TEST QA PRODUCTION

Unit Test Smoke Test System Test Load Test(!)

Integration Test RST UAT

Acceptance Test Regression Test Performance Test

4 Eye Principle System Test

Dev PC

MONITORING

PreProd

MONITORINGMONITORING

PROD

3rd PartyTest Doubles

3rd PartyLive Environment

3rd PartyTest Environment

LOCAL TEST QA PROD

Test Doubles: Fake, Mock, Stub

Linting

• Linting - Running a program that will analyze code for potential errors• Start small with a just a few rules and expand as you go

Unit Test / Integration Test• Run on LOCAL and TEST• Code Coverage – %?• Code Coverage – Shouldn’t deteriorate• Should be fast, if not try to parallelize

Build Test Set 01

Test Set 02

Test Set 03

Build Test Set 01 Test Set 02 Test Set 03

Test Data – Setup and Teardown

Why?End and start tests in a know state will avoid tests failing due to other breaking tests

Initial State

Setup

Execute

Teardown

Return to Initial State

Branches – When to test

A #1

Master #01 Master #02Master #00

Test Deep

Branches – When to test

A #1

Master #01 Master #02Master #00

B #1

Master #03

Test Deep

Test Deep

Tests done in the branchmight not be valid after B #1

Tests done in Mastermight be too late dueto breaking changes

Branches – When to test

A #1

Master #01 Master #02Master #00

A #1

Master #03

1. Test Deep

2. Test Shallowi.e. Smoke Test

Middle of the road approach

Branches – When to test

A #1

Master #01 Master #02Master #00

A #1

Master #03

Master should always be deployableJez Humble, DevOps Research and Assessment LLC

Master should never contain known errorsJakob Alander, Spotify

On test automation (checks)• Writing automated checks is bug prevention, if it’s done by a

developer. I.e. bugs are found while writing the tests.• Let new tests qualify for the CI monitoring (Non-deterministic tests)

Code Review• Review all code, including tests• Don’t just read the code. Run the code and the applicable tests during

the review• Testers can join in on code reviews even if they cannot code

Canarying / Canary Deploy• Deploy / switch on and verify one instance / section at a time• Start with the “cheapest” one…• Monitor closely

Empty board right-to-left i.e. empty Test column before starting a new feature

The Test columnWho owns it?

ToDo In Progress Test Done

!

Join up with someone in the team and do joint test sessionGather the team in front of a big monitor and do a mob session

Pair/Mob Testing

Image source: Taipei Times Dec 24, 2014. AFP / KCNA via KNS

Make sure that there is a DOD before starting on a new feature

Done DoneOr almost done?

In Progress Test Almost Done... Done

Monitor the builds

• The state of the build should be everyone’s primary interest• Must always be green – Avoid the broken window syndrome• Display on a dedicated monitor• All green != no defects…

Monitor your systems

• Look into different strategies for monitoring (RED, USE, Other)• Use the same monitoring cross the different environments• Share usefull monitoring

Monitoring vs. Testing

Testing Monitoring

Cove

rage

Increased risk of defects in PROD →

Monitoring vs. Testing. Why not? If:

ü Possibility to perform A-B testing, Canary Release or equivalentü Hidden PROD environment (AWS Green & Blue)ü Support & Tech organization available to act quicklyü Extremely loyal users

DevOps Toolchain

1. Understand the activity in each stage

2. Identify quality initiatives

3. Prioritize and implement

TeQuILA

Bringing it all together

Modeling

Level 1•Set up test coverage bundles•Set up a continuous build•C lassify your tests as Sm all, M edium and Large•Identify nondeterm inistic tests•C reate a sm oke test suite

Level 2•N o re leases w ith red tests•R equire a sm oke test suite to pass before a subm it•Increm ental coverage by a ll tests >=50%•Increm ental coverage by sm all tests >=10%•At least one feature tested by an integration test

Level 3•R equire tests for a ll nontriv ia l changes•Increm ental coverage by sm all tests >=50%•N ew significant features are tested by integration testsLevel 4•Autom ate running of sm oke tests before subm itting new code.•Sm oke tests should take less than 30 m inutes to run•N o nondeterm inistic tests•Total test coverage should be at least 40%•Test coverage for sm all tests a lone should be at least 25%•A ll s ignificant features are tested by integration tests

Level 5•Add a test for each nontriv ia l bug fix•Actively use available analysis tools•Total test coverage should be at least 60%•Test coverage for sm all tests a lone should be at least 40%

Google Test CertifiedA 5 level model developed at Google for enhancing quality

Modeling

Google Test CertifiedA 5 level model developed at Google for enhancing quality

Level 1•Set up test coverage bundles•Set up a continuous build•C lassify your tests as Sm all, M edium and Large•Identify nondeterm inistic tests•C reate a sm oke test suite

Level 2•N o re leases w ith red tests•R equire a sm oke test suite to pass before a subm it•Increm ental coverage by a ll tests >=50%•Increm ental coverage by sm all tests >=10%•At least one feature tested by an integration test

Level 3•R equire tests for a ll nontriv ia l changes•Increm ental coverage by sm all tests >=50%•N ew significant features are tested by integration testsLevel 4•Autom ate running of sm oke tests before subm itting new code.•Sm oke tests should take less than 30 m inutes to run•N o nondeterm inistic tests•Total test coverage should be at least 40%•Test coverage for sm all tests a lone should be at least 25%•A ll s ignificant features are tested by integration tests

Level 5•Add a test for each nontriv ia l bug fix•Actively use available analysis tools•Total test coverage should be at least 60%•Test coverage for sm all tests a lone should be at least 40%

Personalize the modelA dded•D efine testable requirem ents for a ll new stories•G uarded com m its•Four eyes princip le•O w n m etrics•...

DependenciesSort out the internal dependencies for the identified initiatives

1 2 3 4

...

Each team need a person within the team to drive the inititiative. This can be any team member.

Driver

Implementation

Get started

Set up workshops and go through each step deciding on how to implement them

Team effort!

None of this will be possible without the participation of the whole team

Working software over new functionallity

TeamArchitect

QADeveloper

SEAL | All roles somewhat overlappingSoftware Engineering ALchemist

This is where the magic happens

DevOps Culture

Malum est consilium, quod mutari non potest

It’s a bad plan if it admits no changes

Publilius Syrus (85-43 BC)

Caveat

@testsson_seFam ilyG uyYourself

Thank you!

Temptare necesse est, navigare non est necesse

Q&A

Inspirations and sources

• World Quality Report 17/18 9th edition

• How Google Tests Software by James A. Whittaker, Jason Arbon, Jeff Carollo

• Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation by Jez Humble

• A Practical Guide to Testing in DevOps by Katrina Clokie

• Blog post "Continuous Testing in DevOps…“ by Dan Ashby

• Anna Sundin Almén, AddQ

• Jakob Alander, Spotify

• Henrik Andersson, Betsson Technologies AB

• Sigurdur Birgisson, Avensia

• Richard Bradshaw, Ministry of Test

• Angie Jones, Twitter

Publications Talks & Discussions

Beside all bits and pices picked up on conferences, meetups, tweets and various other places (too many and too scattered to list) that has gone into this presentation there are some that stands out.