moving to continuous delivery without breaking everything

Moving to Continuous Delivery Without Breaking Your Code

Viktor Clerc, 23 June 2015

2 Copyright 2014. Confidential – Distribution prohibited without permission

Agenda

▪The Need For Speed

▪The Two Faces of CD

▪Testing is Changing

▪A Central Hub for Application Quality For Your Pipeline

▪Beyond Test Automation: Active Test Optimization


About Me

▪Product Manager XL TestView for XebiaLabs

▪Traversed through all phases of the software development lifecycle

▪Supported major organization in setting up atest strategy and test automation strategy

▪Is eager to flip the way (most) organizations do testing


About XebiaLabs

We build tools to solve problems around DevOps and Continuous Delivery at scale


The Need For Speed

▪Every business is an IT business− Known as “Software-defined Enterprise”, even traditionally brick and mortar business is becoming

software based

▪Customers demand that you deliver new features faster whilst maintaining high levels of quality

▪If you don’t, your competitor probably will


The Need For Speed

▪What is so compelling about CD?

▪Business initiative with cool technical implementation

▪“CD eats DevOps for breakfast as the business eats IT”


The Two Faces of CD

▪A lot of focus right now is on pipeline execution

▪…but there’s no point delivering at light speed if everything starts breaking

▪Testing (= quality/risk) needs to be a first-class citizen of your CD initiative!


The Two Faces of CD

▪CD = Execution + Analysis


The Two Faces of CD


▪= Speed + Quality


The Two Faces of CD


▪= Speed + Quality

▪= Pipeline orchestration + ..?


SPECIFY INTEGRATE RELEASEREGRESSION

VALUE CHAINCONCEPT CASH

Test

Effo

rt

USER ACCEPTANCEDESIGN BUILD TEST

Testing is Changing


SPECIFY INTEGRATE RELEASEREGRESSION

VALUE CHAINCONCEPT CASH

Test

Effo

rt

USER ACCEPTANCE

USER ACCEPTANCE

AcceptanceDriven Testing

“I add value by sharpening the acceptance criteria of requested

features”Automate ALL

“Test automation serves as the safety net for my new functionality: I focus on running

the appropriate tests continuously during the iterations”

TEST

DESIGN

BUILD

D BT

T D

Development = TestTest = Development

“Testing is transforming to a automation mindset and skill instead of a separate activity”

Testing is Changing


Testing is Changing: Challenges

▪Many test tools for each of the test levels, but no single place to answer “Good enough to go live?”

▪Requirements coverage is not available− “Did we test enough?”

▪Minimize the mean time to repair− Support for failure analysis

JUnit, FitNesse, JMeter, YSlow, Vanity Check, WireShark, SOAP-UI,

Jasmine, Karma, Speedtrace, Selenium, WebScarab, TTA,

DynaTrace, HP Diagnostics, ALM stack AppDynamics, Code Tester for Oracle, Arachnid, Fortify, Sonar, …



▪Thousands of tests makes test sets hard to manage:− “Where is my subset?”

− “What tests add most value, what tests are superfluous?”

− “When to run what tests?”

▪Running all tests all the time takes too long, feedback is too late

▪Quality control of the tests themselves and maintenance of testware

▪Tooling overstretch


Testing is Changing: Best Practices

▪Focus on functional coverage, not technical coverage

▪Say 40 user stories, 400 tests− Do I have relatively more tests for the more important user stories?

− How do I link tests to user stories/features/fixes?

▪Metrics− Number of tests

− Number of tests that have not passed in <time>

− Flaky tests

− Duration


Testing is Changing: Best Practices

▪“Slice and dice” your test code− Responsible team

− Topic

− Functional area

− Flaky

− Known issue

− etc.

▪Radical parallelization− Fail faster!


Making Sense of Test Results

▪Real go/no go decisions are non-trivial− No failing tests

− 5 % of failing tests

− No regression (tests that currently fail but passed previously)

− List of tests-that-should-not-fail

▪Need historical context

▪One integrated view

▪Data to guide improvement


Example Job Distribution

Build Deploy Int. Tests Test Test Test Perf. Tests

Build Deploy Int. Tests Test

Test

Test

Perf. Tests


Example Job Distribution

Build Deploy Int. Tests Test Test Test Perf. Tests

Build Deploy Int. Tests Test

Test

Test

Perf. Tests

Simple pipelines – scattered test results



Executing tests from Jenkins is great, but…

▪Different testing jobs use different plugins or scripts, each with different visualization styles

▪No consolidated historic view available across jobs

▪Pass/Unstable/Fail is too coarse− How to do “Passed, but with known failures”?



▪Ultimate analysis question (“are we good to go live?”) is difficult to answer

▪No obvious solution for now, unless all your tests are running through one service


Test Analysis: Homebrew

24


Test Analysis: Custom Reporting

25


A Central Hub for Application Quality

What is needed:

1. A single, integrated overview of all the test (= quality, risk) information related to your current release



What is needed:


2. …irrespective of where or by whom the information was produced



What is needed:



3. The ability to analyze and “slice and dice” the test results for different audiences and use cases



What is needed:



3. The ability to analyze and “slice and dice” the test results for different audiences and use cases

4. The ability to access historical context and other test attributes to make real-world “go/no-go” decisions


Beyond Test Automation

Can we go further? How about

5. The ability to use the aggregated test results, historical contexts and other attributes to invoke tests more intelligently?



It’s a bit of an open question:

▪ Google: it’s too expensive and time-consuming to run all the tests all the time - automated selection of a subset of tests to run

▪ Dave Farley: if you can’t run all the tests all the time, you need to optimize your tests or you have the wrong tests in the first place



Middle ground:

▪ Label your tests along all relevant dimensions to ensure that you can easily select a relevant subset of your tests if needed

▪ Consider automatically annotating tests related to features (e.g. added/modified in the same commit), or introducing that as a practice

▪ Use data from your test aggregation tool to ignore flaky/”known failure” tests (and then fix those flaky tests, of course ;-))


Summary

▪Testing = Automation− Testers are developers

▪Structure and annotate tests− Conway’s Law for Tests

− Link to functions/features/use cases

▪Radical parallelization− Throwaway environments


Summary

▪CD = Speed + Quality = Execution + Analysis

▪Making sense of scattered test results is still a challenge

▪Need to figure out how to address real world go/no go decisions


Analyzing Test Results


Tagging Tests


Evaluating Go/No-go Criteria


Next steps

▪Next-Generation Testing: The Key to Continuous Delivery

https://xebialabs.com/resources/whitepapers/next-generation-testing-the-key-to-continuous-delivery/

▪An Introduction to XL TestView

https://www.youtube.com/watch?v=_17xKtB3iWU

▪Download XL TestView

https://xebialabs.com/products/xl-testview/community





https://xebialabs.com/products/xl-testview/community

Thank you!

moving to continuous delivery without breaking everything

Software

confidential distribution

faces of cd testing

cd initiative

permission agenda

speed quality

mortar business

business eats

business initiative