continuous delivery of a cloud deployment at a large telecommunications provider

© 2013 IBM Corporation

Continuous Delivery of a Cloud Deployment at a Large Telecommunications ProviderM Kevin McHugh @mkmchu

Eduardo Patrocinio @patrocinio

1717 – February 25, 2014

1

Agenda

Introduction

Continuous Delivery – what is it, how did we do it

Wrap up

Questions

2

Agenda

Introduction

Continuous Delivery – what is it, how did we do it

Wrap up

Questions

3

Solution Architecture

4

Agenda

Introduction

Continuous Delivery – what is it, how did we do it

Wrap up

Questions

5

Definition

Continuous Delivery (CD) is a design practice used in software development to

automate and improve the process of software delivery. Techniques such as automated testing, continuous integration and continuous deployment allow software to be

developed to a high standard and easily packaged and deployed to test environments, resulting in the ability to rapidly, reliably and repeatedly push out enhancements and bug fixes to customers at low risk and with minimal manual overhead. The technique was one of the assumptions of extreme programming but at an enterprise level has developed into a discipline of its own, with job descriptions for roles such as "buildmaster" calling for CD skills as mandatory.

(Source: Wikipedia 1/2014 - http://en.wikipedia.org/wiki/Continuous_delivery)

The ability to build a releasable asset with nearly trivial effort

What about Continuous Deployment?

6

Components

Method

Continuous Integration

Process

Assets

Automated Test

Automated Build of deployable asset

Tools

7

Method: Agile

7

Team Member

Team Member

Customer

Team Member

Team Lead

Agile

�Value-driven lifecycle with regular production of working software

�Small, co-located team developing straightforward software

�Goal is to develop a high-quality system in an evolutionary, collaborative, and self-organizing manner

�Pre-requisite: can build a product backlog

Agile

Scrum Roles (pigs)

�Product Owner

– Owns the product backlog

– Chooses what work to do next

�Team Member

– Creates the solution

– Owns estimation of difficulty of items in the backlog

– Constrains how much work can be done

�Team Lead (Scrum Master)

– Removes obstacles

– Status reporting

8

Agile terms and how the team operates

Backlog Grooming

Sprint Planning

Scrums

Velocity

Sprint Demo

Sprint Retrospective

Sample User Stories and point assignments:

8

Story

Points Summary

13 pts As a Bank Customer, I want to apply for a loan, so that I can…

8 pts As a Bank Customer, I want to deposit cash

5 pts As a Bank Customer, I want to change my PIN.

3 pts As a Bank Customer, I want to withdraw cash

1 pt As a Bank Customer, I check my account balance

5 pts As a Bank Customer, I want to buy postage stamps

AgileMethod: Tenants of Agile and its iterative cycle

Sprint 1 Sprint 2

Story Points DeliveredStory Points Delivered Story Points DeliveredStory Points Delivered

Velocity (rolling average of points delivered)Velocity (rolling average of points delivered)

9

Agile adoption – adoption by execution

Two-week sprints

Single Product Owner

Scrum Master

Rank user stories

Set up the meetings

Method: What we did Agile

Sprint 1 Sprint 2

Story Points DeliveredStory Points Delivered Story Points DeliveredStory Points Delivered

Velocity (rolling average of points delivered)Velocity (rolling average of points delivered)

1010

Sprint by Sprint view of demonstrated progress of feature delivery

Agile

Release 1.0 - Functional Story Point Burndown

0

50

100

150

200

250

300

350

400

450

Sprin

t 1Spr

int 2

Sprin

t 3Spr

int 4

Sprin

t 5Spr

int 6

Sprin

t 7Spr

int 8

Time

Sto

ry P

oin

ts

Backlog Size

Remaining Backlog

Sprint Velocity

Projected Pts

Planned Pts

Method: Sample Burndown – Measured delivery

11

Continuous Integration

Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day.

(Source: Wikipedia 1/29/2014 - http://en.wikipedia.org/wiki/Continuous_integration)

Develop features or fix defects on isolated branches

Merge branches onto the main

Identify points in time on the main that represent a viable code drop

This all drives the ability to have a stable, automatically tested, “main” which can be sent forward for Test, QA, and Production deployment

12

Continuous Integration – what we did

Initially

Work space isolation obtained via Business Process Manager (BPM) locking of a business process

The team is working on the main

This limited / prevented concurrent engineering

Led to Sprint-fall

Evolved into

Paired programmers working on separate tool kits, and backend processes

13

Process: 30,000 ft view

Development Test QA Production

Main

Feature 455

Unit Test

Asset Mgmt deploy deploy deploy

Build

Customer E2E Automated Test

- REST API

Customer Manual Test

- UAT

Automated Test

IBM Automated Test

- REST API

- UI

IBM Automated Test

- REST API

- UI

Customer Manual Test

- UAT

14

Assets

Assets include any element that the team can change on any of the systems supporting the project. For example:

A WAR file

An executable

An environment variable

A database definition

15

Assets – what we did

BPM process application and tool kitTWX files

PatternThe topology of an application

PatchThose changes to SCO software and/or associated configuration changes

ConfigurationThose changes to the systems supporting the SCO product

Configuration of the VMs internally and from a VM mgmt standpoint

ImagesThe building block for a pattern

16

Rational Team Concert

Assets – what we did

Development

Exported as .twxfile, then checked

into RTC along with a change

description

Import .twx from RTC

QATest Production

17

Automated Test

Automated test provides the means to assess the quality of the main branch or the deployed release

Through the use of Automated Test, the Continuous Delivery framework can then decide of a particular release candidate is of sufficient quality to be promoted to the next level

18

Automated Test – what we did

Test Architecture

Automated REST API Test

Deliver a solution that can be accessed via REST API

Verify that, through the REST API, all features work

Automated User Interface (UI) Test

Confirm that the solution works with the developed user interface

19

Automated Build of Deployable Asset

Automated build provides the means to collect the set of assets from the development system such that they can later be deployed to a target system

Automated deploy provides the means to place the assets onto a target system for test and use

20

Automated Build of Deployable Asset – what we did

Use SmartCloud Orchestrator (SCO), REST based export/import feature

Implemented a persistence layer to encapsulate differences between deployment environments

Leverage build and version control tools

Control asset versions

Initiate automated testing

Track test results

21

Tools to integrate and support

Integrate SCM and Build management tools toSpeed the development

Deliver to the customer control systems

Automated TestTest the User Interface – record and replay UI tool

Test the REST Interface – test harness to invoke API, record results

Automated BuildBPM – REST API based extract from SCO

Patches, etc handled more conventionally

Automated DeployBPM – REST API based import to SCO

Patches, etc handled more conventionally

22

Tools to integrate and support – what we did

23

Tools to integrate and support – what we did

24

Agenda

Introduction

Continuous Delivery – what is it, how did we do it

Wrap up

Questions

25

Components

Method

Continuous Integration

Process

Assets

Automated Test

Automated Build of deployable asset

Tools

26

…In Retrospect

Defect zone vs. Story zone (defect backlog vs. story backlog)

Team commitment to Agile and CD – No boycotts

Sprint granularity of CD vs. Story level of CD

Deliver both Product and Services

Difficulty grasping task ownership and delivery

27

Questions

28

Thank you!!