1

Next Gen Continuous Delivery

From Build delivery to Value delivery

2

Eric’s Bio

I’m a Lead Consultant at Urbancode where I helps customers get the most out of their build, deploy and release processes. I have 9 years of automation experience throughout the application life-cycle in roles as a developer, test automation engineer, and support engineer. I’ve been at the forefront of CI & CD for 7+ years

Eric Minickeric@urbancode.com

3

Agenda

• Introduction to Continuous Delivery• Continuous Delivery & Complex Apps• Adapting apps to CD• Adapting CD to complex apps• Q&A

4

What is CD?

• An automated flow from build to “ready to deploy to production”

• Push-button deployment to production

• The execution of many types of tests

• Cultural emphasis on constant shipability

5

“Normal” CD

• Expanding the CI emphasis on quality and automation downstream

6

The Build Pipeline (Build > ? > Prod)

• Perform a build– and execute unit tests

• Promote build to a test environment & test– Repeat N times

• Release

7

We even have a maturity model for it

8

Why are we doing it?

• Small batch sizes:– Reduce risk– Lower the “waste” of unreleased features

• Natural extension of the “test your builds” mentality of CI

• Address the deployment bottleneck in Operations.

9

Continuous Delivery is a “DevOps” Strategy

• Successful implementation requires assistance from developers, operations, and others

• Cooperation and coordination between developers and operations must improve

10

Agenda

• Introduction to Continuous Delivery• Continuous Delivery & Complex Apps• Adapting apps to CD• Adapting CD to complex apps• Q&A

11

The Hard Part is Coordination

Image from wisc.edu

12

Complex apps have related builds

• Builds of one part of the app depend on another

• A change in one “pipeline” could impact another pipeline

• Tests cross-cut builds pipelines

13

Multiple tier apps

• Different tiers, different teams, different builds

• How do align the changes?

14

Modern architectures make it worse

• Dozens of inter-related components

• The promise of “change only one piece” is rarely realized in practice

Image from ischool.tv

15

Prod deployments aren’t of one build

16

Build pipelines in coupled systems

17

Some pieces aren’t built

• Databases

• Infrastructure

• Content

• Reports / ETL

18

CD’s Build Pipeline isn’t Perfect

• It fails to:– Account for deployment time dependencies– Model things that aren’t built– Deal with incremental updates

19

Now what?

Work hard to salvage build pipelines

Or

Use a different model

20

Agenda

• Introduction to Continuous Delivery• Continuous Delivery & Complex Apps• Adapting apps to CD• Adapting CD to complex apps• Q&A

21

Adapting apps to CD: Principals

• Principals– Everything is a “build”– Builds are promoted independent of other builds

• Example techniques– Release build of builds– Enforce backwards compatibility– Expand / Contact pattern for databases

22

A Build of Builds

• Create a mini-release process for each component, and combine them into a big build.

• Mega build contains the binaries from the others

devtest

Comp.Build

devtest

Comp.Build

devtest

Comp.Build

23

A Build of Builds

• Create a mini-release process for each component, and combine them into a big build.

• Mega build contains the binaries from the others

• Challenges: My “mega build” is big, and is always fully deployed. My components don’t know if they went to Prod.

devtest

Comp.Build

devtest

Comp.Build

devtest

Comp.Build

24

Enforce Backwards Compatibility

• Build and immediately deploy to integration testing

• If integration tests fail, the build is rejected and the old build of that component is redeployed to integration testing

25

Enforce Backwards Compatibility

• Build and immediately deploy to integration testing

• If integration tests fail, the build is rejected and the old build of that component is redeployed to integration testing

• Challenges: Good integration tests, extra engineering to support new and old versions, etc.

26

Database Expand / Contract

• Goal: Backwards compatible, zero downtime database deployments.

• Never remove objects old / active users of the database need. Only add new objects. Once all clients are using the new objects, remove the old.

See: http://exortech.com/blog/2009/02/01/weekly-release-blog-11-zero-downtime-database-deployment/

27

Database Expand / Contract

• Goal: Backwards compatible, zero downtime database deployments.

• Never remove objects old / active users of the database need. Only add new objects. Once all clients are using the new objects, remove the old.

• Challenges: a significant and not always easy change to how organizations develop DB updates.

See: http://exortech.com/blog/2009/02/01/weekly-release-blog-11-zero-downtime-database-deployment/

28

Agenda

• Introduction to Continuous Delivery• Continuous Delivery & Complex Apps• Adapting apps to CD• Adapting CD to complex apps• Q&A

29

Adapting CD to our Apps

• Account for deployment time dependencies• Model things that aren’t built• Deal with incremental updates

30

Use the “Build of Builds” model as a start

devtest

Comp.Build

devtest

Comp.Build

devtest

Comp.Build

31

Shift to “Release Sets” or “Snapshots”

• We don’t need a new build, we need a name for a collection of builds.

• Delay the creation of these until integration tests pass, and create based on the successful integration tests

UATSign-off

Staging Prod

Snapshots at “Application” or “System” level.

32

Don’t require “build”

• Extracts from existing systems, artifact repos, or source control are OK to get deployable version.

UATSign-off

Staging Prod

Snapshots at “Application” or “System” level.

33

Support multiple incremental moves

• Incremental requires:– Multiple versions of a component in snapshots– Awareness when tracking what is where– Order awareness when performing rollbacks.

34

Pipeline with Snapshots

UATSign-off

Stage Prod

Web

Mid. Code

Mid. Config

DB

35

In story form

• A change to a component, creates a new version (often by doing a build).

36

In story form

• A change to a component, creates a new version (often by doing a build).

• The new version is vetted, and then tested in an integration environment.

37

In story form

• A change to a component, creates a new version (often by doing a build).

• The new version is vetted, and then tested in an integration environment.

• When the integrated system passes tests, a snapshot of all the component versions in the system is created.

38

In story form

• A change to a component, creates a new version (often by doing a build).

• The new version is vetted, and then tested in an integration environment.

• When the integrated system passes tests, a snapshot of all the component versions in the system is created.

• Snapshot deployments don’t redeploy unchanged components

39

In story form

• A change to a component, creates a new version (often by doing a build).

• The new version is vetted, and then tested in an integration environment.

• When the integrated system passes tests, a snapshot of all the component versions in the system is created.

• Snapshot deployments don’t redeploy unchanged components

• The component version is promoted & released

40

In Summary

• Today, continuous delivery on complex systems is hard to coordinate.

• Two options1. Strict development standards to force our

systems into the build promotion model2. A shift towards snapshot deployments that

accommodate projects “as they are”

41

References

http://urbancode.com/resources• Enterprise CD Maturity Model

• Death to Manual Deployments!• Build & Deployment Automation for the Lean Economy• ITIL Release Management and Automation

Urbancode.com/blogs/Twitter.com/UrbanCodeSoftFacbebook.com/UrbanCodeSoftSlideshare.net/Urbancode

42

Yes, we sell products for this

• AnthillPro– Build automation and build promotion

• uDeploy– Deployment and release management

Upcoming AnthillPro + uDeploy demo:http://web.urbancode.com/using-anthillpro-with-udeploy

43

Questions?

eric@urbancode.com@EricMinick