Deploying and Releasing ApplicationsContinuous Delivery, Chap. 10

Xuebin MA

Introduction• Releasing software into production and deploying to testing

environment

• Can be done at the same deployment process but configuration files and ability of roll back are different

• How to create and follow a deployment strategy

• Blue-green deployments and canary releasing are introduced for zero-downtime releases and rollbacks

• Ae a part of deployment pipeline implementation

• which version in which environment, who authorized the deployment and what changes have be made to application

• Emergency fixes, upgrade plan...

Create a release strategy• How?

• All stakeholders meet up > get common understanding concerning the deployment and operations > create the release strategy > update and maintain

• Followings should be considered at the first version of release strategy at the beginning of project

• Parties in charge of deployments to each environment, as well as in charge of the release

• An asset and configuration management strategy

• A description of the technology used for deployment. This should be agreed upon by both the operations and development teams

• A plan for implementing the deployment pipeline

Create a release strategy

• An enumeration of the environments available for acceptance, capacity, integration, and user acceptance testing, and the process by which builds will be moved through these environments

• A description of the processes to be followed for deployment into testing and production environments, such as change requests to be opened and approvals that need to be granted.

• Requirements for monitoring the application, including any APIs or services the application should use to notify the operations team of its state.

• A discussion of the method by which the application’s deploy-time and runtime configuration will be managed, and how this relates to the automated deployment process.

Create a release strategy

• Description of the integration with any external systems. At what stage and how are they tested as part of a release? How do the operations personnel communicate with the provider in the event of a problem?

• Details of logging so that operations personnel can determine the application’s state and identify any error conditions.

• A disaster recovery plan so that the application’s state can be recovered following a disaster.

• The service-level agreements for the software, which will determine whether the application will require techniques like failover and other high-availability strategies.

Create a release strategy

• Production sizing and capacity planning: How much data will your live application create? How many log files or databases will you need? How much bandwidth and disk space will you need? What latency are clients expecting?

• An archiving strategy so that production data that is no longer needed can be kept for auditing or support purposes.

• How the initial deployment to production works.

• How fixing defects and applying patches to the production environment will be handled.

• How upgrades to the production environment will be handled, including data migration.

Create a release strategy

• Why?

• Be a source of both functional and nonfunctional requirements for both software development and for design, configuration, and commissioning of hardware environment

• Others

• the release strategy should be added and changed during the project progresses

• vital component is the release plan describing how releases are performed

The release plan

• Release plan: automated scripts, documents or other procedures for reliable and repeatable deployment:

• Comparing to deployment strategy

• The steps required to deploy the application for the first time

• How to smoke-test the application and any services it uses as part of the deployment process

• The steps required to back out the deployment should it go wrong

• The steps required to back up and restore the application’s state application’s state

The release plan• The steps required to upgrade the application without

destroying the application’s state

• The steps to restart or redeploy the application should it fail

• The location of the logs and a description of the information they contain

• The methods of monitoring the application

• The steps to perform any data migrations that are necessary as part of the release

• An issue log of problems from previous deployments, and their solutions

Releasing products

• More should be considered for software product

• Pricing model

• Licensing strategy

• Copyright issues around third-party technologies used

• Packaging

• Marketing materials-print, web-based, podcasts, blogs, press releases, conferences

• Production documentation

• Installers

• Preparing sales and support teams

The first deployment

• The first deployment should be in the first iteration

• Principal goals of the first iteration is to get the early stages of our deployment pipeline running and be able to deploy and demonstrate somethings

• deployment pipeline’s commit stage

• production-like environment to deploy to

• An automated process that takes the binaries created by your commit stage and deploys them into the environment

• A simple smoke test that verifies that the deployment worked and the application is running

Production like environment

• Development and production environment

• Try to build the same or similar environment, use virtualization and chicken-counting

• chicken-counting: if 250 web servers, 2 should be enough to represent the significant process boundaries

• Production like environment

• same operating system

• same software installed (none development toolchain like IDE)

• be managed the same way as production environment

• for client-installed software: UAT environment should be representative of clients’ hardware statistics

Modeling your release process and promoting builds

• With the grow of application, deployment pipeline implementation becomes complex and test and release process should be modeled considering the following:

• What stage a build has to go through in order to be released

• What the required gates or approval are

• For each gate, who has the authority to approve a build passing through that gate

Example test and release process diagram

ptg

map for your release process. We discussed value stream mapping as a way to optimize your release process in Chapter 5, “Anatomy of the Deployment Pipeline.”

Acceptance testing

Integration testing

integration testing sign-off

gateQA sign-off

gate

packaged builds of the applicaiton

that pass the automated pipeline

stages

StagingUser

acceptance testing

customer sign-off

gateOperations

sign off-gate Production

Figure 10.1 An example test and release process diagram

Once you’ve created this diagram, you can create placeholders for each part of your release process in the tool you use for managing deployments. Go and AntHill Pro both allow you to do this out of the box, and most continuous inte-gration tools can model and manage this process with some work. Once this is done, it should be possible for the people responsible for approvals to approve, using your tool, a particular build moving through a gate in the release process.

The other essential facility that must be provided by the tool you use to manage your deployment pipeline is the ability, for each stage, to see which builds have passed all the previous stages in the pipeline and are hence ready for the next stage. It should then be possible to choose one of these builds and press a button to have it deployed. This process is known as promotion. Promoting builds at the press of a button is what turns the deployment pipeline into a pull system, giving everybody involved in the delivery process the ability to manage their own work. Analysts and testers can self-service deployments for exploratory testing, showcasing, or usability testing. Operations personnel can deploy any version of their choice to staging or production at the press of a button.

An automated deployment mechanism makes promotion a simple matter of selecting the desired release candidate and waiting for it to deploy to the correct environment. This automated deployment mechanism should be usable by anyone who needs to deploy the application. It should not require any knowledge or understanding of the technicalities of the deployment itself. To this end, it is very helpful to include automated smoke tests that run once the deployment system thinks that the system is ready. This way, we can assure the person requesting a deployment—whether it is an analyst, tester, or operations person—that the system is ready for use and working as it should be, or make it easy to diagnose the cause if it is not.

255Deploying and Promoting Your Application

Managing the test and release process

• Model and manage deployment process

• Create placeholder for each part

• Used for approval of people responsible for approvals

• Manage information about which builds have passed all previous stages and whether are ready for next stage.

• Choose one build and press a deploy button (promotion)

• Get everybody involved in the delivery process to manage their own works after promotion, pull system

Tools

• GO

• Agile Release management

• http://www.thoughtworks-studios.com/go-agile-release-management

• AntHill Pro (DevOps Platform)

• http://www.urbancode.com/html/default.html

• Other Agile management tools

• IceScrum: http://www.icescrum.org/

• Jira: http://www.atlassian.com/ja/software/jira/overview

Test and deploy workflow

• Can select which version to deploy to test environment

• Prepare environment and associated infrastructure

• Deploy the application’s binaries.

• Configure the application

• Prepare or migrate any data managed by the application

• Smoke-test the deployment

• Perform the testing

• Approve promotion if test passed. if not, record why

Promoting configuration

• Configuration of environment and application should be promoted

• Managing promotion of configuration associated with an application is complex

• Don’t want to promote all the configuration after test at SIT database or a test double of external service

• One way is use smoke test to verify that you are pointing at the right things

• In case of service-oriented architectures and componentized applications, all the services and components should be promoted together

Orchestration

• Environment shared between several applications

• Extra care trying not to disturb the operation of any other applications in the environment

• If applications depend on each other, integration testing environment is ensure the new versions of each application cooperate with each other well

Deployments to staging environments

• Ensure your production, capacity testing, and staging environments are commissioned. In particular, on a green field project, have your production environment ready some time before the release, and deploy to it as part of your pipeline.

• Have an automated process for configuring your environment, including networks, external services, and infrastructure.

• Ensure the deployment process is adequately smoke-tested

• Measure the warm-up period for your application. This is especially applicable if your application uses caching. Incorporate this into your deployment plan.

• Test integration with external systems. You don’t want your application’s release to be the first time you run against the real external systems

Deployments to staging environments

• If possible, get your application into its production environment well before release. If “release” can be as simple as reconfiguring some router to direct traffic from a holding page to your production environment, so much the better. This technique, known as blue-green deployment, is described a little later in this chapter.

• If possible, try rolling your system out to a small group of users before you roll it out to everybody. This technique is known as canary releasing, and is also described later in this chapter

• Deploy every change that passes acceptance tests to your staging environment (although not necessarily to production).

Rollback by redeploying previous good version

• Good:

• low risk if you don’t have automated rollback but deploy process

• deployment process is much more well tested than rollback

• Demerit

• Has downtime as time of redeploy the old version is nonzero

• Make it hared to debug what went wrong

• May lose data created since last deployment

Zero-Downtime Releases

• Zero-downtime release, hot deployment

• Key to zero-downtime release is decoupling the various parts of release so they can happen independently as far as possible

• It is easy for static resources and web-based services, but it is harder for resources like DB

Blue-Green Deployments

• Deploy to blue environment, let it warm up -> run smoke tests to check it works -> move the new version by changing the router configuration

• If somethings goes wrong, change router back to green environment

ptg

The key to zero-downtime releases is decoupling the various parts of the release process so they can happen independently as far as possible. In particular, it should be possible to put in place new versions of shared resources your applica-tions depend on, such as databases, services, and static resources, before you upgrade your applications.

With static resources and web-based services, this is relatively easy. You just include the version of the resource or service in the URI, and you can have mul-tiple versions of them available simultaneously. For example, Amazon Web Ser-vices has a date-based versioning system, with the latest version of the EC2 API (at the time of writing) available at http://ec2.amazonaws.com/doc/2009-11-30/ AmazonEC2.wsdl. Of course, they keep the earlier versions of the API working as well at the old URIs. For resources, when you push a new version of your website out, you put the static resources such as images, Javascript, HTML, and CSS to a new directory—for example, you could put the images for version 2.6.5 of your application under /static/2.6.5/images.

Things are a little harder with databases. There is a section dedicated to man-aging databases in a zero-downtime scenario in Chapter 12, “Managing Data.”

Blue-Green Deployments

This is one of the most powerful techniques we know for managing releases. The idea is to have two identical versions of your production environment, which we’ll call blue and green.

Web server

Green slice

Blue slice

Application server

Green slice

Blue slice

RouterUsers

Database server

Green database

Blue database

Figure 10.2 Blue-green deployments

In the example in Figure 10.2, users of the system are routed to the green envi-ronment, which is the currently designated production. We want to release a new version of the application. So we deploy it to the blue environment, and let the application warm up (you can do this as much as you like). This does not in any way affect the operation of the green environment. We can run smoke tests against the blue environment to check it is working properly. When we’re ready, moving to the new version is as simple as changing the router configuration to point to the blue environment instead of the green environment. The blue environment thus becomes production. This switchover can typically be performed in much less than a second.

261Rolling Back Deployments and Zero-Downtime Releases

Database switch at Blue-Green deployment

• Data migration may be needed if schema is changed

• One: Put application into read-only mode before switchover, copy green db, restore into blue db, perform migration, switch, change to read-write mode

• Two: design the application so that you can migrate the db independently of the upgrade process

Blue-Green deployment and budget

• Low budget:

• Have two copies of application runs side by side on the same environment

• use virtualization

• Sufficient budget

• Blue and green environment can be completely separate replicas

• Using staging and production environment as blue and green environment

Canary releasing

• Why canary releasing?

• Only have one version of software in production at a time makes easier to manage bug fixes and indeed infrastructure in general

• However, it presents and impediment to test as defects pop up in production

• To create a meaningful capacity testing environment is impossible if we have extremely large production environment

Canary releasing ptg

Canary Releasing

It is usually a safe assumption that you only have one version of your software in production at a time. This makes it much easier to manage bugfixes, and indeed your infrastructure in general. However, it also presents an impediment to testing your software. Even with a solid and comprehensive testing strategy, de-fects pop up in production. And even with a low cycle time, development teams could still benefit from faster feedback on the new features and whatever they could be doing to make their software more valuable.

Furthermore, if you have an extremely large production environment, it’s im-possible to create a meaningful capacity testing environment (unless your appli-cation’s architecture employs end-to-end sharing). How do you ensure a new version of your application won’t perform poorly?

Canary releasing aims to address these challenges. Canary releasing, as shown in Figure 10.3, involves rolling out a new version of an application to a subset of the production servers to get fast feedback. Like a canary in a coal mine, this quickly uncovers any problems with the new version without impacting the ma-jority of users. This is a great way to reduce the risk of releasing a new version.

Users

Router

Version x Version x+1

Most users Small set of users

Figure 10.3 Canary releasing

Like blue-green deployments, you need to initially deploy the new version of the application to a set of servers where no users are routed to. You can then do smoke tests and, if desired, capacity tests, on the new version. Finally, you can start to route selected users to the new version of the application. Some companies

263Rolling Back Deployments and Zero-Downtime Releases

Canary releasing

• Benefits:

• Make rolling back easy

• A/B testing can be done by routing some users to new version and some to the old

• You can check if the application meets capacity requirement gradually

• Others

• not only way for A/B test; runtime configuration setting to change behavior

• Hard to use for user install soft on own computer

• Impose further constraints on DB upgrades

• Limit few versions of application in production

Emergency fixes

• Do not, under any circumstances, subvert your process, even when critical defect is discovered and has to be fixed

• Have to go through the same build, deploy, test and release process as any other change. In other works, run every emergency fix through standard deployment pipeline

• If not,

• The change will lead to regression and exacerbate the problem

• Change is often not recorded and environment ends up in an unknown state

• Check the emergency fix is wether or not is needed, roll back is another choice

Emergency fixes

• Considerations about dealing with defect in production

• Never do them late at night, and always pair with somebody else.

• Make sure you have tested your emergency fix process.

• Only under extreme circumstances circumvent the usual process for making changes to your application.

• Make sure you have tested making an emergency fix using your staging environment.

• Sometimes it’s better to roll back to the previous version than to deploy a fix. Do some analysis to work out what the best solution is. Consider what happens if you lose data or face integration or orchestration problems.

Continuous deployment

• If it hurts, do it more often => Take the pipeline and make deployment to production automatic

• automated unit tests, component tests, acceptance tests covering entire application

• Continuous deployment can be combined with canary release

• Continuous deployment isn’t for everyone, as some new features are not wanted to be released to production

• By release every change, amount of risk is limited to the risk inherent in one change

• Continuous deployment force we to do the right thing: build, deploy, test and release process

Continuously releasing user-installed software

• Issues to consider for client-installed software

• Managing the upgrade experience

• Migrating binaries, data, and configuration

• Testing the upgrade process

• Getting crash reports from users

Upgrade plan

• Support different versions of software is difficult, so we need to make upgrade as painless as possible

• Upgrade plan

1. Have your software check for new versions and prompt the user to download and upgrade to the latest version

2. Download in the background and prompt for installation

3. Download in the background and silently upgrade the next time the application is restarted

• 1,2 seems attractive but wrong as users can choose not to upgrade

Upgrade plan

• Correct solution:

• Make upgrade process bullet proof and upgrade silently

• Prompt to user if corrective action needs to be taken

• Sometimes, you don’y want your software upgrade silently because of poor network and so on...

• User can turn off automatic upgrades

• Migrating binaries, data and configuration: keep old ones until upgrade successful

• Be able to upgrade from any version to any other version

• Test upgrade process as part of deployment pipeline

• Be able to report crashes back to development team

Tips and tricks

• People who do the deployment should be involved in creating the deployment process

• Log deployment activities: log all files for debug

• Don't delete the old files, move them

• Deployment is the whole team's responsibility: any member should do

• Server application should not have GUIs: the machine must have had a user logged in and the UI showing. Reboot, accidental or due to upgrades will log the user out, and the server will stop.

• Have a warm-up period for a new deployment (canary release)

• Fail fast

• Don't make changes directly on the production environment