Download - Robert Mircea & Virgil Chereches: Our Journey To Continuous Delivery at I T.A.K.E. Unconference 2015
2
“How long would it take your organization to deploy a change that
involved just one single line of code?
Do you do this on a repeatable, reliable basis?”
Mary and Tom Poppendieck, Implementing Lean Software Development
3
“Failure is not an option”
Learning opportunity
Detectable quickly
Restore service quickly
Failure is inevitable
Orange
4
IT Performance and DevOps Practices
“Our data shows that IT performance and well-know DevOps practices, such as those that enable continuous delivery, are predictive of organizational performance.
As IT performance increases, profitability, market share and productivity also increase.
IT is a competitive advantage, not just a utility, and it’s more critical now than ever to invest in IT performance.”
http://bit.ly/1LwY4oB
5
The Need for Continuous Delivery
Agility for our Business
Eliminate wasted time due to hand offs between teams
8
Share the Vision of CD
Spread
the “WHY”
the “WHAT”
ensure they “GET IT”
Started organic, inspired by a few enthusiasts.
Understand that it is about changing the way we work
9
How We Joined the Army
Dev Ops
Enthusiasts
7 x DevOps Wannabes
One project
One opportunity
R&D style
Managers jumped on
board from the start
DevOps
11
JBoss cluster
standalone
JBoss cluster
standalone
v1.0 v1.1
JBoss cluster
standalone
v1.1
…
VMWare VCenter Infrastructure
JBoss cluster
standalone
v1.0
ProdDev/UAT
v1.1
commit
https://www.orange.ro/myaccount
12
From What We Start
Business
requests in the
hundreds/year
200+ apps in
Java, C#, C, a
few PHP. Lots of
PL/SQL
~150 people –
Dev, QA, BA,
Ops, PM
Mélange of
proprietary and
lots of custom
development apps
13
What is Continuous Delivery?
Small, frequent changes delivered constantly
A method to reduce risk of failure
A way to give to our business a glimpse of the shape of product being built
A set of practices enabled by a toolset
14
Many deployments eventually lead to a Product Launch
What is the difference between
a Deployment and a Product Launch?
16
Orange Deployment Pipeline Architecture
(Atlassian Stash Git / SVN)
(Artifactory)
(Jenkins/Teamcity/Sonar) (Selenium)
(Enterprise Tester)
(JMeter)
Rundeck
/Puppet
17
Best Practices for CD
build application only once and use it in all environments
keep application releasable all the time – master is releasable at all times
hide new functionality until it is finished – use feature flags
make small changes incrementally – commit and integrate often
use snapshots (in Maven) with care
19
Blue-Green Deployments
v1.0-WL
App Server
v1.0-Jboss
www.orange.ro/myaccount-corporate
Weblogic
JBoss
20
Feature flags
Integrating with
production is a test
On/Off
Orange IPs
List of Users
Time interval
Release date
Gradual Rollout (%)
Blacklist/whitelist IPs
Enable/disable features
quickly (no restart)
ORO testers check feature
in live
22
Code Based Migration - FluentMigrator
Run migration
.\migrate --conn "server=oracle-server; database=SmsDb" -db OracleManaged -a
"..\Migrations\Migrations.dll" --task migrate:up
Rollback change
.\migrate --conn "server=oracle-server; database=SmsDb" -db OracleManaged -a
"..\Migrations\Migrations.dll" --task migrate:down
23
Automated tests AFTER deploy
Run smoke tests automatically to validate deployment success
Examples:
make a HTTP request to homepage and look for a string
login with a test user
make a recharge with a test PrePay
24
Metrics
Server Metrics
CPU utilization
disk space
Application Metrics
JMS messages sent/received,
memcached hits
Business Metrics
Logins / Registration
Orders
Recharges
Options activations
25
Naming Metrics
Establish consistent rules in naming metrics
EnvApp name
Server
name
Module
name
Message
type
Target
Action
prod.apps.maco.platini.jms.vantive.customers.request.sent
27
Instrumenting Code for Metrics Collection
Trivial counters with Statsd (https://github.com/etsy/statsd/)
Types of counters:
Counters
Timers
Gauges
Histograms
28
Centralized Logging
Apache Logs
App logs
Logstash
LumberjackLogstash
Log shipping
Logstash
(parse log into fields)
Elasticsearch
(index logs)
Statsd + Graphite
(metrics dashboard)
(log search & analysis)
30
“At an abstract level, a
deployment pipeline is an
automated manifestation of
your process for getting
software from version control
into the hands of your users.”
Deployment Pipeline
34
Coding best practices
Data/code separation
r10k
HIERA
Keep code in version controlreaktor
Unit & acceptance testing
Development workflow feature branch
workflow
puppet-rspec
server-spec
35
Node breeds
Several items give
node breed: puppet role class
Foreman host
group (one-to-one
relationship)
stage
(dev/prod/qa/test)
Foreman compute
profile (compute
provider, CPUs,
RAM, Disk)
38
Don’t be afraid to get dirty
Rundeck API Foreman API PuppetDB API
BigIP iControl Atlassian Stash API
40
What’s Next
• Sponsor Rundeckdevelopment for SCM integration and sensitive options encryption
• Evaluate network orchestration
• Containers
• Acceptance testing of puppet code with beaker (rspec+serverspec)