Release Engineering Downstream of an OpenStack

ProjectBest Practices for the Path from Idea to Production with

an Open Source Code Base

Rainya Mosher

@rainyamosher | www.rainyamosher.com

People don’t want…a quarter inch drill. They want a quarter inch hole.

Clayton M. ChristensenThe Innovator’s Solution

About Rainya• 10+ Years Web App Devoplment

(ASP & .NET C#)

• 3+ Years OpenStack Operations

• Master of Science in Technology

Commercialization

• Musical Theatre Nerd, Singer, and

Actor

• Twitter: @rainyamosher

• Blog: rainyamosher.com That about sums it up.

ENFP

What is Release Engineering?

Release Engineering (RelEg / RE) is the sub-discipline of software engineering concerned

with the compilation, assembly, and delivery of source code into finished products or other

software components.

https://en.wikipedia.org/wiki/Release_engineering

Release Engineering

UPSTREAM

PDLC / SDLC TOOLS

CULTURE

What is your relationship with the upstream codebase?

Current Dates & Milestones: https://wiki.openstack.org/wiki/Releases

http://docs.openstack.org/infra/manual/developers.html

• Clone the code from github• Branch your change• Make fixes• Submit review• Submit patch revisions until you

get passing tests and a +2• Change is merged

Average Review Wait Times as of 7/20/2015

• Nova – 22 days• Neutron – 9 days• Glance – 26 days

http://stackalytics.com

Pull OpenStack

Master

Consume Third-Party

Distro

Internal Integrations & Configs

More ownership over release

cadence

More in-house resources for packaging

More flexiblity with to

integration of internal

changes into python source

code

Third party guides release

cadence

Fewer in-house packaging

needs

Less flexibility in integrating

internal changes into

python source code

No matter the upstream relationship, you’ll need to account for your

internal configurations and system integrations!

Even understanding the upstream

landscape of the project, there will be

times when someone – usually a product

or engineering manager – will say “I don’t care about the community, just ship

the feature/bug fix/widget.” What can

you do?

How will you manage the product and software development lifecycles?

PDLC

SDLC

AGILE

Scrum, Kanban, XP, etc.

PDLC is the management of a good or service from inception,

design, engineering, build, sale, maintenance, and disposal

through the integration of people, data, process, and

business systems to make an optimal experience for a

customer (internal or external).

SDLC is a framework of steps and phases providing a model for development teams to follow when creating a piece of software to improve the quality of the software and the overall process.

AGILE SDLCs are a collection of methodologies valuing

flexibility, collaboration, communication, and rapid

iterations in the development process.

Simple Easy

One role / task / concept / dimension

vs Complex

Familiar, part of our skillset, near at handvs Hard

Source Image: Rich Hickey, “Simple Made Easy”, 11/2012

Early on, the easy path gives you speed and a sense of doing.

Over time, easy ignores complexity, slows you down, and leads to painful technical debt.

Being intentional with your PDLC & SDLC approach can help you get from mad baby to successful sand eating baby.

What tools will you use for build, release, and deploy activities?

BUILD RELEASE DEPLOY

SYSTEMS

Pipeline

Config Management Packaging

Orchestration

BUILD RELEASE DEPLOY

SYSTEMS

Pipeline

• Invest in your non-production environments

• Engineers cannot deliver stable software to production if the places they have to develop and validate are unstable

Dev

• Prod-like functionality (as much as possible)

• Unit tests at a minimum

Integration

• First place for Dev to integrate code with upstream packages

• Unit tests • Basic smoke tests

Test

• Full smoke tests

• High level integration testing with tightly coupled services

Pre-Prod

• Mirrors Prod• Full integration testing with all services

• Load and performance testing when major changes

Production

• Run baseline tests prior to an upgrade

• Automate the deploy to keep human error out of the way

• Don’t touch it!

Speed: Find failures quickly Accuracy: Find all failures Stability!

BUILD RELEASE DEPLOY

SYSTEMS

Config Management

• Make all system and software changes through a config management system

• Start with immutable servers and services (it’s harder to give up root later)

• Ansible, Puppet, Chef, Salt all have strengths & weaknesses – take the time to determine which is best for your situation!

BUILD RELEASE DEPLOY

SYSTEMS

Packaging

• Successful packaging has been done with RPMs and DEBs on physical hosts and VMs

• Containers are rapidly maturing into a viable approach for the control plane

• Finding an approach everyone can agree to is perhaps the biggest challenge!

• Configs can be bundled with the source code or distributed separately based on frequency of updates and scale of system

BUILD RELEASE DEPLOY

SYSTEMS

Orchestration

• Thoughtful orchestration for deploy-time activities can overcome many shortfalls of software

• Dedicate time and energy to find the best order of operations for introducing changes to services

• Deploy patterns - blue-green, rolling, shotgun – are limited by the architecture of the software being deployed, and that’s ok!

How will your organization adapt to the changes necessary

for success?

We need to create a culture that reinforces the value of taking risks and learning from

failure and the need for repetition and practice to create mastery.

Gene KimThe Phoenix Project

CI & DevOpsRelease

Management

Engineering Practices

Always code as if the person who ends up maintaining your code will be a violent psychopath who knows where you live. ~ John F. Woods, c. 1991

Until zero-downtime deploys are a reality, CI/CD may as well stand for continual impact and constant downtime. Adopt a release management process to stay sane!

Automation first! This is difficult for many organizations to do. It often means slowing down, taking the long way rather than the easy way, and telling people “not right now.”

Be Intentional

Adapt to Overcome

Simplicity First

References Images

- Gripp & Drouhard. Space Unicorn. (2012). http://youtu.be/17o1OlroNSE

- Miltonious Blog Unicorn of Technical Difficulties, Source Unknown

- Random Internet Meme’s found via Google Search

- Hickey, R. 2012. Simple Made Easy. http://www.slideshare.net/evandrix/simple-made-easy

Quotes

- Christensen, C. M. (2003). The Innovator’s Solution: Creating and Sustaining Successful Growth

- Kim, G., Behre, K., Spafford, G. (2013). The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win

References Useful Links

- OpenStack Dev: http://docs.openstack.org/infra/manual/developers.html

- Review Wait Times: http://stackalytics.com/

- OpenStack Releases: https://wiki.openstack.org/wiki/Releases

- Rel Eng Overview: https://en.wikipedia.org/wiki/Release_engineering

Release Engineering Downstream of an OpenStack

ProjectDiscovering the Path from Idea to Production with an

Open Source Code Base

Rainya Mosher

@rainyamosher | www.rainyamosher.com