Conversational DevOps

By Don Jones

© 2016 Conversational Geek

Conversational DevOps

Published by Conversational Geek Inc.

www.conversationalgeek.com

All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, photocopying, recording, or otherwise, without written permission from the publisher. No patent liability is assumed with respect to the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions. Nor is any liability assumed for damages resulting from the use of the information contained herein.

Trademarks Conversational Geek, the Conversational Geek logo and J. the Geek are trademarks of Conversational Geek™. All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. We cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

Warning and Disclaimer Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied. The information provided is on an “as is” basis. The author and the publisher shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this book or programs accompanying it.

Additional Information For general information on our other products and services, or how to create a custom Conversational Geek book for your business or organization, please visit our website at ConversationalGeek.com

Publisher Acknowledgments

All of the folks responsible for the creation of this guide:

Author: Don Jones

Project Editor: J. Peter Bruzzese

Copy Editor: John Rugh

Content Reviewer(s): Karla Reina

Note from the Author

It’s surprising how long this “DevOps” thing has been lurking around the back alleys of the IT world, because only recently have vendors’ marketing folks gotten hold of it and made it a thing. But marketing hoopla aside, it’s a real thing, and it’s actually pretty cool.

As a founder of The DevOps Collective (DevOpsCollective.org), a US-based 501(c)(3) educational nonprofit, I’ve made it kind of a personal mission to de-myth-ize and clarify what DevOps really is, and how you can really “do” it. Thanks to Conversational Geek for giving me this opportunity to further that mission!

Don Jones

The “Conversational” Method

We have two objectives when we create a “Conversational” book: First, to make sure it’s written in a conversational tone so that it’s fun and easy to read. Second, to make sure you, the reader, can immediately take what you read and include it into your own conversations (personal or business-focused) with confidence.

These books are meant to increase your understanding of the subject. Terminology, conceptual ideas, trends in the market, and even fringe subject matter are brought together to ensure you can engage your customer, team, co-worker, friend and even the know-it-all Best Buy geek on a level playing field.

“Geek in the Mirror” Boxes

We infuse humor into our books through both cartoons and light banter from the author. When you see one of these boxes it’s the author stepping outside the dialog to speak directly to you. It might be an anecdote, it might be a personal experience or gut reaction and analysis, it might just be a sarcastic quip, but these “geek in the mirror” boxes are not to be skipped.

Within these boxes I can share just about anything on the subject at hand. Read ’em!

Is This a Real Thing?

Yes, DevOps is a real thing. Thanks for reading!

Yeah, you probably need a bit more than that. Okay.

DevOps has its origin in the world of Agile software development. To criminally oversimplify it, Agile is all about continuously producing small, incremental releases for your applications or other code. Rather than working and working and working, and then releasing some major version every other year or more, you put in a tiny amount of work in a “sprint,” and then release that into the world.

The idea is that these short sprints make it easier to adjust to changing market conditions, evolving business needs, and so

on. They also make it easier to squash bugs quickly. In theory, because each release changes just a tiny bit from the last one, bugs become easier to identify, because they stand out more.

It’s a great theory, but developers aren’t the only players in this game. Once they finish their code, Operations usually has to physically deploy that code – and Operations hasn’t, traditionally, been willing to keep up with the Agile release cadence. “Produce all the code you want, Developers, but we’re on a strict schedule over here, and we can’t deploy more often than about once a year.”

This kind of bummed everyone out, and so – to make a very long story very, very short – DevOps was invented.

The Phoenix Project, a book by Kevin Behr, George Spafford, and Gene Kim is considered

one of the seminal illustrations of what DevOps looks like from end to end.

DevOps, the Short Version

If you’re not going to have time to read more than this one tiny chapter (of an already short book), then you simply need to know that DevOps is about making Dev and Ops work more smoothly together. It’s about making Ops just as agile as Dev. It’s about everyone working more closely together to achieve the overall goal of getting software and services in front of users more quickly, more efficiently, and more effectively.

There are a lot of things that go into making that happen. Automation, for example, is a big piece of the puzzle in terms of bringing DevOps to life, but automation itself is not DevOps. You can end up building or buying a lot of tools to make

implementing DevOps easier, but tools themselves are not DevOps. DevOps is, more than anything, an IT management approach that prioritizes a rapid release cadence for software and IT services. In fact, it can sometimes prioritize “rapid” higher than “bug free,” as we’ll discuss in a bit.

DevOps as a Culture

“Doing” DevOps is really hard for a lot of organizations. Really, really hard. Sometimes impossible. And it’s important to understand that you can’t do DevOps halfway – it’s the whole philosophy or bust. That means it just won’t work for some organizations, because they won’t be able to shift their culture.

For example, organizations that really nail DevOps often have unusual looking IT teams. Rather than a Dev team and an Ops team, they might have teams dedicated to specific applications. Each team might have developers, Operations people, and whoever else is needed to make that project succeed. The same Operations people might belong to multiple application teams, lending their expertise to the team to make sure the applications can be deployed and managed efficiently.

All the Operations people from all the teams might form a “guild,” meeting occasionally to discuss what’s happening companywide, keeping everyone on track and sharing successes and hurdles from across the different teams.

Not every organization can bend and reorganize themselves that way. It’s probably important to figure out, up front, if your organization can do this or not. Maybe you can pull it off for one kind of project to start with, and see where it goes from there. Or maybe you have operational, political, or cultural constraints that just make DevOps the wrong approach for you. That’s okay – it’s just important to recognize that up front.

I usually suggest getting management – from the executive level – to buy off on trying

DevOps. If they don’t, it’s usually impossible to get things rolling.

By the Numbers

Puppet Labs, one of the bigger players in the DevOps tooling space, issued a “2015 State of DevOps Report.” It contained a whole bunch of fun numbers they’d gleaned from their customers and from surveying the general audience – and these numbers paint a compelling story:

Organizations employing DevOps approaches enjoyed

60 times fewer failures in their environment.

When failures did occur, the agile nature of DevOps

enabled 168x faster recoveries. That’s like going from a

day of downtime to eight minutes of downtime.

Using the DevOps approach enabled 30x more

application deployments, meaning the business was

able to respond more agile-y to changing conditions,

and to address bugs quicker.

The lead time for deploying an application was 200x

shorter. That’s like going from a month of prep time to

under four hours.

Yeah. Jaw-dropping numbers. Check out the whole report at https://puppetlabs.com/2015-devops-report (registration required).

The Big Takeaways

DevOps is an approach to IT management that prioritizes rapid releases for software and services. It requires deep cooperation from the entire technology team, and often involves re-organizing that team into project-centric, cross-discipline units. DevOps can involve tools and techniques, like automation, but is itself more than just tools and techniques.

A DevOpsian Example

What might DevOps actually look like in a for-real-real environment?

Well, let’s take an example that supposes a Microsoft-centric organization. I’m only doing that so I can put some real, concrete names to things, rather than being all vague and wishy-washy. In fact, DevOps is bigger outside the Microsoft space, right now, than in it, although it’s making big inroads.

Here’s the Story

Danny and Donna Developer, a superstar C# husband-and-wife programming team, check in the latest build of the app they’ve been working on. It’s called Amazing App, because, well, it’s amazing. They’ve been testing it locally on their development workstations, running it in a set of virtual machines they keep on hand for that purpose. They work in Visual Studio, and

they’re checking the code into a Team Foundation Server (TFS) repository.

That check-in triggers a runbook that the Ops team has built using System Center Service Management Automation (SMA). SMA talks to System Center Virtual Machine Manager (VMM), and asks it to spin up a couple of virtual machines. The app code is copied from TFS and compiled. The developers also included a Desired State Configuration (DSC) document in their source code, and that’s copied from TFS and injected into the virtual machines.

The virtual machines configure themselves according to whatever’s in the DSC documents, installing support libraries and whatnot. The application itself is installed, and a battery of Microsoft Test Manager (MTM, because yay acronyms, right?) tests start to run. These tests run the app through a variety of scenarios, injecting data and then analyzing the results to make sure it all comes out correct.

If something fails in the test, all the associated test data is logged back into TFS, and Danny and Donna are fired asked to make fixes. The test virtual machines are destroyed, because they’ll be spun up from scratch for the next time.

But if the tests all pass, then SMA runs a PowerShell script that packages the application into a System Center Configuration Manager (ConfigMgr, because Microsoft hates “SCCM” for some reason) installer package, and adds the package to the company’s ConfigMgr distribution points. SMA then opens a ticket in System Center Service Manager, assigning the ticket to someone in Operations who needs to manually assign a deployment time during a maintenance window.

Notice that DevOps doesn’t need to remove all human input from the equation!

Operations gets the ticket, checks on the other activity scheduled for the next maintenance window, and ensures that this package deployment won’t conflict with anything else. Maybe they review the test logs to get an idea of how the application did. They send a notice to the company’s Help Desk, letting them know what’s going to be happening, just in case something goes wrong – better to be prepared, no? Then they go into ConfigMgr and schedule the package for deployment.

Meanwhile, Danny and Donna are hard at work on the next release. Perhaps they’re reviewing feature requests and bug tickets that have been assigned to them in Service Manager, and using those to drive the next round of changes.

Depending on the organization, this might happen once a day! You might have users on a “fast track” deployment schedule, so they get the latest-and-greatest every day. Other users might be on a “standard track” where they only get updated every week or month. The deployment tools – ConfigMgr, in this case – make that easy to manage.

The Big Takeaways

The real takeaway here is all the automation. Development is a creative effort, and it’s something that right now can only be done by humans. But almost everything else in this story was, or could have been, automated. Ops created that automation, and I think that best highlights the role Ops evolves into within

a DevOps environment: creating units of automation that shorten and smooth the path between that creative developer effort and the users of the application.

Team-Building the DevOps Way

As I’ve hinted already, DevOps teams look a lot different than traditional IT teams.

The Traditional Team

IT has usually built discipline-specific teams. Nay, silos is often a better word, because in many organizations these groups of people operate in as much isolation from one another as possible. And that isolation is often preferred by these groups, because they’ve not fostered much of a working relationship.

Things like applications are formally passed between groups, like show tickets passing through that little hole at the bottom of the box office window. There are handoff meetings, and coordination calls, where the walls between the groups are

reinforced and formalized, and where communications occur only through rigid, begrudging channels.

Seriously, it can make the Cold War look positively warm and friendly. I’ve even been in organizations where only designated liaisons – sometimes actually called team ambassadors, and I am not making that up – communicate between teams.

No, really. The Team Ambassador role paid like $80k a year, and all you did was attend meetings all day, every day. Thrill a minute.

That won’t work for DevOps.

The DevOps Team

While this isn’t a hard-and-fast rule, most companies I’ve seen do DevOps well have reorganized into cross-discipline teams that focus on a particular project. Their joint goal is to get that project on track and keep it that way.

I didn’t hear your jaw hit the desk just then, so I’ll rephrase:

The goal of a DevOps team is to deliver its applications and services to users in a speedy and continuous manner. This was not necessarily the actual goal of a traditional IT team, where in many cases frameworks like ITIL® have taught us to try and slow all change as much as possible, ideally to the point of halting molecular motion in the team members’ bodies.

So a typical DevOps team might have a handful of developers, maybe a UI designer, and likely an Operations person. If the team produces an end-user app, then it might also have someone from the help desk organization, since they have to support the app. If it’s a financial app, it might have someone

from the Finance department. Bottom line - everyone who can contribute to the success of that project is on the team.

This does not make your team headcount go up geometrically, because some human beings will actually belong to multiple project teams. An Ops person probably isn’t needed full-time, 2,000 hours a year, on every single application team. So they float between projects, lending their expertise as needed.

Having these disciplines all collaborating on every project is massively beneficial. “Hey,” says the Ops guy, “that’s a very cool security library you’re thinking of using there, but I was playing with it last night and you physically can’t install it on half of our computers. We should really pick something else, so this doesn’t become an enormous fail later on.”

Guilds then form that encompass everyone from a given discipline. So all of the UI designers might get together once a month to talk about whatever UI designers talk about. Maybe they work on companywide UI standards, or talk about the latest in… I dunno, button design. The guilds benefit every project by sharing information across the teams. “Hey, we just found out if you use such-and-such communications library, gravity quits working and people die. So, you know, don’t use that one in your projects.”

Companies that have a massive investment in, and cultural bias toward, more traditional IT org charts will probably hate this DevOps team approach. Those companies cannot “do” DevOps. They just can’t. They can try, but it’ll be painful, and it won’t work.

The Big Takeaways

Rethinking the way your IT department – yeah, the whole department – is organized is not for the faint of heart. But you gotta do it if you’re going to do DevOps. Now, you don’t need to do it all at once, right? As I’ll explain in the last section of this book, you can often start with a single DevOps-ish team, and then expand the concept as you get good at it.

And this is where I re-emphasize the fact that this approach ain’t for everyone. It’s super important that you recognize when DevOps isn’t the right tool for your company, whether it’s because you can’t work in this way, or because – for political reasons – you simply won’t.

Core Capabilities for DevOps Teams

In order to pull off DevOps, your organization needs to develop a few very specific capabilities. And you need all of these, not just the ones that you like or that seem easy.

Agile Development

It all starts with Agile development, which – if you remember the first part of this book – is where DevOps came from in the first place. You need to have your developer teams organized in a way that supports short, rapid “sprints” of coding – often around 6 weeks – followed by a release of whatever got done during that timeframe.

Without going off on an Agile-friendly rant, the idea here is to stop waiting until you think code is “done” or has reached some particularly important milestone. You don’t code up 100

new features and then release; you code one feature and release it. And then code a couple more, and release. And then another, and release.

Automated Infrastructure

You also need the ability to automate provisioning and de-provisioning in your infrastructure. Big words. All they mean is that you need to be able to have virtual machines and/or containers spin up automatically, in response to some triggering event.

You don’t necessarily need to worry about what those events are – just make sure that whatever automation you’ve built is flexible enough to be triggered in a variety of ways. A developer checking in code to your source-control system, for example, might trigger some workflow that results in a new virtual machine coming into existence, so that the new code can be compiled and tested.

The actual capability you’re looking for here is called orchestration. At least that’s what the vendors who make the tooling call it, and you’re almost certainly going to have to buy or build tooling to make this possible.

Automated Software Testing

You also need the ability to run automated software tests. This will, again, require you to buy or build some tools, but it depends a lot on the development languages your team works with. The stuff you need to automate testing in a Microsoft .NET environment, for example, is different from what you’d use in a Java environment.

But tools exist. Microsoft makes some. TeamCity is one I’ve used. Open-source frameworks also exist, such as the Pester framework used to test Windows PowerShell code.

But the basic capability is simple: given a developer checking in code, you need to be able to spin up a fresh test environment, and start running a battery of tests. Those tests need to collect a lot of detailed information, and ship it all back to the developer for action. And, if the tests pass, you’ll be forwarding that code on to production as part of your overall orchestration.

This automated testing is the bit that tends to freak people out the most, to be honest. “What if,” they cry, “you forget to test for something that turns out to be broken?” Yeah, well, you’re going to ship a bug in that case. Maybe a big bug. Maybe a big, hairy, eight-legged – well, you get the idea. But this is no gosh-darn different from what you’ve done in the past, and anyone who says, “no, we’ve never shipped a bug the way we do things now” is lying.

The difference with automated testing is when you do screw up (and you will), the fix is to add another test, or fix an existing test. In other words, you are learning from your mistakes, and doing so in a way that will prevent that exact mistake from happening again, because once you fix the test, the computer will make sure it always happens from then on. You can’t say that about manual testing.

One of my catchphrases about DevOps is “learn to fail with style.” It means that you

WILL have failures, but you’ll get past them, and you won’t repeat them.

Developers Specify Environments

This is a key bit a lot of people miss.

You see, one of the traditional problems in the traditional hand-off from Development to Operations is that the damn developers (sorry, I’m an Ops guy) forget to tell you what the damn app needs to run successfully in the damn production environment. Not that this has made me bitter at a young age, or anything.

It’s because developers tend to work a bit organically. They go to implement some feature and realize there’s a great pre-made library that’ll do some of the heavy lifting. So they install the library, and realize it depends on some other, lower-level library, which they also install.

They then go out drinking, or camping, or whatever developers do in their spare time (Ops people don’t have spare time), and they forget to tell you what they installed. So Ops pushes the application out to a billion users, and it proceeds to crash with prejudice because the libraries are missing. Argh.

The fix for this is to let developers specify the components their application needs, and to do so in some machine-readable way, and to make that an actual part of their source code. Here’s a real world example: Amazon Web Services (AWS) does this in their Elastic Beanstalk offering.

Combined with the previous capabilities I mentioned, this creates a stupid-proof workflow:

1. Developer checks in code, which includes whatever

wacky stuff the application needs to run.

2. Automation spins up a test environment, reads the

configuration instructions from the source code, and

sets up whatever the developer specified.

3. If the developer forgot anything, the test fails, and the

developer is notified to get their life together and try

again. But if the test passes, then you know you’ve got

everything. You can deploy the same app, with the

same environment specs, to production, confident that

it’ll work.

Because that automated test environment gets created fresh from scratch each time, there’s no worry that something will have “snuck in” that you won’t notice. The only things that go into the test environment are the things the developer specified, so the test is a real and valid way of making sure everything was specified.

It’s awesome.

Truly, Elastic Beanstalk is a gorgeous example of DevOps, with AWS playing the role of Ops. Check it out, even if you don’t plan to use it.

DevOps Purely for Ops

Up to now, I’ve kind of taken a very classical approach to explaining DevOps, which is to say I assumed you have in-house developers making in-house applications. If you don’t have your own in-house dev team, then you can’t do DevOps the way I’ve been describing. I’ve never seen anyone successfully pull off a DevOps approach when all they use are applications from an outside vendor – you just can’t have the relationship and proximity in that situation.

But you can still “do” DevOps, even if your only in-house IT assets are your servers or client computers, or even networking devices like routers and switches. This is the whole “infrastructure as code” concept, and it lends itself beautifully to DevOps.

I abhor the phrase “infrastructure as code” because it’s wrong. Read

http://devopscollective.org/2016/01/03/infrastructure-as-code-isnt-the-right-way/ for my

full opinion

The Basic Concept

The idea is to run as much of your infrastructure as possible – ideally, all of it – in a “policy-based infrastructure” approach. What’s that mean?

You stop running around and configuring servers, clients, and network devices by hand. You don’t even write scripts to provision these things or to manage them. Instead, you implement an architecture that lets you write policy documents. These documents – which are often human-readable, at least by a knowledgeable human – get fed to your servers, clients, and whatnot, which then dynamically configure themselves to match whatever the documents say. When you need to change something, you change the policy documents, and the servers and stuff just magically reconfigure themselves to match whatever the document now specifies.

There’s obviously a lot of code going on underneath the magic to make it all happen. Technologies like Microsoft’s Desired State Configuration (DSC) are designed to do this, as are commercial products like Chef and Puppet, and open-source frameworks like Ansible and Salt. All of these let you implement policy-based infrastructure in one way or another.

Those policy documents aren’t necessarily software code in the way we usually think of it. That is, they don’t necessarily contain logical decision-making components, loops, variables, and whatnot. Depending on the system you implement, they might have those things, though. And regardless, they do serve the same purpose as software code, right? You’re using those documents to tell a computer what to do, or more specifically in this case, what to be. And that’s where the DevOps connection happens.

DevOps for Ops

If you’re using code – what I’m calling a “policy document” to be generic – to tell computers how to configure themselves, then that code (er, policy document) lends itself to a DevOps approach.

For example, changes to a policy document might be checked into a source control system. That check-in action might trigger a test environment to be spun up, and the new policy document fed to it. Once the test environment configures itself according to policy, a series of automated tests might run to confirm the configuration is correct. And if those tests pass, then the new policy document might find itself in your production environment. All of that, from the moment of checking in the document, could be automated.

So you’re not using DevOps to smooth and speed the deployment of applications – you’re using it to smooth and speed the deployment of servers. Or client computers, or network devices, or whatever. Your infrastructure becomes more agile, because you can make, test, and deploy infrastructure changes quickly and confidently – just as DevOps lets you do with applications.

This is exceedingly cool.

The Big Takeaways

Since DevOps is a management approach more than anything else, there’s no reason it has to be limited to the world of software applications. The approach can be just as easily used, and can be just as beneficial, to manage IT infrastructure components – provided those components are being provisioned and managed in a code-like fashion.

Getting Yourself Started

So how do you actually start doing this DevOps stuff?

Start small.

Because doing DevOps right can involve serious organizational and cultural changes, it’s worth starting small. Like Agile itself, this gives you the flexibility to make small, incremental changes to your approach as you go. DevOps isn’t some rigid framework like ITIL®, and it isn’t some standard like ISO. One company’s successful DevOps will look very different from another company’s. You can steal borrow ideas from others and try them on for size, but at the end of the day, you’ll make your DevOps work for you.

So pick some project that’s perhaps already on an Agile-style development cadence. Assemble a cross-discipline team. Let the Ops folks help Dev understand what might need to change in the app to facilitate faster roll-outs and more frequent changes. Start automating unit tests, if you haven’t already

done that. Build or buy tools to help automate deployments – just for that one project.

You’ll run into problems – everyone does. You’ll also fix them. And you’ll gather important organizational skills and comfort levels with this new approach, and eventually, one day, you’ll go to work and it’ll be DevOps as far as the eye can see.

Good luck!