Puppet at BackstopAnother year of lessons

1

$ whoami

‣ Bill Weiss <bill@backstopsolutions.com>

‣ @BillWeiss

‣ Probably responsible for 90% of the badness in this talk, 25% of the good

‣ Still not a designer

2

What’s this talk about?aka, who cares?

3

4Image via MemeCrunch. Original owned by the History Channel.

No, wait, that’s not it

(Come talk to me afterward about that)

5

‣ I’m here to talk about things that work (and don’t) for us in our environment

‣ Hopefully most of it is useful to you

‣ Some might just be there so you can point and laugh

6

–You

“What the heck is Backstop?”

7

Super helpful, I know

8

SaaS for institutional investment firms

‣ That’s it.

‣ You’re probably not our target audience

‣ If you are, awesome, let’s talk later

9

Some numbers

‣ ~195 Puppeted machines in production (US)

‣ ~350 total

‣ 2 datacenters, 2 AWS AZs, 1 office closet full of machines

‣ 50+ Engineering folks (IT, dev, QA, etc)

Numbers of machines from MCollective!

10

Logo owned by the CentOS project

Logo also owned by Oracle, of course

Logo owned by Oracle

Logo by David Heinemeier Hansson11

‣ Big application is J2EE on JBoss

‣ Lots of smaller bits (“services”) on Tomcat

‣ Second app is Ruby on Rails

12

How ‘bout that Puppet?

‣ Moved to GitHub 2013/11/18

‣ ~4100 commits since then (~15 commits/day!)

‣ 2700 from people outside sysadmin team

‣ ~1350 pull requests from 21 contributors

13

The move to GitHub was awesome

‣ 4 years of Subversion before that had 29 authors and 6700 commits

‣ Other things at play here beside just s/svn/git/ , but it helped a lot

‣ How?

14

! git workflow‣ Org-owned repo, two branches

‣ master ➡ production

‣ office ➡ testing (pre-prod)

‣ Everyone forks from master and sends pull requests to office

‣ Pull changes from office to master as needed

15

‣ Everyone has their own fork with work branches

‣ Branch names probably are Jira ticket numbers

‣ Everyone bases from master, not office, so we can pull changes independently

16

This is why we have so many contributors

17

‣ If you’re a software shop, you already have a workflow. Do that

‣ Otherwise, this is working well for us

18

" git hooks‣ We had a pretty good set of pre-commit hooks in

svn to test changes

‣ Github doesn’t do that #

‣ Git pre-commit hooks exist, but it’s up to each person to set them up

‣ No awesome answer yet

‣ The answer might be…

19

! Jenkins!

‣ We used to have this “deploy” job on the Puppet masters as a cron job:

• (cd /etc/puppet && svn up)

‣ Nothing could go wrong there, right?

‣ For a while we just did it manually instead

20

‣ Jenkins polls for changes to either branch

‣ Runs a deploy job that grabs repo, runs librarian-puppet, a “build” step, and then pushes it out to the masters

‣ Determines masters via mco call

‣ All the good (and bad) of capistrano for the push

21

22

alias puppet-test='sudo puppet agent --test --noop --environment'

That deploys an environment titled UserBranch, camel-cased

‣ Jenkins also runs a test suite in parallel

‣ Some day that’ll be a precursor to the deploy job. Some day…

23

" Sometimes you just want to do something to prod

‣ All that well-defined procedure means time for quick fixes goes up

‣ Make a branch, make some changes, commit, push, PR to office, merge, PR to prod, merge, run deploy…

24

‣ Possible answers here:

• Pull some things out into their own repos

• Accept that it’s slower

• Tool it up, make the machines do the slow parts

25

! MCollective is still awesome

26

Original image: http://xkcd.com/353/ Cheapo editing courtesy OS X Preview “Annotate” Same graphic as last year’s talk. I love you, mco!

It’s like Frank’s Red Hot

‣ Our deploys (not IT, app teams) now use it to figure out which machines to deploy to

‣ We wrote an agent for our VM system (Proxmox) Which, uh, evidently we haven’t released. Will do.

‣ Stop/start the world for Oracle maintenance

27

" ActiveMQ is hard

‣ MCollective sits on top of ActiveMQ

‣ Easy, right? Just install it (with Puppet, even) and go to town

‣ Second datacenter? Yeah, just get them talking to each other and…

28

A sad example

29

Suddenly, one day…

‣ Pretend x is time

‣ y is memory usage

‣ That’s our activeMQ

• At 2AM

• Every day

30

31

“Sudden Clarity Clarence” via memecreator.eu Original image? Who knows. Might be Clarence.

‣ Long story short: make really sure your broker names are unique across your network

‣ Really really sure

32

That won’t happen to us

‣ Maybe not

‣ Just know you’re adding another service you rely on

‣ Monitor it

‣ Measure it

‣ Have a game plan for what to do when it goes horribly wrong

33

‣ R.I.’s scripts [1]

‣ Erwan Ben Souiden, aka labynocle [2]

‣ A script I butchered to help

34

[1]: https://github.com/ripienaar/monitoring-scripts/tree/master/activemq [2]: https://github.com/labynocle/nagios_plugins_by_labynocle/tree/master/check_activeMQ [3]: https://github.com/BillWeiss/MiscScripts/blob/master/check_activemq.pl

I know, I know, bleh, Nagios

! Cloudstack!

‣ It’s great

‣ Setup was easy

‣ Only using it in test now (so we only kinda care if the machines burn down)

35

That’s the CloudStack monkey, owned by the Apache Software Foundation

‣ Remember that puppet-test alias?

‣ That + CloudStack means people can build machines to test Puppet changes to

‣ I was surprised recently by a service that just showed up ready to run in prod, Puppet and all

36

You don’t have to use CloudStack, but you need something like it

‣ That could be OpenStack, Vagrant, even some AWS credit

‣ You just need fast to build disposable machines

37

" Would be better in prod

‣ We’re not yet running CloudStack in production. It’ll happen someday.

‣ So, while I’m singing its praises, I can’t say we’re all in.

38

! create_resources‣ I have to show off one chunk of code

‣ Let me quote from TFM:

39

‣ That’s vague, right? The example in TFM didn’t inspire me either, so I glossed over it for years.

Converts a hash into a set of resources and adds them to the catalog.

I already said something about Frank’s Red Hot, right?

Almost everywhere I use it looks like this

40

$thing = loadYaml(“${yamlDir}/thing/${environment}.yaml”)create_resources( some::define, $thing )

That same YAML gets used in a bunch of those.

backstopsolutions: lastOctet: 226 clientFqdn: 'www.backstopsolutions.com' seconddn: 'backstopsolutions.com' includeFile: 'true'

‣ One define uses the lastOctet and some knowledge about our systems to set up an interface on the load balancers

‣ Another uses the hostnames to set up a couple of vhosts for the Rails app

‣ Yet another adds it into several DNS zones

41

‣ Another benefit: it’s YAML. You don’t have to write it by hand

‣ Grab some external data source, frob as needed, dump out some YAML. As long as you meet the spec, you’ll be fine

42

Huge gotcha here

‣ Everything you pass that YAML to needs to do something with all the parameters named in the file

‣ Adding a field thus means changing all those defines

‣ We have a lot of ‘$foo = “dontcare”’ in there. Just come up with an idiom and stick to it

‣ You could probably do something clever to filter out params?

43

" Maybe not everything goes in Puppet

‣ Last year I had this slide (included verbatim)

44

Well, I can’t always be right

45

‣ Think to yourself: do I ever want to do this thing without running Puppet?

‣ Does it make sense for this to go through the same test -> prod cycle?

‣ Do the same people need to see this content as the Puppet manifests?

‣ Can it be generated in one place, or do you need to gather data from several?

46

‣ As we’ve grown to more sites and more Puppet masters, this has hurt all over the place

‣ The deploy concept (vs `svn up`) made a mess of the checkin of this derived data

‣ I still think it’s an OK approach, but it needs thought

47

Examples

‣ DNS data: there’s only one actual copy of your DNS, and that’s what the world can see. What does a beta machine do with it?

‣ Load balancer configs: multiple datacenters need to know about each other, but can’t see the other’s database to derive config from

48

</TechnicalContent>

49

Want to help out?We’re hiring

50

‣ Sysadmins

‣ Developers (Rails or Java)

‣ Testers

‣ Lots more

backstopsolutions.com/careers

DevOpsDays Chicago!

‣ October 7th and 8th @ the Sears Tower

‣ CFP open until the 22nd

‣ Registration open now! Use code PUPPET_CAMP for 10% off devopsdays.org/events/2014-chicago/registration/

‣ I might be on the committee

51

I refuse to call it the Willis Tower, y’all

Puppet Users Group

‣ Hosted by this guy, downtown, ~monthly (coming soon!)

‣ meetup.com/Chicago-Puppet-users-group/

52

That’s itQuestions?

53