de-centralise and conquer: masterless puppet in a dynamic environment

De-centralise and Conquer

Masterless Puppet in a Dynamic Environment

Sam Bashton, Bashton Ltd

Who am I?

● Linux guy since Slackware, floppy disks and

root + boot

● Using Puppet since 2007

● Run a company Manchester, North West

England

● We provide outsourced ops for other

companies

● High traffic environments

● Most are now on Amazon Web Services

● #1 reason for moving to AWS? The ability to

scale on demand

Our Environments

Server instances, single day

How we use Puppet

● No Puppetmaster

● Puppet manifests and modules distributed to

all machines

What's wrong with standard Puppet?

● Pets vs Cattle

● Standard Puppet configuration assumes that

servers are pets, not cattle

What's wrong with standard Puppet?

● Standard Puppetmaster/Puppet Client

configuration makes assumptions about

environments○ Machine creation is a manual operation

■ Sign certs

○ No in-built mechanism to automatically clean up old

machines

What's wrong with standard Puppet?

● Puppetmaster is a single point of failure

● When servers are pets, this isn't too much of

a problem○ Existing servers continue to work, but not any

updates

What's wrong with standard Puppet?

● When servers are auto-scaling cattle, new

instances can appear at any time

● New instances require config to become

operational

● Configuration requires Puppet

What's wrong with standard Puppet?

● Our environments span multiple data centres

('availability zones')

● Imagine a data centre fails

● New instances get auto-provisioned to

replace missing capacity

● But these instances need the Puppetmaster

● ..which was in the failed AZ

What's wrong with standard Puppet?

● Resource contention

● Even when Puppetmaster isn't in the failed zone, multiple concurrent connections slow things down

What's wrong with standard Puppet?

● None of these problems are insurmountable

● We could have configured a Puppetmaster a

cluster of Puppetmasters for our needs○ With autosign

○ and some sort of certificate distribution mechanism

○ uuid certificate names

○ And a mechanism for cleaning up old machines

Meanwhile, on the other side of the room...

● Another team was evaluating Pulp

● Provides yum repository management

● To be used for managing security updates

and deploying application code

http://pulpproject.org/

Pulp

● Allows cloning of repos, copying packages

between repos

● Allows us to push packages to clients○ Uses qpid message queue

● Has 'content distribution servers' for easy

replication + clustering

How we deploy code

● Everything managed via the Jenkins

continuous integration server

● Jenkins uses Pulp to install code on remote

machines

How we deploy code

● Jenkins fetches code from source control

(git)

● An RPM is built

● Tests are run

● The RPM is added to the relevant Pulp

repository

● RPM installed on the target machine(s)

How we deploy code

● Jenkins also manages deployment lifecycle

● 'Promoted Builds' plugin used to install

previously built RPMs on staging

● Promoted Builds plugin then used to install

the same RPMs on live once testing is

complete

Deploying configuration as code

● Idea: Why not just build an RPM of our

Puppet manifests + modules?

● Have puppet apply as part of the %

postinst

Deploying configuration as code

● Allowed us to reuse our existing code

deployment infrastructure

● Manage configuration deployment from

Jenkins

How we deploy configuration

● Puppet manifests and modules are checked

into git

● Jenkins builds configuration into an RPM

● Jenkins promoted builds plugin applies the

updates to environments via Pulp

Our system architecture

● Quite AWS specific

● Concepts could be applied to other clouds○ Once they catch up in terms of toolsets..

Separation of Roles

● CloudFormation - defines infrastructure

● Puppet manages configuration

● Pulp manages package versions○ Pulp in turn managed via Jenkins for custom repos

Instance Provisioning

● Minimal images used

● cloud-init the only addition beyond standard

CentOS install

● cloud-init allows us to specify script to be run

at boot

Puppet bootstrap

● cloud-init script adds local Puppet yum repo

and installs the Puppet configuration RPM

● Installing the RPM installs Puppet and

applies the configuration

Machine metadata

● cloud-init also sets some variables in

/etc/environment

● $HOST_TYPE - the type of machine this is, eg

web, cache

Machine metadata

● Also set facts to be used by facter, eg RDS

database hostname○ Values from CloudFormation

● $FACTER_DBHOSTset via cloud-init too, eg /root/.my.cnf

Defining machine roles

● For each machine type there is a manifest /etc/puppet/manifests/$HOST_TYPE.pp

● This file looks something like this:node default {

import global

...

}

Building the RPM

● Puppet manifests and modules are all

packed into an RPM

● Owner set to root, mode 600

● %postinst creates an at job set for now + 1

minute to run puppet apply

Deploying configuration

Free wins!

Free wins

● Greater control over the timing of Puppet

runs

● Improved visibility - for ops and devs

● Configuration changes now have to be

deployed to testing/staging first

More free wins

● Puppet configs now have a version

● Easy to find config version on the machine

itself

● Config changelogs accessible on every

machine○ (Git changelog added to RPM)

Cheap wins

Cheap wins

● Jenkins performs syntax checks with puppet parser validate

● Jenkins also runs puppet-lint on

manifests

Cheap wins

● Config change required for new code?○ Make the Puppet RPM version a dependency

The downsides

● Puppet manifests and modules on all

machines○ Potentially a security issue?

● No reporting*

Alternative implementations

● Don't want to use Pulp?

● Could do basically the same thing with yum

s3 plugin

https://github.com/jbraeuer/yum-s3-plugin

Questions? Comments?

Sam Bashtonsam@bashton.comTwitter: @bashtoni