Candy Canes, Elves and Duck Tape

Patrick LaRoche Co-Founder patrick@toplog.io

aka how I learnt to stop worrying and love puppet

Candy Canes, Elves, Duck Tape

I think all products have three main components that make up the product’s infrastructure… candy canes, elves and duct tape

• Candy Canes = Defining “chunks of infra” (some call these micro servers"

• Elves = “They put together the micro services”

• Duck Tape = “The messy bits that make it actually work”

History

• First thought ever: • need a way to build up our product on my laptop

• At this point, topLog was a shell script

• Next step, putting a UI in front of it

• LAMP stack (I’m old…. school…..)

• Someone told me about this vagrant thing….

First stab

Vagrant

Vagrant gives you this great way to build up a VM of your choosing, on pretty much any virtual machine platform (VMWare, VBox, AWS, Rackspace, Docker, etc)

All you need is Vagrant, the provider (VBox by default) and a vagrant file.

Sounds awesome!

Vagrant

Our first Vagrant file (sucked)

• Set box up

• Run a provisioner

• do some inline stuff to get code in place

Vagrant.configure("2") do |config| config.vm.box = "toplog" config.vm.box_url = "http://files.vagrantup.com/precise64.box"

config.vm.network :private_network, ip: "192.168.56.101" config.vm.network :forwarded_port, guest: 80, host: 8080 config.ssh.forward_agent = true

config.vm.provider :virtualbox do |v| v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"] v.customize ["modifyvm", :id, "--memory", 2048] v.customize ["modifyvm", :id, "--name", "demobox_toplog"] end

config.vm.provision :puppet do |puppet| puppet.manifests_path = "manifests" puppet.module_path = "modules" puppet.options = ['--verbose'] end config.vm.provision :shell, :inline => "git clone -b master git@repo:toplog.main.git www/toplog.main/; git clone -b master git@repo:ui.git www/ui/; git clone -b master git@repo:x.git www/x/; chown www-data:www-data -R www/"

end

Things got messy (Duct Tape)

Vagrant lets you use different ways of provisioning, chef, puppet, bash, inline scripts, pre made base boxes, whatever you want…

I’m old school and lazy.. so bash scripts and inline scripts it is!

Added a shell “magic bash script” ^

That got big and complex fast… so lets try out this puppet thing…

dev.vm.provision :shell, :path => "./shared/setStuffUp", :args => "-m master -u master"

Why Puppet?

Google’s fault: “lamp stack vagrant puppet” had a better top hit then “lamp stack vagrant chef”

Seriously… thats all… 2 years later, loving puppet, I’m sure Chef is awesome too

What is puppet: awesome provisioner, you define modules, and init scripts, it uses them to deploy (provision) your box. With Chef this is “recipes and cookbooks”

First init.pp# Class: toplog # # Installs toplog class toplog { service { toplog: ensure => "running", enable => true, hasstatus => false, hasrestart => true, provider => "debian", require => [ File ["/etc/init.d/toplog"]],

}

file {“/etc/init.d/toplog.sh": ensure => present, mode => 755, owner => root, group => root, source => [ “puppet:///modules/toplog/init.d", ], }

}

Okay, so starting to get somewhere

• Start by having vagrant use a puppet manifest (init.pp) and some modules to start building boxes with the right components / services (The Candy Canes)

• puppet puts it all together (Elves)

• BUT:

• This is all in one box

• And requires vagrant (does it?)

• And a LOT of variables, etc are hardcoded into those init.pp files (DUCT TAPE)

At this point

• Actually don’t need vagrant if we already have a box! Pull down our vagrant repo, and just run puppet directly:

puppet apply --modulepath=/root/dev/modules init.pp

• This lets us deploy this all in one box anywhere! Canarie, Rackspace, AWS, Grandma’s server…

• But this is incredibly brittle (hard coded variables, all in one, requires different init.pp’s for different deployments, etc)

How to make this better

• HEIRA

• Uses yaml files to define variables

• PuppetMaster (santa clause ? too much ?)

• A central server that has all manifests and modules that can be used, uses keys etc to authenticate, magic!

• Node definitions:

• Lets the PuppetMaster know which type of node to build and only deploys the services needed on that node

Example HIERA

#hostname.regex ###### WORKER ###### - ‘^worker-\d+$': modules: - 'wget' - 'gcc' - 'mysql' - 'redis' - 'toplog::supervisor_install' - 'toplog::directories' - 'toplog::hhvm_install' - 'toplog::env_set' - 'toplog::common'

toplog::supervisor_install::name: 'toplog_jobs' toplog::supervisor_install::td_name: 'toplog__jobs' toplog::supervisor_install::numprocs: 10 toplog::supervisor_install::numprocs_start: 1 toplog::supervisor_install::command: 'hhvm artisan queue:listen --timeout=86400 --tries=1'

Example HIERA

#common.yaml #base username baseUser: ‘toplog'

#These are used when we are pulling the repo artisan_env: - "DB_HOST=db.rds.amazonaws.com" - “DB_NAME=BLAH" - “DB_USER=BLAH_USER" - "DB_PASSWORD=PASSWORD"

#name, tag or hash of the branch main app is on revision_app: 'tags/v4.2.2'

#some env file stuff environment: - "PHPLIB=pecl" - "ENV=staging" - “SERVER=HOST.IP"

Now what do we have?• Node definitions

• Each one can have different components of topLog in them

• uses simple hostname rules (worker, app, scheduler, etc)

• reuses manifests across nodes when needed

• high level env / config options passed down to modules as needed

• Different environments

• We have staging, prod, dev and standalone

• they all have a different node def thats simply says what components to put where

• Can be deployed in many places, in minutes

Example our latest Vagrantfile

#two boxes, dev and logs, on `vagrant up` both are made and talk to each other

logs_config.vm.provision "puppet_server" do |puppet| puppet.puppet_server = "puppet" #puppet.client_cert_path = "./ssl/certs/logs.pem" #puppet.client_private_key_path = "./ssl/private_keys/logs.pem" puppet.puppet_node = "logs" puppet.options = "--verbose --environment=development" end

dev_config.vm.provision "puppet_server" do |puppet|

puppet.puppet_server = "puppet" #puppet.client_cert_path = "./ssl/certs/bitnami.pem" #puppet.client_private_key_path = "./ssl/private_keys/bitnami.pem" puppet.puppet_node = "dev" puppet.options = "--verbose --environment=development" end

Putting all the Elves together…$hostname worker.toplog

$cat /etc/puppet/puppet.conf server = puppet environment = staging

#all it takes to build the box $puppet agent -t #forces puppet to run the agent, if left alone, this runs every 30 minute anyway

• We use hostnames rules to define the node the server will be

• Want to change the node type? Change the hostname, run puppet, PuppetMaster will just give you back what you need to switch

Current Duct Tape

• Puppet is awesome, but can break, a lot..

• At the root of a lot of manifests is exec commands, which you have to be careful, especially when upgrading pieces

• Heira has been a life saver, but you have to be careful to understand variable scope and what you can pass to manifests

• Testing of full infra deploy is still very hands on and challenging. Real example: Elastic search didn’t get the logging file written correctly, so ES failed to start, but since not logging, VERY HARD to debug

Cons!

• A deploy goes from scratch, so can take longer then you like. Defining AMIs / containers / base boxes is our next step so that puppet is used for updates, not “from bare bones”

• We don’t keep puppet agent running on prod, we manually run it (which, kind of defeats the purpose).

• To overcome this, are building out better CI / testing environment so we can trust that puppet won’t break prod

• Updating code: puppet can’t pick up new hashes inside a git repo, so you have to give it specific hashes and update it that way…… at least that we can tell…..

BUT! (CANDY CANES)• We can deploy topLog with one Vagrant up inside our own

laptops

• We can give partners and enterprise customers custom deployments based on node definitions within minutes

• We moved from rackspace to AWS in 2 days

• Small Note on the move: we also are very careful to make sure our product does not depend on any specific provider features. And if we do take advantage of them (say, S3 buckets) we make sure we have a standard “backup” method that can be used. Puppet + Hiera lets us switch between all of these different platforms fairly easily.

Thanks… email bellow for any specific questions or if you want more details

Patrick LaRoche Co-Founder patrick@toplog.io