dev to delivery with puppet, vagrant and aws

DEV TO DELIVERY WITHPUPPET, VAGRANT AND AWS

SAM BASHTON, BASHTON LTD

ABOUT MELinux guy since Slackware, floppy disks and root + bootUsing Puppet since 2007Run a company in Manchester, North West EnglandWe provide outsourced ops for other companies

TOOLS FOR THE DAY

WHAT IS THE POINT OF THISTALK?

WHAT WE HAVEDevIntegrationQAStageLive

WHICH ENVIRONMENTS AREMANAGED BY PUPPET?

DevIntegrationQAStageLive

WHAT WE'RE AFTERConfidence that everything will work correctly inproductionConsistency between environments

OPS AND DEVS CO-OPERATING

Previously:Devs built stuffLater, Ops came and built production infrastructure

This caused many IT problemsThe solution?

OPSVELOPMENT

OPS AND DEVS WORKINGTOGETHER

Ops need to be involved in development planning processPuppet modules and manifests should be selected/built aspart of the development process

DEVELOP ON PUPPETPROVISIONED

ENVIRONMENTSAs early as possible, all dev should be done on systemsbuilt from PuppetPuppet manifests get tested as part of the developmentprocess

VAGRANTBuilds virtual machines, optionally from Puppet manifestsMakes it easy to spin up short-lived dev instancesQuick to get working

Avoid ops being a blocker for dev

A WORKFLOWDevelopment happens on Vagrant VM(s)Deployment to all shared environments happens viaJenkins

PUPPET CONFIGThere should be only one set of Puppetmanifests/modules

Tested deployed and merged through software testenvironments

ONE SET OF MANIFESTS,MANY ENVIRONMENTS

Different environments need different configResource locationsSettings

DEALING WITH DIFFERENTENVIRONMENTS

HieraRemoves the need for ugly if/else blocksPut anything that differers by environment in a separatefileCan encrypt with hiera-gpg if data sensitive

HIERA.YAML:hierarchy: - %{environment} - common

VAGRANTFILEVagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "centos64-lxc" config.vm.hostname = "puppetconf-example" config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests" puppet.manifest_file = "site.pp" puppet.module_path = "puppet/modules" puppet.hiera_config_path = 'puppet/hiera.yaml' puppet.options => ["--environment", "localdev"] endend

TO DEVELOP:Start of the day, dev runs vagrant up and gets the latestenvironmentCode/objects sit in a shared vagrant volumeEnd of the day, or when new Puppet manifests/modulesare available, vagrant destroy is run

VAGRANT PROVISIONERSAvoid VirtualBox wherever possibleSlow, prone to taking down host machineOn Linux, vagrant-lxc is speedyVMWare Fusion for non-free fruit-based Unix

VAGRANT AND AWSUse Vagrant to bring up machines in AWS usingvagrant-aws plugin

Makes it easy to share work in progressMeans VirtualBox doesn't crash your laptopHas cost implications

QA/STAGINGENVIRONMENTS IN AWS

Merge to appropriate branch in gitJenkins takes over

ADVANTAGE OF AWSGreat thing about AWS - we don't need to run our testenvironments all the timeHave the environments only when you need them

TESTING VS LIVEUse the money saved to build better environmentsMinimise differences between testing and liveIn particular, test on environments with relevant HA asearly as possible

SPEEDING UP THE PROCESSSome resources, in particular DBs can be slow to provision(30 mins plus)Could just run 24/7One approach: pilot light provisioning

PILOT LIGHT PROVISIONINGTiers built using autoscaling groupsMinimum instance count is 0Jenkins sets desired capacity appropriately on deployReset to 0 via a recurring scheduled operation on ASGand/or Jenkins job

CONCLUSIONSInfrastructure development should run in parallel tosoftware dev

This means devs + ops must co-operate

Minimise differences from production at all stagesIf a dev can't see the problem in their environment,you're much more likely to get woken up by it

QUESTIONS? COMMENTS?Sam Bashton

sam@bashton.com

Twitter: @bashtoni

(Psst.. )http://www.bashton.com/jobs/

REFERENCES, LINKSVagrantvagrant-lxchiera-gpgMasterless Puppet + AWS