Assembling an Open Source Tool Chain for the Hybrid Cloud

Isaac Christoffersen Bit Herder & Cloud Wrangler

@ichristo

Bit Herder, Cloud Wrangler

●  Geek,  Insomniac,  Tinkerer  

●  Wearer  of  Many  Hats  

●  Delivering  Open  Source  Solutions  for  10+  years  

●  Solution  Architect  at  Vizuri  

Why  the  Hybrid  Cloud?  

Our  App  is  going  to  change  the  world  as  we  know  it!  

“The  best  way  to  predict  the  future  is  to  invent  it”  -­‐-­‐  Ed  Catmull  

Unleash the System Administrators!

source  -­‐  http://commons.wikimedia.org/wiki/  

Word of Mouth Spreads. Popularity Rises.

source  -­‐  http://commons.wikimedia.org/wiki/File:Ford_assembly_line_-­‐_1913.jpg  

More Sys Admins and a Little Automation

SlashDot / Oprah / Twitter Effect ...

We Need More DevOps!

Your App Has Gone Global!

All Hail the Fail Whale!

source  -­‐  http://www.whatisfailwhale.info/  

But We Had DevOps?!

source  -­‐  http://upload.wikimedia.org/wikipedia/commons/e/e0/Nagasakibomb.jpg  

Cloud Bursting?

Assembling  the  Hybrid  Cloud  Tool  Chain  

Automatically create machine images for multiple platforms from a single blueprint

AWS: AMIVMware: VMX + disks VirtualBox: OVF + disks DigitalOcean: Snapshots and many more … http://www.packer.io/docs/templates/builders.html

BoxCutter (github.com/boxcutter) •  Build VirtualBox, VMWare, and Parallels Images •  Choose your desired CM tool (Puppet, Chef,

Ansible)

Bento (https://github.com/chef/bento) •  Used by Chef Release Team for Testing •  Chef specific Configuration Managment

AutomationNo human interaction. Great for Continuous Integration / Deployment

StandardizationUse Puppet, Chef, Ansible, Bash to configure the image

RepeatabilityTemplate goes into version control Image creation knowledge is now in code Anyone can build / rebuild the base images

EXAMPLECentOS Image in both AWS

& Digital Ocean

{ "builders": [ { "type" "access_key" "secret_key" "region" "source_ami" "security_group_id" "instance_type" "ssh_username" "ssh_timeout" "ami_name"

}, …

: "amazon-ebs", : "{{user `aws_access_key`}}", : "{{user `aws_secret_key`}}", : "us-east-1", : "ami-8997afe0", : "sg-8f7e24e4", : "t1.micro", : "ec2-user", : "5m", : "centos-baseline {{timestamp}}"

… {

"type" "api_key" "client_id" "image_id"

: "digitalocean", : "{{user `do_api_key`}}", : "{{user `do_client_id`}}", : "562354",

"snapshot_name" : "centos-baseline {{timestamp}}" } ] …

}

$ packer validate base-image.json Template validated successfully. $ packer build base-image.json amazon-ebs output will be in this color. digitalocean output will be in this color. … ==> amazon-ebs: Deleting temporary keypair... Build 'amazon-ebs' finished. ==> digitalocean: Destroying droplet... ==> digitalocean: Deleting temporary ssh key... Build 'digitalocean' finished. ==> Builds finished. The artifacts of successful builds are: --> digitalocean: A snapshot was created: 'centos-baseline 1396457723' in region 'New York 1'

EXAMPLEBoxCutter Provisioning of

Centos to VMWare

Isaacs-MBP-2:centos-vm isaac$ make list Prepend 'vmware/', 'virtualbox/', or 'parallels/' to build only one target platform: make vmware/centos66 Targets: … centos66-desktop centos66-docker centos66-i386 centos66 centos70-desktop centos70-docker centos70 Isaacs-MBP-2:centos-vm isaac$

isaac$ make vmware/centos70-docker rm -rf output-vmware-iso mkdir -p box/vmware packer build -only=vmware-iso -var 'cm=nocm’ -var 'headless=' -var 'update=' -var 'version=1.0.10' -var 'ssh_username=vagrant' -var 'ssh_password=vagrant' -var 'install_vagrant_key=true' -var "iso_url=http://mirrors.sonic.net/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1503-01.iso" centos70-docker.json vmware-iso output will be in this color. …

… ==> vmware-iso: Downloading or copying ISO vmware-iso: Downloading or copying: http://mirrors.sonic.net/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1503-01.iso ==> vmware-iso: Creating floppy disk... vmware-iso: Copying: floppy/vmware9.compat_dcache.h.patch ==> vmware-iso: Creating virtual machine disk ==> vmware-iso: Building and writing VMX file ==> vmware-iso: Starting HTTP server on port 8597 ==> vmware-iso: Starting virtual machine... ==> vmware-iso: Waiting 10s for boot... ==> vmware-iso: Connecting to VM via VNC ==> vmware-iso: Typing the boot command over VNC... ==> vmware-iso: Waiting for SSH to become available.. …

+

Mature, stable, proven. Development since Jan 2010. Used by thousands of companies.

Deploy to Multiple Providersvagrant up --provider=foo

AWS, DigitalOcean, HP Cloud, Joyent, KVM, libvirt, lxc, OpenStack, Rackspace, Vmware, VirtualBox

Leverage your own Packer Images

•  Base image as starting point. •  Integrates with multiple

“provisioners” including Puppet, Chef, Ansible, Bash

AutomationNo human interaction. Great for Continuous Integration / Deployment

StandardizationUse Puppet, Chef, Ansible, Bash to configure the image

RepeatabilityTemplate goes into version control Image creation knowledge is now in code Anyone can build / rebuild the base images

Vagrant.configure("2") do |config| config.vm.box = "dummy" config.vm.box_url = "https://github.com/mitchellh/

vagrant-aws/raw/master/dummy.box"

config.vm.provider :aws do |aws, override| aws.access_key_id = "YOUR KEY" aws.secret_access_key = "YOUR SECRET KEY" aws.keypair_name = "KEYPAIR NAME” aws.ami

= ”ami-9baa9cf2” override.ssh.username = “ec2-user” override.ssh.forward_agent = true override.ssh.private_key_path = “YOUR PRIVATE KEY”

end

EXAMPLEMultiple CentOS Images

created in AWS from base AMI

$ vagrant up --provider=aws

Use `vagrant plugin` commands to manage plugins. This warning will be removed in the next version of Vagrant.

Bringing machine 'test-broker' up with 'aws' provider... Bringing machine 'test-node-01' up with 'aws' provider... Bringing machine 'test-node-02' up with 'aws' provider…

•  Configuration Management tool like Puppet, Chef, CFEngine

•  Quick to get started •  Builds on familiar tools •  Run commands over SSH. No

additional agents required

… config.vm.provision :ansible do |ansible|

ansible.sudo = true ansible.playbook = "provisioning/ansible/playbook.yml” ansible.verbose = true

end

- hosts: all tasks: - name: ensure ntpd is at the latest version yum: pkg=ntp state=latest notify: - restart ntpd

handlers: - name: restart ntpd service: name=ntpd state=restarted

EXAMPLEProvision a CentOS LAMP

Stack in AWS

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| boxes.each do |box| config.vm.define box[:name], primary: box[:primary] do |config|

config.vm.box = "aws-centos" config.vm.box_url =

https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box …

config.vm.provision :shell, :privileged => false, :inline => "sudo yum -y install screen"

config.vm.provision :ansible do |ansible|

ansible.sudo ansible.playbook

lamp.yml" ansible.verbose

end …

= true = "provisioning/ansible/playbook-

= true

- name : Install LAMP Stack user: ec2-user hosts: all tasks: - name: Install mysql yum: name=mysql-server state=latest

- name: install httpd yum: name=httpd

-  name: Install php for mysql yum: name=$item with_items: - php - php-mysql - mysql-server

Only  Scratching  the  Surface  

So many categories, so little time …

● Cloud Management (Scalr, ManageIQ) ● Log Aggregation (ELK) ● Monitoring Tools (nagios, zenoss) ● Version Control (Git, Subversion) ● CI Servers (Jenkins, TravisCI) ● … and many more

OSS DevOps Tools Ecosystem

Check  Out  -­‐  http://www.devopsbookmarks.com/  

Thank You

Isaac Christoffersen www.vizuri.com ichristoffersen@vizuri.com @ichristo

vagrantup.com packer.io ansible.com