cloud foundry bosh where the platform rubber meets the infrastructure road - chefconf

Cloud Foundry BOSHWhere the Platform rubber

meets the Infrastructure road

Patrick ChanezonSenior DirectorDeveloper [email protected]://twitter.com/chanezon ChefConf, San Francisco, May 2012

with slides from Martin Englund

Cloud Foundry Engineering

@pmenglund

Wednesday, May 16, 12

mailto:[email protected]


http://twitter.com/chanezon

http://twitter.com/chanezon

What is Cloud Computing?

2

Cloud According to my daughter Eliette


Cloud Stack - Classic Pyramid

3

Platform As A Service

Infrastructure As A Service

SoftwareAs A Service


Cloud Stack - Developer Perspective

4

InfrastructureAs A Service

SoftwareAs A Service

Platform As A Service


5

Infrastructure


IaaS/Virtualization getting mainstream

§ AWS, Joyent, Rackspace,...

§ Open Source projects: OpenStack, DeltaCloud, Eucalyptus

§ Automation: Chef, Juju

§ Standardization? DMTF

§ Inside the Firewall, Virtualization: VMware, Microsoft, Xen, KVM

§ 50% of workloads are virtualized

§ Easy to provision, manage instance...BUT

§ Still need to manage backups, software stacks, monitor, upgrades

6


With Infrastructure, you still need to build your own platform

§ Need to build a distributed platform on top of you infrastructure

§ Story of the AWS meltdown from last summer• http://blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html

• http://www.readwriteweb.com/cloud/2010/12/chaos-monkey-how-netflix-uses.php

• http://news.ycombinator.com/item?id=2477296

• http://stu.mp/2011/04/the-cloud-is-not-a-silver-bullet.html

§ Twilio, Smugmug, SimpleGeo survived it because they built their own distributed platform on top of IaaS

§ Enterprise customers want to consider Infrastructure like CDNs• Multi Cloud usage

• Based on Open Source de facto standards, or full standards whenever that happens

7


http://blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html

http://blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html

http://www.readwriteweb.com/cloud/2010/12/chaos-monkey-how-netflix-uses.php




http://news.ycombinator.com/item?id=2477296

http://news.ycombinator.com/item?id=2477296

http://stu.mp/2011/04/the-cloud-is-not-a-silver-bullet.html

http://stu.mp/2011/04/the-cloud-is-not-a-silver-bullet.html

8

Platforms

Let the wild rumpus begin!


Platforms

§ Raise the Unit of currency to be application & services instead of infrastructure

§ Google App Engine, Cloud Foundry, SalesForce Heroku, Cloudbees, Amazon Elastic Beanstalk, Microsoft ex-Azure, AppFog

§ Single or a few languages, services

§ Start of Multi language Polyglot platforms

§ Enabler for Agile Developers -> Create Business value faster

§ Lack of standards: risk, vendor lock-in

§ Enterprise needs:•Control, customizability

•Private/Hybrid Cloud

•Avoid lock-in

9


Main Risk: Lock-In

10

Welcome to the hotel californiaSuch a lovely placeSuch a lovely facePlenty of room at the hotel californiaAny time of year, you can find it here

Last thing I remember, I wasRunning for the doorI had to find the passage backTo the place I was before’relax,’ said the night man,We are programmed to receive.You can checkout any time you like,But you can never leave!


Cloud Foundry: The Open PaaS

11

Clou

d Pr

ovide

r Int

erfa

ce

Application Service Private Clouds

PublicCloud

MicroCloud

Data Services

Other Services

Msg Services

vFabric Postgres

vFabric RabbitMQTM

• Open Source: Apache 2 Licensed

• multi language/frameworks

• multi services

• multi cloud


12

Open Source


Open Source Advantage: moves faster

§ http://code.google.com/p/googleappengine/issues/detail?id=13

13

• https://github.com/cloudfoundry/vcap/pull/25


http://code.google.com/p/googleappengine/issues/detail?id=13

http://code.google.com/p/googleappengine/issues/detail?id=13

https://github.com/cloudfoundry/vcap/pull/25

https://github.com/cloudfoundry/vcap/pull/25

Open Source Advantage: diverse communities

14

§ Juju Charms to setup a multi node Cloud Foundry in 10 minutes

§ See Brian Thomason & Juan Negron’s UDS 2011 session

http://cloud.ubuntu.com/2011/09/from-zero-to-drawbridge-via-ubuntu-server-ensemble-and-cloudfoundry-in-less-than-10-minutes/






Open Source Advantage: more innovation

15


16

CloudFoundry.org


17

Cloud Foundry Open Source Workflow

Reviewers +2/-2

test verification score ✓ ✗ git push triggers test execution

Gerrit Code Review

reviews.cloudfoundry.org

Jenkins CI

ci.cloudfoundry.org

github

github.com/cloudfoundry

+1/-1 Committers

on +2 and ✓ change pushed to github

cloudfoundry.com

production updated ~2x/week


Cloud Foundry is 1 year old

18

§ Services• Caldecott – tunnel into your services, explore with standard client tools

• PostgreSQL, RabbitMQ

§ Frameworks, Runtimes, and Tools• Java and Ruby Auto-Reconfiguration

• Scala, node.JS 0.6.*, Erlang, JRuby, PHP, Python, .NET, Spring 3.1, Grails 2.0, Play 2.0

• Multi-Node Chef based deployment tools

• Maven Plugin, Eclipse Integration

• VMC manifests, Java Debugging, Rails Console

• Standalone applications

§ Micro Cloud Foundry x 3


Cloud Foundry Ecosystem

19


Cloud Foundry Ecosystem

20


Register today

21

http://cloudfoundry.com/signup

ChefConfUse Promo Code

To avoid approval queue waiting time




Logical View

22

VMC client STS plugin Browser(user app access)

Routers

CloudControllers App

Services

App

HealthManager

DEA Pool

Messaging


23

Cloud Foundry BOSH


24

Chef at VMware

§ Chef is used internally in several projects at VMware

§ To manage the Cloud Foundry deployment and Life Cycle, we developed a more specific system: BOSH


25

§ 500 – 5,000 VMs

§ 40+ unique node types

§ 75+ unique software packages

§ 75+ unique environments

§ 2x/week cf.com updates

§ 24x7x365 non-stop operation

§ No-downtime deployments

§ Reliable, robust, repeatable deployments, updates, capacity adjustments

§ Small teams manage many instances

Production Grade Cloud Foundry Clusters

Google style problem è Google style solution

cloudfoundry.com

production, staging, stress, qa, dev


26

§ Cloud Foundry BOSH is an open source tool-chain for release engineering, deployment, and lifecycle management of large scale distributed services• Prescriptive way of creating releases and managing systems and services

• It is not a collection of shell scripts, not a pile of Perl

§ Built to deploy and manage production-class, large scale clusters• Production grade Cloud Foundry clusters: 500+ VMs, 40+ jobs, 75+ packages

• Multi-node, multi-tier, complex clusters: e.g., our Gerrit/Jenkins Cluster

§ Built for devops usage and scale by a crack team of veterans• A project, not a product: command line interface, YAML, etc.

• Continuous improvement, iterative development, rough edges

Cloud Foundry BOSH

github.com/cloudfoundry/bosh


27

§ Background

§ Concepts

§ Demo

§ Workflow

§ Under the hood

BOSH Agenda


Background

§ Built from the need to operate cloudfoundry.com• Over 40 different types of VMs

• Hundreds of running instances of VMs

• Pushing at least 2 changes per week

§ End-to-end management• Release engineering

• Deployment

• Lifecycle management

§ Generic solution• Any service

• Any cloud

• Any operating system

28


29

§ Packages

§ Jobs

§ Releases• http://github.com/cloudfoundry/bosh-sample-release

§ Stemcells

§ Deployments

Concepts


30

§ Source / blobs• Don’t commit large objects in the SCM

§ Dependencies• Compile time dependencies

§ Versioning• Auto-increment

§ Pre-packaging• Preparation for compilation

§ Packaging• Compiling the source

Packages


Packages (nginx)

31

src/nginx-rw-r--r-- 1 martin staff 25490 Apr 11 19:17 headers-more-v0.15rc1.tgz-rw-r--r-- 1 martin staff 691501 Apr 11 19:17 nginx-1.0.11.tar.gz-rw-r--r-- 1 martin staff 1556329 Apr 11 19:17 pcre-8.21.tar.gz


Packages (nginx)

32

packages/nginx/spec---name: nginxfiles: - nginx/headers-more-v0.15rc1.tgz - nginx/nginx-1.0.11.tar.gz - nginx/pcre-8.21.tar.gz


Packages (nginx)

33

packages/nginx/packaging# abort script on any command that exit with a non zero valueset -e

echo "Extracting pcre..."tar xzf nginx/pcre-8.21.tar.gz

echo "Extracting headers-more module..."tar xzf nginx/headers-more-v0.15rc1.tgz

echo "Extracting nginx..."tar xzf nginx/nginx-1.0.11.tar.gz

echo "Building nginx..."cd nginx-1.0.11./configure \ --prefix=${BOSH_INSTALL_TARGET} \ --with-pcre=../pcre-8.21 \ --with-http_ssl_module \ --add-module=../headers-more-v0.15rc1makemake install


34

§ Packages• Runtime dependencies

§ Templates• Converting a generalized form to an instance specific file

§ Versioned• Auto increments

§ Supervision• Through monit

§ Monitoring• Reported to the health monitor

§ Lifecycle hooks• E.g. drain workload from job

Jobs


Jobs (nginx)

35

jobs/nginx/spec---name: nginx

templates: nginx_ctl: bin/nginx_ctl nginx.conf.erb: config/nginx.conf mime.types: config/mime.types

packages: - nginx


Jobs (nginx)

36

jobs/nginx/monitcheck process nginx with pidfile /var/vcap/sys/run/nginx/nginx.pid start program "/var/vcap/jobs/nginx/bin/nginx_ctl start" stop program "/var/vcap/jobs/nginx/bin/nginx_ctl stop" group vcap


Jobs (nginx)

37

jobs/nginx/templates/nginx.conf.erbuser root root;worker_processes <%= properties.nginx.workers || 1 %>;

error_log /var/vcap/sys/log/nginx/nginx.error.log;pid /var/vcap/sys/run/nginx/nginx.pid;

events { worker_connections 8192; use epoll;}... upstream wordpress { <% properties.wordpress.servers.each do |server| %> server <%= server %>:<%= properties.wordpress.port %>; <% end %> }

server { listen 80; server_name "<%= properties.wordpress.servername %>"; server_name_in_redirect off;...


38

§ Packages

§ Jobs

§ Versioned

§ Self contained

§ Incremental

§ Published

Releases

» bosh create release» bosh upload release» bosh delete release» bosh releases


39

§ VM template

§ BOSH Agent

§ Versioned

§ Built using vmbuilder

§ IaaS Plugin

Stemcells

» bosh upload stemcell» bosh delete stemcell» bosh stemcells


40

§ Release

§ Networks

§ Resource pools

§ Compilation workers

§ Jobs

§ Update concurrency

§ Properties

§ Cloud properties

Deployments


Deployment (wordpress)

41

wordpress.yml---name: bosh-wordpressrelease: name: wordpress version: 3

compilation: workers: 4 network: default cloud_properties: ram: 2048 disk: 8096 cpu: 2

update: canaries: 1 canary_watch_time: 60000 update_watch_time: 30000 max_in_flight: 4 max_errors: 1



42

wordpress.yml (continued)networks: - name: default subnets: - reserved: - 172.31.220.2 - 172.31.220.10 static: - 172.31.220.11 - 172.31.220.100 range: 172.31.220.0/22 gateway: 172.31.220.1 dns: - 172.30.22.153 cloud_properties: name: VLAN2220 - name: dmz subnets: - static: - 172.28.5.209 - 172.28.5.210 range: 172.28.5.208/28 dns: - 172.22.22.153 cloud_properties: name: VLAN3093



43

wordpress.yml (continued)resource_pools: - name: infrastructure network: default size: 6 stemcell: name: bosh-stemcell version: 0.4.6 cloud_properties: cpu: 1 disk: 8192 ram: 4096



44

wordpress.yml (continued)jobs: - name: mysql template: mysql instances: 1 resource_pool: infrastructure persistent_disk: 16384 networks: - name: default static_ips: - 172.31.220.20

- name: wordpress template: wordpress instances: 4 resource_pool: infrastructure networks: - name: default static_ips: - 172.31.220.30 - 172.31.220.33



45

wordpress.yml (continued) - name: nginx template: nginx instances: 1 resource_pool: infrastructure networks: - name: default default: [dns, gateway] static_ips: - 172.31.220.40 - name: dmz static_ips: - 172.28.5.209



46

wordpress.yml (continued)properties: wordpress: admin: [email protected] port: 8008 servers: - 172.31.220.30 - 172.31.220.31 - 172.31.220.32 - 172.31.220.33 servername: wordpress.cloudfoundry.com db: name: wp user: wordpress pass: w0rdpr3ss auth_key: random key secure_auth_key: random key logged_in_key: random key mysql: address: 172.31.220.20 port: 3306 password: rootpass nginx: workers: 1




BOSH Demo

47


Dev Workflow

48

bosh create release

bosh upload release

bosh deployment~/dev.yml

bosh deploy

write code

run tests

git commit

bosh target dev


QA Workflow

49

bosh create release

bosh upload release

bosh deployment~/qa.yml

bosh deploy

git pull

run tests

bosh createrelease --final

bosh target qa

bug reports to dev

git commit


Ops Workflow

50

bosh upload release

bosh deployment~/prod.yml

bosh deploy

git pull

run tests

bosh target prod

bug reports to dev

done!


51

§ Repeatable deployments

§ Predictable rollout

Control


52

§ Dev / Staging / Production

§ BOSH targets

Consistency


53

§ Director / Workers

§ Agent

§ Monitor

§ CLI

§ NATS

§ Redis

§ Database

§ Blobstore

Components


54

BOSH: under the hood

active jobs

diskIaaS CPI

workers

bosh cli

redis nats

stemcellagent

db

blobs

director healthmon

“BOSH is deployed by BOSH”

cloudfoundry.com

BOSH User


55

IaaS neutral by design

CPI: code completefunctional status: “work in progress”

vSphere: battle tested implementation, thousands of deploymentsvCloud Director: “work in progress”, 2H 2012

contribute: github.com/cloudfoundry/bosh

Cloud Provider Interface (CPI)

Cloud Foundry BOSH

github.com/piston/openstack-bosh-cpi


https://github.com/piston/openstack-bosh-cpi




56

Stemcellcreate_stemcell(image, cloud_properties)delete_stemcell(stemcell)

VMcreate_vm(agent_id, stemcell, resource_pool, networks, disk_locality, env)delete_vm(vm)configure_networks(vm, networks)

Diskcreate_disk(size, vm_locality)delete_disk(disk)attach_disk(vm, disk)detach_disk(vm, disk)

Cloud Provider Interface


57

BOSH and Chef?

§ How should they work together?

§ Talk to us on public mailing lists:• http://groups.google.com/a/cloudfoundry.org/group/bosh-users• http://groups.google.com/a/cloudfoundry.org/group/bosh-dev


http://groups.google.com/a/cloudfoundry.org/group/bosh-users




http://groups.google.com/a/cloudfoundry.org/group/bosh-dev








58

§ Mailing lists• http://groups.google.com/a/cloudfoundry.org/group/bosh-users• http://groups.google.com/a/cloudfoundry.org/group/bosh-dev

§ File a Bug• http://cloudfoundry.atlassian.net

§ Browse, Fork & Contribute• https://github.com/cloudfoundry/bosh• gem install gerrit-cli• gerrit clone ssh://reviews.cloudfoundry.org:29418/bosh

Reference / Q&A














http://cloudfoundry.atlassian.net




https://github.com/cloudfoundry/bosh




BOSH Community Resources

@DrNichttp://drnicwilliams.com/2012/04/16/creating-a-bosh-from-scratch-on-aws/

59

@BrianMMcClainhttp://www.brianmmcclain.com/2012/05/08/using-bosh-with-vsphere-part-2/


https://groups.google.com/a/cloudfoundry.org/group/bosh-users/topics



http://drnicwilliams.com/2012/04/16/creating-a-bosh-from-scratch-on-aws/














Acknowledgement

§ Drawings from my daughters Eliette

§ Slides from Mark Lucovsky, Dave McCrory, Derek Collison, Martin Englund

60


cloud foundry bosh where the platform rubber meets the infrastructure road - chefconf

Technology