deploying infrastructure with opscode chef

INFRASTRUCTURE AS CODEManaging System Complexity with Chef

Robert J. Berger - CTO Runa, [email protected] http://blog.ibd.com

1Friday, April 22, 2011Image: http://techcrunch.com/2010/03/16/big-data-freedom/

mailto:[email protected]

mailto:[email protected]

MOORE’S LAW

2Friday, April 22, 2011Graph: http://www.gotw.ca/publications/concurrency-ddj.htm

MOORE’S LAWHITS ASPEED BUMPCPU speed

growthhas

STOPPED

2Friday, April 22, 2011Graph: http://www.gotw.ca/publications/concurrency-ddj.htm

BUT WE GET: GROWTH OF CORES &

FOOTBALL FIELDS OF SERVERS3Friday, April 22, 2011

Data Center Image: http://www.datacenterknowledge.com/inside-microsofts-dublin-mega-data-center/Chart: http://news.cnet.com/8301-13924_3-10101987-64.html

VON NEUMANN SEQUENTIAL TRANSFORMS TO...

4Friday, April 22, 2011Image: http://www.necessarysufficient.com

ACTORS,

5Friday, April 22, 2011Carl Hewitt Image: http://people.csail.mit.edu/psz/LCS-75/languages.htmlMap Reduce Diagram: http://horicky.blogspot.com/2008/11/hadoop-mapreduce-implementation.htmlNanites: http://www.startrek.com/database_article/nanitesRuna Infrastructure Diagram: Robert J. Berger

AnalyticsReporting

Monitor & Recovery

Data Collectors

Hadoop / HBaseMap / ReducePetabyte Store

SharedSessionMemory

HTTPDispatchers

RedisMem

CacheRedisMem

CacheRedisMem

CacheRedisMem

CacheRedisMem

Cache

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

HBaseHBase

Cheshire / Swarmiji

Dynamic Runtime

Queue

PARALLELISM!ACTORS, MAP/REDUCE,NANITE...

5Friday, April 22, 2011Carl Hewitt Image: http://people.csail.mit.edu/psz/LCS-75/languages.htmlMap Reduce Diagram: http://horicky.blogspot.com/2008/11/hadoop-mapreduce-implementation.htmlNanites: http://www.startrek.com/database_article/nanitesRuna Infrastructure Diagram: Robert J. Berger

COMPLEXITY EXPLOSION

6Friday, April 22, 2011Complexity Background: http://www.c0d3m0nk3y.com/gallery/

COMPLEXITY EXPLOSION

Not just in the LAMP stack

Lots of moving heterogeneous parts

Horizontal scalingAuto scaling

Monitoring & Management

Clouds

6Friday, April 22, 2011Complexity Background: http://www.c0d3m0nk3y.com/gallery/

CODE YOUR INFRASTRUCTURE

7Friday, April 22, 2011Background: http://www.flickr.com/photos/jason-samfield/4398151919/

CODE YOUR INFRASTRUCTURE

Programming is all about managing complexity

Apply same techniques to complexity of infrastructure

Revision control

Keep it DRY

Build on the shoulders of others

7Friday, April 22, 2011Background: http://www.flickr.com/photos/jason-samfield/4398151919/

CHEF: THE RUBY WAY TO MANAGE INFRASTRUCTUREPlain Old Ruby programming

External DSL adds language constructs for service deployment

Services to support Configuration Management

Community Cookbooks

8Friday, April 22, 2011Background Image: http://www.flickr.com/photos/steveritchie/3779779586/

CHEF: THE RUBY WAY TO MANAGE INFRASTRUCTUREPlain Old Ruby programming

External DSL adds language constructs for service deployment

Services to support Configuration Management

Community CookbooksENABLES YOU TO CODE YOUR INFRASTRUCTURE

8Friday, April 22, 2011Background Image: http://www.flickr.com/photos/steveritchie/3779779586/

WHAT’S IN THE KITCHEN

Chef-C

lient

Chef-C

lient

Chef-C

lient

PowerBook G4

Chef Server

Your Dev Environment

Deployed Infrastructure

AuthoritativeData bagsRolesCookbooksNode states

Local Project Repo of CookbooksDo your cooking (editing)

Chef Knife clientPush Data Bags, Roles, CookbooksLaunch/Bootstrap instances

Remote Repo

9Friday, April 22, 2011

CHEF SERVER

• Manage & Authenticate clients (nodes, dev, admin)

• Assign roles & Configurations to nodes

• Track states of Nodes

• Support search on Data Bags, Roles and Node states.

• Serve a RESTful API for humans and machines

• Can run your own or use Opscode Platform


PowerBook G4

PROJECT REPO

Data Bags-App Level Attributes

Roles-A Nodes Purpose

Environments-Staging, Production-New in 0.10.x

Cookbooks-"Standards" from cookbooks.Opscode.com

Site-Cookbooks-Your own cookbooks-Over-rides of Standard Cookbooks


COOKBOOKS

Attributes-Key-Value pairs associated with a node-Hierarchy of overridesFiles-Static Files to be copied to node

Library-Place to DRY up Ruby code used in Recipes

Recipes-Where the Action is-Specifies resources that the chef client should take action on

Templates-ERB files to build config files on node

Providers / Resources-DIY DSL Extensions

12Friday, April 22, 2011You only need Recipes. Most cookbooks are just Recipe[s] and TemplatesBackground: Clipart Library

PLETHORA OF EXISTING COOKBOOKS

13Friday, April 22, 2011These are the ones that Opscode has in their “official” repo. I always search on Github for others before I write one

activemq erlang maven pxe_dust stompserverant fail2ban memcached python subversionapache2 gems mercurial quick_start sudoapparmor git munin rabbitmq teamspeakapplication glassfish mysql rabbitmq_chef teamspeak3apt gnu_parallel nagios radiant thriftaws god nanite rails tmuxbluepill gunicorn nginx rails_enterprise tomcatboost hadoop nscd redmine tomcat6build-essential haproxy ntp reprepro traccapistrano heartbeat ohai resolver transmissionchef imagemagick one-shot riak ubuntuchef-client instiki openldap rsync ucspi-tcpcloudkick iptables openssh rsyslog unicorncouchdb java openssl ruby userscron java_sun openvpn ruby_enterprise varnishdaemontools jetty ossec rubygems vimdatabase jira packages runit wordpressdjango jpackage pacman rush xfsdjbdns keepalived passenger_apache2 samba xmldmg kickstart passenger_enterprise sbuild zenossdrbd logrotate pdns screen zlibdynect logwatch perl snort zshdynomite lvm php solrec2 man postfix sqliteemacs maradns postgresql ssh_known_hosts

PLETHORA OF EXISTING COOKBOOKS

13Friday, April 22, 2011These are the ones that Opscode has in their “official” repo. I always search on Github for others before I write one

MOST IMPORTANT WIKI PAGEWIKI.OPSCODE.COM/DISPLAY/CHEF/RESOURCES

14Friday, April 22, 2011Whole range of Directives. Everything from copying files from file systems, URLs, etcto deploy from Git/SVN... Similar to Capistrano. But its pull instead of push

MOST IMPORTANT WIKI PAGEWIKI.OPSCODE.COM/DISPLAY/CHEF/RESOURCES

Chef DSL

Directives

14Friday, April 22, 2011Whole range of Directives. Everything from copying files from file systems, URLs, etcto deploy from Git/SVN... Similar to Capistrano. But its pull instead of push

A SIMPLE COOKBOOK: RABBITMQ


package "rabbitmq-server" do action :installend

service "rabbitmq-server" do supports :status => true, :restart => true, :reload => true action [ :enable, :start ]end

RECIPES: default.rb




Install Packages (Knows about many package formats)

RECIPES: default.rb




Defines a service that other recipes can trigger

(Knows about many init style mechanism)

RECIPES: default.rb


case node[:platform]when "ubuntu" template "/etc/rabbitmq/rabbitmq.conf" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") endelse template "/etc/rabbitmq/rabbitmq.config" do source "rabbitmq.config.erb" owner "root" group "root" mode 0644 notifies :restart, resources(:service => "rabbitmq-server") endend

RECIPES: default.rb pt 2


ERB: Embedded Ruby


Tweak based on target OS(Its just Ruby!)



ERB: Embedded Ruby



Chefisms tend to be blocks with optional parameters


ERB: Embedded Ruby



Creates files using ERB(ERB Params could be set here)


ERB: Embedded Ruby



Using the service defined earlier


ERB: Embedded Ruby

ATTRIBUTES: default.rb

default[:rabbitmq][:nodename] = "rabbit"default[:rabbitmq][:address] = "0.0.0.0"default[:rabbitmq][:port] = "5672"default[:rabbitmq][:erl_args] = "+K true +A 30 \-kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] \-kernel inet_default_connect_options [{nodelay,true}]"default[:rabbitmq][:start_args] = ""default[:rabbitmq][:logdir] = "/var/log/rabbitmq"default[:rabbitmq][:mnesiadir] = "/var/lib/rabbitmq/mnesia"default[:rabbitmq][:cluster] = "no"default[:rabbitmq][:cluster_config] = "/etc/rabbitmq/rabbitmq_cluster.config"default[:rabbitmq][:cluster_disk_nodes] = []




Sets the default value




Precedence-default-set-override-File type




AutomaticOhai discovers OS & HW

instance attributes at runtime


Templates:rabbitmq.config.erb

NODENAME=<%= node[:rabbitmq][:nodename] %>NODE_IP_ADDRESS=<%= node[:rabbitmq][:address] %>NODE_PORT=<%= node[:rabbitmq][:port] %>SERVER_ERL_ARGS=<%= node[:rabbitmq][:erl_args] %>CLUSTER_CONFIG_FILE=<%= node[:rabbitmq][:cluster_config] %>LOG_BASE=<%= node[:rabbitmq][:logdir] %>MNESIA_BASE=<%= node[:rabbitmq][:mnesiadir] %>SERVER_START_ARGS=<%= node[:rabbitmq][:start_args] %>


Creates configuration files using attributes




node object created when chef-client runs(attribute precedence resolved)




NODENAME=rabbitNODE_IP_ADDRESS=0.0.0.0NODE_PORT=5672SERVER_ERL_ARGS=+K true +A 30 -kernel inet_default_listen_options [{nodelay,true},{sndbuf,16384},{recbuf,4096}] -kernel inet_default_connect_options [{nodelay,true}]CLUSTER_CONFIG_FILE=/etc/rabbitmq/rabbitmq_cluster.configLOG_BASE=/var/log/rabbitmqMNESIA_BASE=/var/lib/rabbitmq/mnesiaSERVER_START_ARGS=


Final File is rendered and copied to location specified by the Template Directive


ROLE: rabbitmq.rbname "rabbitmq"description "Deploy rabbitmq instance"recipes "ulimits", "rabbitmq::default"override_attributes({ :rabbitmq => { :port => "8888" }, "ulimits_list" => [ { :domain => "rabbitmq", :type => "soft", :item => "nofile", :value => 32768 } ]})


Specify recipes needed

ROLE: rabbitmq.rbname "rabbitmq"description "Deploy rabbitmq instance"recipes "ulimits", "rabbitmq::default"override_attributes({ :rabbitmq => { :port => "8888" }, "ulimits_list" => [ { :domain => "rabbitmq", :type => "soft", :item => "nofile", :value => 32768 } ]})


ROLE: rabbitmq.rb

Attribute Overrides

name "rabbitmq"description "Deploy rabbitmq instance"recipes "ulimits", "rabbitmq::default"override_attributes({ :rabbitmq => { :port => "8888" }, "ulimits_list" => [ { :domain => "rabbitmq", :type => "soft", :item => "nofile", :value => 32768 } ]})


A TASTE OF A SPICIER COOKBOOK:

APPLICATION/DATABASE

21Friday, April 22, 2011Just touch on the most powerful Cookbook pair

Can use the same scafolding for many applicationsImage: http://www.theepochtimes.com/n2/content/view/46924/

DATA BAG DRIVEN:MAP RECIPES TO ROLES IN BAG

{ "id": "my_app", "server_roles": [ "my_app" ], "type": { "my_app": [ "rails", "postfix", "unicorn" ] }, ...

22Friday, April 22, 2011Snippet of a JSON representation of a Data Bag



Application specific role(s), typically the name of the app




Recipes in applicationcookbook to run for this role


DEFAULT RECIPE:

search(:apps) do |app| (app["server_roles"] & node.run_list.roles).each do |app_role| app["type"][app_role].each do |thing| node.run_state[:current_app] = app include_recipe "application::#{thing}" end endend

node.run_state.delete(:current_app)

23Friday, April 22, 2011This happens when chef-client runs on an instance and node has all resolved attributes

Node may have one role or many roles.

For instance having the web server, rails app and db roles on one server.

Or have a each role on their own server.

Can even have one role on many servers like the rails app servers for horizontal scaling.

DEFAULT RECIPE:



ITS KIND OF META






DEFAULT RECIPE:



Search thru all ‘apps’ databagEXECUTES ON EACH NODE BY CHEF-CLIENT






DEFAULT RECIPE:



Loop thru roles thatmatch running node’s run_list

EXECUTES ON EACH NODE BY CHEF-CLIENT






DEFAULT RECIPE:



Loop thru & run recipesfor each role

railspostfix

nginxunicorn

EXECUTES ON EACH NODE BY CHEF-CLIENT






CHEF-CLIENT(RUNTIME)


CHEF-CLIENT(RUNTIME)

• All work done by client on instances (not server)

• Runs on each instance regularly (every 1/2 by default)

• Roles, Cookbook, Data Bag, Attribute changes propagate from Chef Server to instances via the chef-client

• Client sends attributes to Chef Server making them searchable


PowerBook G4

THE AMAZING KNIFECLI for Chef Server


PowerBook G4

THE AMAZING KNIFECLI for Chef Server

Manage Cookbooks-Create-Upload-Syntax Check-Generate Metadata-Access Public Cookbooks-Git Vendor Pulls-Publish

Manage Data BagsNode Mgmt & StatusRole ManagementClient ManagementSearch-Nodes, Roles, Data Bags


PowerBook G4


PowerBook G4

AND THAT’S NOT ALL!

Manage Knife ConfigMulti-ssh-Select by Search

Bootstrap Servers-From Bare Basic OS

Create, List, Delete Cloud Instances- EC2- Rackspace- Slicehost- Terremark


PUSH YOUR COOKBOOKSTO THE CHEF SERVER


PUSH YOUR COOKBOOKSTO THE CHEF SERVER

knife cookbook upload -a

Pushes all cookbooks in your repo to chef server


PUSH YOUR ROLESTO THE CHEF SERVER


PUSH YOUR ROLESTO THE CHEF SERVER

knife role from file roles/rabbitmq.rb

Pushes the rabbitmq role to the chef server


LAUNCH RABBITMQ ON EC2

knife ec2 server create \'role[production]' 'role[base]' role[rabbitmq]' \-S my-aws -I ~/.ssh/my-aws.pem -x ubuntu \-G production,rabbitmq -i ami-88f504e1 -f m1.large


LAUNCH RABBITMQ ON EC2Uses Fog to issue EC2 api calls to configure

and create an instance with these parameters




“Run List” specifies roles for this instance




“Run List” specifies roles for this instance


Works with other Clouds:Rackspace, Terramark, Slicehost, Bluebox


LIKE BEING AT BENIHANA’S

31Friday, April 22, 2011Image: http://ocdeals.ocregister.com/2010/11/01/30-on-your-birthday-at-benihana/73372/benihana/

LIKE BEING AT BENIHANA’SLaunches ec2 instance[s] with AMIs you specifiedInstalls Ruby & packages for building RubyGemsInstalls RubyGems from sourceInstalls Chef from RubyGemsCreates the /etc/chef directoryWrites validation cert to /etc/chef/validation.pemWrites an /etc/chef/client.rb config fileWrites a JSON file, /etc/chef/first-boot.json with roles and recipes specified as the run listExecutes chef-client with the first-boot.json, connecting to the chef server in the client.rb

31Friday, April 22, 2011Image: http://ocdeals.ocregister.com/2010/11/01/30-on-your-birthday-at-benihana/73372/benihana/

KEEP YOUR INFRASTRUCTURE UNDER CONTROL WITH CHEF!

32Friday, April 22, 2011Image: http://forums.brickarms.com/yaf_postst18328_The-Evil-Chefs-Are-Back-and-With-Even-More-Weapons.aspx

deploying infrastructure with opscode chef

Technology

h http

complexity background

background image

infrastructure diagram

data center image

system complexity

complexity explosionlots

complexity explosionfriday