squashing the heisenbugs
DESCRIPTION
Squashing the Heisenbugs (really Bohrbugs) is a talk that I gave at the Chariot CI Event on December 1st, 2010. Feel free to ask me any questions about it via email or twitter. Also, please hire us if you're looking for any devops help.TRANSCRIPT
SQUASHING THE HEISENBUGSTrotter Cashion / Chariot CI Event 2010
http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/
SQUASHING THE HEISENBUGSTrotter Cashion / Chariot CI Event 2010
http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/
BOHRBUGS
SQUASHING THE BOHRBUGSTrotter Cashion / Chariot CI Event 2010
http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/
THIS WILL BE ON THE INTERNET!
http://www.charlesheflin.com/wp-content/uploads/2010/06/fail-whale.jpg
@CASHION
http://gems.github.com/octocat.png
github.com/trotter
AGENDA
HEISENBUGS AND BOHRBUGS
THE MANUAL PROBLEM
CHEF
TYING CHEF INTO CI
QUESTIONS (AQAFT)
HEISENBUGS
http://upload.wikimedia.org/wikipedia/commons/8/84/Hindenburg_burning.jpg
OBSERVER EFFECT
EventYou
http://en.wikipedia.org/wiki/File:Bohr-atom-PAR.svg
BOHRBUGS
MANDELBUGS
SCHRÖDINBUGS
NOOBBUGS
THE MANUAL PROBLEM
http://www.dolphinsc.com/images/Dolphin.jpg
MYSQL
http://images.buycostumes.com/mgen/merchandiser/32494.jpg
POSTGRES
http://www.wallpapers-free.org/34/-/Apache_Longbow_Helicopter/
APACHE
http://www.allbestwallpapers.com/travel-kremlin_and_red_square_fireworks,_moscow,_russia_wallpapers.html
NGINX
Memcached
MemcachedRedis
MemcachedRedis
Mongodb
MemcachedRedis
Mongodb Riak
MemcachedRedis
Mongodb Riak
Cassandra
MemcachedRedis
Mongodb Riak
Cassandra
Project Voldemort
MemcachedRedis
Mongodb Riak
Cassandra
Project Voldemort
Ubuntu
MemcachedRedis
Mongodb Riak
Cassandra
Project Voldemort
Ubuntu
Redhat
MemcachedRedis
Mongodb Riak
Cassandra
Project Voldemort
Ubuntu
RedhatSolaris
MemcachedRedis
Mongodb Riak
Cassandra
Project Voldemort
Ubuntu
RedhatSolaris
iptables
MemcachedRedis
Mongodb Riak
Cassandra
Project Voldemort
Ubuntu
RedhatSolaris
iptables
sshd
MemcachedRedis
Mongodb Riak
Cassandra
Project Voldemort
Ubuntu
RedhatSolaris
iptables
sshd
DNS Servers
MemcachedRedis
Mongodb Riak
Cassandra
Project Voldemort
Ubuntu
RedhatSolaris
iptables
sshd
DNS Servers
TCP/IP internals
We Idolize the Artisan Sysadmin
... but what do we end up with?
http://www.enviro-bc.ca/wp-content/uploads/2009/04/bowlake-aval_clair-israelson.jpg
SNOWFLAKES... that become avalanches
http://upload.wikimedia.org/wikipedia/commons/6/6d/Niels_Bohr.jpg
BOHRBUGS
http://assets.troubledteenswizard.com/uploaded/WoW/nomoney1.jpg
NO MORE MONEY!
We’re really left with two choices
Automate... or die.
Automation is not hard!
BASH
http://hackedgadgets.com/wp-content/2/LEGO_V8_Engine_1.jpg
CFENGINE
http://www.flickr.com/photos/willemvelthoven/381608678/sizes/l/
PUPPET
http://www.flickr.com/photos/venndiagram/4891510333/sizes/l/
CHEF!
http://www.prestonlee.com/wp-content/uploads/2008/09/ruby.png
WRITTEN IN
Two Delicious Flavors
http://upload.wikimedia.org/wikipedia/commons/e/e0/Google%E2%80%99s_First_Production_Server.jpg
SERVER
HOW IT WORKS
HOW IT WORKS
Your Computer
HOW IT WORKS
Your Computer
Box to Provision
HOW IT WORKS
Your Computer
Chef ServerBox to Provision
HOW IT WORKS
Your Computer
Chef ServerBox to Provision
Prepare
HOW IT WORKS
Your Computer
Chef ServerBox to Provision
Prepare
What now?
HOW IT WORKS
Your Computer
Chef ServerBox to Provision
Prepare
What now?
Cookbooks
HOW IT WORKS
Your Computer
Chef ServerBox to Provision
Prepare
What now?
Cookbooks
Run!
http://www.pbs.org/parents/supersisters/han_solo-thumb-454x654.jpg
SOLO
HOW IT WORKS
HOW IT WORKS
Your Computer
HOW IT WORKS
Your Computer
Box to Provision
HOW IT WORKS
Your Computer
Box to Provision
Prepare
HOW IT WORKS
Your Computer
Box to ProvisionCookbooks
Prepare
HOW IT WORKS
Your Computer
Box to ProvisionCookbooks
Prepare
Go!
Chef Structure
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
ROLES ARE REUSABLE!
config/dev.json config/db.json
roles/db.rb
http://www.flickr.com/photos/seandreilinger/959864706/sizes/o/
GETTINGGOING
WHAT YOU NEED
WHAT YOU NEED
Ruby & Rubygems
WHAT YOU NEED
Ruby & Rubygems
`gem install chef` Chef Gem
WHAT YOU NEED
Ruby & Rubygems
`gem install chef` Chef Gem
`git clone http://github.com/opscode/chef-repo.git`
Chef Repo
http://cookbooks.opscode.com/
COOKBOOKS.OPSCODE.COM
TERMINAL JUNKIES
TERMINAL JUNKIES
•spatula search mysql
TERMINAL JUNKIES
•spatula search mysql
•spatula install mysql
TERMINAL JUNKIES
•spatula search mysql
•spatula install mysql
•spatula show mysql
TERMINAL JUNKIES
•spatula search mysql
•spatula install mysql
•spatula show mysql
•... knife has similar functionality
GET THE BOX READY
spatula prepare db-one.host
ssh db-one.host “install ruby”
ssh db-one.host “install chef”
=
GO GO GO!
spatula cook db-one.host db ssh db-one.host “sudo \ chef-solo -c config/solo.rb \ -j config/db.json”
rsync repo to db-one.host
=
WITH CHEF SERVER
install dependencies
turn on client
send cookbooks to client
http://media.photobucket.com/image/minecraft/bluekabal/roma/minesofroma.jpg
DIGGING DEEPER
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
// ./config/db_one.json{ "run_list": ["role[db]"] }
A NODE
Recipes and Roles go here
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
# ./roles/db.rbname "database"description "Be a DB"run_list [ "apache2", "mysql::server" ]
A ROLE
# ./roles/db.rbname "database"description "Be a DB"run_list [ "apache2", "mysql::server" ]
A ROLE
Recipes to run
# ./roles/db.rbname "database"description "Be a DB"run_list [ "apache2", "mysql::server" ]
A ROLE
./cookbooks/apache2/recipes/default.rb
# ./roles/db.rbname "database"description "Be a DB"run_list [ "apache2", "mysql::server" ]
A ROLE
./cookbooks/mysql/recipes/server.rb
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
# ./cookbooks/mysql/attributes/server.rbdefault[:mysql][:server_debian_password] = secure_passworddefault[:mysql][:server_root_password] = secure_passworddefault[:mysql][:server_repl_password] = secure_passworddefault[:mysql][:bind_address] = ipaddressdefault[:mysql][:datadir] = "/var/lib/mysql"
# Tunablesdefault[:mysql][:tunable][:key_buffer] = "250M"default[:mysql][:tunable][:max_connections] = "800"default[:mysql][:tunable][:wait_timeout] = "180"
ATTRIBUTES
// ./config/db.json{ "mysql": { "server_root_password": "beefcake" }, "recipes": [ "mysql" ] }
OVERRIDING ATTRIBUTES
# ./roles/db.rbname "database"description "Be a DB"run_list [ "mysql::server" ]default_attributes :mysql => { :server_root_password =>
"beefcake" }
Node
Role
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
# ./cookbooks/mysql/recipes/server.rbinclude_recipe "mysql::client"
THE RECIPE... PART 1
package "mysql-server" do action :installend
THE RECIPE... PART 2
template "/etc/mysql/my.cnf" do source "my.cnf.erb" owner "root" group "root" mode "0644" notifies :restart, resources(:service => "mysql"), :immediatelyend
THE RECIPE... PART 3
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
user = mysqlpid-file = /var/run/mysqld/mysqld.pidsocket = /var/run/mysqld/mysqld.sockdatadir = <%= @datadir %>bind-address = <%= @node[:mysql][:bind_address] %>
A TEMPLATE SAMPLE
user = mysqlpid-file = /var/run/mysqld/mysqld.pidsocket = /var/run/mysqld/mysqld.sockdatadir = <%= @datadir %>bind-address = <%= @node[:mysql][:bind_address] %>
A TEMPLATE SAMPLE
OMG, ERB!!
user = mysqlpid-file = /var/run/mysqld/mysqld.pidsocket = /var/run/mysqld/mysqld.sockdatadir = <%= @datadir %>bind-address = <%= @node[:mysql][:bind_address] %>
A TEMPLATE SAMPLE
From Attributes
user = mysqlpid-file = /var/run/mysqld/mysqld.pidsocket = /var/run/mysqld/mysqld.sockdatadir = <%= @datadir %>bind-address = <%= @node[:mysql][:bind_address] %>
A TEMPLATE SAMPLE
From Template
template "/etc/mysql/my.cnf" do source "my.cnf.erb" owner "root" group "root" mode "0644" variables :datadir => “/var/lib/mysql”end
ASSIGNING VARIABLES
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
define :apache_conf do dir = node[:apache][:dir] template "#{dir}/mods-available/#{params[:name]}.conf" do source "mods/#{params[:name]}.conf.erb" notifies :restart, resources(:service => "apache2") end end
# In your recipeapache_conf "mod_ssl"
DEFINITIONS
TYING IT INTO CI
USE CHEF TO PROVISION CI
http://images.nationalgeographic.com/wpf/media-live/photos/000/148/custom/heart-ngk0207_14824_470x300.jpg
attribution
CHEF AS PART OF TEST SCRIPT
CONTINUOUS DEPLOYMENT
http://www.flickr.com/photos/andresthor/3946773501/sizes/l/in/photostream/
LET’SWRAPIT UP
http://www.flickr.com/photos/kenfagerdotcom/3409813881/
AUTOMATE EVERYTHING!!
Deploying == Provisioning
http://www.flickr.com/photos/martin_borjesson/4083726805/sizes/o/
USE CHEF!!!!!
ON THE INTERNETS
ON THE INTERNETS
•twitter - @cashion
ON THE INTERNETS
•twitter - @cashion
•github - http://github.com/trotter
ON THE INTERNETS
•twitter - @cashion
•github - http://github.com/trotter
•email - cashion on the gmailz
http://www.flickr.com/photos/dancoulter/21042744/sizes/o/
THANK YOU!
RESOURCES
RESOURCES
•http://wiki.opscode.com/display/chef/Resources
RESOURCES
•http://wiki.opscode.com/display/chef/Resources
•#chef on freenode
RESOURCES
•http://wiki.opscode.com/display/chef/Resources
•#chef on freenode