real world django deployment using chef
DESCRIPTION
TRANSCRIPT
Who am I?
• Django hacker
• Ruby user
• Developer
• Sys admin
Who are you?
• System administrators?
• Designers?
• Developers?
• “Business” People?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
What are we talking about?
http://www.flickr.com/photos/peterkaminski/2174679908/
Agenda
• How’s and Why’s
• Getting Started with Chef
• Python-specific Tools
• Case Study: Packaginator
http://www.flickr.com/photos/koalazymonkey/3590953001/
Infrastructure as Code
A technical domain revolving around building and managing infrastructure programmatically
Enable the reconstruction of the business from
nothing but a source code repository, an application
data backup, and bare metal resources.
Configuration Management
System Integration
http://www.flickr.com/photos/opalsson/3773629074/
n-Tier Infrastructure
Database Master
Load Balancer
App Server App Server
• Provision
• Configure
• Integrate
{{
The Chef Framework
With thanks (and apologies) to Stephen Nelson-Smith
The Chef Framework
• Reasonability
• Flexibility
• Library & Primitives
The Chef Tool(s)
With thanks (and apologies) to Stephen Nelson-Smith
The Chef Tool(s)
• ohai
• chef-client
• chef-server
• knife
The Chef API
With thanks (and apologies) to Stephen Nelson-Smith
The Chef API
• Client/Server
• RESTful API w/ JSON
• Search Service
• Derivative Services
The Chef Community
With thanks (and apologies) to Stephen Nelson-Smith
The Chef Community
• Apache License, Version 2.0
• 400+ Individual contributors
• 90+ Corporate contributors
• Dell, Rackspace,VMware, RightScale, Heroku, and more
• 240+ cookbooks
• http://community.opscode.com/
Chef Enables Infrastructure as Code
• Resources
• Recipes
• Roles
• Source Code
package "haproxy" do action :installend
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
service "haproxy" do supports :restart => true action [:enable, :start]end
Chef Resources
• Have a type.
• Have a name.
• Have parameters.
• Take action to put the resource in the declared state.
• Can send notifications to other resources.
package "haproxy" do action :installend
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
service "haproxy" do supports :restart => true action [:enable, :start]end
Resources take action through Providers
package “haproxy” { yum install haproxyapt-get install haproxypacman sync haproxypkg_add -r haproxy
Chef Providers
Common Resources
package "apache2"
template "/etc/apache2/httpd.conf"
cookbook_file "/etc/apache2/key.pem"
user "www-data"
execute "/etc/init.d/apache2 restart"
Idempotence
• Convergence
• Guard clauses
execute "createdb myapp" do not_if "psql -c \list | grep myapp"end
execute "createdb myapp" do only_if do db.query("SELECT ...").first == 0 endend
Recipes are collections of Resources
Chef Recipes
• Recipes are evaluated for resources in the order they appear.
• Each resource object is added to the Resource Collection.
package "haproxy" do action :installend
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
service "haproxy" do supports :restart => true action [:enable, :start]end
Chef Recipes
• Recipes can include other recipes.
• Included recipes are processed in order.
include_recipe "apache2"include_recipe "apache2::mod_rewrite"include_recipe "apache2::mod_deflate"include_recipe "apache2::mod_headers"include_recipe "apache2::mod_php5"
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
%w{ php5 php5-dev php5-cgi }.each do |pkg|
package pkg do action :install end
end
Chef Recipes
• Extend recipes with Ruby.
• Dynamic configuration through search.
pool_members = search("node", "role:mediawiki")
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]"end
Chef Roles
• Roles describe nodes.
• Roles have a run list.
• Roles can have attributes.
name "mediawiki"description "mediawiki app server"run_list( "recipe[mysql::client]", "recipe[application]", "recipe[mediawiki::status]")
name "mediawiki_load_balancer"description "mediawiki load balancer"run_list( "recipe[haproxy::app_lb]")override_attributes( "haproxy" => { "app_server_role" => "mediawiki" })
Track it like source code...
% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <[email protected]>
Import nagios version 1.0.0
commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <[email protected]>
installation and usage instruction docs
commit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <[email protected]>
Import haproxy version 1.0.1
commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <[email protected]>
add mediawiki cookbook
commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <[email protected]>
multiple environments in data bag for mediawiki
Other Chef Terms
• Cookbooks are collections of recipes
• Environments are pegged cookbook versions
• Servers are Nodes
• Data bags are JSON blobs
Python Cookbook
• python::package, python::source — Install Python
• python::pip, python::virtualenv — Make it dance
• python::default — All of the above!
• python_pip, python_virtualenv
Installing a package
python_virtualenv "/srv/myapp/env" do action :createend
python_pip "django" do action :upgrade version "1.3" virtualenv "/srv/myapp/env"end
Gunicorn Cookbook
• gunicorn::default
• gunicorn_config
Supervisor Cookbook
• supervisor::default
• Debian-style for now
• supervisor_service
Packaginator
Role: base
• Install users
• Configure sudo
• apt-get update
• Install gcc
name "base"description "Base role applied to all nodes."run_list( "recipe[users::sysadmins]", "recipe[sudo]", "recipe[apt]", "recipe[build-essential]")override_attributes( :authorization => { :sudo => { :users => ["ubuntu"], :passwordless => true } })
Recipe: packaginator
• The good stuff
• General parameters
• Sub-resources
application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup" repository "https://github.com/coderanger/packaginator.git" revision "master" migrate true packages ["libpq-dev", "git-core", "mercurial"]
django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "settings.py.erb" debug true database do database "packaginator" engine "postgresql_psycopg2" username "packaginator" password "awesome_password" end database_master_role "packaginator_database_master" collectstatic "build_static --noinput" end
gunicorn do only_if { node['roles'].include? 'packaginator_application_server' } app_module :django port 8080 end
celery do only_if { node['roles'].include? 'packaginator_application_server' } config "celery_settings.py" django true celerybeat true celerycam true broker do transport "redis" end end
nginx_load_balancer do only_if { node['roles'].include? 'packaginator_load_balancer' } application_port 8080 static_files "/site_media" => "site_media" end
end
Application Resource
• Who are we?
• Where are we going?
application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup"
Application Resource
directory "/srv/packaginator"
directory "/srv/packaginator/shared"
deploy_revision "packaginator" do deploy_to "/srv/packaginator" owner "nobody" group "nogroup"
Application Resource
• Source code
• Migration status
• Base packages
repository "https://github.com/..."revision "master"migrate truepackages ["libpq-dev", "git-core","mercurial"]
Django Resource
• Python packages
• pip requirements
• Settings file
• Debug mode
• Static files
django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "settings.py.erb" debug true collectstatic "build_static --noinput"
Django Resource
python_pip "redis"
execute "pip -E ... -r ..."
template "shared/local_settings.py"
execute "manage.py build_static"
Folder Layout
• /srv/myapp
• /srv/myapp/shared
• /srv/myapp/shared/releases
• /srv/myapp/shared/releases/4faedb2ee...
• /srv/myapp/shared/releases/9457be295...
• ...
• /srv/myapp/current
Folder Layout
• /srv/myapp
• ...9457be295/local_settings.py -> ../shared/local_settings.py
• /srv/myapp/current -> .../releases/9457be295...
Django Resource
• Database
• Where?
database do database "packaginator" engine "postgresql_psycopg2" username "packaginator" password "awesome_password"enddatabase_master_role "packaginator_database_master"
Django Resource
• Check local first
• Cloud IP!
if node["roles"].include? ... nodeelse search(:node, "roles:...").firstend
Gunicorn Resource
• Only on frontends
• Use Django mode
• Internal port
gunicorn do only_if do node["roles"].include? ... end app_module :django port 8080end
Gunicorn Resource
• Write out config
• Symlink it in
• Load service
gunicorn_config ...
supervisor_service ...
Celery Resource
• Where?
• Django-mode
• Enable all three
• Broker and results
celery do only_if ... config "celery_settings.py" django true celerybeat true celerycam true broker do transport "redis" endend
Nginx Resource
• Internal port again
• Static map
• URL => path
nginx_load_balancer do only_if ... application_port 8080 static_files({
"/site_media" => "site_media"})
end
https://github.com/coderanger/djangocon2011
