real world django deployment using chef

Real-world Django Deployment with Chef

DjangoCon 2011

Noah Kantrowitznoah@opscode.com

@kantrn

Who am I?

• Django hacker

• Ruby user

• Developer

• Sys admin

Who are you?

• System administrators?

• Designers?

• Developers?

• “Business” People?

http://www.flickr.com/photos/timyates/2854357446/sizes/l/

What are we talking about?

http://www.flickr.com/photos/peterkaminski/2174679908/

Agenda

• How’s and Why’s

• Getting Started with Chef

• Python-specific Tools

• Case Study: Packaginator

http://www.flickr.com/photos/koalazymonkey/3590953001/

Infrastructure as Code

A technical domain revolving around building and managing infrastructure programmatically

Enable the reconstruction of the business from

nothing but a source code repository, an application

data backup, and bare metal resources.

Configuration Management

System Integration

http://www.flickr.com/photos/opalsson/3773629074/

n-Tier Infrastructure

Database Master

Load Balancer

App Server App Server

• Provision

• Configure

• Integrate

{{

The Chef Framework

With thanks (and apologies) to Stephen Nelson-Smith

The Chef Framework

• Reasonability

• Flexibility

• Library & Primitives

The Chef Tool(s)

With thanks (and apologies) to Stephen Nelson-Smith

The Chef Tool(s)

• ohai

• chef-client

• chef-server

• knife

The Chef API

With thanks (and apologies) to Stephen Nelson-Smith

The Chef API

• Client/Server

• RESTful API w/ JSON

• Search Service

• Derivative Services

The Chef Community

With thanks (and apologies) to Stephen Nelson-Smith

The Chef Community

• Apache License, Version 2.0

• 400+ Individual contributors

• 90+ Corporate contributors

• Dell, Rackspace,VMware, RightScale, Heroku, and more

• 240+ cookbooks

• http://community.opscode.com/

Chef Enables Infrastructure as Code

• Resources

• Recipes

• Roles

• Source Code

package "haproxy" do action :installend

template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end

service "haproxy" do supports :restart => true action [:enable, :start]end

Chef Resources

• Have a type.

• Have a name.

• Have parameters.

• Take action to put the resource in the declared state.

• Can send notifications to other resources.

package "haproxy" do action :installend

template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end

service "haproxy" do supports :restart => true action [:enable, :start]end

Resources take action through Providers

package “haproxy” { yum install haproxyapt-get install haproxypacman sync haproxypkg_add -r haproxy

Chef Providers

Common Resources

package "apache2"

template "/etc/apache2/httpd.conf"

cookbook_file "/etc/apache2/key.pem"

user "www-data"

execute "/etc/init.d/apache2 restart"

Idempotence

• Convergence

• Guard clauses

execute "createdb myapp" do not_if "psql -c \list | grep myapp"end

execute "createdb myapp" do only_if do db.query("SELECT ...").first == 0 endend

Recipes are collections of Resources

Chef Recipes

• Recipes are evaluated for resources in the order they appear.

• Each resource object is added to the Resource Collection.

package "haproxy" do action :installend

template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end

service "haproxy" do supports :restart => true action [:enable, :start]end

Chef Recipes

• Recipes can include other recipes.

• Included recipes are processed in order.

include_recipe "apache2"include_recipe "apache2::mod_rewrite"include_recipe "apache2::mod_deflate"include_recipe "apache2::mod_headers"include_recipe "apache2::mod_php5"

template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end

%w{ php5 php5-dev php5-cgi }.each do |pkg|

package pkg do action :install end

end

Chef Recipes

• Extend recipes with Ruby.

• Dynamic configuration through search.

pool_members = search("node", "role:mediawiki")

template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]"end

Chef Roles

• Roles describe nodes.

• Roles have a run list.

• Roles can have attributes.

name "mediawiki"description "mediawiki app server"run_list( "recipe[mysql::client]", "recipe[application]", "recipe[mediawiki::status]")

name "mediawiki_load_balancer"description "mediawiki load balancer"run_list( "recipe[haproxy::app_lb]")override_attributes( "haproxy" => { "app_server_role" => "mediawiki" })

Track it like source code...

% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <joshua@opscode.com>

Import nagios version 1.0.0

commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <joshua@opscode.com>

installation and usage instruction docs

commit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <joshua@opscode.com>

Import haproxy version 1.0.1

commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <joshua@opscode.com>

add mediawiki cookbook

commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <joshua@opscode.com>

multiple environments in data bag for mediawiki

Other Chef Terms

• Cookbooks are collections of recipes

• Environments are pegged cookbook versions

• Servers are Nodes

• Data bags are JSON blobs

Python Cookbook

• python::package, python::source — Install Python

• python::pip, python::virtualenv — Make it dance

• python::default — All of the above!

• python_pip, python_virtualenv

Installing a package

python_virtualenv "/srv/myapp/env" do action :createend

python_pip "django" do action :upgrade version "1.3" virtualenv "/srv/myapp/env"end

Gunicorn Cookbook

• gunicorn::default

• gunicorn_config

Supervisor Cookbook

• supervisor::default

• Debian-style for now

• supervisor_service

Packaginator

Role: base

• Install users

• Configure sudo

• apt-get update

• Install gcc

name "base"description "Base role applied to all nodes."run_list( "recipe[users::sysadmins]", "recipe[sudo]", "recipe[apt]", "recipe[build-essential]")override_attributes( :authorization => { :sudo => { :users => ["ubuntu"], :passwordless => true } })

Recipe: packaginator

• The good stuff

• General parameters

• Sub-resources

application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup" repository "https://github.com/coderanger/packaginator.git" revision "master" migrate true packages ["libpq-dev", "git-core", "mercurial"]

django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "settings.py.erb" debug true database do database "packaginator" engine "postgresql_psycopg2" username "packaginator" password "awesome_password" end database_master_role "packaginator_database_master" collectstatic "build_static --noinput" end

gunicorn do only_if { node['roles'].include? 'packaginator_application_server' } app_module :django port 8080 end

celery do only_if { node['roles'].include? 'packaginator_application_server' } config "celery_settings.py" django true celerybeat true celerycam true broker do transport "redis" end end

nginx_load_balancer do only_if { node['roles'].include? 'packaginator_load_balancer' } application_port 8080 static_files "/site_media" => "site_media" end

end

Application Resource

• Who are we?

• Where are we going?

application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup"

Application Resource

directory "/srv/packaginator"

directory "/srv/packaginator/shared"

deploy_revision "packaginator" do deploy_to "/srv/packaginator" owner "nobody" group "nogroup"

Application Resource

• Source code

• Migration status

• Base packages

repository "https://github.com/..."revision "master"migrate truepackages ["libpq-dev", "git-core","mercurial"]

Django Resource

• Python packages

• pip requirements

• Settings file

• Debug mode

• Static files

django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "settings.py.erb" debug true collectstatic "build_static --noinput"

Django Resource

python_pip "redis"

execute "pip -E ... -r ..."

template "shared/local_settings.py"

execute "manage.py build_static"

Folder Layout

• /srv/myapp

• /srv/myapp/shared

• /srv/myapp/shared/releases

• /srv/myapp/shared/releases/4faedb2ee...

• /srv/myapp/shared/releases/9457be295...

• ...

• /srv/myapp/current

Folder Layout

• /srv/myapp

• ...9457be295/local_settings.py -> ../shared/local_settings.py

• /srv/myapp/current -> .../releases/9457be295...

Django Resource

• Database

• Where?

database do database "packaginator" engine "postgresql_psycopg2" username "packaginator" password "awesome_password"enddatabase_master_role "packaginator_database_master"

Django Resource

• Check local first

• Cloud IP!

if node["roles"].include? ... nodeelse search(:node, "roles:...").firstend

Gunicorn Resource

• Only on frontends

• Use Django mode

• Internal port

gunicorn do only_if do node["roles"].include? ... end app_module :django port 8080end

Gunicorn Resource

• Write out config

• Symlink it in

• Load service

gunicorn_config ...

supervisor_service ...

Celery Resource

• Where?

• Django-mode

• Enable all three

• Broker and results

celery do only_if ... config "celery_settings.py" django true celerybeat true celerycam true broker do transport "redis" endend

Nginx Resource

• Internal port again

• Static map

• URL => path

nginx_load_balancer do only_if ... application_port 8080 static_files({

"/site_media" => "site_media"})

end

https://github.com/coderanger/djangocon2011

Noah Kantrowitz@kantrn

noah@opscode.com