python deployment with fabric 3401

8/3/2019 Python Deployment With Fabric 3401

1/30

Prodution Architectureand Deployment

with Fabric

- Andy McCurdy -@andymccurdy


2/30

Whiskey Media


3/30

Whiskey Sites


4/30

Your First Django App


5/30

Basic Config (web)

Apachemod_wsgi

use daemon mode

threads more efficientprocesses if you're unsure of thread safety

WSGIDaemonProcess my-site python-path=/home/code/processes=2 threads=150 maximum-requests=5000

WSGIProcessGroup my-site

WSGIScriptAlias / /home/code/my-site/deploy/wsgi/my-site.wsgi


6/30

Basic Config (media)

NginxUse Nginx to proxy traffic to ApacheMeanwhile Nginx serves media

upstream my-site {server 127.0.0.1:8000;

}server {

listen 80;

location ~ ^/media/ {root /home/code/my-site;expires 30d;

}location / {

proxy_pass http://my-site;

proxy_set_header X-Real-IP $remote_addr;}


7/30

Basic Config (db)

DatabasesPostgreSQL

Frank Wiles @ www.revsys.com

MySQLPercona @ www.mysqlperformanceblog.com


8/30

Basic Config (cache)

Use Memcached! Even 16mb willsignificantly help against a Digg orbeing Slashdot'edIt's incredibly easy...

# settings.py

MIDDLEWARE_CLASSES = ('django.middleware.cache.UpdateCacheMiddleware','django.middleware.common.CommonMiddleware','django.middleware.cache.FetchFromCacheMiddleware')

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'CACHE_MIDDLEWARE_SECONDS = 60*5 # 5 minutes


9/30

Basic Deployment

Copy Code & Media (rsync or scp)Run DB migrations / syncdbBounce wsgi daemons


10/30

Managing Growth (db)

Move DB to its own serverAs much RAM as possibleWrite-heavy (>10%)? Get fastdisksTune your config file


11/30

Managing Growth (web)

Add more web serversUse a resource monitoring tool likeMunin to understand if your app isCPU or memory bound


12/30

Even More Growth

Replicated or sharded DatabasesMultiple load balancers forredundancyMessage queuesCrons

Search daemons (Solr, Sphinx)etc...


13/30

(not so) Basic

Deployment

image credit:Brad Fitzpatrick


14/30


15/30

Deployment Options

Capistrano

+ Out of box support for common use cases

+ Hooks to customize tasks+ Source control integration+ Threaded deployment to multiple hosts

- Ruby :(


16/30

Deployment Options

Fabric

+ Very simple, tasks are just Python functions

+ Easy to chain together tasks to create complexscripts out of bite size pieces

- No source control integration- No out of box support- Some bugs, although fairly easy to work around,and new maintainer is working on fixes


17/30

Fabric Basics

sudo easy_install fabricneed a fabfile.pyfrom fabric.api import *

be mindful of tasks that may fail

each remote command starts freshchanging directories


18/30

Core Functionality

local() - Run a command locallyrun() - Run a command remotelysudo() - Run a command remotely asanother user

put() - Copy a file from local to remoteget() - Copy a file from remote to localmany more helper-ish commands


19/30

Authentication

Relies on SSH modelUse SSH keysControl access to root user viasudoersWhen you have to revoke access,

you just turn off their SSH account


20/30

Configuration

Fabric environment (env) -- it'sjust a dictionary

Hosts and Roles

Code RepositoriesWhatever you need

~/fabricrc

Global settings or all Fabric deploymentsSSH username


21/30

Example Config

# fabfile.pyfrom fabric.api import *

env.roledefs = {'web' : ['10.1.1.1', '10.1.1.2'],

'db' : ['10.1.1.3'],

'lb' : ['10.1.1.4'],

}

env.repositories = {...}


22/30

Tasks

# fabfile.pydef uptime():

run('uptime')

$> fab uptime -H 10.1.1.3

[10.1.1.3] run: uptime

[10.1.1.3] out: 05:20:39 up 88 days,

12:00, 0 users, load average: 0.03,0.03, 0.00


23/30

Mapping Roles to Tasks

# fabfile.py

@roles('web')

def uptime():run('uptime')

$> fab uptime

[10.1.1.1] run: uptime[10.1.1.1] out: 05:20:39 up 88

days...

[10.1.1.2] run: uptime


24/30

Decorator ProblemsSome problems with Fabric's role managementCan't override decorated tasks at command lineas docs suggest

def default_roles(*role_list):

def selectively_attach(func):

if not env.roles and not env.hosts:

return roles(*role_list)(func)

else:if env.hosts:

func = hosts(*env.hosts)(func)

if env.roles:

func = roles(*env.roles)(func)

return func


25/30

All better now

#fabfile.py

@default_roles('web', 'db')

def uptime():run('uptime')

$> fab uptime

# runs on all hosts in the 'web' and'db' roles

$> fab uptime --roles lb


26/30

Dealing with Failures

By default Fabric dies if a task failsUse a context manager whenfailures are anticipated

# fabfile.py

from __future__ import with_statement #

py2.5

def symlink_me():

with settings(warn_only=True):

run('rm /path/to/symlink')

run('ln -s /home/andy /path/to/symlink')


27/30

Easy sys-admin

Make an "invoke" commandGreat for sys-admin and one-offtasks

# fabfile.py

@default_roles('all')def invoke(command):

"Invoke an arbritrary command"

sudo(command)

# install new acka es on all hosts in one command

l ld k


28/30

Real World Tasks$> fab --list

Available commands:

bounce_wsgi_procs Bounce the WSGI procs by touching the filesdeploy Full deploymentdeploy_media Push media to S3invoke Invoke an arbritrary commandmigrate Run any migrations via Southreload_nginx Update Nginx's running configsplash_off Configure Nginx to serve the site

splash_on Configure Nginx to serve a downed-site pageupdate_repositories Push code to serversupdate_dependencies Update dependencies to third party libs

hi k ' l


29/30

Whiskey's Deployment

def deploy(splash='no'):"Full deployment"

deploy_media()

update_cached_repositories()

update_dependencies()

generate_releases()if splash == 'yes':

splash_on()

_symlink_code()

migrate()

bounce_wsgi_procs()if splash == 'yes':

splash_off()

$> fab deploy:splash=yes


30/30

Questions?

python deployment with fabric 3401

Documents