reliably deploying web apps with database migrations

Reliably Deploying Web Apps with Database

MigrationsPratik Vyas

To deploy new code

with a database migration

that takes 10 seconds to run per site

The Problem

in batches of five sites at a time

Then the 600th site would be running new code with

unpatched database for 20 minutes!

If we have to migrate 600 sites

run migrationfor 600th site

code deployed

danger zone

Deployments need to be handled more carefully

Create a new environment with the new code

Migrate sites one at a time by moving them to the new environment

An Intuitive Solution

build x

build x+1

site c

site d

site e

site b

site a

site f

site g

site h

site i

site j

● move to build x+1● run migrations

maintenance area(No HTTP requests)

site c

The Two Environments

The Solution - Part 1

Building an Environment

Use virtualenv and pip?

pip install would also need to compile C extensions

eg. lxml takes a lot of cpu time and memory

Makes creating new environmentsEXPENSIVE

Wheels are the new standard of python distribution (PEP 427)

Unzip the wheel in site-packages and the package is installed … like magic!

Build wheels on a different machine and watch a new environment get deployed

in a few seconds

Python Wheels

Cost of creating virtualenv is mostly unzipping

No compiler required on production server

PyPI availability is not a factor during deployments

Benefits of Build Step

buildfile.tar.gz├── scripts│ ├── virtualenv.py│ └── virtualenv_support│ ├── pip-1.5.2-py2.py3-none-any.whl│ └── setuptools-2.1-py2.py3-none-any.whl├── sites│ └── statics│ ├── css│ └── *.css│ ├── img│ └── *.png│ └── js│ └── *.js└── wheels └── *.whl

The Solution - Part 2

Running Processes

The new code will require us to start

a new wsgi process + workers

use Supervisor

Supervisor

Use Jinja and generate

process list per virtualenv

Protip: For adding/removing a group of processes, use reread and update instead of reload. It prevents non-related processes

from restarting.

The Solution - Part 3

Serving sites, both old and new

Serve unmigrated sites using old code,

migrated sites using new code.

nginx to the rescue

Configuring nginx awesomeness

Use Jinja to generate configuration and reload nginx

Fallback to old environment based on error condition

For performance, 404s should be cheap

upstream upstream-1 { server 127.0.0.1:8000 fail_timeout=0; }

upstream upstream-2 { server 127.0.0.1:8001 fail_timeout=0; }

server {

. . . .

location / {

error_page 404 = @upstream2handler;

proxy_redirect off;

proxy_intercept_errors on;

proxy_pass http://upstream-1;

}

location @upstream2handler {

proxy_redirect off;

proxy_intercept_errors on;

proxy_pass http://upstream-2;

}

}

Background Workers?

Stop executing scheduled jobs when in handover state

Preferably, use a fresh broker for every build

Track pending jobs per site(we prepend sitename to queue name for this)

The Final Picture

build x

build x+1

site c

site d

site e

site b

site a

site f

site g

site h

site i

site j

● Delay enqueuing scheduled tasks.● Continue if no pending jobs, else

retry after 60 seconds.

● Run migrations.● Move to build x+1.

site c

maintenance area(No HTTP requests)

ध यवाद@pdvyas, @frappe_io