modern, scalable deployment for plone

Scalable, modern deployment for Plone

Christian Theunect@gocept.com

Tuesday, October 16, 12

Tuesday, October 16, 12

Tuesday, October 16, 12

Tuesday, October 16, 12

Tuesday, October 16, 12

Infrastructure

Tuesday, October 16, 12

EntropyTuesday, October 16, 12

Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Mon

itori

ng

Hardware

Data center

Application

Supp

ort

Tuesday, October 16, 12

Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application

Tuesday, October 16, 12

Data center

Tuesday, October 16, 12

Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Application

Data center

Tuesday, October 16, 12

Hardware

Tuesday, October 16, 12

Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Data center

Application

Hardware

Tuesday, October 16, 12

• Virtualization is very affordable

• Platform creates interface between app and infrastructure

Virtualization

Components

OS

Tuesday, October 16, 12

Platform

Man

agem

ent

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application

Virtualization

Components

OS

Tuesday, October 16, 12

• convergence

• idempotency

• versioning

• everything as code

• continuous repairs

Automation

Tuesday, October 16, 12

Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application

Tuesday, October 16, 12

• Take care of updates and “production-ready” configuration

• Backup and other day-to-day operations

• OS and regularly used components

• Gradual process moving components from user-space installations to platform-wide features

Management

Tuesday, October 16, 12

Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application

Tuesday, October 16, 12

• Nagios :(

• (Un-)fortunately: #monitoringsucks

Monitoring

Tuesday, October 16, 12

Platform

Man

agem

ent

Virtualization

Components

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application

Tuesday, October 16, 12

• Request tracker

• custom glue code

• Daily review

• Interrupt shield

Support

Tuesday, October 16, 12

Platform

Components

Man

agem

ent

Virtualization

OS

Aut

omat

ion

Supp

ort

Mon

itori

ng

Hardware

Data center

Application

Tuesday, October 16, 12

Application deployment and operations

Tuesday, October 16, 12

Deployment

• Quick

• Repeatable

• Platform-independent

• single-click

Tuesday, October 16, 12

Batou

• Model-based

• Use everywhere

• No effect on runtime

• Orchestration

• Convergency

• Idempotency

• Reusability

Tuesday, October 16, 12

class Solr(Component):

def configure(self): self.address = Address(self.host.fqdn, 9000) self.provide('solr:server', self.address)

self += Buildout('solr', python='2.7')

self += Program('solr', command='java', command_absolute=False, args='-jar start.jar', directory=self.expand( '{{component.workdir}}/parts/instance'), restart=True)

Tuesday, October 16, 12

[hosts]oshaweb00 = nginx, www, varnish, supervisor, haproxy, docsoshaweb01 = zope, hwthemes, supervisor...oshaweb20 = zope, hwthemes, supervisoroshaweb21 = zeo, openldap, supervisoroshaweb22 = nginx, media2oshaweb23 = nginx, birt, lms, ugm, supervisoroshaweb24 = solr, memcached, supervisoroshaweb26 = mailout

[component:haproxy]strategy = production_strategybots = 3whitelist = 8default = 3...

Tuesday, October 16, 12

Components

goodnginx

varnish

haproxy

PostgreSQL

memcached

supervisor

postfix

badapache

mysql, BDB

OpenLDAP

Tuesday, October 16, 12

Minimal "production-ready" checklist

• Sane front-end configuration

• System startup/shutdown

• Log rotation

• Database maintenance

• Monitoring: processes and ports

• Service user

Tuesday, October 16, 12

Secrets

• One AES-encrypted config file per environment

• Transparent decryption

• Safety-belt to avoid accidental checking in unencrypted files

• Future: PGP-encryption

Tuesday, October 16, 12

Environments

• Production vs. Staging

• Structurally identical but smaller

• Only one of many (dev, integration, ...)

• "Fly what you test, test what you fly."

Tuesday, October 16, 12

Web applications

• Shared nothing in your app!

• Many small processes

• Do not listen early

• Never store runtime config in the database

Tuesday, October 16, 12

Web applications II

• Log long running requests

• Avoid synchronous external requests

• Avoid accidental write requests.

• Use feature switches.

Tuesday, October 16, 12

Process control

• only one component to integrate

• turn anything into a daemon

• use superlance (httpok, memmon) for active restarts

• use 3.0 release candidate

Tuesday, October 16, 12

Python: virtualenv and buildout

• use both, combined

• buildout might be a hammer, but not everything is a nail

Tuesday, October 16, 12

more buildout

• use extends to keep your configs clean and structured

• allow-picked-versions=False

• -t 3

• buildout >= 1.6

• mr.developer

• allowed-hosts = ...

Tuesday, October 16, 12

and even more buildout

• multiple small better than one big

• different Python versions

• different subsets of egg versions

• use mirrors - see pypi-mirrors.org

• latency to the PyPI server is the biggest factor in buildout performance

Tuesday, October 16, 12

Databases

• As much RAM as you can afford - seriously

• Low latency network (<0.1ms)

• Multi-core if necessary

• Disk usage < 40%

Tuesday, October 16, 12

story: more hardware = slower

• Application is slow.

• Customer buys more hardware. Specifically: RAM.

• System is slower now.

• System runs many components. Hard to see through.

Tuesday, October 16, 12

story: more hardware = slower

• One number stands out: process exit rate is 20/sec. PosgreSQL. What?

• Customer creates new connection for each query.

• PostgreSQL forks master for new connection.

• Fork went from few 10ms to few 100ms

Tuesday, October 16, 12

story: more hardware = slower

• Diagnose. Find root cause.

Tuesday, October 16, 12

HTTP/S termination

• HTTP and SSL

• Virtual hosting

• Redirects

• host normalization

• Proxy to load balancer

• Static files

Tuesday, October 16, 12

Load balancing

• LB + HA

• Classify requests

• redispatch + retries

• balance: leastconn

• minconn 1, maxconn 2, threads +2

• Live statistic UI

Tuesday, October 16, 12

Monitoring

• again :(

• processes, ports, logs

• model dependencies

• check_webpage.rb

• Business processes (BPI)

Tuesday, October 16, 12

Reporting

• report + phone call

• incidents

• changes

• availability

• performance

• trends

Tuesday, October 16, 12

SLAs

• Availability

• more keeps your business safe

• less reduces costs

• Friends don't let friends think in "nines".

• Alignment of technical and organisational measures

Tuesday, October 16, 12

References

• "The practice of system and network administration" (Limoncelli, et. al.)

• Mark Burgess on config management

• "The twelve-factor app" (Heroku)

• gocept.net/doc

Tuesday, October 16, 12

Conclusion I

• Use infrastructure to build platforms

• Platforms should be transparent.

•Deploy applications on platforms.

Tuesday, October 16, 12

Conclusion II

• Do not cobble together random stuff from the internet.

• Careful work pays. Buzzwords don't.

• If you hear "enterprise". Run.

• Sapere aude.

Tuesday, October 16, 12

Conclusion III

• I don't think IaaS and PaaS in the current extremes are a good answer to many projects.

• Many people experiment with ideas. Everything is becoming more flexible.

• Let's play!

Tuesday, October 16, 12

Questions?Tuesday, October 16, 12