OpenStack upgrades

OpenStack Summit 2013Hong Kong

Whoarewe• Sébastien Han• Frédéric Lepied• Mehdi Abaakouk

Working for eNovanceCompany blog: http://techs.enovance.com/

Worldwide offices coverageWe design, build and run clouds – anytime -

anywhere

ProblemsState of arts

Key principles• do not log in to the servers• do not manually install packages• do not manually edit configuration files• do not manually restart daemons

Puppet, in combination with GIT is the only one managing the configuration.

It’s all about architecture designWe need redundancy to perform upgrades without downtime:

• Active / active setup – Loadbalancer• Active / passive setup• Databases must be replicated : Galera / MongoDB

RollbackEven with a good QA system, problems might rise in production thus we need a rollback mechanism.

SolutionWell, ours

Breaking pointPuppet doesn’t install packages anymore.

It only manages configurations. So operating systems are shipped with all the packages installed.

eDeploy solution• Change the abstraction level• Manage updates using sub-trees• 2 kinds of sub-trees : data and programs• Data is not updated only programs

Example :• Data: /var/lib/mysql, /var/log...• Program : /usr, /lib...

Consequences• Prepare trees before installation or upgrade• Debootstrap/yum + chroot magic

• Install in 3 phases :• Hardware detection• Hardware configuration• Tree copy

• Update :• Rsync Prog sub-trees.• Script to adapt Data and restore config

eDeploy - Overview• Manage system provisioning by software role and

hardware profile• Reproduce provisioning easily• Manage upgrades and rollbacks• Efficient in term of expressiveness and performance

QATesting systems

PrinciplesEverything is versioned:

• Jenkins jobs• Puppet modules/manifests• eDeploy system images• Ansible recipes

This is what we getThe upgrade process becomes:

• Reproducible• Automated• Testable

MethodologyUpgrades with (almost) no downtime

Things that you must consider• Architecture design• MySQL schemas• Do backups!

Configuration management and OrchestrationPuppet is responsible for the configuration of a node:

• Upgrade the configuration files only (no packages upgrades!)• Restart services

Ansible orchestrates the process upgrade.

Components dependency

Base of the process

Database schemaBut what if the database schema needs to be updated?

Just upgrade the database schemas at the end of the orchestration! … Not yet (Icehouse?)

But with DB schema upgrade

Summary• Follow best practices• Architecture matters• Automation is mandatory• Tests, tests, tests

Many thanks!Questions?

Contact: sebastien.han@enovance.comfrederic.lepied@enovance.commehdi.abaakouk@enovance.com