openstack hk-summit-upgrades-talk
TRANSCRIPT
OpenStack upgrades
OpenStack Summit 2013Hong Kong
Whoarewe• Sébastien Han• Frédéric Lepied• Mehdi Abaakouk
Working for eNovanceCompany blog: http://techs.enovance.com/
Worldwide offices coverageWe design, build and run clouds – anytime -
anywhere
ProblemsState of arts
Key principles• do not log in to the servers• do not manually install packages• do not manually edit configuration files• do not manually restart daemons
Puppet, in combination with GIT is the only one managing the configuration.
It’s all about architecture designWe need redundancy to perform upgrades without downtime:
• Active / active setup – Loadbalancer• Active / passive setup• Databases must be replicated : Galera / MongoDB
RollbackEven with a good QA system, problems might rise in production thus we need a rollback mechanism.
SolutionWell, ours
Breaking pointPuppet doesn’t install packages anymore.
It only manages configurations. So operating systems are shipped with all the packages installed.
eDeploy solution• Change the abstraction level• Manage updates using sub-trees• 2 kinds of sub-trees : data and programs• Data is not updated only programs
Example :• Data: /var/lib/mysql, /var/log...• Program : /usr, /lib...
Consequences• Prepare trees before installation or upgrade• Debootstrap/yum + chroot magic
• Install in 3 phases :• Hardware detection• Hardware configuration• Tree copy
• Update :• Rsync Prog sub-trees.• Script to adapt Data and restore config
eDeploy - Overview• Manage system provisioning by software role and
hardware profile• Reproduce provisioning easily• Manage upgrades and rollbacks• Efficient in term of expressiveness and performance
QATesting systems
PrinciplesEverything is versioned:
• Jenkins jobs• Puppet modules/manifests• eDeploy system images• Ansible recipes
This is what we getThe upgrade process becomes:
• Reproducible• Automated• Testable
MethodologyUpgrades with (almost) no downtime
Things that you must consider• Architecture design• MySQL schemas• Do backups!
Configuration management and OrchestrationPuppet is responsible for the configuration of a node:
• Upgrade the configuration files only (no packages upgrades!)• Restart services
Ansible orchestrates the process upgrade.
Components dependency
Base of the process
Database schemaBut what if the database schema needs to be updated?
Just upgrade the database schemas at the end of the orchestration! … Not yet (Icehouse?)
But with DB schema upgrade
Summary• Follow best practices• Architecture matters• Automation is mandatory• Tests, tests, tests
