zero downtime deployment with ansible
TRANSCRIPT
Zero Downtime Deployment with AnsibleDevOpsCon Berlin 2016
@steinim
OPEN
Slideshttp://steinim.github.io/slides/devopscon/zero-downtime-ansible
Source [email protected]:steinim/zero-downtime-ansible.git
Full tutorialhttp://steinim.github.io/slides/devopscon/zero-downtime-
ansible/tutorial.html
What's a provisioning framework?Automated setup of serversConfiguration as codeNot immutable (in itself)
ExamplesCreate usersInstall softwareGenerate and manipulate config filesStart/stop/restart processesSet up dependencies between operations
Describe what to do (imperative)#!/bin/bash if $( command -v vim >/dev/null 2>&1 ); then echo "vim is already installed." else apt-get install vim fi if $( grep -Fxq "filetype indent off" /etc/vim/vimrc ); then echo "set filetype indent off is already in /etc/vim/vimrc." else echo "filetype indent off" >> /etc/vim/vimrc # TODO: Do not continue if this fails. fi # TODO: Rollback if something fails.
Describe state (declarative)- name: ensure installed vim apt: pkg=vim state=installed - name: set filetype indent off for vim lineinfile: dest=/etc/vim/vimrc line='filetype indent off' state=present
ProsIn source control.Self documenting (it's code!).Refactoring.Less differences between environments.Deterministic.Prevents manual steps.Fast and easy to configure up a new environment.Easier to test server setup.
AnsibleSSH-basedClient only (no server)YAML configurationPush (and pull)Supports more than setup and provisioning:
Application deploymentRemote command execution
Layout (convention over configuration)├── ansible.cfg ├── hosts ├── site.yml ├── group_vars │ └── <group name> ├── host_vars │ └── <host name> ├── roles │ ├── <role> │ │ ├── files │ │ └── <file> │ │ └── templates │ │ └── <template>.j2 │ │ ├── handlers │ │ │ └── main.yml │ │ ├── tasks │ │ │ └── main.yml │ │ ├── vars │ │ │ └── main.yml
FactsAnsible by default gathers “facts” about the machines undermanagement.These facts can be accessed in Playbooks and in templates.ansible -m setup app1.local
The taskAn app user 'devops', with:
Home directory: /home/devopsssh-key
A PostgresSQL database.Nginx as a reverse proxy.An init script installed as a service.Deploy an application that uses the provisioned infrastructure.
Help!http://docs.ansible.com/list_of_all_modules.html
Task1: Install and configure softwareModify roles/common/tasks/apt.yml.Install Vim.Insert the line 'filetype indent off' in /etc/vim/vimrc
Help:
http://docs.ansible.com/apt_module.htmlhttp://docs.ansible.com/lineinfile_module.html
Task1: Solution- name: Ensure installed vim apt: pkg=vim state=present update_cache=no tags: - vim - name: Set filetype indent off lineinfile: dest=/etc/vim/vimrc line='filetype indent off' state=present tags: - vim
Run it!ansible-playbook site.yml --tags vim
ProTip: Use '--tags', '--skip-tags', '--limit' and/or 'gather_facts: False'to reduce execution time.
VariablesUse variables! → Infrastructure as data.Where should variables be defined? Ansible has very many options.
Inventory (./hosts)group_vars and host_varsPlaybook (site.yml)Facts (local or server)Command line (as arguments)
Access variables from playbooks: "{{ variable }}"
http://docs.ansible.com/playbooks_variables.html
Task2: Create an application userCreate roles/users/tasks/main.ymlHome directory: /home/devopsssh-keyUse variables! (group_vars)
Help: (create a directory)
(.ssh/authorized_keys)
http://docs.ansible.com/group_module.htmlhttp://docs.ansible.com/user_module.htmlhttp://docs.ansible.com/file_module.htmlhttp://docs.ansible.com/lineinfile_module.htmlhttp://docs.ansible.com/playbooks_best_practices.html#group-and-host-variables
Define some variables# group_vars/appservers
user: devops group: devops user_name: "Devops app user"
Task2: Solution# roles/users/tasks/main.yml
- name: Create group group: name={{ group }} state=present tags: - users - name: Create user user: name={{ user }} comment="{{ user_name }}" group={{ group }} createhome=yes home=/home/{{ user }} shell=/bin/bash state=present tags: - users - name: Ensure ssh directory exists for user
Run it!ansible-playbook site.yml --limit appservers --skip-tags apt,vim,java
Try it!ssh [email protected]
Task3: Install and configure PostgreSQLroles/postgresql ├── files │ └── postgresql.conf ├── handlers │ └── main.yml ├── tasks │ ├── main.yml │ └── ... └── templates └── pg_hba.conf.j2
Use variables (group_vars/dbservers).
Template: roles/postgresql/templates/pg_hba.conf.j2
Use handler to restart postgresql upon notification
Help: (pg_hba.conf.j2)
http://docs.ansible.com/template_module.htmlhttp://docs.ansible.com/postgresql_user_module.htmlhttp://docs.ansible.com/postgresql_db_module.htmlhttp://docs.ansible.com/playbooks_intro.html#handlers-running-operations-on-change
Define some variables# group_vars/dbservers
postgresql: version: 9.4 repo: 'deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main' user: postgres group: postgres data_dir: /var/lib/pgsql/data address: 192.168.101.0/24 db: name: devops user: devops password: devops123
Create a template# roles/postgresql/templates/pg_hba.conf.j2
# TYPE DATABASE USER ADDRESS METHOD local all all trust host {{ db.name }} {{ db.user }} {{ postgresql.address }} md5
Create a handler# roles/postgresql/handlers/main.yml
- name: Restart postgresql service: name=postgresql state=restarted
Task3: Solution# roles/postgresql/tasks/postgresql.yml
- name: Install Postgresql and dependencies apt: pkg={{ item }} state=installed update_cache=yes with_items: - postgresql-{{ postgresql.version }} - python-psycopg2 tags: - pg_install - name: Install postgresql.conf copy: src=postgresql.conf dest=/etc/postgresql/{{ postgresql.version }}/main/postgresql.conf owner={{ postgresql.user }} group={{ postgresql.group }} notify: - Restart postgresql tags: - pg_install
Run it!ansible-playbook site.yml --limit dbservers --tags pg_install
Try it!$ vagrant ssh db vagrant@db:~$ psql -d devops -U devops -W devops=> \q
ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-keyInstalled and configured a database and a db user
Task4: Deploy!roles/app ├── files │ └── init.sh ├── tasks │ └── main.yml └── templates └── config.properties.j2
NB! Use variables (./hosts).
Set 'serial: 1' for appservers in the playbook (site.yml).
Help:
http://docs.ansible.com/service_module.html
Browse to
Run it!ansible-playbook site.yml --limit appservers --tags deploy
Try it!http://app1.local:1234/
What just happened?/home/devops ├── config.properties ├── current -> /home/devops/devops_1416228023.jar ├── previous -> /home/devops/devops_1416221573.jar ├── devops_1416221573.jar ├── devops_1416228023.jar └── logs ├── stderr.log └── stdout.log
/etc/init.d └── devops
ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-keyInstalled and configured a database and a db userDeployed an application to two appservers and enabled it as aservice
Task5: Deploy databaseroles/db ├── files │ └── migrate_db.sql └── tasks └── main.yml
Help: http://docs.ansible.com/command_module.html
Task5: Solution- name: Copy db migration script copy: src=migrate_db.sql dest=/tmp/migrate_db.sql become_user: postgres tags: - deploy - name: Run db migration script command: psql -d {{ db.name }} -q -f /tmp/migrate_db.sql become_user: postgres tags: - deploy
Browse to
Run it!ansible-playbook site.yml --limit dbservers --tags deploy
Try it!$ vagrant ssh db vagrant@db:~$ psql -d devops -U devops -W devops=> \dt devops=> select * from hello; devops=> \q
http://app1.local:1234/
ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-keyInstalled and configured a database and a db userDeployed an application to two appservers and enabled it as aserviceMigrated the database schema and fetched data from it through theapplication
Task6: Set up proxyroles/nginx ├── handlers │ └── main.yml ├── tasks │ ├── config_nginx.yml │ ├── install_nginx.yml │ └── main.yml └── templates └── devops.conf.j2
Help:
http://wsgiarea.pocoo.org/jinja/docs/loops.html
Task6: Solutionupstream backend { {% for appserver in groups.appservers %} server {{ appserver }}:{{ app_port }} fail_timeout=1s; {% endfor %} } server { listen 80; server_name _; access_log /var/log/nginx/devops-access.log; location / { proxy_pass http://backend; proxy_redirect off; proxy_next_upstream error timeout invalid_header http_500 http_502; proxy_connect_timeout 2; } }
Browse to # refresh me many times
Run it!ansible-playbook site.yml --limit proxies --tags nginx
Try it!http://proxy.local/
ProgressInstalled softwareManipulated filesCreated a user and set up a ssh-keyInstalled and configured a database and a db userDeployed an application to two appservers and enabled it as aserviceMigrated the database schema and fetched data from it through theapplicationSet up a reverse proxy for automatic failover between the twoappservers
The Expand/Contract pattern
Expand ContractAdd tablesAdd columnsTweak indexes
Remove tablesRemove columnsRemove/add constraints
Play time :-)Suggestions:
Change database table name from HELLO to MESSAGES anddeploy a new version without downtime.Implement automated rollback.