dynamic inventory: no more host lists!

Dynamic inventory:No more host list!

#ansibleja Ansible勉強会#1

2014/06/10@tagomoris

14年6月10日火曜日

TAGOMORI Satoshi (@tagomoris)LINE Corp.


Ansible

Batch server operations

add routing table entry

deploy ssh keys

update openssl

Install/update middleware

Hadoop, Presto, Fluentd (and its config/code ...)


Ansible: good points

Minimal managed node requirements

Simple tool chain and files

YAML

and many built-in modules

Less stress of shell scripts

Dynamic inventory


Ansible: bad points

Less japanese case-studies

Less japanese books and blog entries

Less japanese users? -> No!


Inventory

Hosts and groups

Host variables / Group variables

Group of groups

‘group_vars/’ and ‘host_vars/’


Inventory: example

mail01.local globalip=192.168.0.11

[blog]www[01:30].blog.local virtualip=203.0.113.63app[01:50].blog.localmaster01.db.blog.local server_id=....slave0[1:4].blog.local server_id=..????

[linenantoka]www[01:10].nantoka.line.local


ROLEs and TYPEs

Roles (web, app, db, hadoop, ...)

Groups (service-names, ...)

OSs, Server models, Disk types (HDD, SSD, ioDrive, ...)

IP addrs (local, global, virtual)

... and many more


YabitzOur host management tool/server:

WebUI and WebAPI (json)

hostname[s], status, service, CPU, memory, disk, rack position, size, local/global/virtual IPs, virtual/parent server, ....

tagschanges(history, diff)anything

links to other internal web tools


Dynamic inventory

Dynamic inventory plugins

Cobbler external inventory script

Example for linux install server

AWS EC2 external inventory script

Digital Ocean, Google Compute Engine, Linode, Zabbix, ...

Mix inventories (static and dynamic)


Custom dynamic inventory

Original scripts AS dynamic inventory

“Developing Dynamic Inventory Sources”http://docs.ansible.com/developing_inventory.html

executable program

implement `--list`

implement `--host HOSTNAME`


http://docs.ansible.com/developing_inventory.html

http://docs.ansible.com/developing_inventory.html

JSON: for `--list`

Dynamic inventory: output{ "databases": { "hosts": [ "host1.local", "host2.local" ], "vars": { "a": true } }, "webservers": [ "host2.local", "host3.local" ], "tokyo": { "hosts": [ "host1.local", "host4.local", "host5.local" ], "vars": { "b": false }, "children": [ "shibuya", "ebisu" ] }, "shibuya": [ "host6.local" ], "ebisu": [ "host7.local" ]}


JSON: for `--host HOSTNAME`

Dynamic inventory: output

{ “localip”: “192.0.2.53”, “virtualip”: “203.0.113.53”, “global_fqdn”: “tagomoris.example.com”,}


Dynamic inventory reference

1. inventory --list

2. inventory --host host1.local



5. ...


Playbook execution

$ ansible-playbook PLAYBOOK_PATH.yml


Yabitz inventory

ansible-playbook

bin/yabitz_inventory

Yabitz

playbooks/PLAYBOOK.yaml

Servers


Role based playbooks

Flat playbook directory

playbooks/*.yamlplaybooks/SERVICE_NAME/*.yaml

“hosts” are always roleshosts: centoshosts: centos6hosts: hadoop-slavehosts: hadoop-all-nodes:fluentd-all-nodes


Yabitz tags and ROLEsTags

Server-type-tag (by studio3104 crawler)

centoscentos5, centos6

Role-based-tag

hadoop-all-nodeshadoop-master, service-hadoop-master2hadoop-slave


Implementation examples

ansible.cfg[defaults]hostfile=/path/to/bin/yabitz_inventory

commands$ ansible-playbook playbooks/update-openssl.yaml

$ TARGET_HOSTS=host1.local,host2.local \

ansible-playbook playbooks/SERVICE/foo.yaml


No more host lists!

Ansible inventory is very simple and understandable

But we cannot use it for real world operations

especially for hundreds or thousands of hosts...

Use dynamic inventory!


Role based playbook testing

Serverspec

testing without breaking operations

providing double check (beside of ansible)


Serverspec directoriesfrom http://serverspec.org/tutorial.html

host based spec directory treespec/SERVERNAME/function_spec.rb

and exec (`rake spec`)

“How to share serverspec tests among hosts”http://serverspec.org/advanced_tips.htmlstatic file based test sharing


http://serverspec.org/tutorial.html

http://serverspec.org/tutorial.html

http://serverspec.org/advanced_tips.html

http://serverspec.org/advanced_tips.html

Specs and roles

Role based provisioning

roles-to-hosts resolution MUST be done dynamically

Helper command to get roles from playbooks

and role based spec directorybin/kick-spec

spec/ROLE/*_spec.rb


Implementation example(2)

$ bin/kick-spec playbooks/BOOK.yaml

$ bin/kick-spec playbooks/BOOK.yaml \

--host host1.local --host host2.local

$ bin/kick-spec --role latest_openssl \

--host host5.local


Conclusion

DO NOT write same data in 2 or more files

USE master data from anywhere

Dynamic inventory makes it possible!


NO MORE HOST LISTS!!!!!!!!

Thanks!


Appendix: “_meta”(1)Special attribute of JSON for `--list`

To include host variables in single JSON result

No more exec: `--host HOST`

{ “shibuya”: [ “host6.local” ], “ebisu”: [ “host7.local” ], “_meta”: { “hostvars”: { “host6.local”: { “var1”: “value1”, “var2”: “value2” }, “host7.local”: { .... } } }}


Without `_meta`:

`--list` x1 and `--host` x1000 !

With `_meta`:

`--list` x1 only!

Appendix: “_meta”(2)


Benchmark: time before task startsw/ yabitz_inventoryfor thousands of hosts

Appendix: “_meta”(3)

w/o _meta w/ _meta

Elapsed time 18min! 10sec!!!!


dynamic inventory: no more host lists!

Technology