beyond static configuration

Beyond static configuration management An overview of current developments Dr. Stefan Schimanski

Upload: sttts

Post on 19-Aug-2014




0 download


Current development around Docker orchestration, service discovery and cluster operation systems


Page 1: Beyond static configuration

Beyond static configuration managementAn overview of current developments !Dr. Stefan Schimanski

Page 2: Beyond static configuration

About meOpenSource developer since mid of 90ies

Freelance Developer & Consultant

1st DevOpsDays

Ex-head of Linux Competence Center @ German Air Traffic Controlpushing Puppet, DevOps, CI, OpenStack & agile

Before spent 5 years @ LMU Munich to come up with essentially this:



@the1stein,, [email protected], schimmi@freenode

Page 3: Beyond static configuration

Long long time ago….

Page 4: Beyond static configuration

Long long time ago, in 2011:

The DevOps world was simple. !

Essentially, choose between: !

Cfengine/Puppet/Chef Ubuntu/Debian/Red Hat/Suse

implement monitoring, metrics, orchestration, log-management and deploy apps, preferably from

some CI/CD pipeline, usually on bare metal or IaaS

Page 5: Beyond static configuration

– In some OpenSpace @ DevOpsDays Berlin May 2013

„Who thinks that Docker changes everything?“

Page 6: Beyond static configuration

16 months later

Page 7: Beyond static configuration
Page 8: Beyond static configuration

Claim: DevOps tool chains are being disrupted

Page 9: Beyond static configuration

Claim: System engineering is being disrupted

Page 10: Beyond static configuration

Claim: Linux/Unix userland is being disrupted

Page 11: Beyond static configuration

Exciting times

Page 12: Beyond static configuration

Development Linux Distributions change,

become very special-purpose oriented. !A container doesn’t need - multi-user/tenant support - an init system - initrd, kernel, boot loader, installer,

hardware detection - application separation - a strict file system hierarchy standard - a lot of daemons.

Page 13: Beyond static configuration

Development Linux Distributions change,

become very special-purpose oriented. !A host for containers doesn’t need - multi-user/tenant support - a lot of user customizations - an extensive packaging system - an in-place&time update system - a strict file system hierarchy standard - a lot of traditional daemons - a desktop environment.

Page 14: Beyond static configuration

CoreOS an OS for cluster nodes running containers

based on systemd

fault-tolerant distributed key-value store: etcd

cluster layer above systemd: fleet and fleedctl

clear separation of state and OS

update = reboot

under heavy development

Page 15: Beyond static configuration

Container Orchestration

Page 16: Beyond static configuration

figFast, isolated development environments using Docker

like a vagrant for docker


fig.yml describing multiple containers, with links, environment variables, etc.

Page 17: Beyond static configuration

maestro-ng Container orchestration for Docker environments

multi-host supportwith static host assignment

powerful container dependencies and linking features

service discovery via env variables and integration with zookeeper

Page 18: Beyond static configuration

fleet in CoreOS

Normal systemd primitives for dependencies

Page 19: Beyond static configuration

libswarm by Docker A minimalist toolkit to compose network services

libswarm, libchan

Page 20: Beyond static configuration
Page 21: Beyond static configuration

Development static to dynamic system configuration

Page 22: Beyond static configuration





Page 23: Beyond static configuration


Page 24: Beyond static configuration

etcdInitialization • app1: etcd -peer-addr -addr -data-dir /tmp/etcd -name app1 -bind-addr • app2: etcd -peer-addr -addr -data-dir /tmp/etcd -name app2 -bind-addr

-peers, • app3: etcd -peer-addr -addr -data-dir /tmp/etcd -name app3 -bind-addr

-pears, !Distributed Key/Value store • etcdctl set /message Hello • etcdctl get /message

Hello • curl -L

{"action":"get","node":{"key":"/message","value":"Hello","modifiedIndex":4,"createdIndex":4}} !Watches • app1: etcdctl watch /foo-service —recursive • app2: etcdctl set /foo-service/container2 localhost:2222

app1: localhost:2222 !TTLs • etcdctl set /foo "Expiring Soon" --ttl 20 # gone after 20 seconds

Page 25: Beyond static configuration


/etc/confd/conf.d/myapp-nginx.toml /etc/confd/templates/nginx.tmpl

Key/Values in etcd (or consul):

confd -verbose -onetime -node '' -confdir /etc/confd/conf.d

Page 26: Beyond static configuration

Serfserf agent & serf join app2:7946 app3:7946 serf event user:deploy !






serf members -tag role=load-balancer alive role=db

Page 27: Beyond static configuration

Consulapp1:# consul agent -server -data-dir /tmp/consul -bootstrap-expect 1

app2:# consul agent -server -data-dir /tmp/consul

app3:# consul agent -server -data-dir /tmp/consul

consul join app2 app3

consul members

Page 28: Beyond static configuration

Consul – service discoveryapp1: /root/consul.d/web.json

consul agent -server -data-dir /tmp/consul -config-dir /root/consul.d

dig @ -p 8600 web.service.consul

apt-get install dnsmasq echo server=/consul/ >> /etc/dnsmasq reload dnsmasqdocker run —dns ubuntu ping -c 1 web.service.consul

Page 29: Beyond static configuration

Consul – dynamic service announcement

Early integration of Mesos/Marathon and Consul – not today

curl -i localhost:8500/v1/agent/services


curl -i -X PUT localhost:8500/v1/agent/service/register -d ‚{"ID":"web2","Name":"web2","Service":"web","Tags":["prod"],"Port":31774}'

curl -i -X DELETE localhost:8500/v1/agent/service/deregister/web2

Page 30: Beyond static configuration

Consul – service announcement from dockerconsul agent -server -data-dir /tmp/consul -config-dir /root/consul.d -client # so docker can access consul API

Dockerfilewith code:

Page 31: Beyond static configuration

Consul – discovery in action

app1: docker run -e CONSUL_SERVICE='{"Name":"web2","Service":"web","Tags":["prod"],"Port":1080}' -e RUN="nodejs /webserver.js" -p 1080:80 -i -t consul-ubuntu

app3: curl web2.service.consul:1080



What about dynamic ports assigned by Docker?

Page 32: Beyond static configuration

Vulcand – web proxy with native etcd supportChanges to configuration take effect immediately without restarting the service

Page 33: Beyond static configuration

An operation system for a computer - manages hardware resources - offers services to programs - isolates running programs

for a cluster of machines


Page 34: Beyond static configuration
Page 35: Beyond static configuration

Mesos a cluster meta operation system

„Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared pool of servers.“

Fault-tolerant replicated master using ZooKeeper

Scalability to 10,000s of nodes

Isolation between tasks with Linux Containers

Quite OS agnostic

developed since 2010, OpenSource

Marathon = init daemon on Mesos

Chronos = cron daemon on Mesos

Deimos = Docker integration

Page 36: Beyond static configuration

– Abdur Chowdhury Former Chief Scientist, Twitter Inc. Claim:

Mesosphere brings Google-scale compute to everybody.

Page 37: Beyond static configuration


Fault Tolerance

Page 38: Beyond static configuration


Page 39: Beyond static configuration

Google Kubernetes also a cluster operation system„Kubernetes is a system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.“

released into OpenSource in June

very early stage

IBM, Mesosphere, Red Hat promise to support it

Mesos promise support as Framework ontop of Mesos

„Main contribution: abstractions for system engineering which are proven within Google“

runs on CoreOS with etcd

runs Docker containers

Page 40: Beyond static configuration

Development App operation will split into - platform operation - app operation

Platform will be more&more off-the-shelf or PaaS

Page 41: Beyond static configuration

Deis – Heroku-style PaaS on top of CoreOS, prototype on Mesos

• Still in (quite) early development • depends a lot on CoreOS,

e.g. on Fleet’s non-existing scheduling • similar project: Flynn

Page 42: Beyond static configuration

Where are !- classical configuration management, - IaaS solutions like OpenStack, - monolithic PaaS like OpenShift and CloudFoundry, - classical Linux distributions?

Page 43: Beyond static configuration
Page 44: Beyond static configuration

Exciting times