opinionated paas on docker · media partner organizers top media partner general partner supporters...

Media Partner

Organizers

Top Media Partner

General Partner

Supporters

Ilkka AnttonenAccenture

Opinionated PAAS on DockerOctober 17, 2014

Getting the demo

• The demo can be checked out from https://github.com/SirIle/openslava-docker

• README.md file contains instructions to get it running

https://github.com/SirIle/openslava-docker

About me

• Started with basic and assembler on C64 almost exactly 30 years ago

• 16 years work experience• Emerging Technology Nordics Lead at

Accenture

• I can be contacted at [email protected]

mailto:[email protected]

Ask questions or rate the Speaker

www.sli.do/openslava

Docker• Containers are nothing new, but Docker has popularized them

• CHROOT -> VMware/VBox, XEN, KVM -> Containers, LXC -> Docker• Containers share the same kernel, but run isolated processes on partitioned

resources• Very light and quick to start

• Docker runs (after 0.9) on a multitude of execution environments• libcontainer (default), LXC, OpenVZ, libvirt-lxc, qemu/kvm, Solaris

Zones, chroot...• From a developer perspective Docker

• Allows easily running lots of containers on a single VM• Allows for simulation of large distributed services• Container can be built using CI and shared using a private registry

Approaching PAAS• Running multiple processes in a container

• Supervisord (init.d, upstart, launchd)• Service registration and discovery

• Consul (etcd)• Log aggregation

• Rsyslog + Logstash + ElasticSearch + Kibana• Advanced networking

• Weave (Pipeworks, iptables)• Orchestration

• MaestroNG (Kubernetes, Fleet, Stampede, …)

http://www.mindmeister.com/389671722/docker-ecosystem

http://www.mindmeister.com/389671722/docker-ecosystem

Multiple Processes per Container• Running some services (as processes) helps

• SSHD for connectivity and running remote commands • nsenter would also work, but is more complex• You can ssh directly to containers from outside the host if needed• You get security log entries for accesses

• Rsyslog for log aggregation• Consul for service registration, discovery and local DNS service

• Supervisord is simple and does the trick

[supervisord]

nodaemon=true

[program:rsyslogd]

command=/usr/sbin/rsyslogd -n

autorestart=true

[program:sshd]

command=/usr/sbin/sshd -D

stdout_logfile=/var/log/supervisor/%(program_name)s.log

stderr_logfile=/var/log/supervisor/%(program_name)s.log

autorestart=true

[program:consul]

command=/bin/bash /usr/local/bin/startConsul.sh

stdout_logfile=/var/log/supervisor/%(program_name)s.log

stderr_logfile=/var/log/supervisor/%(program_name)s.log

autorestart=true

[program:cassandra]

command=/bin/bash /usr/local/bin/startCassandra.sh

Example of supervisord.conf

Service Registration…• Containers enable quick and easy scaling• Applications in containers should be agnostic about the environment

• Helps with portability and ease of building• Consul provides distributed framework with an agent on every container• Configuring with a file per service or dynamically with API• Supports multiple failure detection methods• Core container is found through container linking for joining

{

"service": {

"name": "cassandra",

"tags": ["database"],

"port": 9160

}

}

Example service configuration:

…and Discovery• Support for DNS based discovery which makes things really simple

• Supports also REST API and other record types than A record for DNS for extra information if default ports can be used

• Datacenter aware, routes between datacenters only with explicit requests• List of matching nodes is randomized on each query to provide simple load

balancing• Services and nodes are named with specific syntax:

• <node>.node.<datacenter>.<domain>• [tag].<service>.service[.datacenter].<domain>

Inside the datacenter:ping cassandra.service.consul

ping cassandra1.node.consul

In a specific datacenter:ping cassandra.service.east.consul

ping cassandra4.node.east.consul

Example service discovery queries

Log Aggregation• Unless explicitly configured, container file system isn’t preserved

• In case of failures accessing the logs may be difficult• Easy solution is to ship the logs to a central place for storage and analysis• Logstash backed by ElasticSearch is simple yet powerful• One core container per datacenter that stores the logs• Rsyslog used to ship the logs

• Not too much overhead, robust system tolerant of temporary failures

$ModLoad imuxsock # for reading local syslog messages

$ModLoad imfile # Load the imfile input module

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

$ActionResumeInterval 1

$ActionResumeRetryCount -1

# Cassandra log

$InputFileName /var/log/cassandra/system.log

$InputFileTag cassandra:

$InputFileStateFile state-cassandra

$InputRunFileMonitor

# Send everything to a logstash server named 'log.service.consul'

*.* @@log.service.consul:5000

auth,authpriv.* /var/log/auth.log

*.*;auth,authpriv.none -/var/log/syslog

Example of rsyslog configuration

Orchestration• Orchestration handles container lifecycle• Fetches (some systems can also build) containers into local registry• Starts containers taking into account the dependencies and start order• Kubernetes has lots of traction, but currently works well only on GAE• Lots of options, some works across multiple nodes

name: cassandra

ships:

vagrant-docker: {ip: 10.10.10.30}

services:

core:

image: registry.local/core:latest

instances:

core:

ship: vagrant-docker

ports: {ssh: '22:1000', elasticsearch: 9200, kibana: 80 , consului: 8500}

volumes:

/etc/localtime: /etc/localtime

/etc/timezone: /etc/timezone

lifecycle:

running: [{type: tcp, port: consului}]

dns: localhost

Example service definition for MaestroNG

Advanced Networking• Docker networking is evolving rapidly as people are building more complex

environments• Basic networking gives changing IP addresses from a configurable range

• In a single node with service registration this is good enough• In multi-node setups greater control of addresses and ranges and the

ability to join the networks between nodes is required• Several projects have emerged to help with advanced networking

• Stampede supports inter-node secure communication, but is very alpha• With pipework and iptables doing complex stuff is possible

• Weave does the same with simplicity replacing the normal docker command

sudo weave launch 10.0.0.1/16

# Launch the container

sudo weave run 10.0.1.2/22 -t -p 9160:9160 -p 1021:22 \

--dns 127.0.0.1 -h cassandra1 --link core:core \

–v /etc/localtime:/etc/localtime:ro \

-v /etc/timezone:/etc/timezone:ro -e DC=west -i

registry.local/cassandra

Example of container start with Weave networking

Example on a single node• Modified MaestroNG used for orchestration

• Added support for DNS parameter as Consul provides a local DNS server for every container

• YAML based readable configuration• One core container

• Core container is discovered through container linking• LogStash + ElasticSearch• Consul server for the node

• Datastax Opscenter container for monitoring Cassandra instances

• Three Cassandra containers• Node container for running

application to demonstrate Cassandra connectivity

• All containers have Consul agents• Connect to Consul server at

the start• Gossip from there on

Host

<Insert picture>

Node1 (10.10.10.30)

CoreConsul + UI

Logstash etcrsyslog

sshd (1000)

Cassandra3Consulrsyslog

sshd (1023)Cassandra





OpscenterConsulrsyslog

sshd (1121)Opscenter

NodeConsulrsyslog

sshd (1001)

Maestro

NG

Example on two nodes• Weave is used to provide advanced networking between nodes

• Orchestration tools don’t support it yet, so shell scripts used• Private registry used to transfer images, no need to build twice

Host

Node1 / West (10.10.10.30)

10.0.0.1Core

10.0.1.1Consul + UI

Logstash etcrsyslog

sshd(1000)

Cassandra310.0.1.4Consulrsyslog

sshs(1023)Cassandra


sshd(1022)Cassandra


sshd(1021)Cassandra

Opscenter10.0.1.10

Consulrsyslog

sshd(1121)Opscenter

Node10.0.1.20

Consulrsyslog

sshd(1001)

Node2 / East (10.10.10.40)

10.0.0.2Core

10.0.2.1Consul + UI

Logstash etcrsyslog

sshd(1000)


sshd(1022)Cassandra


sshd(1021)Cassandra

Registry (10.10.10.100)

Q & AThank you for listening!

Questions?

opinionated paas on docker · media partner organizers top media partner general partner supporters...

Documents