Common primitivesCommon primitivesin Dockerin Docker

environmentsenvironmentsAlex Giurgiu (alex@giurgiu.io)

DockerDockerisisgreat!great!

Until you want to deploy yourUntil you want to deploy yournew application in production...new application in production...on multiple machineson multiple machines

You thought you have thisYou thought you have this

When in fact you have thisWhen in fact you have this

We are trying to get hereWe are trying to get here

This problem is intenselyThis problem is intenselydebated at the moment...debated at the moment...

with many competingwith many competingprojects...projects...

that that approachapproach it in one it in oneway or another...way or another...

Just look atJust look atMesosGoogle's OmegaKubernetesCoreOSCenturionHeliosFlynnDeisDokkuetc.

What do they have inWhat do they have incommon?common?

they abstract a set of machines, makingit look like its one machine

they provide a set of primitives that dealwith resources on that set of machines

From this

To this

Why not use one of the mentionedsolutions?

Most of them require you to write yourapplication/workload in a custom way. Tototally give in to their way of doing things.

But (I)we want to run the old/legacyapplications, while gaining the same

advantages

Our goals are similar

standardize the way we interact with our infrastructuretreat all machines in a similar wayachieve reliability, through software and not throughhardwareachieve reproducible infrastructurereduce manual labor

Our building blockOur building block

ContainerInputs

(binaries,code,

packages, etc)

External services

Build process

State

Common primitivesCommon primitives"common enough that a generalized solution can be devised"

"should be applicable to both in-house or external applications"

Common primitivesCommon primitivespersistenceservice discoverymonitoringloggingauthentication and authorizationimage build serviceimage registry

(state) Persistence(state) Persistence

PersistencePersistenceone of the hardest problems to solve in a clean andscalable wayshould be transparent for the applicationmost people just avoid Docker-izing services thatrequire persistence

LocalLocal- bring the state locally,relative to where thecontainer runs- should be taken care byyour deployment/PaaSsolution- advantages: write/readspeeds, reliability- disadvantages: potentiallyslow deploys, complexorchestration

RemoteRemote- keep state remotely and"mount" it where theapplication is deployed- can be done by your PaaSsolution or by the containeritself- advantages: simpler toorchestrate, fast deploys- disadvantages: write/readspeeds, (un)reliability

Projects that try toProjects that try tosolve persitencesolve persitence

Flocker - https://github.com/ClusterHQ/flocker

?

Flocker way(local)Flocker way(local)

Service discoveryService discoveryand registrationand registration

Service discoveryService discovery

most worked on aspect of Docker orchestration quite a few different open source projects that tacklethis problemmultiple approaches: environment variables,configuration files, key/value stores, DNS,ambassador pattern etc.

Open source projectsOpen source projects

Consul (my personal favorite)etcd (CoreOS's favorite)ZooKeeper (many people's favorite)Eureka (Netflix's favorite)Smartstack (Airbnb's favorite)...

(service discovery)

choose a solution that can accommodate both legacyand custom applications: discovery using DNS orHTTPchoose a solution that can be manipulated using acommon protocol: HTTPmake sure to remove died out applications from yourSD systemIdeally it should have no single point of failureConsul satisfies all the above requirements

How to do itHow to do it(service discovery)

ConsulConsul(service discovery)

can be queried over DNSand HTTPdistributed key:value storeconsistent and faulttolerant(RAFT)fast convergence(SWIM)Service checks

Service registrationService registration

Can be done

by your application - simple HTTP call to Consula separate script/application inside your containeranother container that inspects running containers -progrium/registrator

Most importantly, each container should provide metadataabout the service its running.

MonitoringMonitoring

MonitoringMonitoring

2 perspectives

service monitoring - can be done as in pre-Dockertimescontainer monitoring

Service monitoringService monitoring(monitoring)

can be done with tools like Nagiosyour monitoring system should react dynamically toservices that start and stopcontainers should define what needs to be monitoredservices should register themselves in the monitoringsystemConsul supports service checks

Container monitoringContainer monitoring(monitoring)

monitor container state(up/down) - Docker event APIprovides this informationgather performance and usage metrics about eachcontainer - Google's cAdvisor provides this

cAdvisor provides an API to pull the data out, so youcan feed it to your trending system

Monitoring principlesMonitoring principles(monitoring)

have a layer of system monitoring - that trusts humanshave a layer of behavior tests - doesnt trust humans.Used to make sure that a certain environment is upreduces manual labor

enables detailed insights inside the kernel andapplicationsthey have a new "cloud" versionsame thing can be achieved on your private Dockerplatform

SysdigSysdig(DTrace for Linux)

LoggingLogging

LoggingLogging

logs will be used by engineers to troubleshoot issues... but now your application is a distributed movingtargetthe need for centralized log aggregation is big

How to do itHow to do it(logging)

Multiple approaches

applications write logs to STDOUT and you pick up thelogs using the Docker API or client. Logspout can be usedto ship the logs remotelyapplications write logs inside the container and a loggingdaemon inside the container(RSYSLOG) ships the logs to acentralized locationapplications write logs in a volume which is shared withanother container that runs a log shipping daemon

How to do itHow to do it(logging)

Choose an approach that fits your needs and sendthe logs to a centralized locationlogstash-forwarder is a great to forward yourlogs(please dont choose python-beaver)elasticsearch is a great way to store your logsKibana is a great way to visualize your logs

What do we do aboutlog ordering?

Authentication andAuthentication andauthorizationauthorization

AuthentificationAuthentification

how can you prove that a container/service is who itsays it is?useful to have a generalized way of authenticatingall your containersthat way you can count on the reported identitywhen allowing access to certain resources

How to do itHow to do it(authentication)

Largely unsolvedDocker 1.3 tries to check image signatures if theycome from the public registry and if they aremarked as an "official repo" A PKI setup fits the problem, with a uniquecertificate for every container(not image)Docker promised some PKI based solution in futurereleases - I would wait for that

AuthorizationAuthorization

builds on top of authenticationwill keep track of what resources a container/service canaccessshould hand over details like user/pass pairs, API tokensand ssh keys

How to do itHow to do it(authorization)

Do NOT bake in credentials and ssh keys into images (securityand coupling)

Easy way- mount external volume that contains credentials, ssh keys oreven ssh agent sockets- doesn't require authentication- increases the complexity of your deployment solution

Hard way- store credentials in a centralized service- requires some form of authentication- decreases complexity in your deployment solution

How to do itHow to do it(authorization)

Crypt and Consul(or etcd)

tries to solve the problem by using OpenPGPeach container needs access to a private key. Can be madeavailable through volumecredentials are stored encrypted in Consulcredentials get retrieved and decrypted in container

Image build serviceImage build service

Image build serviceImage build service

Build gets triggered when code gets changed and committed toyour repositoryCan perform basic checks to make sure the image complies withsome basic rulesCommits image to image registryIf other images depend on it, a build job should be triggered forthose imagesExtra tip: more control over the input sources for your imageswill in turn improve the reliability of your builds

How to do itHow to do it(image build service)

Git and Jenkins?

probably any vcs and CI tool will workbut Git and Jenkins work great

Simple workflow

commits code

Git post commit hook

Github webhook

Jenkins test

and build

Push to

registry

ContainerInputs

(binaries,code,

packages, etc)Build process

Basic build process

Image registryImage registry

Image registryImage registry

a central place to store your Docker imagesDocker Hub is the public oneyou can easily run a private registry

Open source projectsOpen source projects

Docker registryhttps://github.com/docker/docker-registry

Artifactory http://www.jfrog.com/open-source/

(image registry)

How to do itHow to do it(image registry)

USE a registry and dont rely on building images onevery machinetag your images with specific versionsmake version requirements explicit

Image registryImage registry

Where are we now?Where are we now?a lot of hype, experience needs to followthe sheer number of projects and work put in the ecosystemis impressivethis momentum fuels on itself and ignites rapid developmentin projects that are required to achieve certain thingscan you program?

Some conclusionsSome conclusionsreduce coupling between componentsthink about your platform as a functional program with sideeffects - identify the logic and identify the statearchitect your system in a service oriented way - this way anyrequired service can be placed inside a containeravoid running services on your Docker hostall container operations should be programmable, and ideallyidempotent

The network is the last bastion ofinflexibility.

trade-off between flexibility and performance(throughput,latency)detailed analysis of performance?

Questions?Questions?