[nyc meetup] docker at nuxeo
DESCRIPTION
In this talk, Damien describes the infrastructure Nuxeo has built around Docker containers, which is mainly based on CoreOS and Docker, and how it provides a way to generically run applications not only on a single host, but across a whole cluster of hosts. The resulting architecture can be used to implement a PaaS approach for any application.TRANSCRIPT
Docker at Nuxeo
Who we are
What is Nuxeo platform
• A platform to develop you own content apps
• Highly pluggable thru extension points
• Model you content with Nuxeo Studio
• Point and click UI to define your business model
• with hot-reload
Nuxeo tech. stack
• Java stack : Seam + JSF for Nuxeo DM and DAM
• Rest API to build your JS Apps
• Nuxeo Studio (Saas) to model your business
• Thru data model (doc types, schemas)
• Thru Automation chains
Nuxeo in the cloud ?
Goals
• Customers ask to host their app
• Provide quick access to test instances (onboarding)
• Provide onDemand dev or CI instances
Goals
• Customers ask to host their app
• Provide quick access to test instances (onboarding)
• Provide onDemand dev or CI instances
This is the nuxeo.io promise
First attempt
Manual OnDemand
• Based on AWS CloudFormation
• Manually deployed
• Works but :
• not scalable
• manual admin & monitoring
• difficult upgrade process
Second attempt
AWS Automation
• Provisionning may be automated
• Working mockup (python, celery, boto) but…
• Strong adherence to AWS
• difficult to test
• slow provisionning (one EC2 instance per Nuxeo instance)
• upgrade process still difficult
• scaling out is difficult
• sensible to failures
This leads to our challenge
Current challenge
• Build a scalable infrastructure
• Quick provisioning (less than a minute)
• Easy setup (no admin needed)
• Easy upgrade process
• Automated healing
• Easy to test
How Docker solves some of these issues ?
Quick provisioning
• We don’t have to start a server
• No VM overhead
• Start time is Nuxeo start time
Data free containers
• Docker container doesn’t hold data
• Upgrading is just using a new version of the image
• Healing means killing and restarting
Nuxeo v5.9.3
S3Postgres
Data free containers
• Docker container doesn’t hold data
• Upgrading is just using a new version of the image
• Healing means killing and restarting
S3Postgres
Data free containers
• Docker container doesn’t hold data
• Upgrading is just using a new version of the image
• Healing means killing and restarting
S3Postgres
Nuxeo v5.9.4
Several containers per server
• Scaling is just a matter of starting several containers
• But means infinite CPU / RAM and IO
Nuxeo.io server
io container
NXIO-0001
io container
NXIO-0002
io container
NXIO-0003
io container
NXIO-XXXX
…
Challenge status
• Build a scalable infrastructure
• Quick provisioning (less than a minute)
• Easy setup (no admin needed)
• Easy upgrade process
• Automated healing
• Easy to test
How to solve the other issues ?… with a little help from my friends
The infinite resource computer…
… is a cluster
Docker is not sufficient
• Docker handles processes at the OS level
• On the cluster level we need a companion
• To load balance the processes
• To check the statuses of the processes
• We need a docker ps at the cluster level
Introducing CoreOS
• A minimal Linux distribution (small footprint)
• Designed to handle cluster
• Providing cluster tooling
• Docker
• etcd
• fleet
https://coreos.com/
etcd : service discovery
• etcd is
• small distributed key value store
• keys may have TTL
• using Raft protocol to elect cluster leader
• suitable for :
• shared configuration
• service discovery
fleet : clustered init system
• fleet is :
• an init system at the cluster level
• compatible with systemd unit files
• suitable for :
• launching docker containers
• distributing load across cluster
• get the status of the jobs
fleet demo
• Deploy our nuxeo container in the cluster
Challenge status
• Build a scalable infrastructure
• Quick provisioning (less than a minute)
• Easy setup (no admin needed)
• Easy upgrade process
• Automated healing
• Easy to test
What are the admin tasks ?
• Launch the fleetctl command
• Setup DNS and reverse proxies
• Link container to external services (S3, Postgres etc…)
DNS and reverse proxy
• docker start containers and link host port to container port
• The cluster has to redirect calls to mycustomDNS.io to the running container meaning knowing
• the ip of the host
• the port on the host
Introducing Gogeta
!/services/NXIO-0001/location:{host:172.31.51.1,port:49153}! /status/current:started! /expected:started!!! ! ! ! ! /alive:1!/domains/mydomain.nuxeo.io/type:iocontainer! /value:NXIO-0001
Given this etcd configuration
ClusterCoreOS
172.31.51.20
etcd
io container
NXIO-0098
systemd
CoreOS 172.31.51.1
etcd
io container
NXIO-0001 Gogeta
systemd
… … … …
Fleet
…
Gogeta
…
Introducing GogetaRound robin load Balancer
etcd !/services/NXIO-0098/location:{host:172.31.51.20,port:49153}! /status/current:started! /expected:started!!! ! ! ! ! /alive:1!/domains/mydomain.nuxeo.io/type:iocontainer! /value:NXIO-0001
ClusterCoreOS
172.31.51.20
etcd
io container
NXIO-0098
systemd
CoreOS 172.31.51.1
etcd
io container
NXIO-0001 Gogeta
systemd
… … … …
Fleet
…
Gogeta
…
Introducing GogetaRound robin load Balancer
etcd !/services/NXIO-0098/location:{host:172.31.51.20,port:49153}! /status/current:started! /expected:started!!! ! ! ! ! /alive:1!/domains/mydomain.nuxeo.io/type:iocontainer! /value:NXIO-0001
ClusterCoreOS
172.31.51.20
etcd
io container
NXIO-0098
systemd
CoreOS 172.31.51.1
etcd
io container
NXIO-0001 Gogeta
systemd
… … … …
Fleet
…
Gogeta
…
Introducing GogetaRound robin load Balancer
etcd !/services/NXIO-0098/location:{host:172.31.51.20,port:49153}! /status/current:started! /expected:started!!! ! ! ! ! /alive:1!/domains/mydomain.nuxeo.io/type:iocontainer! /value:NXIO-0001
ClusterCoreOS
172.31.51.20
etcd
io container
NXIO-0098
systemd
CoreOS 172.31.51.1
etcd
io container
NXIO-0001 Gogeta
systemd
… … … …
Fleet
…
Gogeta
…
Introducing GogetaRound robin load Balancer
etcd !/services/NXIO-0098/location:{host:172.31.51.20,port:49153}! /status/current:started! /expected:started!!! ! ! ! ! /alive:1!/domains/mydomain.nuxeo.io/type:iocontainer! /value:NXIO-0001
ClusterCoreOS
172.31.51.20
etcd
io container
NXIO-0098
systemd
CoreOS 172.31.51.1
etcd
io container
NXIO-0001 Gogeta
systemd
… … … …
Fleet
…
Gogeta
…
Introducing GogetaRound robin load Balancer
etcd !/services/NXIO-0098/location:{host:172.31.51.20,port:49153}! /status/current:started! /expected:started!!! ! ! ! ! /alive:1!/domains/mydomain.nuxeo.io/type:iocontainer! /value:NXIO-0001
ClusterCoreOS
172.31.51.20
etcd
io container
NXIO-0098
systemd
CoreOS 172.31.51.1
etcd
io container
NXIO-0001 Gogeta
systemd
… … … …
Fleet
…
Gogeta
…
Introducing GogetaRound robin load Balancer
etcd !/services/NXIO-0098/location:{host:172.31.51.20,port:49153}! /status/current:started! /expected:started!!! ! ! ! ! /alive:1!/domains/mydomain.nuxeo.io/type:iocontainer! /value:NXIO-0001
Introducing Gogeta
• Easy to start as a docker container
!
!
• Written in Go
• Available at : http://github.com/nuxeo/gogeta
/usr/bin/docker run --rm --name gogeta -p 7777:7777 nuxeo/gogeta
Application startup unit!
[Unit]!
Description=Sample startup script!
!
[Service]!
ExecStartPre= Set starting status in etcd!
ExecStart= Launch docker container!
ExecStartPost= Wait container to be launched and set app location in etcd!
ExecStop= Stop docker container!
ExecStopPost= Set stopped status in etcd!
Linking services
• Docker recommend to use ambassadors to link services
• Service may be hosted on other servers or outside of the cluster
• We’ve implemented dynamic ambassadors
• http://coreos.com/blog/docker-dynamic-ambassador-powered-by-etcd/
ClusterCoreOS
172.31.51.20
etcd
Postgres
systemd
CoreOS 172.31.51.1
etcd
io-container NXIO-0001
systemd
… … … …
Fleet
postgres-amb
Dynamic ambassador
etcd !/services/postgres/location:{host:172.31.51.20,port:49159}!
https://github.com/nuxeo/etcd-netfw
ClusterCoreOS
172.31.51.20
etcd
Postgres
systemd
CoreOS 172.31.51.1
etcd
io-container NXIO-0001
systemd
… … … …
Fleet
postgres-amb
Dynamic ambassador
etcd !/services/postgres/location:{host:172.31.51.20,port:49159}!
https://github.com/nuxeo/etcd-netfw
ClusterCoreOS
172.31.51.20
etcd
Postgres
systemd
CoreOS 172.31.51.1
etcd
io-container NXIO-0001
systemd
… … … …
Fleet
postgres-amb
Dynamic ambassador
etcd !/services/postgres/location:{host:172.31.51.20,port:49159}!
https://github.com/nuxeo/etcd-netfw
Dynamic ambassador
• Very easy to start
!
• Uses https://github.com/nuxeo/etcd-netfw
docker run --rm --name postgres-amb -P nuxeo/service-amb postgres-service
Launching commands
• A manager web app now launches commands :
• sets up basic etcd values (like domain to service resolution)
• launches fleet commands (start & stop apps)
• polls statuses
Challenge status
• Build a scalable infrastructure
• Quick provisioning (less than a minute)
• Easy setup (no admin needed)
• Easy upgrade process
• Automated healing
• Easy to test
Let’s resume
From nuxeo.io…
• We’ve built a cluster infrastructure that :
• knows how to deploy Docker containers (with fleet)
• Expose those containers thru domain name resolution (with gogeta)
• Link dependent services to the container (with etcd-netfw)
From nuxeo.io…
• Given that a container should represent an application
• Running this apps means :
• Launch it with runtime parameters
• Link it to its dependent services
• Then we can build a metamodel around it
…
name: io-container!version: 1.0!params:! - env_tech_id: ! - description: The id of the environment to run! - type:string! - clid:! - description: a registered connect client ID! - type:string!!buildimage:! path: "./docker"!!dependencies:! - postgres.service! - s3.service!!
… to arken
• to be able to run any application
!
• it will take care of :
• Submitting the model to the cluster
• Building the image if needed
• Launching the container
• Registering everything in etcd
arkenctl start iocontainer.ark —domain myapp.mycluster.io
Arken status
• We are extracting things from http://github.com/nuxeo/nuxeo.io
• http://github.com/arkenio
• home of gogeta and etcd
• provide cluster init instruction
• arken and arkenctl tools needs to be done
• new opensource projects (not in nuxeo repo)
• pull requests are welcome
Conclusion
• Docker helped us to enter the micro containers world
• Lots of new features : multi-repo, automatic /etc/hosts etc…
• CoreOS provided us cluster deployment
• Arken wants to provide a higher deployment model
Time for questions@damienmetzler