docker at devtable

Docker at DevTable

What is DevTable?

DevTable is a browser-based, hosted, collaborative IDE

Develop in the cloud with the same power as your desktop applications

Code

Collaborate

Debug and Test

Deploy

● Google App Engine● REST● SCP● Git (Heroku and other providers)

Sealed evil in a can

There are a lot of neat things that we run for our users, but they are all potentially very dangerous:

● App Engine Development Server● Debuggers and Emulators● REPLs (Python, etc)● Terminal support (which means all of the

above as well)

Why this is a problem

● Without a containment system of some kind, any of these awesome features would allow users to cause mayhem:

○ A REPL use could open any file○ A DevServer can execute arbitrary code○ A terminal could allow anything to happen

Why not simply use permissions?

● Permissions solve the file access problem● Permissions do not prevent users from

causing other system issues: instability, exhaustion, escalation, etc

Solution: containers!

To contain the insecurity of running live code, we run all non-custom code in a container, with only the user’s project mounted and available

Evil (not to scale)

Project data

Container

Ideal container properties

● Lightweight● Secure● Easy to manage● FAST

Originally we used LXC...

● Lightweight (sort of…)● Secure● Easy to manage (sort of...)● FAST

In the beginning, there was LXC...

… and it was slow.

● Typical startup times for our containers were on the order of minutes

● Starting a debugger or shell is not fun at those speeds

● Getting the security and management just right was quite painful

Then the community said “let there be Docker”...

Yo!

… and it made things amazing.

Our average startup time for a container has dropped from over a minute to just under

four seconds.

LXC

Docker

Go make a cup of coffee and play swords on office chairs

Go!

Before Docker

But, but Docker is just... LXC...

Almost, Docker does some things that make starting up single processes lightning quick:

● Incremental by default● Replace distro init process with lightweight

version● No DHCP, upstart, dnsmasq, etc.● Aufs seems to be faster than OverlayFS● Build process is MUCH better (Dockerfiles)

Docker at DevTable

The fun technical details!

DevTable overview

Clients

Web browsers

Clients - Web

browsers

Frontends

Python

Clients - Web

browsers

Backends

C#

WebSocket Socket

DFS

Clients - Web

browsers

Container Servers

Python

Thrift

SSH

HTTP

?

Images

Things we’ll discuss today

Clients

Web browsers

Clients - Web

browsers

Frontends

Python

Clients - Web

browsers

Backends

C#

WebSocket Socket

DFS

Clients - Web

browsers

Container Servers

Python

ThriftHTTP

SSH

?

Images

How we use docker now

● Python Docker API bindings● Run a single instance per project● Mount only the files relevant to the project in

the container● Run an SSH “command and control” process● Execute user processes through SSH● Dynamic version of Docker port forwarding

Backend <-> Container server

Backends

C#

Container Servers

Python

Thrift

Container server

The container server is the server in charge of managing all aspects related to the Docker containers

● Written in Python● Conforms to a Thrift interface● Called by the Backends to start containers,

stop containers, run commands, mount file systems in containers, etc

Container server

startContainer Starts a new container for a project.

runCommand

Runs a command inside a container

stopCommand

Stops a command inside a container

notifyFilesModifed

Notifies a container that a file has been modified by the backend

stopContainer

Stops a container

Handling file changes

● Changes made by the container or the backend to the DFS are propagated automatically

● However, both sides have code that depends on notification of changes

● Each server notifies the other about changes that occur via a notification service

DFS change notifications

Backend

C#

Container Server

Python

Hey, a user added file “test.txt” in container 1234

Backend

C#

Container Server

Python

Hey, the user changed file “foo.py” in container 1

How we handle file changes in Docker

● The container server watches changes inside the container using inotify, and reports changes to the backend

● The backend reports changes to the container server which will touch files that have been added or changed

Container server <-> Docker

Clients - Web

browsers

Container Servers

Python

SSH

Container server <-> Docker

We use the Python Docker bindings to create a new image and load it with a temporary ssh key

New container requests bring up the container with the known session SSH key and issue commands to the container via SSH

Much better than LXC issuing commands via subprocess

Docker <-> Outside world

For many services we run (such as the App Engine Development Server), we need to expose the server running inside Docker to the outside world

Docker <-> Outside world

HTTPClients

Web browsers

HTTP

Container Server

HAProxy

Docker <-> Outside world

Services inside of Docker as exposed via dynamic port mapping to a HAProxy running on the container server

The HAProxy exposes the port by remapping it to the external port and a custom subdomain

Docker <-> Outside world

Container Server

93nx83ndsc34mn.c4.devtable.io:80Clients

Web browsers

Port 38563

HAProxy

Example: running a dev server

1. Backend requests a container from the server

Backend

C#

Container Server

Python

I need a container for project “testapplication”

Container “container1234” started for project

Example: running a dev server

2. Backend registers for file notification events

Backend

C#

Container Server

Python

Let me know if any files change

Duly noted

Example: running a dev server

3. Backend asks for the dev server to be started and port 80 to be forwarded

Backend

C#

Container Server

Python

Please start the dev server and forward port 80

Dev server started and port is forwarded at subdomain foobarbaz

Example: running a dev server

1. Container server tells Docker to start a container

Container Server

Python

create_container, mount_filesystem, forward_port, start_ssh

Done. Port exposed: 84639

Example: running a dev server

2. Container server tells HAProxy to forward the port returned by docker

Container Server

Python

Forward port 84639 as subdomain foobarbaz

HAProxy

Example: running a dev server

3. Container server tells Docker to run the dev server

Container Server

Python

ssh command_for_devserver

Summary

Docker has allowed DevTable to run amazing tools securely and fast, without a large management overhead

Future opportunities

Docker presents some amazing new opportunities for DevTable and the community:

● Ability to quickly load (and save) complete development environments, securely

● Ability to quickly write custom plugins and run them in our IDE (want to analyze and build Go? just give us a URL or a Dockerfile!)

But wait…

There’s something that has been bugging us…

How should we distribute our private images in production?

Quay Demo

At this point in the live talk we unveiled and gave a demo of our hosted private docker registry called Quay.io.

Questions? Comments? Witty anecdotes?

devtable.com

Jacob Moshenko - jake@devtable.comJoseph Schorr - jschorr@devtable.com