understanding and building your own docker

Building blocks of Linux Containers

Motiejus [email protected]

@mo kelione

2016-11-18

Table of Contents

IntroductionWhy meA container in Linux is...

NamespacesIsolation in LinuxWhat did we just do

File systems and COW

What did we forget?Leftover elephants in the room

The End

Conclusion!

Devil Hides in The Details

I Many use Docker.I We lack time to understand.

I You need to understand infra to successfullytroubleshoot infra.

I There are trade-offs in the configuration.

I Make container engine in 30 minutes.

I Details!

→ You will still pick existing tools.

Conclusion!

Devil Hides in The Details.

I Details!

Conclusion!

Devil Hides in The Details?

I Details!

Conclusion!

I Many use Docker.

I We lack time to understand.

I Details!

Conclusion!

I Details!

Conclusion!

I Details!

Conclusion!

I Details!

Conclusion!

I Details!

Conclusion!

I Details!

Conclusion!

I Details! → You will still pick existing tools.

Why me

My resume: oncall experience.

I 2009− 2012 Telecom (Dev + Ops).

I 2012− 2014 Online Gaming (Dev + Ops).

I 2014− 2016 Amazon (Dev + Ops).I 2016− now Uber (Dev + Ops):

I From 2016.02: Dev.I From 2016.11: SRE.

I had to understand how exactly infrastructureworks.

Why me

My resume: oncall experience.

I 2009− 2012 Telecom (Dev + Ops).

I 2012− 2014 Online Gaming (Dev + Ops).

I 2014− 2016 Amazon (Dev + Ops).I 2016− now Uber (Dev + Ops):

I From 2016.02: Dev.I From 2016.11: SRE.

I had to understand how exactly infrastructureworks.

A container in Linux is ...

Fork/exec with bells & whistles:

I Fancy tarball for distribution.

I COW filesystem to make it start fast.

I Cgroups for fairness.

I Namespaces for isolation.

Table of Contents

The End

We will cover

I User namespaces.

I Pid namespaces.

I Mount namespaces.

I Network namespaces.

I There are more, but not today.

We will cover

I User namespaces.

I Pid namespaces.

I Mount namespaces.

We will cover

I User namespaces.

I Pid namespaces.

I Mount namespaces.

We will cover

I User namespaces.

I Pid namespaces.

I Mount namespaces.

We will cover

I User namespaces.

I Pid namespaces.

I Mount namespaces.

We will cover

I User namespaces.

I Pid namespaces.

I Mount namespaces.

User namespace

Become container-local root.unshare --map-root-user

Mount namespace

Hide container mounts.unshare --mount

Pid namespace

Hide other pids.unshare --pid --mount-proc --fork

Network namespace

Demonstrate this:

I Create namespace.

I Activate loopback (lo).I Create pair of devices veth1a and veth1b:

I veth1b will go to the namespace.I veth1a will stay in default.

I Add ip addresses.

I curl and ping.

I lsof, bind on ports separately.Ever wanted to run tcpdump on an application?

Network namespace

Demonstrate this:

I Create namespace.

I Add ip addresses.

I curl and ping.

Network namespace

Demonstrate this:

I Create namespace.

I Activate loopback (lo).

I Create pair of devices veth1a and veth1b:

I Add ip addresses.

I curl and ping.

Network namespace

Demonstrate this:

I Create namespace.

I Add ip addresses.

I curl and ping.

Network namespace

Demonstrate this:

I Create namespace.

I veth1b will go to the namespace.

I veth1a will stay in default.

I Add ip addresses.

I curl and ping.

Network namespace

Demonstrate this:

I Create namespace.

I Add ip addresses.

I curl and ping.

Network namespace

Demonstrate this:

I Create namespace.

I Add ip addresses.

I curl and ping.

Network namespace

Demonstrate this:

I Create namespace.

I Add ip addresses.

I curl and ping.

Network namespace

Demonstrate this:

I Create namespace.

I Add ip addresses.

I curl and ping.

I lsof, bind on ports separately.

Ever wanted to run tcpdump on an application?

Network namespace

Demonstrate this:

I Create namespace.

I Add ip addresses.

I curl and ping.

Network namespace

default

127.0.0.1

192.0.2.1

127.0.0.1

veth1a veth1bveth1a

10.0.0.1

veth1b

10.0.0.2

Network namespace

default

127.0.0.1

192.0.2.1

127.0.0.1

veth1a veth1bveth1a

10.0.0.1

veth1b

10.0.0.2

Network namespace

default

127.0.0.1

192.0.2.1

127.0.0.1

veth1a veth1bveth1a

10.0.0.1

veth1b

10.0.0.2

Network namespace

default

127.0.0.1

192.0.2.1

127.0.0.1

veth1a veth1b

veth1a

10.0.0.1

veth1b

10.0.0.2

Network namespace

default

127.0.0.1

192.0.2.1

127.0.0.1

veth1a veth1bveth1a

10.0.0.1

veth1b

10.0.0.2

What did we just do

Created a container:

User namespace apt-get, iptables, mount, etc.

Isolated pids no nobody, isolate from each other.

Isolated mounts e.g. for /tmp.

Isolated network safely bind to :80.

An improvement over ”run and hope it doesn’taffect anything else”.

What did we just do

Table of Contents

The End

A container:

I Needs a file system.

I Starts quickly regardless of size.

Do not want to copy 1GB with every startup.Copy On Write!lvm? zfs? btrfs?

A container:

Do not want to copy 1GB with every startup.

Copy On Write!lvm? zfs? btrfs?

A container:

Do not want to copy 1GB with every startup.Copy On Write!

lvm? zfs? btrfs?

A container:

A quick demo

I Create tank/images/debian@latest

I Create tank/containers/t1 from @latest

I unshare --mount --pid --fork chroot . bash

Table of Contents

The End

Leftover elephants in the room

I Trivial to escape this ”container”.

I Sec: no leftover file descriptors.

I Resource fairness.

I Sec/DoS: shared kernel resources.

I Supervision, daemonization and cleanup.

I Logging.

I Collect zombie processes.

I Image management.

Should someone else do it?

I Logging.

I Image management.

I Logging.

I Image management.

I Logging.

I Image management.

I Logging.

I Image management.

I Logging.

I Image management.

I Logging.

I Image management.

I Logging.

I Image management.

I Logging.

I Image management.

I Logging.

I Image management.

We almost have a container engine

I But look at my conclusions again.

I Devil hides in the details.

I Tooling companies (Docker, CoreOS, etc)raised > $108.

To recap

I Easy to understand kernel facilities.

I Either spend a lot of time and headache, orre-use existing tools.

To recap

Table of Contents

The End

We’re hiring!

Uber SRE locations: SF, NYC, Seattle, Vilnius.

I Check out join.uber.com

I Also, contact me at [email protected]

understanding and building your own docker

Software