building docker containers @ scale

58
Building containers Go-Toronto Nov/2016 @lxfontes

Upload: lxfontes

Post on 23-Jan-2018

532 views

Category:

Engineering


4 download

TRANSCRIPT

Page 1: Building Docker Containers @ Scale

Building containers

Go-TorontoNov/2016

@lxfontes

Page 2: Building Docker Containers @ Scale

#bfcm

Page 3: Building Docker Containers @ Scale

Continuous Delivery Pipeline 1-1

BuildGit Push Test Deploy

Page 4: Building Docker Containers @ Scale

Your company keeps growing

Page 5: Building Docker Containers @ Scale

Growing

Page 6: Building Docker Containers @ Scale

Growing MOAR...

Page 7: Building Docker Containers @ Scale
Page 8: Building Docker Containers @ Scale

Clone repository

docker build

- Install RubyGems

- Install NPM Packages

- Compile static assets

- Seal Docker image

docker push

- Upload to Docker Registry

Building a Rails Container

Page 9: Building Docker Containers @ Scale

InertiaYou get used to it

Your build time only goes up

Number of workers only goes up

“Our build is complex”

“We already use Docker, the Cloud. This is as fast as it gets!”

“It is automated, we don’t care it takes 30 minutes”

“The XYZ team should fix it”

Page 10: Building Docker Containers @ Scale

it was a reality check20 minutes per build

Flakiness

Resource Starvation

Really expensive

* where do you put secrets??????

Page 11: Building Docker Containers @ Scale

Why is it slow?

push webhook

worker bootstrap

docker build cache not reused between builds

cold asset precompilation

2nd build

Page 12: Building Docker Containers @ Scale

peed

Page 13: Building Docker Containers @ Scale

Where are we: Daily stats

+200Devs

+700Builds

+42kTests

per build

~3Images

per build

Page 14: Building Docker Containers @ Scale

Commitment

Container

BuildGit Push

Automated

TestsDeploy

5s 90s 200s 300s

From Git Push to Deployed in ~10 minutes

Every developer can deploy to productionGoals:

Testing Rails at

Scale by Emil

How We Deploy

Shopify by KatThis right here

Page 15: Building Docker Containers @ Scale

push 30 sec POLL

Pull / Build

Coordinator+ API

zookeeper

workers

registry

branch affinity

Page 16: Building Docker Containers @ Scale

ohai bootstrap

Page 17: Building Docker Containers @ Scale

Clone repository

docker build

- Install RubyGems

- Install NPM Packages

- Compile static assets

- Seal Docker image

docker push

- Upload to Docker Registry

Building a Rails Container

Page 18: Building Docker Containers @ Scale

Dockerfile

Page 19: Building Docker Containers @ Scale

mo layers, mo problems

* apparently docker 1.13can squash these

Page 20: Building Docker Containers @ Scale

Look ma, no Dockerfilesprepare

Install OS packages

precompile

Load (ejson) secrets

Populate artifact cache

compile

Combine app + artifact cache

Seal image

Daily seed 20min

Builds 2minInheritCache

InheritCache

Page 21: Building Docker Containers @ Scale

Docker LayersBase

Ubuntu-ish

Prepare

OS packages

Intermediate (Daily seed)

App + Libraries

Final

Delta between Intermediate … Current

Page 22: Building Docker Containers @ Scale

Locutus trade offsFAST

Local caches

Secure

Stable

But needs love

We need to maintain:

Infrastructure

Orchestration

Web UI

Scripts Copy & Pasta

Dockerfiles / Buildpacks :(

Troubleshooting :(

Page 23: Building Docker Containers @ Scale

Pipa* Kite in Portuguese(and sounds cool)

Page 24: Building Docker Containers @ Scale

Provider agnostic

Disposable

Secure

Sandboxed Docker Daemon (dind)

Pre-made recipes

Buildpack

Dockerfiles

Locutus Assembly

Pipa: RequirementsBuild whatever you want

Cache whatever you want

Repeatable

Run locally if needed

As fast as Locutus

Page 25: Building Docker Containers @ Scale

Is there an app fo dat?Orchestration / UI

Docker first

On-premise

Worker fully sandboxed

Concurrency Primitives

“Only 3 builds for app X at same time”

Parallelism

“Send job to 30 workers”

Page 26: Building Docker Containers @ Scale

BuildkiteOn-premise

Orchestration *only*

Github Hooks

UI / Reporting

API

Agent is a single binary (Go)

Also distributed as docker container

Concurrency Primitives

Parallelism

Per Build artifacts

Per Build shared key-value store

Switch workers!!

No batteries included … on purpose

Page 27: Building Docker Containers @ Scale

Buildkite Pipeline Sample - buildkite/sample-pipelines

Waits for previous stepto complete

Requires InteractionUI/API/slack

Branch filtering

Page 28: Building Docker Containers @ Scale

Can it scale?

Page 29: Building Docker Containers @ Scale

BuildkiteGitHub BK PipaDocker

RegistryBK Tests

Webhook Build Container

Switch Worker

Start Tests

Fetch Image

Tests DoneShip It!

Tests

Upload Image

100 cores 8820 coresPipa: Goal

Page 30: Building Docker Containers @ Scale

Pipa: Architecture goalsEverything exposed via environment variables

PIPA_SSH_KEY / PIPA_APP_COMMIT_ID

Chainable (simplify IF/ELSE ops)pipa build --unless v1 -- docker build -t myimage:v1 .

Configurable via environment, config file, argument switchesPIPA_APP_NAME=myapp pipa build ...

pipa build --app-name myapp …

Isolated Docker Daemon (docker-in-docker)Reset after each buildKubernetes: external SSD / Local: give it a directory

Page 31: Building Docker Containers @ Scale

Environment / Process Tree

pipa

wrapper

AWS / GCS / SSH Keys

Kubernetes secretsenv vars with paths

APP Name

Buildkite / Jenkins / etcGIT (local)

APP SHA

Buildkite / Jenkins / etcGIT (local)

Your code

Signals

Page 32: Building Docker Containers @ Scale

Prototype: I’m going to write this all in bash

Page 33: Building Docker Containers @ Scale

Single Binary

Shell files as Assets

using go-bindata

docker / aws-cli / rsync

Page 34: Building Docker Containers @ Scale
Page 35: Building Docker Containers @ Scale

LOL

Page 36: Building Docker Containers @ Scale

BUT, normalized environment is solid!

Page 37: Building Docker Containers @ Scale

Provider agnostic

Disposable

Secure

Sandboxed Docker Daemon (dind)

Pre-made recipes

Buildpack

Dockerfiles

Locutus Assembly

Pipa: RequirementsBuild whatever you want

Cache whatever you want

Repeatable

Run locally if needed

As fast as Locutus

Page 38: Building Docker Containers @ Scale

Toolbox: spf13/cobraCLI handlingdocker, kubernetes, rkt, etcd

ABUSE: func init()

Page 39: Building Docker Containers @ Scale

Toolbox: kelseyhightower/envconfig *spf13/cobra deficiency

Fills struct with env vars

Page 40: Building Docker Containers @ Scale

Why both?

spf13/cobra

kelseyhightower/envconfig

Page 41: Building Docker Containers @ Scale

Result

Google Cloud Storage

Amazon S3

JSON Asymmetric Encryption

Shopify/ejson

Registry Interaction

push / pull / exists

mmap diff 2 dirs

burke/treediff

Environment normalizer (entrypoint)

Page 42: Building Docker Containers @ Scale

Using itEverything is namespaced!

pipa cache pull /tmp/artifacts

pipa build --if base -- docker build -t $PIPA_IMAGE_FULL_NAME .

pipa image exists --local --tag basepipa image exists --remote --tag base

Downloads s3://artifacts/app_group/app_env/app_name/cache.tar to /tmp/artifacts

Only runs command if registry/app_group/app_env/app_name:base exists

Checks if registry/app_group/app_env/app_name:base exists (locally/remotely)

Page 43: Building Docker Containers @ Scale

Pipeline InferenceHow should pipa build your app?

Customize it in your repo; or

Select a template; or

Let pipa figure it out!

Page 44: Building Docker Containers @ Scale

Built-in pipelines

Page 45: Building Docker Containers @ Scale

Pipeline Output

Page 46: Building Docker Containers @ Scale

PipelineParallelization

3 parallel builds

Page 47: Building Docker Containers @ Scale

Pipeline SelectionCustom script in repo

Custom pipeline in repo

Default pipeline in repo

User requested specific pipeline

Check for common file locationsrepo/Dockerfile? -> dockerrepo/borg? -> borg

Default to Heroku Buildpack

PIPA_PIPELINE_CMD

PIPA_PIPELINE_FILE

PIPA_PIPELINE_TEMPLATE

repo/.buildkite/pipeline.yml

Page 48: Building Docker Containers @ Scale

High Level

push webhook

Agent pollSSL

dockerimages

cacheartifacts

QueueAPI

User Interface

Page 49: Building Docker Containers @ Scale

ssh-keygcloud-keybuildkite-tokendocker-auth

Sec: Kubernetes Layout

EJSONpriv/pub keypairs

/builder-secrets

/app-secrets

/var/lib/docker (dind)

/builds (git clone)

/cache (artifacts)

6x nodes

24 pods

/mnt/disks/ssd0/builder1/docker

Page 50: Building Docker Containers @ Scale

Provider agnostic

Disposable

Secure

Sandboxed Docker Daemon (dind)

Pre-made recipes

Buildpack

Dockerfiles

Locutus Assembly

Pipa: RequirementsBuild whatever you want

Cache whatever you want

Repeatable

Run locally if needed

As fast as Locutus?

Page 51: Building Docker Containers @ Scale

Gems

Page 52: Building Docker Containers @ Scale
Page 53: Building Docker Containers @ Scale

Tar + Gzip

Page 54: Building Docker Containers @ Scale

Go + Docker = fsouza/go-dockerclient

Respect environment variables

Reads $HOME/.docker/config.json or $HOME/.dockercfg

Page 55: Building Docker Containers @ Scale

Rocker - grammarly/rockerDockerfile on steroids

Page 56: Building Docker Containers @ Scale

Herokuish - gliderlabs/herokuishIf you like Heroku, you will want this

Packaged as docker container or single binary

Page 57: Building Docker Containers @ Scale

alpinelinux.org

based on musl libc and busybox

Lots of *up-to-date* packages (yes debian, looking at you)

Page 58: Building Docker Containers @ Scale

Demo Time