docker bestpractices - rainfocus · 2019-06-27 · docker hub 7 04.04.2019 docker best practices...

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA

HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Docker BestPracticesfor MicroServices

Thomas Brö[email protected]

@tvdtb

Agenda

Docker Best Practices for MicroServices - Oracle Code2 04.04.2019

1. Docker Best Practices

2. MicroServices in Docker

3. Security

4. Pitfalls


Docker Best Practices

It‘s all about containers


Docker Basics: The two ways to use docker


docker run -d ...

Use Docker as a daemon

Docker isolates resources

Easy reset

docker run --rm -ti \

-v ${PWD}:/work -w /work ...

Use Docker as CLI tool wrapper

Don‘t install software any more, use images

Run everything in docker!

^P^Q to detach (if started using -ti)

^C to stop

Dockerfile: CMD Dockerfile: ENTRYPOINT

Single Responsibility


Docker container

daemon1

1=

Docker Hub https://hub.docker.com


Docker Hub is a great starting point

Most of these sources can be found at github

„docker inspect <image>“ and

„docker history --no-trunc <image>“

show the details of the image

Best Practices

Important images should

controlled by the project team

– everything which is running in

production

– base images

https://hub.docker.com/

Docker filesystem


Base Linux

5-100 MB Base distribution

Java Installation

90MB JDK 12

Base Image

50MB Runtime/Libraries

Application Image

1-x MB Application

Container

0-x MB Log files ...

Frozen

Image

Layers

Any change goes here

Docker filesystem


Changes apply to the container only

Changes are copy-on write

Opening the file for write access is

sufficient

Initial write to large files seem to block

the container

Best Practices:

Don‘t change image contents at

runtime

➢No software updates

➢Container contains what you expect

➢Less disk usage

Create well-prepared images

➢Better startup time

➢Less disk usage

Docker versions


Docker allows implicit versioning

:latest if no version is given

Hard to determine what‘s in the image

if :latest tag changed

Best Practices:

Use latest only if you don‘t care

Explicitly version your images

Correlate versions to SCM

➢Create tags

➢Use SCM commit id

➢Git history „distance“

FROM nginx

FROM nginx:1.15.5

Docker history (demo)


Handling a 7.74 MB file

IMAGE CREATED CREATED BY SIZE

COMMENT

abf4100168ab About an hour ago /bin/sh -c #(nop) ENV ENTRY2=two 0B

593d5b0661f0 About an hour ago /bin/sh -c #(nop) ENV ENTRY1=one 0B

cc23f0b1af4a About an hour ago /bin/sh -c rm /test2.zip 0B

6867ccb1d83b About an hour ago /bin/sh -c mv /test.zip /test2.zip 7.74MB

56eb7570f1e3 About an hour ago /bin/sh -c #(nop) COPY file:41e18487354e1a3f… 7.74MB

5cb3aa00f899 3 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B

<missing> 3 weeks ago /bin/sh -c #(nop) ADD file:88875982b0512a9d0… 5.53MB

Docker images


Frozen

Image

Layers

Base Linux

5-100 MB Base distribution

Java Installation

90MB JDK 12

Base Image

50MB Runtime/Libraries

Application Image

1-x MB Application

Container

0-x MB Log files ...

ADD - 50 MB

RUN – 5MB

RUN – 1kB

ENV- 0 MB

Docker images – command chaining


ENV chaining

RUN multiple commands

RUN echo "my first command" && \

echo "second command"

ENV JAVA_HOME=/opt/jdk12 && \

PATH=/opt/jdk12/bin:$PATH

Docker images – Build Cache


Docker builds bottom up

Each Dockerfile command (ADD,

COPY, ENV, RUN, ...) results in an

additional layer

Docker uses the build cache

The lowest layer which changes

causes a rebuild of all subsequent

layers

Best Practices:

Correct ordering of layers

➢Add frequently changing layers the

latest possible

Use (and don‘t clear) the build cache

➢ It saves time and disk space

Check caching behaviour

➢Only create new images if contents

change

Build Cache - Java


JAR plugin

copy-dependencies plugin

WAR plugin

FAT Jar Plugins

Best Practices:

copy dependencies

– no fat jars

create extracted WAR directories

– Do not use .war file

Jlink

– Reduces JDK size even when using

all modules

– MultiStage Build

Smiley Images from pixabay.com, user Pixaline

Docker software installation


Centos – don‘t yum update!

Always install & clean! – useless without chaining!

Debian – don‘t apt-get upgrade!

RUN apt-get update && \

apt-get install –y ...... && \

apt-get clean

RUN yum install –y ..... && \

yum clean all

Docker images – how to install software


Task:

Install software from files (non-

package manager)

– Archives

– Downloaded packages

– Installation files not in image

Best Practices:

Install software using trusted

repositories

Use all options of Docker commands

– ADD (extracts archives)

– COPY (e.g. chown)

Docker – install via download


Simple Web-Server:

Installation via Download (curl) and delete installation file!

RUN curl -s -o /tmp/oracle-instantclient.rpm "http://... " && \

yum -y install /tmp/oracle-instantclient.rpm && \

yum clean all && \

rm -f /tmp/oracle-instantclient.rpm && \

echo done installing software

docker run -d --name downloadserver -p 80:80 \

-v /path/to/your/downloads:/usr/share/nginx/html/ \

nginx:1.15.5

Docker – install - using MultiStage Build


Finally build (in the same Dockerfile) the final image

Create a temporary build image ...

FROM my-image:version as builder

ADD ...

RUN ...

RUN ...

...

FROM my-image:version

COPY --from=builder --chown=tomcat:tomcat \

/opt/tomcat $CATALINA_HOME

Why small images


Keep in mind

Small images might not be reusable

Many small images are bad as well

Making small images can be hard

Reduce disk storage usage for CI/CD

– Faster builds

– Less push & pull network transfer

Deployments run faster

– Shared files on the host

– Shared Layers for less effort

Less cleanup

Docker is daemon software


„docker build“ creates a build context (tar file) which is sent to the docker daemon

Everything is included - unless .dockerignore hides it

All arguments to launch a container refer to the docker host (normall local)

Docker CLI uses Docker REST api

Everything refers to the host

target/*

my-temp-file

*.tmp

temp?.tmp


MicroServices in Docker

Docker & MicroServices


Split your domain into Bounded

Contexts

Run multiple services

One image for each service

– Run on multiple stages and

machines

Best Practices

It‘s all about automation & tests

– Keep it simple

– Everything as code

Use Docker in development

– Test database

– Mock services

Build in Docker (agent)

Test in Docker

– Docker-Compose

– Load Tests

Docker MicroServices


Baseimage Java

MicroService

MicroService

MicroService

MicroService

Baseimage Java

Baseimage Java

Baseimage Java

>60MB

>60MB

>60MB

>60MB

Docker MicroServices


Baseimage Java

MicroService

MicroService

MicroService

MicroService

60MB

Baseimage Java

Baseimage Java

Baseimage Java

<1 MB

60MB

Dependencies

Dependencies

Dependencies

Dependencies

>60MB

Docker layout for MicroServices


Baseimage Java Dependencies

>60MB90MB70MB


>60MB90MB70MBMicroService

MicroService

MicroService

MicroService

<1 MB

Docker layout for MicroServices



Larger size

Higher frequency of change

60MB90MB70MB

MicroService

MicroService

MicroService

MicroService

<1 MB

MicroServices Development


Everything in Docker

Multi-Branch Development

Every „git push“ creates an image

Deploy automatically

Images are selected manually and

promoted to „production“ registry

– Including tagging the sources

Test & deploy to production

Image types

Backend - Java (in MicroService

layout)

Frontend – HTML5 (nginx)

Build Agents

Database including test data

Mock Containers (nginx)

Load Test Containers

Infrastructure (Login, Loadbalancer, ..)

CI / CD environment


Security

Container and security?


“Gartner asserts that applications deployed in

containers are more secure than applications

deployed on the bare OS [...] as long as a kernel

privilege escalation vulnerability does not exist on

the host OS”

(Joerg Fritsch, Research Director, Gartner, 2016)

Container and security?


Attacking process

Vulnerability in exposed service

Attack

– Privilege escalation

– Attack network

Common OS kernel


Host and container share the kernel

Vulnerabilities in the host kernel apply

to all containers

Best Practices:

Hardened systems

– Minimum software

Keep your docker host up-to-date

Update your docker base images from

a trusted source

Dangerous defaults & obvious leaks


Use secrets, no cleartext passwords

– No ENV entries

Use non-root users for execution, limit

privileges

Use cgroups to limit resource access (DoS)

Use only official images, better:

– Build your own, updated images

Minimize toolset – use different containers for

MultiStage build

Do not open your docker daemon

Build your own images!


Starting with the base image

– https://docs.docker.com/develop/develop-images/baseimages/

– Centos:

rpm --root <new-root> -ivh <centos-release>.rpm

rpm --root <new-root> \

--import $centos_root/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

yum -y --installroot=<new-root> --setopt=... install \

<place your package names here>

https://docs.docker.com/develop/develop-images/baseimages/

Build your own images!


Dockerfile

Docker Images are just directory contents

tar -C <new-root> cf centos.tar .

FROM scratch

ADD centos.tar

Docker MicroServices updates


Baseimage Java

MicroService

MicroService

MicroService

MicroService

Dependencies

Higher probability of known vulnerabilities

Java release cadence

8 releases/year

git push

frequency of non-security relevant changes

HOST


Pitfalls & Solutions

Docker – Pitfalls Terminal


Interactive session in installation

– „docker build“ runs without terminal connection

– Add „-y“ option to commands if available

„docker run“ without „-ti“

– Command silently exits

– No input / no output

„-ti“ but no terminal

– E.g. In cron jobs – results in an error

Pitfalls and BestPractices: Docker install


Pitfalls

Build Caches:

Dockerfile command is the build cache

key for RUN

Command chaining may result in large

and costly builds

Best Practices

Use unique names which change

when the contents change

No git clone without tag, Curl urls with

version identifier

Clear unnecessary files as soon as

possible

Use all options of ADD and COPY

Automate everything – no manual

steps

Docker – Pitfalls Java


Until Java 8u131 you have to specify maximum heap size

– Java unaware of cgroups limits

– Container will be killed by host OS due to memory overload

– Worse with more memory

– Worse with more containers

Especially critical when using RESTART=always

https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-

memory-limits

https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits

Docker – Pitfall 8.8.8.8


About: --dns

“…If the container cannot reach any of the IP addresses

you specify, Google’s public DNS server 8.8.8.8 is added,

so that your container can resolve internet domains.…”

Source:

https://docs.docker.com/config/containers/container-networking/#dns-services

https://docs.docker.com/config/containers/container-networking/#dns-services

Docker – Pitfall 8.8.8.8


Simple DNS:

Provide a DNS service – remember DNS is „AP“ in „CAP theorem“

$cat /etc/docker/daemon.json

{

"dns": ["192.168.12.100"]

}

yum install –y dnsmasq

systemctl enable dnsmasq

systemctl start dnsmasq

Docker – Pitfalls bind mounts


Bind mounts to non-existing host files/directories will create them

– by default as a directory

Changes to bind-mounted files on the host might not be visible in the container

– Bind mounts refer to their inode

– Some editors (depending on their config) change the file‘s inode

• Nano: OK

• Vi(m): Not OK

• ~/.vimrc: set backupcopy=yes


Summary

Summary


Single Responsibility

Create well-prepared images

Command chaining

Build Cache

Small but reusable images

Install

– Packages

– Via download

– Using MultiStage build

MicroServices image layout

Security Updates – host & image

Trusted image source

– Build your own


Questions and answers

Thomas Bröll

Principal Consultant

Trivadis

[email protected]

docker bestpractices - rainfocus · 2019-06-27 · docker hub 7 04.04.2019 docker best practices...

Documents