solving real world production problems with docker
TRANSCRIPT
SOLVING REAL WORLD PRODUCTION PROBLEMS WITH DOCKER
OCTOBER 11TH, 2016
DOCKER MEETUP, LOS ANGELES
MARC CAMPBELL@mccode
GOALS
• Review Docker features that enable a more reliable, secure production environment
• Present a secure build-deliver-execute process that includes Docker in production
• Provide solutions you can start using today
“The only difference between a process in a container and a process not in a container is a few labels on top of a process that say ‘this is in container X’”
Jérôme Petazzoni, Docker July 06, 2015
SECURE DELIVERY PIPELINE
BUILD DELIVER EXECUTE
BUILD DELIVER EXECUTE
BUILD DELIVER EXECUTE
BUILD Choosing and creating container images that will run in your production environment.
THREE DIFFERENT ROLES, EQUALLY IMPORTANT
Operations
Development
Security
Does it work?
Can it be supported?
Can it be safely run?
Is it Elasticsearch? I don’t want “elasticsearch-like”, I want Elasticsearch and I want version 2.4.1.
Will it send alerts when it breaks? Does it support zero-downtime upgrades?
There are 2,532 Elasticsearch containers in DockerHub. Why this one?
BUILD DELIVER EXECUTE
• Development images do not have to be the same as production images
• Prefer library (official) images when possible
BUILD DELIVER EXECUTE
• Always look at the Dockerfile, regardless of pull count
• Be cautious when bind mounting the docker.sock file
BUILD DELIVER EXECUTE
Best practices- whitelist (or choose) base images - don’t trust “pull count” from DockerHub, find and read the dockerfile - use the most specific tag possible
redis:sha256@abc > redis:3.2.4 > redis:3.2 > redis:3 > redis:latest - adopt a tagging pattern for your own images - use security scanning (coreos clair or dockerhub) - use docker content trust
BUILD DELIVER EXECUTE
Monitor images with DockerHub Security Scanning or CoreOS Clair
The current nginx container on DockerHub has:
13 Critical CVEs 23 Major CVEs
Including 1 CRITICAL OpenSSL CVE
BUILD DELIVER EXECUTE
BUILD DELIVER EXECUTE
DELIVER Ensure the images you want to run are the images you are running
BUILD DELIVER EXECUTE
I typed `docker run redis` so now i’m running redis…right?…right???
BUILD DELIVER EXECUTE
What happens when you type `docker run redis`
DOCKER RUN REDIS
REDIS:LATEST IMAGE EXISTS?
CREATE REDIS CONTAINER
PULL REDIS:LATEST
START REDIS CONTAINER
NO
YES
BUILD DELIVER EXECUTE
BUILD DELIVER EXECUTE
DOCKER RUN REDISDOCKER CLI
DOCKER ENGINE
DOCKER HUB
CREATE NO IMAGE PULL
GET /V2
PARSE HEADER
Trust Boundaries
401 AUTH REQUIRED
POST /LOGIN
GET /V2/…/MANIFEST
GET /V2/…/LAYER
IMAGE COMPLETE CREATE START
Connect to a trusted host
Deliver the content over a secure channel
Sent the content you requested
Verify the author of the content
A.
B.
C.
D.
To securely download data from the Internet
BUILD DELIVER EXECUTE
HTTPS
TLS
Content Addressable IDs
Signed Images
The problems The solutions
Downloading and executing software from the Internet is dangerous
Don’t download from untrusted hosts. e.g.: `docker pull [—insecure-registry] 52.207.178.113/redis:latest` Don’t download on insecure channels. e.g.: `docker pull [—insecure-registry] registry.mycompany.com/redis:latest`
Don’t trust the remote server to look up the content. e.g.: `docker pull redis:latest`
Don’t trust content that isn’t signed by the publisher. e.g.: `docker pull --disable-content-trust redis:latest`
1.
2.
3.
4.
BUILD DELIVER EXECUTE
Docker Content Trust
“Content trust gives you the ability to verify both the integrity and the publisher of all the data received from a registry over any channel”
BUILD DELIVER EXECUTE
$ docker pull redis Using default tag: latest latest: Pulling from library/redis
6a5a5368e0c2: Pull complete <...> 2bcdfa1b63bf: Pull complete Digest: sha256:38e873a...912 Status: Downloaded newer image for redis:latest
WITHOUT TRUST: PULL BY TAG
BUILD DELIVER EXECUTE
$ export DOCKER_CONTENT_TRUST=1 $ docker pull redis Using default tag: latest Pull (1 of 1): redis:latest@sha256:c4365e...680 sha256:c4365ec...680: Pulling from library/redis
6a5a5368e0c2: Pull complete <...> 58e3d55f4ce5: Pull complete Digest: sha256:c4365e...680 Status: Downloaded newer image for redis@sha256:c4365e...680 Tagging redis@sha256:c4365e...680 as redis:latest
WITH TRUST: PULL BY SHA
BUILD DELIVER EXECUTE
BUILD DELIVER EXECUTE
DEMO
•Create a signed image •Run a signed image •Update the image from an untrusted source •Pull and run the new image
BUILD DELIVER EXECUTE
EXECUTE Provide a consistent, secure environment with continuous auditing
BUILD DELIVER EXECUTE
Center For Internet Security• Use AppArmor / SELinux • Enable Kernel Auditing • User namespaces • /var/lib/docker volume • Enable an authorization plugin • Use a centralized log driver • Prevent registry v1 access
https://benchmarks.cisecurity.org/tools2/docker/CIS_Docker_1.11.0_Benchmark_v1.0.0.pdf
BUILD DELIVER EXECUTE
Docker Bench for Securityhttps://dockerbench.com/
docker run -it --net host --pid host --cap-add audit_control \ -v /var/lib:/var/lib \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /usr/lib/systemd:/usr/lib/systemd \ -v /etc:/etc --label docker_bench_security \ docker/docker-bench-security
BUILD DELIVER EXECUTE
2.1 - Restrict network traffic between containers 2.2 - Set the logging level 2.3 - Allow Docker to make changes to iptables 2.4 - Do not use insecure registries 2.5 - Do not use the aufs storage driver 2.6 - Configure TLS authentication for Docker daemon * Docker daemon not listening on TCP 2.7 - Set default ulimit as appropriate * Default ulimit doesn't appear to be set 2.8 - Enable user namespace support 2.9 - Confirm default cgroup usage 2.10 - Do not change base device size until needed 2.11 - Use authorization plugin 2.12 - Configure centralized and remote logging 2.13 - Disable operations on legacy registry (v1)
[WARN] [PASS] [PASS] [PASS] [WARN] [INFO] [INFO] [INFO] [INFO] [WARN] [PASS] [PASS] [WARN] [WARN] [WARN]
BUILD DELIVER EXECUTE
Review☑ Choose images carefully ☑ Scan your Dockerfiles ☑ Enable Docker Content Trust ☑ Run Docker Benchmark for Security