containers debugging docker - meetupfiles.meetup.com/19687526/debugging docker containers.pdf ·...
TRANSCRIPT
Debugging Docker ContainersTroy Folger, ISS Inc • 04.27.2016
Docker!The Brave New World of Containers
● Microservice approach- faster delivery- granular configuration and deployment- scalability, reliability, maintainability
● Image layering
● Container security- sandboxing / process protection- filesystem abstraction / mapping
● Platform independence
Features of Docker Containers
Docker!
Oh, wait ...
Debugging Challenges
Debugging Approaches
Contributions / Questions
● Indirect access to container processes and files
● Image layering (inheritance) can obscure issues in certain situations
● Inter-container communication- network implementations- varying, changing methods
● Platform differences (subtle and not-so-subtle)
● Loss of functionality / change of established practices“Who moved my cheese?”
Docker Debugging Challenges
General understanding of various container debugging approaches may be the single most helpful thing to know.
A. Container issue emergesB. DismayC. Revisit candidate debugging approaches …D. Google, baby, GoogleE. …F. Profit!
Docker Debugging Approaches
First rule: learn and apply docker image building and container running “best practices”.
Including, but not limited too:
● Microservice philosophy - one process / service per container● Simple Dockerfiles - use image layers to your advantage!● Consistent build approach● Understand volume mounting and use appropriately (!)● Understand and utilize “industry-standard” tools and practices (docker-compose,
docker-machine, etc)
Docker Debugging Approaches
“Best practices”, continued:
● Keep deployment / environment information separate from docker images● Keep in mind that not all deployments / use cases are the same - avoid over-
designing, use an iterative approach● Putting a poorly-designed application in a docker image does not magically make
the application elegant and reliable● Learn to love the command-line
Docker Debugging Approaches
Eclipse / IntelliJ / Other IDE tools
● Wait for it … wait …
Google. Seriously, these tools and setups change so frequently, the vendor web site and stackoverflow.com are going to be consulted at some point, anyway.
Docker Debugging Approaches
docker logs
● Interactive:$ docker logs CONTAINER | less$ docker logs CONTAINER > logfile
● Follow:$ docker logs -f CONTAINER
● Default log driver: JSON
Docker Debugging Approaches
docker attach
● Attach:$ docker attach [OPTIONS] CONTAINER
● Only really useful for containers running formatted stdout applications (think ‘top’)● Detach from the process via the default `ctrl-p ctrl-q` sequence
Docker Debugging Approaches
docker exec
● Console inside the container:$ docker exec -it CONTAINER /bin/bash
● Useful for running scripts INSIDE a container - example:$ docker run --name "platform" -v /usr/local/outer_dir/certs:/opt/issinc/certs \ -v /etc/localtime:/etc/localtime "platform/example_container" \ /bin/bash -c "cd /opt/issinc/certs ; /opt/scripts/test_certs.sh \ ca/DigiCertCA.crt host.key host.pem keystore.jks \ password "*.2thedge.com" truststore.jks password CA"
Docker Debugging Approaches
docker events
● Start a separate console running ‘docker events’● Docker containers report the following events:
attach, commit, copy, create, destroy, die, exec_create, exec_start, export, kill, oom, pause, rename, resize, restart, start, stop, top, unpause, update
● Docker images report the following events:delete, import, pull, push, tag, untag
● Docker volumes report the following events:create, mount, unmount, destroy
● Docker networks report the following events:create, connect, disconnect, destroy
Docker Debugging Approaches
docker events (continued)
● Filtering - currently supported filters are:container (container=<name or id>)event (event=<event action>)image (image=<tag or id>)label (label=<key> or label=<key>=<value>)type (type=<container or image or volume or network>)volume (volume=<name or id>)network (network=<name or id>)
Docker Debugging Approaches
docker diff
● Inspect changes on a container's filesystem:$ docker run --name ubuntu ubuntu:14.04 /bin/touch /tmp/difftest$ docker diff ubuntu C /runA /run/secretsC /tmpA /tmp/difftest
Docker Debugging Approaches
docker stats
● Display a live stream of one or more containers' resource usage:console #1: $ docker run -it --name ubuntu ubuntu:14.04 watch -n 2 ls /console #2: $ docker stats ubuntuCONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O2cfe1cb26035 0.38% 5.738 MB / 33.68 GB 0.02% 508 B / 508 B 3.49 MB / 0 B
Docker Debugging Approaches
Override the ENTRYPOINT
● Default ENTRYPOINT specified in Dockerfile● No ENTRYPOINT specification will cause the container to use ‘/bin/sh -c’● Usage:
$ docker run -d -p 80:80 --entrypoint /bin/sh CONTAINER
● Often useful to ‘exec’ a container and then attempt the default ENTRYPOINT command in the resulting shell:$ docker exec -it CONTAINER /bin/bash$ cd /scripts$ ./run_me.sh
Docker Debugging Approaches
Add new runtime options with CMD
● Dockerfile CMD parameters will be passed to the container ENTRYPOINT and interpreted by that command (assuming the ENTRYPOINT is a command that interprets command-line arguments)
● ‘docker run’ treats parameters after the CONTAINER specification as CMD arguments:$ docker run -d -p 80:80 --entrypoint /bin/sh CONTAINER option1 option2
Docker Debugging Approaches
docker top
● $ docker top CONTAINER
Docker Debugging Approaches
docker inspect
● $ docker inspect CONTAINER
● JSON output with details of running container: mount points, log file, env vars, ports, etc.● The format option is useful:
$ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $INSTANCE_ID
Docker Debugging Approaches
docker history
● docker history CONTAINERff4d8bf9cd6c 4 months ago /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/opt/s 0 B ef89804b623a 4 months ago /bin/sh -c chmod -R +x /opt/scripts 14.78 kB f940c4b6fe31 4 months ago /bin/sh -c #(nop) ADD dir:f0ee20b44c1a3df8c25 14.78 kB c79487890f41 8 months ago /bin/sh -c apt-get update && apt-get install 10.37 MB …
● Use the ‘--no-trunc’ flag to see the entire command line for each of the image layers
Docker Debugging Approaches
docker commit
● In object-oriented parlance, images are ‘classes’, containers are ‘instances’ of images
● Committing a docker container causes a docker image to be committed to the registry
● Often the quickest way to make configuration changes, or do artifact replacement (jars, wars, property files, go classes, python scripts, etc)
● THIS IS NOT A REPLACEMENT FOR SOURCE CONTROL!!! You’re welcome.
Docker Debugging Approaches
Pause / Unpause the container
● Usage:$ docker run -d --name=pausetest alpine /bin/sh -c “while true; do sleep 2; date; done”$ docker pause CONTAINER$ docker unpause CONTAINER
Docker Debugging Approaches
Docker Debugging Approaches
Using (Abusing) volume mount directories
● Facilitates file copying / modifying● Very useful for debugging web applications● Keep in mind that many applications will cache files at start-up, so the container
may need to be restarted to notice your change● Note that read-only volume mounts will limit what you can achieve with this
technique● Not limited to ‘text files’ - sockets, pipes, other dev nodes can be volume mounted ...
Docker Debugging Approaches
Abusing docker container mapped directories
● Direct examination of container file system● DANGER DANGER … this is definitely a very efficient way to corrupt your container,
potentially eliminating the possibility of any future run of the container’s image ever running successfully ...
Restarting / Rebuilding Docker Infrastructure
● Restart docker daemon● Restart host networking● Remove docker0 bridge interface● Clean (‘rm -rf’) docker run-time directories:
# systemctl stop docker# rm -rf /var/lib/docker# systemctl start docker
- WARNING - last resort. This will remove all images and containers!
Docker Debugging Approaches
Network sniffing tools, process trace tools
● Containers typically take on host /proc/sys● Volume mount individual /dev nodes● Network tools may need to watch bridge0 as well default host interface
Docker Debugging Approaches
Docker daemon debug options
● Docker docs: ‘Enable debug output: -D’● Keep this in mind - startup problems are often at the daemon level, not so much at
the container level
Docker Debugging Approaches
Scripts / Source Instrumentation
● YourKit● Zipkin● Custom log driver for containers● Docker-specific logging
Docker Debugging Approaches
Docker Debugging Approaches
Debug / Instrumentation Containers
● Various monitor containers (Google cAdvisor)● Go nuts - create a mongo debug container that you deploy, start and stop as
required, for example
https://docs.docker.com
https://medium.com/@betz.mark/ten-tips-for-debugging-docker-containers-cde4da841a1d
https://opencredo.com/debugging-java-applications-running-in-docker/
https://github.com/google/cadvisor
References