Logging & Metrics With DockerA Comprehensive Monitoring Solution

Stefan Zier

June 13th, 2015

whoami

Infrastructure, Backend Dev/Architect

Chief Architect, Sumo Logic, since 2010

Server & Infrastructure, ArcSight (HP), 2001-

2010

Mandatory Slide Showing Shipping Containers

Docker – What’s making debugging hard?

One more layer of abstraction

Container per app = File system

per process

File systems short lived, transient

Resource schedulers = no

container affinity to host

What Our Customers Are Telling Us

We have one process per container

We like to log to stdout

We have multiple processes per container

We run the Sumo Logic collector on the Docker host

We are looking into using Beanstalk with Docker

We are using Amazon ECS

Everyone here loves Docker

We are logging straight from the application

We are using /dev/log for Syslog

We want immutable infrastructure

GoalGet logs from our containerized applications to a centralized logging platform.

How do apps emit logs

Append to a file

Use syslog()

Use log4j, log4net, slf4, etc.

printf() to stdout

Getting logs out of the container - Files

Use VOLUME to mount a host directory

Collect files from the host

Collect files from another container sharing the VOLUME

Need to manage disk space, i.e. rotate logs

App (where supported)

Host

Yet another container with logrotate

docker run -v /tmp/clogs:/tmp/clogs -d --name="sumo-logic-collector" sumologic/collector:latest-file [Access ID] [Access key]

Getting logs out - Syslog

VOLUME /dev/log from host and use host

syslogd

Run a syslogd inside the container

Emit TCP/UDP

Write to a file using VOLUME

Emit syslog TCP/UDP directly from the app

docker run -d -p 514:514 -p 514:514/udp \ --name="sumo-logic-collector” \ sumologic/collector:latest-syslog [Access ID] [Access key]

Getting logs out – Logging frameworks

Sumo Logic blog on official collector imageshttp://www.sumologic.com/blog/company/an-official-docker-image-for-the-sumo-logic-collector

https://github.com/SumoLogic/sumologic-collector-docker

Rainer Gerhards on Rsyslog’s file input modulehttp://www.slideshare.net/rainergerhards1/using-wildcards-with-rsyslogs-file-monitor-imfile

OWASP Log Injectionhttps://www.owasp.org/index.php/Log_injection

Getting logs out – Logging frameworks

Directly to network destinations

HTTP/HTTPS

Also support files, stdout, etc.

Getting logs out – Logging frameworks

Various application stackshttp://help.papertrailapp.com/

Log4Jhttps://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SyslogAppender.html

Apache Web Server http://httpd.apache.org/docs/trunk/mod/mod_syslog.html

https://raymii.org/s/snippets/Apache_access_and_error_log_to_syslog.html

Nginxhttp://nginx.org/en/docs/syslog.html

Postgreshttp://www.postgresql.org/docs/9.1/static/runtime-config-logging.html

Sumo Logic blog on official syslog collector imagehttp://www.sumologic.com/blog/company/an-official-docker-image-for-the-sumo-logic-collector

https://github.com/SumoLogic/sumologic-collector-docker

Getting logs out – stdout

Simply printf()

Logging framework to console

Symlink to /dev/stdout or /dev/stderr

Configure paths to /dev/stdout or /dev/stderr

RUN ln -sf /dev/stdout /var/log/nginx/access.logRUN ln -sf /dev/stderr /var/log/nginx/error.log

Docker Logging Drivers

What Docker provides

Captures stdout/stderr

Feeds it to logging drivers

docker logs command

Returns the entire log every time

Works with json-file driver only

Can tail logs

docker logs –tf –-tail 0 [ID]

Docker Logging Drivers

Configured on docker run

stdout and stderr dispatched to drivers

json-file (default pre 1.6)

syslog

journald

No stats, no events

json-file driver

Output unbounded, can fill up the host disk

Requires logrotate on the Docker host

https://github.com/docker/docker/issues/7333

Stats

Docker Stats

Per-container cgroups metrics (like docker

stats)

Memory

CPU

Block I/OCONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/Ocollector 2.23% 232.6 MiB/2 GiB 11.36% 191.9 KiB/636.3 KiB

RequirementsHow would we want it to work?

What information do we want to collect?

Timestamp

Log message

Docker host info

Container ID

Image ID

Process ID

How should it work?

Use docker logging infrastructure

Minimal moving parts

Containerized - don’t touch the host

Complete – pick up all available data

Automatically discover new containers

Docker APIThe solution maybe?

Docker API

Docker daemon has a REST API

TCP or unix socket

Streaming APIs

Docker Events (container lifecycle updates)

Container Stats (CPU, memory used, …)

App Logs (container stdout/stderr)

Collecting via Docker API

Discover new containers via events

Start streaming their logs and stats

When they go away, stop

Do all of this via the API

Send all of it to centralized log management

Collecting via Docker API, continued

Single component to do it

Zero footprint on the host

Follows Docker standard way of logging

One more thing…

Introducing: Sumo Logic Docker Source

Sumo Logic Docker Source

Active development

Early access expected later this year

Demo Time

fin.Questions?

@stefanzier