docker monitoring webinar
TRANSCRIPT
Docker Monitoring Webinar
20% OFF 201509WNR20S 201509WNR20Lsematext.com/spm sematext.com/logsene
Housekeeping / Questions
Intro
SPM: Performance monitoring,Anomaly Detection and Alerting
Logsene: Centralized Log Management
Search and Big Data Consulting Support for Solr and Elasticsearch
Agenda
● Docker Monitoring ○ Docker Monitoring Basics○ Deployment Options for Docker Agents
● Application Monitoring on Docker○ Monitoring Containerized Applications○ Deployment Options for
Application Monitoring Agents
Why we like Blue Whales
● Portable packaging ● Optimized resource usage● Configurable resource limits● Isolated environment for apps
What is Docker made of?
Docker Platforms
Docker Challenges
● New workflows for deployments○ from developer laptop straight to the Cloud
● Orchestration of applications● Logging
○ Webinars 29/30 September, @sematext● Monitoring
Docker Monitoring Basics
Let’s understand what we monitor!
Why Docker Monitoring
You can’t fix problems you don’t see!● Tuning
○ Resources / Container Limits○ Application Performance
● Quality assurance ● Capacity planning● Availability / SLA’s
What to monitor?
Cluster
Server 1
Container 1
Container 2
Container N
Container 1 Elasticsearch
Container 2 Node.js - web service
Container N NGINX - reverse proxy
Server N
JVM Runtime Env
JS Runtime Env
The Metrics Stack Changed!
Server Metrics
Runtime Metrics
App Metrics
Server Metrics
Runtime Metrics
App Metrics
Container Metrics
Runtime Metrics = JVM Metrics or Node.js Metrics, …
Without Docker With Docker
Poll
What is the result of ‘vmstat’ when started in a container?
A) OS Metrics limited to the containerB) OS Metrics from the docker host
Docker Stats Command> docker stats $(docker ps -q)
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
19a9f4c9ecb7 0.00% 1.421 MB/3.944 GB 0.04% 0 B/0 B
7f043a32307c 0.00% 1.413 MB/3.944 GB 0.04% 0 B/0 B
889646bd003f 0.11% 4.235 MB/3.944 GB 0.11% 0 B/0 B
9a29f494bf7d 0.27% 156.8 MB/3.944 GB 3.98% 0 B/0 B
Too simple!
Docker API for Metrics> echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0\r\n"
| nc -U /var/run/docker.sockHTTP/1.0 200 OK
Date: Thu, 30 Jul 2015 08:37:31 GMT
Content-Type: text/plain; charset=utf-8
{"read":"2015-07-30T04:37:31.13789208-04:00","network":{"rx_bytes":636450,"rx_packets":1944,"rx_errors":0,"rx_dropped":0,"tx_bytes":495897,"tx_packets":2415,"tx_errors":0,"tx_dropped":0},"precpu_stats":{"cpu_usage":{"total_usage":0,"percpu_usage":null,"usage_in_kernelmode":0,"usage_in_usermode":0},"system_cpu_usage":0,"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}},"cpu_stats ...
Too verbose! More than 70 Metrics are exposed
Watching a Metric> echo -e "GET /containers/3236f3328bbd/stats HTTP/1.0\r\n"
| nc -U /var/run/docker.sock
| logagent -y
| grep rx_bytes
rx_bytes: 667326
rx_bytes: 667354
rx_bytes: 667413
rx_bytes: 667538
github.com/sematext/logagent-js
Which Metrics are most relevant?
Docker Key Metrics
Server Metrics
● CPU ● Memory Usage● Disk Space
○ Docker Images consume a lot of disk space
Container Metrics● Containers CPU
+ Throttled CPU time / CPU limits!● Memory Limits
+ Memory Allocation Fail Counters● Disk I/O + Wait Times● Network I/O + Network Errors
Docker Key Metrics in SPM
Beyond Metrics: Events
● Docker Events○ container auditing○ create, destroy, die, export, kill,
pause, restart, start, stop,unpause, oom
Beyond Metrics: Logs
● Logs○ Docker collects console output
of each container○ API or Log Drivers for forwarding
● Correlation with Metrics○ Faster troubleshooting
blog.sematext.com/2015/08/11/docker-logging-webinar/
Docker Logging Webinar
Deploying Docker Monitoring Agents
Directly on each Docker Host
● The traditional way!● Full access to the operating system ● Limitations in ’Container only’ systems
- CoreOS- RancherOS- Many PaaS Solutions, e.g., DEIS
One Container on each Docker Host
● Yes. The Docker Way!● Portable to any ‘Docker platform’ ● Requirements:
- Access to the Host Metrics + Docker API- Extended privileges (e.g., for SELinux, GCE,...)
Containerized Monitoring Agents
SPM for Docker Monitoring
apps.sematext.com/spm-reports/registerApplication.do
hub.docker.com/r/sematext/spm-agent-docker/
github.com/sematext/spm-agent-docker
Example 1: Run SPM for Dockerdocker run -d --name spm-agent
-v /var/run/docker.sock:/var/run/docker.sock
-e SPM_TOKEN=fe31fc3a-xxxx-47c6-b83c-be376bfxxx
-e HOSTNAME=$HOSTNAME
-e SPM_MONITORING_TAGS="project:dev-test,role:demo"
-e LOGSENE_TOKEN=53a6c7e7-xxxx-4725-962e-ea47cebxxx
--privileged
sematext/spm-agent-docker
Example 2: CoreOS + DEIS
● CoreOS has no package manager● systemd - init system ● fleet - as distributed init system● etcd - for central configurations
How to distribute a Monitoring Agent to all CoreOS servers in a cluster?
Example 2
Example 2: Distribute the Agent# Configure SPM app token
etcdctl set /sematext.com/myapp/spm/token YOUR_SPM_TOKEN
# Download the fleet unit file for SPMwget https://raw.githubusercontent.com/sematext/spm-agent-docker/master/coreos/spm-agent.service \
-O spm-agent.service
# Start SPM Agent in the whole clusterfleetctl load spm-agent.servicefleetctl start spm-agent.service
Example 3: RANCHER OS
● Minimalistic (~25 MB !)● No package manager● All system services are containers● No cluster manager (by default)● Use restart policy ‘always’ for services
docker run --restart=always … sematext/spm-agent-docker
Application Monitoring
layers.next()
Application Metrics
How are Applications monitored?
● Standalone Agents○ Remote interface: JMX, MySQL, NGINX, Apache○ No restart required, when agent gets updated
● In-Process Agents○ Resource friendly○ Node.js○ JVM Apps (Elasticsearch, Solr, Kafka, ...)
- Function Transaction/Call Tracing possible
Transaction Tracing + AppMap
● Only possible with in-process monitoring
Deployment Options for Application Monitoring
… on Docker
Everything in Each Container
Standalone in Separate Container
In-Process in Separate Container
Setup for Elasticsearch Imagespm-client:
image: sematext/spm-client # Exports Volume /opt/spm
container_name: spm-client
environment:
- SPM_CFG="${SPM_TOKEN} es javaagent jvmname:ES1"
ELASTICSEARCH:
image: elasticsearch
ports:
- "9200:9200"
volumes_from:
- spm-client
environment: # inject in-process monitor from SPM client volume /opt/spm
- ES_JAVA_OPTS="-Dcom.sun.management.jmxremote \
-javaagent:/opt/spm/spm-monitor/lib/spm-monitor-es.jar=${SPM_TOKEN}::ES1"
Setup for node.js Apps
Dockerfile:FROM nodejs
RUN git clone my/app
RUN npm i
RUN npm i spm-agent-nodejs
CMD node -r spm-agent-nodejs app.js
---
docker build -t myapp .
docker run -e SPM_TOKEN=43A... myapp
Application + Docker Metrics
SPM Logsene
METRICS + LOGS ⇒ BETTER TOGETHER
Summary
Stefan ThiesTwitter: @[email protected]
[email protected]/spmhub.docker.com/r/sematext/spm-agent-docker/github.com/sematext/spm-agent-docker
Docker Monitoring Webinar
20% OFF 201509WNR20S 201509WNR20Lsematext.com/spm sematext.com/logsene
Thank you for your attention