how to monitor microservices
TRANSCRIPT
Jorge Salamero Sanz@bencerillo @sysdig
How toMonitor Microservices
How to Monitor Microservices?
Apps
Infra
Health Checks
JVM/JMXCustom metrics
Metrics Processing Unicorns, rainbows And cute dashboards
% whoami
Jorge Salamero Sanz
• Working on OSS last 12 years
• Working on monitoring last 3 years
• Containers gamer @sysdig
@bencerillo
@sysdig
Agenda
• Challenges of container infrastructures
• Traditional monitoring limitations
• Best practices monitoring Microservices
• Sysdig, container native monitoring & troubleshooting
Breaking traditional model
Microservices and containers break traditional monitoring and troubleshooting models
Traditional deployment
Full host OS
kernelsystemdsyslogd
App services
MySQLNginx
OpenSSLJava
App A
App B
App C
Ops Devs
Application workflow
Database App Cache
backend middleware frontend
The Docker revolution
Servers Virtual Machines Containers
Unit: machine
Orchestration: no
Architecture: monolithic
Unit: machine
Orchestration: externalArchitecture: monolithic
Unit: (micro)service
Orchestration:nativeArchitecture: distributed
Containerized deployment
Full host OS
kernel
+Docker
MySQL App A
Ops DevOps
Nginx + OpenSSL App B
Java 8.0 build XXX App C
… but in reality
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node
… but in reality
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node
Application workflow
Database App Cache
backend middleware frontend
New organization structures
Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure
Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure
1. Metric collection
• We containers, because:
– are simple
– are small
– are isolated
– less dependencies
• … but they are an opaque blackbox
“Workarounds”
Agent in the Docker container
Agent in the Kubernetes pod
Export metrics through an external agent
App Agent App Agent
App
Agent
App
App
App
1. Complex instrumentation (x2 because just the monitoring) plus service monitoring configuration
2. Limited and pre-established metric collection (Docker API, etc)
Kernel instrumentation
Kernel
Docker
Container1
Container2
Container3
App Apprkt LXC
Sysdig
Docker
Why this is cool?
• Just one instrumentation per host:– spawning or destroying a container is instrumentation-less
• Full visibility: all the system calls:– automatic service discovery
– all metrics collection (no filtering)
– application monitoring without instrumentation (magic of
decoding protocols)
Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure
Remember... but in reality:
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node
2. Information aggregation
• Infrastructure monitoring should be transparent and
automatic (no instrumentation no configuration)
• You should handle your custom/biz metrics
• All metrics should be tagged automatically
• All metrics should be aggregated and segmented on a
service level basis
Application workflow
Database App Cache
backend middleware frontend
Orchestration platforms knows already...
Demo!
Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure
Real example
https://github.com/kubernetes/kubernetes/issues/14051
3. Analysis & troubleshooting
• Imagine:
strace + wireshark + htop + lsof + iostat + vmstat + *
• Not available on containers, don’t understand
namespaces
• Metrics and logs can bite your in the ass, system
calls have all the truth
• Infrastructure gets more complex and volatile
Demo!
Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure
Teams on Microservices
Francesc Zacarias, SRE @ Spotify
4. Teams by service
• Tags/Metadata from the orchestration platform, eg
Kubernetes:
– namespaces (dev, prod)
– services, deployments, RCs, pods
– custom tags
• ACLs out of the box (dashboards, alerts, etc) on
multi-tenant/PaaS scenarios
Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure
Sysdig
• 100% open-source
• 1M+ downloads
• Host analysis
• sysdig.org
• SaaS & on-prem
• 200+ customers
• Cluster analysis
• Dashboards, alerts,
events, teams
• sysdig.com
