you're monitoring kubernetes wrong

You're Monitoring Kubernetes WrongLoris Degioanni, CEO, Sysdig

Data

Application HealthChecks

System/Process

Infrastructure

CustomMetrics

JVM

Data Big Honkin’ Data Engine

Unicorns and RainbowsDashboards

Loris DegioanniFounder and CEO of sysdigPast: WinPcap, Wireshark

http://www.sysdig.org@lorisdegio

Me

Today• Monitoring Kubernetes

• Few best practices• Some ideas• Some proposals

Using• Sysdig

• https://github.com/draios/sysdig • Sysdig Cloud

• https://sysdig.com/

In a Vendor Neutral Way

We Are in the Middle of a Big Change

PCs/Servers Virtual Machines Containers

We Are in the Middle of a Big Change

PCs/Servers Virtual Machines Containers

Unit: machineOrchestration: noneArchitecture: monolithic

Unit: machineOrchestration: externalArchitecture: monolithic

Unit: app/serviceOrchestration: nativeArchitecture: service-based

Things Are More Complex NowCacheWebserverDatabase

Computing Node

Computing Node

Computing Node

Service1Service2Service3

Computing Node

Computing Node

Computing Node

Things Are More Complex Now

Computing Node

Computing Node

Computing Node

Computing Node

Computing Node

Computing Node

Service1Service2Service3

Things Are More Complex

This Time Is More Than Servers and Processes

How Does This Impact Monitoring?

4 Things That Are Harder Now1. Getting the data2. Making sense of the data3. Troubleshooting4. People

Getting The Data• What makes containers great…

• Simple• Small • Isolated • Few dependencies

• … Also makes them more opaque

Getting The Data: Pods

OS

Container1

Docker

Container2

Docker

PodPod

Getting The Data: Pods

OS

Container1

Docker

Container2

Docker

PodPod

Getting The Data: Pods

OS

Container1

Docker

Monitoring

Docker

Container2

Docker

Monitoring

Docker

Getting The Data: Sysdig

OS

Container1

Docker

Container2

Docker

Container3

DockerAppApp

Getting The Data: Sysdig

OS

Container1

Docker

Container2

Docker

Container3

DockerAppApp

kernel module

Getting The Data: Sysdig

OS

Container1

Docker

Container2

Docker

Container3

DockerAppApp

sysdig

Docker

Capture andanalysis

Getting The Data: Sysdig

OS

Container1

Docker

Container2

Docker

Container3

DockerAppApp

sysdig

Docker

Capture andanalysis

Getting/Making Sense of the Data: Guidelines• You should not be involved in monitoring instrumentation• You should not be involved in producing anything that is not a

custom metric• Every metric should be tagged

• Tagging must be integrated with Kubernetes• You should not be involved in tagging metrics

• You should collect everything, with no filters

Demo – monitoring and segmenting data with sysdig monitoring

Troubleshooting

(https://github.com/kubernetes/kubernetes/issues/14051)

Demo – troubleshooting with open source sysdig

People• Microservices are about people• Monitoring microservices should be about people too

Introducing:Sysdig Teams

Sysdig Teams: Service-based access control• Designed to reflect teams of people• Integrated with Kubernetes• Support isolation and customization • Self service• Centralized

Use Cases• “dev vs prod” split• Team per Microservice• PaaS• Restricted environments

Links• Sysdig

• https://github.com/draios/sysdig • Sysdig Cloud

• https://sysdig.com/• Teams

• https://sysdig.com/blog/introducing-sysdig-teams/