how and why we got prometheus working with docker swarm
TRANSCRIPT
How and why we got Prometheus working with
Docker SwarmLuke Marsden
@lmarsden
Luke’s goal: Make Weave Cloud accessible to Docker
Swarm users. Let’s get Prometheus working with Docker Swarm!
Justin’s goal: Everyone kept asking me “how do I
Prometheus?”
So, how to integrate?
+
V1 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
swarm API
Net A+B
V1 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
swarm API
Net A+B
prom-wrapper writes service discoveryinfo to Prom config, starts Prom on
the right Docker networks…
V1 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
swarm API
Net A+B
V1 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
swarm API
Net A+B
Service CNetwork C
V1 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
Service CNetwork C
swarm API
V1 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
Service CNetwork C
swarm API
Net A+B+C
Downsides to V1 design• Assumption: can’t dynamically add container to new
networks • Therefore, need to restart Prometheus every time a
service or network comes or goes • Services & networks can be re-deployed frequently,
especially in a busy cluster + continuous delivery • Means Prometheus can miss scrapes • Losing data :-(
V2 architecture
Service ANetwork A
Service BNetwork B
swarm API
Net P
prom-proxy
prom-proxy
Net P+A
Net P+B
prom-wrapper
Service AService A
Service BService B
V2 architecture
Service ANetwork A
Service BNetwork B
Service CNetwork C
swarm API
Net P
prom-proxy
prom-proxy
Net P+A
Net P+B
prom-wrapper
Service AService A
Service BService B
Service CService C
V2 architecture
Network A
Network B
Network C
swarm API
Net P
prom-proxy
prom-proxy
prom-proxy
Net P+A
Net P+B
Net P+C
prom-wrapper
Service AService A
Service BService B
Service CService C
What is prom-proxy?
prom-proxy
• socat? • No, needs to aggregate stats from
multiple instances, concatenate them, discover them…
• Write a custom proxy? • Hmm it’s starting to look a look like
Prometheus… • Eureka! Use Prometheus in federated
mode!
V2 architecture
Service ANetwork A
Service BNetwork B
Service CNetwork C
swarm API
Net P
Net P+A
Net P+B
Net P+C
prom-wrapper
Service AService A
Service BService B
Service CService C
Downsides to V2 design• It’s bloody complicated! • “You’re not going to do that, are you?”
• Lots of moving parts • Oh well, let’s give it a go
Containers can be dynamically added to networks!
V3 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
swarm API
Net A+B
V3 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
swarm API
Net A+B
prom-wrapper writes service discoveryinfo to Prom config, starts Prom on
the right Docker networks…
V3 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
swarm API
Net A+B
V3 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
swarm API
Net A+B
Service CNetwork C
V3 architecture
Service A
prom-wrapper
Network A
Service BNetwork B
Service CNetwork C
swarm API
Net A+B+C
prom-wrapper dynamically addsPrometheus to the right
networks, rewrites config whichProm hot-reloads
So that’s what we did :-)• github.com/containersolutions/
prometheus-swarm-discovery • Now you can monitor Docker Swarm with
Weave Cloud :-) • Demo!
Conclusion• Weave Cloud is an ops platform for app devs,
help you go round the loop faster • Includes hosted scalable Prometheus, works
with Swarm • Try the prototype at github.com/
containersolutions/prometheus-swarm-discovery • Try Weave Cloud at weave.works
Join the Weave user group for talks & trainings!
• Continuous delivery • Network policy for security • Deep dive on Prometheus monitoring
meetup.com/pro/Weave
and then we went to the pub!