Containing Graphite(And Herding Cats)

Trying to Make the World at TWC a Better Place

*Drinker of Good Beer

Phil Zimmerman*

@phil_zimmerman

Graphite: OverviewLegos and Herding CatsShow Me Some Cool Shit

Meet The Players

Registrator

A Self-Serve Interface to Your Cluster

Distributed init For Long-running Services

You Know The Company Behind Consul(Think Vagrant)

Key/Value StoreService Discovery (HTTP and DNS)

Failure DetectionMulti Datacenter

Queries Consul and Updates SpecifiedTemplates on the Filesystem

Registrator

https://github.com/gliderlabs/registrator

Auto Register/De-register Docker Containers

Graphite Overview

GraphiteCarbon-cache

WhisperGraphite Web

Carbon-relay

Carbon-cacheReceives and Writes Incoming Metrics

Caches and flushes data

2 Receiver ports, 1 Query Port

Whisper DatabaseTime-series Data

Graphite WebQueries carbon-cache

Reads whisper dataAND

Carbon-relayLoad Balances and Proxies Metrics

Can Write To Multiple carbon-caches

For High Volume of Metrics

Carbon Relay [:2003]

Metrics Metrics Metrics

Carbon Cache [:2003] [:2004] [:7002]

Whisper Data

Carbon Webapp [:80]

postgresql (dashboards)

[:5432]

Typical Single-node Setup

Does Anyone See Containers??

Master Webapp [:80]

pgsql[:5432]

Clustered Setup

Carbon Relay [:2003]

Metrics Metrics Metrics

Carbon Relay [:2003]

Carbon Relay [:2003]

Carbon Relay [:2003]

Carbon Cache [:2003] [:2004] [:7002]

Whisper Data

Carbon Cache [:2003] [:2004] [:7002]

Carbon Webapp [:80]

MemcachedMemcachedMemcached [:11211]

Carbon Relay [:2003]

Carbon Cache [:2003] [:2004] [:7002]

Whisper Data

Carbon Cache [:2003] [:2004] [:7002]

Carbon Webapp [:80]

And You Thought You Were Too Old

To Play With Legos

How Do We Orchestrate All Of This??

Without Service Discovery

Containers

Service AnnouncementService Discovery

Configuration

What We NeedWhen Containers Start/Stop, They Register/

De-register Their ServicesWhen Dependent Service Endpoints Change,

Config Files Auto-Update and Trigger Service Restart

Registrator

The Service ContainersCarbon-cache: Dockerfile

The Service ContainersCarbon-relay: Dockerfile

The Service ContainersCarbon-webapp: Dockerfile

The Service Containerspostgresql: Dockerfile

The Service Containersmemcached

The Data Volume ContainersWhisperCreate a carbon-cache container without

running the cmd (use entrypoint)

The Data Volume Containersgraphite-dbCreate a postgresql container without

running the cmd (use entrypoint)

The Data Volume Containersconsul-templateCreate a consul-template container

without running the cmd (use entrypoint)

Service Announcement/Registration

RegistratorRuns in a container Pass consul url as argument for docker run

Service Announcement/Registration

docker run -d -P -e SERVICE_5432_NAME=graphite-db

--volumes-from dashboard-data twcable/postgresql93:0.2

Start & Register postgresql

Service Discoverygraphite-master-webapplocal_settings.ctmpl (consul-template)

Service Discoverygraphite-master-webapptmpl.json - config for consul-template

Services Registered In

Consul

Great, Now How Do You Launch All of These

Containers??

Registrator

Mesos Master !

Mesos Marathon

Zookeeper

Mesos Master !

Mesos Marathon

Zookeeper

Mesos Master !

Mesos Marathon

Zookeeper

Mesos Slave !

Mesos Docker Consul

Mesos Slave !

Mesos Docker Consul

Mesos Cluster

Marathon JSON Deploy Definition

Post JSON To Running Marathon Instance

Show Me Something Cool

Helpful Linkshttp://mesos.apache.org/https://mesosphere.github.io/marathon/https://consul.io/https://github.com/hashicorp/consul-templatehttps://github.com/gliderlabs/registrator

https://grey-boundary.io/the-architecture-of-clustering-graphite/

http://philzim.com/2014/11/12/service-discovery-orchestration-with-mesos-and-consul/