graphite, an introduction
DESCRIPTION
An introduction to graphite and why it's so great.TRANSCRIPT
![Page 1: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/1.jpg)
Graphite:An Introduction
Scaling real-time monitoring
![Page 2: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/2.jpg)
The purpose today
![Page 3: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/3.jpg)
What is graphite
![Page 4: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/4.jpg)
Why it’s so great
![Page 5: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/5.jpg)
How to graph(It’s really easy!)
![Page 6: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/6.jpg)
How we use graphite
![Page 7: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/7.jpg)
First, a definition
![Page 8: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/8.jpg)
Alerts+Metrics=Monitoring
Graphite Cacti Munin
NagiosIcinga
BothZenoss Hyperic ZabbixPNP4Nagios
Alerting Metrics
![Page 9: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/9.jpg)
What is graphite
![Page 10: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/10.jpg)
![Page 11: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/11.jpg)
![Page 12: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/12.jpg)
About graphite
● Django web application consisting of 3 parts:○ carbon (relays, caches, aggregates metrics)○ whisper (graphite’s equivalent of RRD files)○ Web UI (graph composer, simple dashboard)
![Page 13: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/13.jpg)
Why graphite?
![Page 14: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/14.jpg)
Why graphing?
Discover trends and patternsWhat time of the day do we get the most users?When x happened, what was the effect on y?How many hits am I getting per hour? How does this compare to last week? last month?
Predict future eventsWhen will we need to add more servers? Databases?
Negative feedbackDid the release into production fix problem x?
![Page 15: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/15.jpg)
Cacti SUCKS
A few reasons:
Ancient user interface (no javascript/ajax), terrible workflow, cannot push metrics, no
formulas, no graph introspection, cannot push metrics, cannot feed out of sequence
metrics, ugly graphs, no API, expose system/os metrics on host via snmp, no graph
composer, no custom graphs, predefine metrics, predefine graphs, static polling interval,
unscalable, tons of work to create one graph, no 3rd party ecosystem, etc.
![Page 16: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/16.jpg)
Graphite ++
![Page 17: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/17.jpg)
Simple
![Page 18: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/18.jpg)
Powerful
![Page 19: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/19.jpg)
Functions(sum, derivatives, integrals, timeshift, mostDeviant, scale,
averages, etc.)
![Page 20: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/20.jpg)
API(Nagios integration, 3rd party custom dashboards)
![Page 21: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/21.jpg)
![Page 22: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/22.jpg)
Scalable
![Page 23: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/23.jpg)
Easy to feed data
![Page 24: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/24.jpg)
Wide ecosystem of 3rd party tools and dashboards
http://graphite.readthedocs.org/en/latest/tools.html
![Page 25: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/25.jpg)
Tools
![Page 26: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/26.jpg)
StatsD
![Page 27: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/27.jpg)
Logster
![Page 28: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/28.jpg)
Skyline
![Page 29: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/29.jpg)
Collectd
![Page 30: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/30.jpg)
Dashboards
![Page 31: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/31.jpg)
![Page 32: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/32.jpg)
![Page 33: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/33.jpg)
![Page 34: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/34.jpg)
![Page 35: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/35.jpg)
![Page 36: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/36.jpg)
Graphite --
![Page 37: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/37.jpg)
No poller
![Page 38: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/38.jpg)
No all in one solution
![Page 39: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/39.jpg)
No easy backups
![Page 40: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/40.jpg)
It probably will become business critical
![Page 41: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/41.jpg)
How to graph
![Page 42: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/42.jpg)
There are tons of ways to feed graphite your data
![Page 43: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/43.jpg)
Bash
#!/bin/bash
timestamp = `date +%s`
value = 10
echo "dot.delimited.metric.name $value $timestamp" | nc -w 1 graphite.
host.name 2003
Python
def send_msg(message, HOST, PORT):
sock = socket.create_connection((HOST, PORT))
sock.send(message)
sock.close()
Python using graphite-pymetrics
from metrics import timing
@timing("heavy.task")
def heavy_task(x, y, z):
# do heavy stuff here
![Page 44: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/44.jpg)
Ruby
require 'socket'
Host = 'somegraphitehost'
conn = TCPSocket.new Host, 2003
conn.puts 'Metrics value timestamp'
conn.close
Java
import java.io.DataOutputStream;
import java.net.Socket;
Socket conn = new Socket("somegraphitehost" , 2003);
DataOutputStream dos = new DataOutputStream(conn .getOutputStream());
dos.writeBytes("metrics value timestamp" );
conn.close();
![Page 45: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/45.jpg)
How we use graphite
![Page 46: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/46.jpg)
700K + metrics per minute
![Page 47: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/47.jpg)
A Common Graphite Stack
Graphite-web
Collectd
Poller(s)
Applications
Carbon Whisper
Dashboards
Statsd
Scripts
Nagios
![Page 48: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/48.jpg)
Collectd
Agent for system/hardware level metricsGrowing repository of plugins for a wide variety of applications:
disk i/o, disk space, cpu, memory, mysql, JMX, java, Redis, file sizes, load, etc.https://collectd.org/wiki/index.php/Table_of_Plugins
Write your custom plugin in python
![Page 49: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/49.jpg)
Nagios integration
You can write Nagios plugins that can alert off of metrics valuesNagios can also feed graphite
performance data, events (ie: update counter each time email is sent), etc.
![Page 50: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/50.jpg)
What to collect?
![Page 51: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/51.jpg)
Hardware/OS metrics
![Page 52: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/52.jpg)
Load
![Page 53: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/53.jpg)
Disk space
![Page 54: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/54.jpg)
Disk I/O
![Page 55: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/55.jpg)
Network data
![Page 56: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/56.jpg)
Application metrics
![Page 57: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/57.jpg)
How often function x is called
![Page 58: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/58.jpg)
Average value of function x
![Page 59: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/59.jpg)
Average running time of function x
![Page 60: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/60.jpg)
Database/Datastore
![Page 61: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/61.jpg)
performance metrics
![Page 62: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/62.jpg)
number of records with value == ?
![Page 63: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/63.jpg)
number of slow queries
![Page 64: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/64.jpg)
Events
![Page 65: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/65.jpg)
Deployments
![Page 66: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/66.jpg)
send a 1, draw as infinite
![Page 67: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/67.jpg)
Log files
![Page 68: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/68.jpg)
http access logs (2xx, 3xx, 4xx, 5xx)
![Page 69: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/69.jpg)
Application logsException counts, results, important events, hits
![Page 70: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/70.jpg)
Final Musings
![Page 71: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/71.jpg)
Treat graphite like ‘Big Data’
![Page 72: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/72.jpg)
You don’t know what metrics you need until you need it
![Page 73: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/73.jpg)
Get Raid 10 SSD’s once you decide to scale
![Page 74: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/74.jpg)
More devopsy
![Page 75: Graphite, an introduction](https://reader034.vdocuments.site/reader034/viewer/2022051015/554f92e6b4c9052a518b549f/html5/thumbnails/75.jpg)
You can start graphing today!