Download - Metrics stack 2.0
![Page 1: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/1.jpg)
![Page 2: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/2.jpg)
Credit: user niteroi @ panoramio.com
![Page 3: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/3.jpg)
vimeo.com/43800150
![Page 4: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/4.jpg)
![Page 5: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/5.jpg)
![Page 6: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/6.jpg)
![Page 7: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/7.jpg)
![Page 8: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/8.jpg)
![Page 9: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/9.jpg)
![Page 10: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/10.jpg)
![Page 11: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/11.jpg)
1 Metrics 2.0 concepts
2 Implementation
3 Advanced stuff
![Page 12: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/12.jpg)
“Dieter” ?
![Page 13: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/13.jpg)
Peter Deter→
![Page 14: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/14.jpg)
Terminology sync
![Page 15: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/15.jpg)
(1234567890, 82)
(1234567900, 123)
(1234567910, 109)
(1234567920, 77)
db15.mysql.queries_running
host=db15 mysql.queries_running
![Page 16: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/16.jpg)
![Page 17: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/17.jpg)
How many pagerequests/s is vimeo.com doing?
![Page 18: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/18.jpg)
● stats.hits.vimeo_com
● stats_counts.hits.vimeo_com
![Page 19: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/19.jpg)
![Page 20: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/20.jpg)
stats.<host>.requesthostport.vimeo_com_443
![Page 21: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/21.jpg)
stats.timers.dfs5.proxyserver.object.GET.200.timing.upper_90
![Page 22: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/22.jpg)
O(X*Y*Z)X = # apps
Y = # people
Z = # aggregators
![Page 23: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/23.jpg)
How long does it take to retrieve an object from swift?
![Page 24: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/24.jpg)
stats.timers.<host>.proxyserver.<swift_type>.<http_method>.<http_code>.timing.<stat>
stats.timers.<host>.objectserver.<http_method>.timing.<stat>
target=stats.timers.dfs*.object*GET*timing.mean ?
target=groupByNode(stats.timers.dfs*.proxyserver.object.GET.*.timing.mean,2,"avg")
target=stats.timers.dfs*.objectserver.GET.timing.mean
![Page 25: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/25.jpg)
swift_type=object stat=mean timing GET avg by http_code
![Page 26: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/26.jpg)
![Page 27: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/27.jpg)
![Page 28: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/28.jpg)
O((DxV)^2)D = # dimensions
V = # values per dim
![Page 29: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/29.jpg)
collectd.db.disk.sda1.disk_time.write
![Page 30: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/30.jpg)
![Page 31: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/31.jpg)
![Page 32: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/32.jpg)
What should I name my metric?
![Page 33: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/33.jpg)
101001000
100001000001000000
![Page 34: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/34.jpg)
![Page 35: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/35.jpg)
Metrics 2.0
![Page 36: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/36.jpg)
Old:● information lacking
● fields unclear & inconsistent
● cumbersome strings / trees
● forbidden characters
New:● Selfdescribing
● Standardized
● all dimensions in orthogonal tagspace
● Allow some useful characters
![Page 37: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/37.jpg)
stats.timers.dfs5.proxyserver.object.GET.200.timing.upper_90
{ “server”: “dfvimeodfsproxy5”, “http_method”: “GET”, “http_code”: “200”, “unit”: “ms”, “target_type”: “gauge”, “stat”: “upper_90”, “swift_type”: “object” “plugin”: “swift_proxy_server”}
![Page 38: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/38.jpg)
Main advantages:● Immediate understanding of metric meaning (ideally)
● Minimize time to graphs, dashboards, alerting rules
![Page 39: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/39.jpg)
github.com/vimeo/graphexplorer/wiki
![Page 40: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/40.jpg)
SI + IEC
B Err Warn Conn Job File Req ...
MB/s Err/d Req/h ...
![Page 41: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/41.jpg)
{
“site”: “vimeo.com”,
“port”: 80,
“unit”: “Req/s”,
“direction”: “in”,
“service”: “webapp_php”,
“server”: “webxx”
}
![Page 42: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/42.jpg)
![Page 43: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/43.jpg)
Carbontagger:
... service=foo.instance=host.target_type=gauge.type=calculation.unit=B 123 1234567890
…
Statsdaemon:
..unit=B..unit=B... unit=B/s→
..unit=ms..unit=ms.. unit=ms stat=mean→
→ unit=ms stat=upper_90
→ ...
![Page 44: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/44.jpg)
![Page 45: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/45.jpg)
![Page 46: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/46.jpg)
GraphExplorer queries 101
site:api.vimeo.com unit=Req/s
requesthostport api_vimeo_com
![Page 47: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/47.jpg)
![Page 48: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/48.jpg)
Smoothing
avg over 10M
avg over ...
![Page 49: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/49.jpg)
![Page 50: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/50.jpg)
Aggregation, compare port 80 vs 443
avg by <dimension>
sum by <dimension>
sum by server
![Page 51: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/51.jpg)
![Page 52: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/52.jpg)
Compare 80 traffic amongt servers
site:api.vimeo.com unit=Req/s port=80 group by none avg over 10M
![Page 53: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/53.jpg)
![Page 54: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/54.jpg)
GraphExplorer queries 201
proxyserver swift server:regex upper_90 unit=ms from <datetime> to <datetime> avg over <timespec>
![Page 55: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/55.jpg)
![Page 56: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/56.jpg)
![Page 57: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/57.jpg)
![Page 58: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/58.jpg)
![Page 59: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/59.jpg)
Compare object put/get
Stack .. http_method:(PUT|GET) swift_type=object avg by http_code,server
![Page 60: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/60.jpg)
![Page 61: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/61.jpg)
Comparing servers
http_method:(PUT|GET) avg by http_code,swift_type,http_method group by none
![Page 62: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/62.jpg)
![Page 63: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/63.jpg)
Compare http codes for GET, per swift type
http_method=GET avg by server group by swift_type
![Page 64: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/64.jpg)
![Page 65: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/65.jpg)
transcode unit=Job/s avg over <time> from <datetime> to <datetime>
![Page 66: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/66.jpg)
Note: data is obfuscated
![Page 67: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/67.jpg)
Bucketing
!queue sum by zone:apsoutheast|euwest|useast|uswest|saeast|vimeodf|vimeolv group by state
![Page 68: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/68.jpg)
Note: data is obfuscated
![Page 69: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/69.jpg)
Compare job states per region (zones bucket)
group by zone
![Page 70: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/70.jpg)
Note: data is obfuscated
![Page 71: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/71.jpg)
Unit conversion
unit=Mb/s network dfvimeorpc sum by server
![Page 72: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/72.jpg)
![Page 73: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/73.jpg)
![Page 74: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/74.jpg)
unit=MB
![Page 75: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/75.jpg)
![Page 76: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/76.jpg)
![Page 77: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/77.jpg)
{
server=dfvimeodfs1
plugin=diskspace
mountpoint=_srv_node_dfs5
unit=B
type=used
target_type=gauge
}
![Page 78: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/78.jpg)
server:dfvimeodfs unit=GB type=free srv node
![Page 79: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/79.jpg)
![Page 80: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/80.jpg)
unit=GB/d group by mountpoint
![Page 81: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/81.jpg)
![Page 82: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/82.jpg)
![Page 83: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/83.jpg)
![Page 84: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/84.jpg)
![Page 85: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/85.jpg)
![Page 86: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/86.jpg)
![Page 87: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/87.jpg)
Dashboard definition
queries = [
'cpu usage sum by core',
'mem unit=B !total group by type:swap',
'stack network unit=b/s',
'unit=B (free|used) group by =mountpoint'
]
![Page 88: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/88.jpg)
![Page 89: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/89.jpg)
stats.dfvimeocliapp2.twitter.error
{
“n1”: “dfvimeocliapp2”,
“n2”: “twitter”,
“n3”: “error”,
“plugin”: “catchall_statsd”,
“source”: “statsd”,
“target_type”: “rate”,
“unit”: “unknown/s”
}
![Page 90: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/90.jpg)
Two hard things in computer science
![Page 91: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/91.jpg)
stats.gauges.files.
id_boundary_7day
stats.gauges.files.
id_boundary_ceil
![Page 92: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/92.jpg)
unit=File id_boundary_7d
{
“unit”: “File”,
“n1”: “id_boundary_7d”,
}
![Page 93: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/93.jpg)
{
“intrinsic”: {
“site”: “vimeo.com”,
“unit”: “Req/s”
},
“extrinsic”: {
“agent”: “diamond”,
“processed_by”: “statsd1”,
“src”: “index.php:135”,
“replaces”: “vimeo_com_reqps”
}
}
![Page 94: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/94.jpg)
site=vimeo.com unit=Req/s \
processed_by=statsd1 \ src=index.php:135 added_by=dieter \
123 1234567890
![Page 95: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/95.jpg)
![Page 96: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/96.jpg)
Equivalence
servers.host.cpu.total.iowait “core” : “_sum_”→
servers.host.cpu.<corenumber>.iowait
servers.host.loadavg.15
![Page 97: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/97.jpg)
Rollups & aggregation
![Page 98: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/98.jpg)
/etc/carbon/storageaggregation.conf[min]
pattern = \.min$
aggregationMethod = min
[max]
pattern = \.max$
aggregationMethod = max
[sum]
pattern = \.count$
aggregationMethod = sum
[default_average]
pattern = .*
aggregationMethod = average
![Page 99: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/99.jpg)
![Page 100: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/100.jpg)
2 kinds of graphite users
![Page 101: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/101.jpg)
Selfdescribing metrics
stat=upper/lower/mean/...target_type=counter..
![Page 102: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/102.jpg)
● stats.timers.render_time.histogram.bin_0.01● stats.timers.render_time.histogram.bin_0.1● stats.timers.render_time.histogram.bin_1 unit=Freq_abs bin_upper=1→
● stats.timers.render_time.histogram.bin_10● stats.timers.render_time.histogram.bin_50● stats.timers.render_time.histogram.bin_inf● stats.timers.render_time.lower unit=ms stat=lower→
● stats.timers.render_time.mean unit=ms stat=mean→
● stats.timers.render_time.mean_90 ...→
● stats.timers.render_time.median● stats.timers.render_time.std● stats.timers.render_time.upper● stats.timers.render_time.upper_90
![Page 103: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/103.jpg)
Also..
● graphite API functions such as "cumulative", "summarize" and "smartSummarize"
● Graph renderers
![Page 104: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/104.jpg)
![Page 105: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/105.jpg)
From: dygraphs.com
![Page 106: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/106.jpg)
![Page 107: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/107.jpg)
![Page 108: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/108.jpg)
![Page 109: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/109.jpg)
![Page 110: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/110.jpg)
![Page 111: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/111.jpg)
Facet based suggestions
![Page 112: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/112.jpg)
![Page 113: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/113.jpg)
Metric types
● gauge● count & rate● counter● timer
![Page 114: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/114.jpg)
![Page 115: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/115.jpg)
![Page 116: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/116.jpg)
![Page 117: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/117.jpg)
![Page 118: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/118.jpg)
gauge
● Multiple values in same interval● “sticky”
![Page 119: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/119.jpg)
![Page 120: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/120.jpg)
Count & Rate
![Page 121: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/121.jpg)
Counter
![Page 122: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/122.jpg)
Timer..
![Page 123: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/123.jpg)
![Page 124: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/124.jpg)
http://janabeck.com/blog/2012/10/12/lessonslearnedfrom100/
![Page 125: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/125.jpg)
Timer..
![Page 126: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/126.jpg)
● What should a metric be?● Stickyness?● Behavior on no packets received● Behavior on multiple packets received
![Page 127: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/127.jpg)
My personal takeaways
![Page 128: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/128.jpg)
Conclusion● Building graphs, setting up alerting cumbersome● Esp. changing information needs (troubleshooting, exploring, ..)● Esp. Complicated information needs
→ PAIN
● Structuring metrics● Selfdescribing metrics● Standardized metrics● Native metrics 2.0
● → BREEZE
![Page 129: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/129.jpg)
Conclusion
● Metrics can be so much more usable and useful. Let's talk about tagging, standardisation, retaining information throughout the pipeline.
● Converting information needs into graph defs, alerting rules● GraphExplorer, carbontagger, statsdaemon, …● Graphiteng (native metrics 2.0)● Metrics 2.0 in your apps, agents, aggregators?● Build out structured metrics library
![Page 130: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/130.jpg)
github.com/vimeo
github.com/Dieterbe
twitter.com/Dieter_be
dieter.plaetinck.be
![Page 131: Metrics stack 2.0](https://reader034.vdocuments.site/reader034/viewer/2022042714/54c6a6e64a7959e4208b45cd/html5/thumbnails/131.jpg)