Which watcher watches cloudwatch?

David Lutz@dlutzy

about me

sysadmin

about me

sysadmindevops

about me

sysadmindevopsoperations

about me

sysadmindevopsweb operations

about me

sysadmindevopsweb operations

about me

#monitoringlove

#infracoders

I “drive” website infrastructure

What do I mean by “drive”?

Operate, anything not build related.

● Gather telemetry or metrics on running system● Use metrics to make decisions about capacity and

architecture● Know what the bottlenecks are and when we’ll hit

them● Advise the business about cost vs performance

tradeoffs● Fix things when the break

Work with my friends the developers…

They build the things I drive.

What the developers think they built

What they actually built

How operations see themselves

How dev sees ops

The Perfect Dashboard

How fast are we going?(km/h)

How hard is the engine working

(RPM)

Is anything broken?(alerts)

How fast can we go before redline

How far can we go before running out of fuel

(Thresholds)

How fast are we going? latency (ms)

How hard are the servers working? throughput (rpm)

Is anything broken?

What about thresholds?

You must load test to find the breaking points and bottlenecks yourself.

Load testing is hard.

AWS CloudWatch pros and cons.Comparison of three SAAS products that consume CloudWatch metrics.

NewRelic StackdriverLibrato

CloudWatch limitations

CloudWatch limitations...

● two week data retention● can’t create custom dashboards● UI is better than it used to be…● can’t time shift metrics to compare last

week to this week

However in CloudWatch’s defense

● good for ad hoc exploration of data● most up to date● can set thresholds and alert on them● can push custom metrics (does anyone use

this?)

Setting up access

Create IAM group and user

Give Access Key and Secret Key to Librato

Setting up access

Create IAM group and user

Launch t1.micro AMI from marketplace (or run it on your own server) Configure and run Sync process (is a ruby gem)

Setting up access

Create role for “cross account access” for “3rd party”, “read only”

Give stackdriver the Role ARN.

NewRelic Stackdriver Librato

3 months 13 months(Elite tier)

12 months (rolled up to 1 hour resolution)

Data retention

Features

NewRelic Stackdriver Librato

Custom dashboards ✓ ✓ ✓

Real time update ✓ ✓ ✓

Metric lag time display ✓

Time shift overlay ✓

Server agent ✓ ✓

Mobile client ✓

Custom metrics ✓ ✓ ✓

Events (like deployments) ✓ ✓ ✓

NewRelic Stackdriver Librato

integration with existing system

open up access to those who wouldn’t otherwise have access

easily define and clusters and aggregate

imports CloudTrail data, who did what when?

anomaly detection

integration with existing system

flexibility of data collection and retention

Key benefits IMHO

NewRelic Stackdriver Librato

Free if you already use NewRelic

But you need a server to run the collector

$12 per resource (Elite tier)

AWS Billing $0.60DynamoDB $1.10EBS $0.50EC2 $0.50ELB $0.65ElastiCache $1.95EMR $1.30OpsWorks $0.75RDS $0.70Redshift $0.75Route53 $0.05SNS $0.20SQS $0.40...

(per instance)

Costs (per month)

Say g’day to me

@dlutzy