it's all about telemetry

It’s all about telemetryMonitoring what matters in a useful way.

Tuesday, June 26, 12

Theo Schlossnagle @postwait

I write software

I write books

I give talks

I participate in the industry

I speak frankly about industry issues


Data, data, everywhere.

A billion pageviews / month.

100k database queries / second.

1MM memcache queries / second.

500k MQ messages / second.

10MM I/O operations / second.


Most new big data problems are

solvable

Big Data


Most new big data problems arecreated by our solutions, and thussolvabledespite their ROI

Most new big data problems are

solvable

Big Data


That’s a whole lot of data

Think in terms of logs (too many do)

About 26 trillion log lines / month

@ 40 bytes compressed: 1PB / month

Just because it is possibledoes not mean it will return on investment(and does not mean it won’t)


It’s all “useful”; which data?

Think in terms of cost/benefit.

Sure the data is useful, but it costs money to store

Does it cost you more to have it or not to have it?

Maybe the right approach is to keep that level of detail for a few days?


Double-edged sword.

Eroding granularity over timekeeps storage under control


Double-edged sword.

Eroding granularity over timekeeps storage under control

MISTAKE


1 yearat a glance


1 weeklooks normalish


1 dayconfidence of normalcy increases


1 weekthat looks different


1 dayyup, that’s not at all like that other week


Other methods

What do you store?

How do you store it?

Why is it useful?

Winning the cost benefit game byreducing costs more significantly thanreducing benefits


0 0.5 1 1.5 2 2.5 3

0.25

0.5

0.75

1

Benefit

Cost

Positive ValueBe in the green.

monitoring activity ➠


0 1 2 3 4 5 6 7 8 9 10

2.5

5

7.5

10

Benefit

Cost

There’s a bigger pictureIt’s not as easy as you think.



0 0.5 1 1.5 2 2.5 3

0.25

0.5

0.75

1

Benefit

Cost

Value is difference, not areaGreen can be misleading



0.5 1 1.5 2 2.5 3

-1

-0.75

-0.5

-0.25

0.25

0.5

Value = Benefit - CostGreen means we have positive return



0.5 1 1.5 2 2.5 3

-1

-0.75

-0.5

-0.25

0.25

0.5

It’s not about returnWell, it’s not only about return



0.5 1 1.5 2 2.5 3

-1

-0.75

-0.5

-0.25

0.25

0.5

It’s about maximizing returnThis is a bit like black magic



Technique 1: text

Store changes


Technique 2: numericStore rollups(i.e. statistical aggregates over fixed windows)

over 1 minute store

min/max/avg/stddev/covariance/50%/95%/99%

lots of information

heavy lossy compression of high-frequency data

loses population distribution information


Database replicationLag (green) and rate of lag change (purple)


Storage UsageWe can see growth.More useful, we can use this to project.


With simple numeric data


With simple numeric dataUnknowns can be predicted


With simple numeric dataIn sane ways with confidence


Full Disclosure

You see awesome examples of predictive analytics

Like the real-world one on the previous slide

In practice, almost all data streams predict one thing:

they have no fucking clue.


Technique 3: histograms

Store histograms

over 1 minute store

counts of datapoints seen in various buckets

retains complete population distribution

loss of precision


Histograms 101This.

This is a histogram.

It shows the frequency ofvalues within a population.

Height represents frequency


Histograms 101This.



Now, height and colorrepresents frequency


This.



Now, only colorrepresents frequency

Histograms 101


This.




Histograms ➠ time series


This.




Histograms ➠ time series

at a single time interval


API Service TimesWe can see a full population shiftof several milliseconds


Combining techniques

In our system (as a reference point)

Arbitrary numbers of numeric data pointson a single streamoccupy 32 bytes of space for statistical aggregates andoccupy about 2k of space for a histogram

These means we can store these transforms on numeric data in perpetuity


Combining techniques

Text is a bit harder

You need to be careful

Some data sources can be constantly changing

Producing gobs of change data

You’re doing it wrong

Find these and fix them


Correlating EventsChange Management vs. Performance


What to monitor?

Most people don’t monitor the things that matter most


Monitor the Business

Financials:

Revenues. Costs. Margins. AR. Account delinquency.

Marketing:

Web analytics. Campaigns. Costs. Returns. Convergence.


Monitor the Support

Customer Service:

Problems. Time investment. Customer satisfaction. Resolution time.


Monitor the Engineering

Engineering:

Deployments. Test coverage.Bug reports. Bug fixes. Effort spent.

Operations:

Faults. Pages. Escalations. Provisioning time. Equipment defect rates. 3rd party failure rates.


Monitor the Service

Systems:

Networks. Systems. Storage.

Databases:

Performance. Error rates. Backups.

Middleware:

Herein lies the magic and room for awesomeness


Monitor the Middleware

Your systems are complex

Monitor their interactions

Messaging, APIs, etc.


Monitor all the things.

But, perhaps most importantly...


Monitor all the things.

But, perhaps most importantly...

USE UNIFIED TOOLING


What we use...

reconnoiter

SNMP, nad, resmon, statsd, HTTP traps, jdbc, etc.

statsd (clients)

javascript beacons


Middleware mixAPI service times, traffic, user signup rates.


Thank you!


it's all about telemetry

Technology

frequency data

histograms store histograms

frequency of values

histograms time series

population shiftof

data streams

monitoring activity

numeric store rollups