Metrics

You are what you measure

@robdcrowley

robdcrowley

Empower you with new ideas to bring your team together!

Metrics. What are they? What to measure? How to visualise?

Tooling. Some secret sauce to make our lives easier

Metrics as a catalyst for cultural change

Goals for Session

Metrics 101What metrics are and why we measure

To informTo compareTo motivate To understandTo improveTo protectNot: Just because successful companies do!

Why do we measure

A quantifiable measure of any component or process used to gauge the performance of your business

– Application– Business– People– Process

– System

What is a metric

Basis of measurementAssumptionsLevel and usageDesired trendWhen to useWhen to stopHow to gameWarnings

Facets of a metric

Red is good, Green is worthless

Metrics 201What should you measure

“Lack of direction, not lack of time, is the problem. We all have twenty-four hour days.”

- Zig Zigler

Pivot: Base conversations on metrics instead of non actionable opinions

Use stakeholder input as the basis for your initial set of metrics

– “What do you want out of this?”– “How quickly do you want this?”

Metrics can help guide us

Awesome book, poor title

Core Principles

– Minimum Viable Product (MVP)– Continuous Deployment– Split Testing (aka A/B Testing)– Actionable Metrics– Pivot– Innovation Accounting– Build-Measure-Learn Loop

Acquisition – finding new usersActivation – getting users to give your product a tryRetention – making sure those users stick aroundReferral – have your loyal users invite othersRevenue – making some money from all this

Note: These steps are not strictly sequential

Pirate Funnel (AARRR!)

Do not focus solely on metrics that focus on reducing errors

Focus on achieving excellence and positive outcomes will ensue

DevOps State of the Union focuses on these four core metrics

– Deployment Frequency– Lead Time for Changes

– Mean Time to Recover (MTTR)– Failed Change Rate

Example Metrics

Fact: We only realise business value from the work we do when it’s released into production

The bottleneck or constraint is the limiting factor on our ability to deliver more quickly

Strengthening any link other than the weakest is a waste of time and effort

Use metrics to identify current constraint

Theory of Constraints

Metrics 301How to visualise and share metrics

Display all metrics on a dashboard

– Business (i.e. Application dropouts)– Dev (i.e. Performance metrics)– Ops (i.e. Web Server CPU Usage)

Single URL for all data

Make it easy for colleagues to access dashboard

Visualisation

Use a tool that can handle different kinds of metrics

A few standouts are:

– StatsD/Graphite/Grafana – Heka

– Logstash/Elastic Seach/Kibana

Tools

StatsD WhisperCarbon

GraphiteDashboardApps

UDP

TCP

HTTP

Graphite and StatsD

Metrics 401Potential pitfalls when using metrics

Case Study: US Health Service

In the 1990s the US Health Service decided to make hospital mortality rates public. What better metric for hospital performance?

Observed Behaviour: Best way to improve rating was not to admit critically ill patients in the first place

You are what you measure

“The actual company values, as opposed to the nice-sounding values, are shown by who

gets rewarded, promoted or let go”

- Patty McCord

Case Study: Enron

Enron which went bankrupt from fraud and whose leaders went to jail, had the following company values:

Integrity, Communication, Respect, Excellence

Where these really the core values at Enron?

You are what you measure

“In the absence of clarity around an objective, any measure will do”

- Bob Paladino

Too Many Metrics

Depict rosiest picture possible but do not accurately reflect the key drivers of your business

– Inaccessible– Not auditable– Not actionable

Vanity Metrics

Metrics based on output rather than outcomes

Aim to maximise outcomes and minimise output

– Total lines of code– Number of bugs fixed– Total hours worked– Lines of code per developer

Useless Metrics

“We’re not doing this data analysis thingjust because. We’re gathering data points

because we’ve seen it work to greateffect in other organisations and industries.”

- Anonymous

Cargo Cult Metrics

Metrics being used for evil rather than for good

Confronting this organisational anti-pattern can be a good first step

Information sharing is the best defense

Break down knowledge silos in the organisation

Weaponized Metrics

Questions