kpis for apis (and how api calls are the new web hits, and you may be measuring all wrong)

KPIs for APIs

or, why API Calls are the new Web Hits

and you may be measuring all wrong

John Musser / @johnmusser / API Science / apiscience.com Business of APIs Conference, NYC, 2014

1995

Why do we need a Web site?

2000 2005 2010

Of course we have a Web site

Of course we have an API

Why do we need an API?

1995

Why do we need a Web site?

2000 2005 2010

Of course we have a Web site

Of course we have an API

Why do we need an API?

You are here

Remember hit counters?

Hits Page Views Visitors Conversions

Evolution of web site metrics

Calls ? ? ?

Evolution of API metrics?

Great APIs get measured

But measure what?

Well, it depends…

Why do you have an API?

Extend product

BizDev / LeadGen Marketing channel

Upsell opportunity

Content acquisition

Device and mobile support

API as Product Distribution channel

Increase stickiness

New line of business

Accelerate internal projects

Drive innovation

User acquisition

Partner opportunities Drive traffic

Increase footprint

Who is your API for?

API consumer continuum

You Your partners & customers

Everyone else

Internal Open Private

The answers to who & why

should drive your KPIs

Traffic Total calls

Top methods Call chains Quota faults

Developers Total developers

Active developers Top developers Trending apps

Retention

Service Performance Availability Error rates

Code defects

Marketing Dev registrations Dev portal funnel Traffic sources Event metrics

Support Support tickets Response times

Business Direct revenue

Indirect revenue Market share

Costs

Many ways to think about API KPIs…

Customer Satisfaction

NPS Churn

Community Forum activity Social media

(both for API and apps built on it)

Quality Defects

Error prone APIs Repair rate

API Maturity Stability

Maturity metric Change rate

Innovation Number of products

Number of apps Speed to market

More ways to think about API KPIs…

Channel Call volume: Mobile,

web, others Revenue by channel

There are a lot of API KPIs

you could measure

Don’t drown in API KPIs

Great APIs prioritize

what to measure

KPI SECRET #1

Choose your “KPI lens” CFO lens CMO lens CEO lens Ops lens

QA lens CRO lens PM lens PR lens

CFO lens: financial metrics Total direct and indirect revenue

Margin ROI

Marginal cost Capex / Opex investment

CLTV per developer

DevOps lens: operational metrics Performance Availability Error rates

Faults by API type Defect backlog

API traffic trends

CMO lens: marketing metrics Developer acquisition cost Developer demographics

Portal visitor count & traffic sources SEO / SEM metrics

Social media metrics Evangelism and event metrics

Know your Developer Funnel

KPI SECRET #2

Know your Developer Funnel

KPI SECRET #2

(for public APIs)

Web sites have funnels

Acquisi9on

Ac9va9on

Referral

Revenue

Reten9on

How do users find you?

Does the user have a good first experience?

Do users return?

How do you make money?

Do users tell others?

Dave McClure’s “Startup Metrics for Pirates”

APIs have funnels too

Acquisi9on

Ac9va9on

Referral

Revenue

Reten9on

How do developers find you?

Does the developer have a good first experience?

Do developers return?

How do you and developers make money?

Do developers tell others?

Potential KPIs

Acquisi9on

Ac9va9on

Referral

Revenue

Reten9on

Dev portal site: sources, traffic vol, uniques

# dev registraCons, API keys issued

# acCve devs, API call volume by dev, # apps

$ per dev, $ per app, # end users

# dev referrals, dev/app growth, NPS

Measure your API ecosystem

KPI SECRET #3

Apps Developers App Users

API 1

API 2

API 3

Your APIs You


API 1

API 2

API 3

Your APIs You

Total app users Revenue per user User growth rate

User churn


API 1

API 2

API 3

Your APIs You

Total apps Trending apps

Revenue per app Channel: mobile/web


API 1

API 2

API 3

Your APIs You

Total developers Active developers

Revenue per developer Top developers

Stuck developers


API 1

API 2

API 3

Your APIs You

Total API calls Top API calls

Top API packages Internal usage


API 1

API 2

API 3

Your APIs You

Total revenue Partnerships Market share

Innovation

Most APIs obey the

Power Law

KPI SECRET #4

The Power Law of APIs

Developers

API c

all vo

lume

The Power Law of APIs Most of your API traffic, revenue, success, etc….

Developers

API c

all vo

lume

Think quality, not just quantity Use your API KPIs, to find who, how, where, when.

Developers

API c

all vo

lume

API KPIs need buy-in

KPI SECRET #5

Don’t let API KPIs live in a vacuum

Who are your stakeholders?

Who’s your sponsor?

Get executive buy-in on your top 3-5 KPIs

Measuring internal API usage often just

as important as external

KPI SECRET #6

API usage from internal apps

External usage

Guardian: 70% internal calls

Evernote: 99% internal calls

API usage from internal apps

External usage

API usage from devices

External usage

Netflix: 99.9% internal calls

Netflix: 99.9% internal calls

Why Enable Netflix experience

Who Netflix device and UI teams

Beware of API vanity metrics

KPI WARNING #1

API Billionaire Club

5 billion API calls / day (April 2010)

1 billion API calls / day (Q1 2012)

5 billion API calls / day (October 2009)

13 billion API calls / day (May 2011)

1 billion API calls / day (January 2012)

1.1 billion API calls / day (April 2011)

1 billion API calls / day (May 2012)

5 Billion API calls/day

“What if, all things being equal, instead of 30 billion API calls a month,

we served 5 billion?” Daniel Jacobson, Netflix

Beware of API KPI traps

KPI WARNING #2

Not having a metrics process (have a metrics plan & a feedback loop)

Using the wrong metrics (don’t be afraid to change what you track)

Unrealistic expectations (“Absolutely, we’ll have X developers on-board by X date”)

3 common API KPI traps

One important metric you may have missed...

What’s your TTFHW? Time To First “Hello World” aka: how long from zero to 60?

What’s your TTFHW? Developer experience (DX) can make or break your API.

What’s your TTFHW? Too subjective to measure?

What’s your TTFHW? Don’t forget about qualitative metrics

What’s your TTFHW? User testing.

Surveys. Forums.

Social media…

Real world API KPIs

“We are seeing a trend where a person who uses another app with Evernote is

50% more likely to move on to our premium (paid) service”

KPI: Conversions

“Apps that have an API integration with Evernote are seeing much more usage:

Pocket readers normally read 15% of what they save, but Evernote users read 80% of

what they save in Pocket.”

KPI: Engagement

API now accounts for 20-25%

of the charity’s annual revenue

KPI: Revenue

KPI: Photo Edits

KPI: Apps

3,500 Apps Aviary, March, 2013

Goal: “more eyeballs on our content”

API calls (requests) Response volume (ex: # stories) Impressions (via image beacon) Loyalty

MoreKPIs

KPI: Impressions

“We sell Vodka. But people buy drinks.”

Goal: “ensure top quality drinks consistently delivered to people

regardless of where people encounter Absolut”

KPI: Recipe delivery

“We sell Vodka. But people buy drinks.”

Goal: “ensure top quality drinks consistently delivered to people

regardless of where people encounter where people encounter

Absolut”

KPI: Recipe delivery APIs let you be where your customers are.

Measure how.

Calls ? ? ?

Evolution of API metrics?

Who & why drives KPIs Have a KPI lens & process

Prioritize & repeat

Thank You

[email protected] @johnmusser

Race car: hCp://www.flickr.com/photos/lim_lik_wei/3270522646/ Stopwatch: hCp://www.flickr.com/photos/purplemaMish/3020016417/ Stop sign: hCps://www.flickr.com/photos/9toperez/5791194358 Power law: hCp://en.wikipedia.org/wiki/Power_law

Photo Credits

kpis for apis (and how api calls are the new web hits, and you may be measuring all wrong)

Internet