Download - Lean Product Analytics by Dan Olsen
Lean Product Analytics Dan Olsen Olsen Solutions Feb 5, 2014
Copyright © 2014 Olsen Solu7ons
My Background n Educa7on
n BS, Electrical Engineering, Northwestern n MS, Industrial Engineering, Virginia Tech n MBA, Stanford n Web development and UI design
n 20 years of Product Management Experience n Managed submarine design for 5 years n 5 years at Intuit, led Quicken Product Management n Led Product Management at Friendster n CEO & Cofounder of YourVersion, “Pandora for your news” n Consultant: Box, YouSendIt, Chartboost, One Medical
Will post slides at hUp://slideshare.net/dan_o
Copyright © 2014 Olsen Solu7ons
What does “Lean” mean? n Lean Startup
n Achieving product-‐market fit n Tes7ng hypotheses & learning n Valida7ng MVP with users n Improving & itera7ng your product quickly
n Minimizing waste = using resources effec7vely
Copyright © 2014 Olsen Solu7ons
What’s the Formula for Product-‐Market Fit?
n A product that: n Meets customers’ needs n Is beUer than other alterna7ves n Is easy to use n Has a good value/price
Dan’s Model for the Causality Underlying Product-‐Market Fit
Copyright © 2014 Olsen Solu7ons
Target Customer
Product
Customer Needs
Customer has needs
You design & build product to meet needs
Customer decides how well product meets needs (sa7sfac7on)
What are Customers Reac7ng To When They Use Your Product?
Feature Set
UX Design Messaging
Copyright © 2014 Olsen Solu7ons
Valida7ng New vs. Exis7ng Products New Product Qualita7ve interviews
Exis0ng Product Quan7ta7ve data
Oprah Spock
How to be a Lean Product Ninja
+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+
+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+
slideshare.net/dan_o/
Copyright © 2014 Olsen Solu7ons
Iden7fy highest ROI idea
Design and Implement
Analyze How the Metric Changes
Brainstorm Ideas to
Improve Metric
Copyright © 2014 Olsen Solu7ons
Lean Product Analy7cs Process
Iden7fy What Your Metrics Are
Measure Metrics Baseline Values
Evaluate Metrics Upside Poten7al
Global Level
Metric Level
Select Top Metric
Learn & Iterate
Copyright © 2014 Olsen Solu7ons
n Net Promoter Score
Valida7ng Product-‐Market Fit: Surveys
Key follow-‐up ques7ons: • Why did you give the score you did? • What do we need to do to improve?
Qualita7ve Compliments Quan7ta7ve
Copyright © 2014 Olsen Solu7ons
QualWhy?
QuantWhat?
Copyright © 2014 Olsen Solu7ons
n Survey.io / Qualaroo.com n “How would you feel if you could no longer use Product X?”
n Very disappointed n Somewhat disappointed n Not disappointed
n General guideline: 40% or more “very disappointed” = product-‐market fit
Valida7ng Product-‐Market Fit: Surveys
Copyright © 2014 Olsen Solu7ons
n Asking a user ques7ons in an interview or survey n Valuable, but… n They’re telling you what they think they would do n Measurement bias (because you’re with them)
n Observing behavior n See what users actually do n Without you there
n Behavioral metrics for Product-‐Market Fit: n Prospects sign up = High conversion rate n They keep using it = High reten7on rate n They use it omen = High frequency of use n They’re deeply engaged with it = Long session 7mes n They pay for it = Revenue per customer
Product-‐Market Fit: Actual User Behavior Trumps Opinions
Valuable to Have a Holis7c Analy7cs Framework
Dave McClure’s “Startup Metrics for Pirates”
AARRR
Focus on right metric at right 7me
Using Analy7cs for Op7miza7on
n In addi7on to Product-‐Market Fit, you can apply the Lean Product Analy7cs Process to op7mize: n Your Business Results n Your User Experience
Copyright © 2014 Olsen Solu7ons
Profit = Revenue -‐ Cost
Unique Visitors x Ad Revenue per Visitor
Impressions/Visitor x Effec7ve CPM / 1000
Visits/Visitor x Pageviews/Visit x Impressions/PV
New Visitors + Returning Visitors
Invited Visitors + Uninvited Visitors
# of Users Sending Invites x Invites Sent/User x Invite Conversion Rate
Define the Equa7on of your Business “Peeling the Onion”
Adver7sing Business Model:
Copyright © 2014 Olsen Solu7ons
Copyright © 2014 Olsen Solu7ons
( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Rate
Paying Users x Revenue per Paying User
New Paying Users + Repeat Paying Users
Previous Paying Users x ( 1 – Cancella7on Rate )
Trial Users x Conv Rate
Profit = Revenue -‐ Cost
Equa7on of your Business: Subscrip7on Business Model
How to Track Your Metrics n Track each metric as daily 7me series
n Create ra7os from primary metrics: X / Y n Example: How good is your registra7on page? n Okay: # of registered users per day n BeUer: registra7on conversion rate = # registered users / # uniques to reg page
Date
Unique Visitors
Page views
Ad Revenue
New User Sign-‐ups …
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
…
Copyright © 2014 Olsen Solu7ons
Registra7on Page Conversion Rate
Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10
Dai
ly S
ignu
p Pa
ge Y
ield
Changedmessaging
Added questionsto signup page
Started requiringregistration
Copyright © 2014 Olsen Solu7ons
Registration Page Conversion Rate vs. Time
Reg
istra
tion
Pag
e C
onve
rsio
n R
ate
View Each Business Metric as a Gauge
Copyright © 2014 Olsen Solu7ons
Minimum Possible Value
Maximum Possible Value
Current Value
Copyright © 2014 Olsen Solu7ons
Priori7zing Product Ideas by ROI
Investment (developer-‐weeks)
Return (V
alue
Created
)
Idea C
Idea B
Idea D
Idea A
Idea F
1
1
2 3 4
2
3
4 ?
Iden7fying the “Cri7cal Few” Metrics n What is the upside poten7al of each metric? n How many resources will it take to “move the needle”?
n Developer-‐days, 7me, money n How much will the needle move? Revenue impact? n Which metrics have highest ROI opportuni7es?
Return
Investment
Return
Investment Re
turn
Investment
Metric A Good ROI
Metric B Bad ROI
Metric C Great ROI
Copyright © 2014 Olsen Solu7ons
Case Study from Intuit
q Improving UX q Improving Business Results
-‐> Sign-‐up Conversion Rate
Copyright © 2014 Olsen Solu7ons
Abandonment Rate (7 Day Moving Average)
0%
10%
20%
30%
40%
50%
60%
70%
80%
10/7
/02
10/1
4/02
10/2
1/02
10/2
8/02
11/4
/02
11/1
1/02
11/1
8/02
11/2
5/02
12/2
/02
12/9
/02
12/1
6/02
12/2
3/02
12/3
0/02
1/6/
03
1/13
/03
1/20
/03
Aba
ndon
men
t Rat
e (7
Day
Mov
ing
Ave
rage
)
Steps 1-2
Copyright © 2014 Olsen Solu7ons
Case Study: Account Signup Process Redesign
Copyright © 2014 Olsen Solu7ons
Analyzed Drop-‐Off at Each Major Sec7on
100%
62.3%58.8%
50.9%
34.4% 32.7%
0%
20%
40%
60%
80%
100%
% of U
sers
Sign in / Registra7on
Account Type Cash vs. Margin
5 Partner Pages
3 Partner Pages
Focus on biggest drop
Copyright © 2014 Olsen Solu7ons
Open Account
Sign in
Account Selec7on
Register
56%
44%
Forget Password
Registra7on Process
45% drop off (20% of total)
36% overall drop off for this step
70% (32% of Total)
17% drop off (10% of total)
20% drop off (6% of total)
30% (14% of Total)
80% (26% of Total)
55% (24% of Total)
64% of Total
Analysis of Sign In/Registra7on Flow
Change Password
83% (46% of Total)
Abandonment Rate (7 Day Moving Average)
0%
10%
20%
30%
40%
50%
60%
70%
80%
10/7
/02
10/1
4/02
10/2
1/02
10/2
8/02
11/4
/02
11/1
1/02
11/1
8/02
11/2
5/02
12/2
/02
12/9
/02
12/1
6/02
12/2
3/02
12/3
0/02
1/6/
03
1/13
/03
1/20
/03
Aba
ndon
men
t Rat
e (7
Day
Mov
ing
Ave
rage
)
Steps 1-2
Copyright © 2014 Olsen Solu7ons
Redesigned User Flow Improved Registra7on Conversion Rate
37% improvement in conversion rate
Released New Design
Case Study from Friendster
q Improving Business Results -‐> Viral New User Growth
Copyright © 2014 Olsen Solu7ons
• Which metric has highest ROI opportunity?
Case Study: Op7mizing Friendster’s Viral Loop
Active Users
Prospective Users
Invite Click
Succeed
Invite click-through rate
Conversion rate
Don’t Click
Fail
Invites per sender
% of users sending invites
• Mul7plied together, these metrics determine your viral ra7o
Users
% of users who are active
= 15% = 2.3
= 85%
Registration Process
Copyright © 2014 Olsen Solu7ons
The Upside Poten7al of a Metric
0
100%
0
100%
0
?
Registra7on Process Yield
% of users sending invita7ons
Avg # of invites sent per sender
2.3
85%
15%
Max possible improvement
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
Copyright © 2014 Olsen Solu7ons
Okay, so how can we improve the metric?
n How do we increase the average number of invites being sent out per sender?
n For each idea: n What’s the expected benefit? (how much will it improve the metric?)
n What’s the expected cost? (how many engineer-‐days will it take?)
n You want to iden7fy highest ROI idea
Copyright © 2014 Olsen Solu7ons
Amer Launching Address Book Importer…
Copyright © 2014 Olsen Solu7ons
Amer Launching Address Book Importer…
Copyright © 2014 Olsen Solu7ons
Amer Launching Address Book Importer…
Copyright © 2014 Olsen Solu7ons
If you could only track 1 metric to measure your Product-‐Market Fit:
Which metric would it be?
Copyright © 2014 Olsen Solu7ons
Reten7on Rate n Reten7on rate tracks what % of your customers are s7ll ac7ve over 7me
Copyright © 2014 Olsen Solu7ons
~80% never use app again
Curve eventually flattens out
Cohort Analysis
Copyright © 2014 Olsen Solu7ons
Cohort Analysis: Data
Copyright © 2014 Olsen Solu7ons
Improving Reten7on Rate Over Time= Increasing Product-‐Market Fit
David Skok, Matrix Partners hUp://www.forentrepreneurs.com/saas-‐metrics-‐2/
Alternate Ways to Track Reten7on n Having lots of cohort curves is hard to read n Would be great to have a 7me series metric = one metric we can track over 7me
n % Users Retained who signed up X days ago n Can use single or mul7ple X (30 & 90 days)
n Another metric: Returning users n Good summary metric: # of users “locking in” n Gives a sense of scale (not a %) n Recommend 7-‐day average (can do others too)
Copyright © 2014 Olsen Solu7ons
Profitability, anyone?
Two key metrics: • Customer Life7me Value (LTV) • Customer Acquisi7on Cost (CAC)
You want:
LTV – CAC > 0
Profitability, anyone?
Profitability, anyone?
Two key metrics: • Customer Life7me Value (LTV) • Customer Acquisi7on Cost (CAC)
You want:
LTV – CAC > 0
Life7me Value (LTV) n Life7me value of a customer = how much value your average customer will generate
n LTV = ARPU x Avg Customer Life7me x Gross Margin n ARPU (Avg Revenue / User) = Total Revenue / # of Users n Average Customer Life7me
n How long your average customer generates revenue n Equals 1 / churn rate (5% monthly churn = avg life 20 months)
n Gross Margin: the % of revenues lem over amer subtrac7ng the cost of providing the product/service
Copyright © 2014 Olsen Solu7ons
Note: for simplicity, this LTV equa7on ignores the “cost of capital”
Customer Acquisi7on Cost (CAC)
n CAC is the average cost for you to obtain a revenue-‐genera7ng customer
n So it takes into account both your cost of acquiring a prospect and your conversion rate for conver7ng prospects to revenue-‐genera7ng customers
n CAC=Cost per Acquisi7on / Conversion Rate
Copyright © 2014 Olsen Solu7ons
What You’d Like to See Over Time
Copyright © 2014 Olsen Solu7ons
n LTV increasing as you improve your value proposi7on, customer reten7on, & pricing
n CAC decreasing as you op7mize your marke7ng: segments, channels, messaging
Ra7o of LTV to CAC: Real data from HubSpot
Copyright © 2014 Olsen Solu7ons
Iden7fy highest ROI idea
Design and Implement
Analyze How the Metric Changes
Brainstorm Ideas to
Improve Metric
Copyright © 2014 Olsen Solu7ons
Lean Product Analy7cs Process
Iden7fy What Your Metrics Are
Measure Metrics Baseline Values
Evaluate Metrics Upside Poten7al
Metric Level
Select Top Metric
Learn & Iterate
Global Level
Questions? olsensolutions.com
linkedin.com/in/danolsen98
@danolsen
Copyright © 2014 Olsen Solu7ons