daily active users don't matter! | raj singh, tempo ai
DESCRIPTION
Raj Singh, CEO, Tempo AI A blueprint for collecting & analyzing actionable mobile metricsTRANSCRIPT
Daily Active Users Don’t MatterA Blueprint for Collecting & Analyzing Actionable Mobile Metrics
Raj SinghCEO, Tempo AI, Inc.510-282-4229 (M)[email protected]@mobileraj
About Me – Mobile First Dude
• 20+ mobile titles since ‘99• 7+ apps hit #1 in the App Store/Catalog (Vanity Metric!)• MobiTV, Kodak Mobile, Dell Mobile, Skyfire, Recipes, Tempo• Mobile blogger for O’reilly; guest blogger at numerous
2 Kinds of Metrics
Vanity or Actionable?
Examples# of webpages indexed by Google% of users uploading a video on YouTubeDAU/MAU (Daily Actives / Monthly Actives)Registration vs skip registration CTR (click-thru rate) over time
Growth Hacking
The process is EXACTLY the same on mobile or any screen for that matter!
• Test• Measure• Repeat
But Mobile Has Caveats
• 3-4 week App Review cycles suck!• No HTML; native code is slow!• Disconnected data across screens; stats from web to App Store to app are
hard to connect
Prioritize your A/B tests!
Start With the Framework
AARRR (Dave McClure’s Pirate Framework)• Acquisition• Activation• Retention• Referral• Revenue
Other frameworks like Acquisition, Activation, Engagement, Virality are the same.
Identify What Type of App You Are
• Free / Advertising• 1-time Paid• In-App (Trending)• Subscription (Trending)
Why App Type Matters?
Time
Rete
ntion
Time
Rete
ntion
Time
Rete
ntion
Time
Rete
ntion
Productivity Smile Curve?
Network Effect?
Common…Paid App?
Case Study: Tempo Smart Calendar
App Type: SubscriptionI care about longer-term retention
App Category: ProductivityI care about unique acquisition strategies since we are not social
Goal for Launch: MVPMVP = Minimum Viable ProductViable = Until 50% of our Alpha group were DAUs
Tempo’s Timeline
Brainstorming
12/10 8/11
Founded
3/12
First Proto
2/13
Public Launch
Seeking Core Product Value
7/12
Optimize Activation & Engagement
6/13
1. Digest Data2. Peaks & Valleys3. Acquisition4. Activation
Engagement / Retention
AARRR is a great framework but you don’t necessarily need to attack it in that order as demonstrated with our timeline.
Seeking Core Product Value
• Created an Alpha Group: ~150 users‒ Initially 50 users; moved to iOS Enterprise acct (~150 was what we could manage)‒ Friends, advisors, investors (too biased)‒ Added friends of friends, non-tech types, high school buddies, former colleagues
• Test, learn & iterate w/ Alpha Group‒ Monthly builds for 9 months (until Launch)‒ Interviewed each member of the Alpha group regularly‒ Except for identifying peaks and valleys, data wasn’t useful (sample too small)‒ Interviews were a MUST; lots of inductive learning
• Searching for Hooks and Must-Haves‒ Hook = 3 Days (of active use); initial delight and activation‒ Must-Have = 3 Weeks (of active use)
Seeking Core Product Value (Cont)
• Conducted surveys to identify Must-Haves‒ Started with fill-in-the-blank Qs (eg What do use Tempo for)‒ Ended with “Choose from the list” and rank 1-5‒ Must-Haves = When >50% of people in our survey said they “Can’t live without this feature”
Survey #11. Why do you use Tempo? [Fill in the Blank]I like how it looks; The auto-dialing feature is great; 100+ responses2. Why don’t you use Tempo? [Fill in the Blank]It doesn’t have the date on the icon; I can’t set 2 alerts; 100+ responses
Survey #2• Distill results to just 5-8 things1. What features you can’t live without (Rate 1-5)
Seeking Core Product Value (Cont)
• Perform Craigslist UX testing‒ 50/50 split between target demographic and random‒ Testing focused on Hooks & Education‒ Always completed a test with a simple Q&A and a word game (ie What words amongst the
following adjectives best describe Tempo for you)
Accessible Desirable Gets in the way Patronizing Stressful
Appealing Easy to use Hard to use Personal Time-consuming
Attractive Efficient High quality Predictable Time-saving
Busy Empowering Inconsistent Relevant Too technical
Collaborative Exciting Intimidating Reliable Trustworthy
Complex Familiar Inviting Rigid Uncontrollable
Comprehensive Fast Motivating Simplistic Unconventional
Confusing Flexible Not valuable Slow Unpredictable
Connected Fresh Organized Smart Usable
Consistent Frustrating Overbearing Stimulating Useful
Customizable Fun Overwhelming Straight Forward Valuable
We Identified Core Product Value?
Tempo Alpha Group Users loves Auto-Conference Calls and I’m Late!But… Alpha data can be misleading• Small # of Users = Small Learnings• Big # of Users = Big Learnings• Goal for Launch was to get Big Users (10K) to learn big things• Why 10K?
‒ That’s a big # in the productivity category!‒ Productivity apps are generally not social‒ We thought we could achieve it!
• Actionable Stats‒ Funnel drop-off rate by screen?‒ Time spent on each screen (speed is important)‒ Download to conversion rate
• Alpha group wasn’t good data b/c they all activated! (Don’t let friends test apps)
So how did we test activation pre-launch?
Performed an out-of-market test in Canada• Used a different application name: Teempo Smart Calendar• Spent $500 (total) to acquire ~2-3K users with mobile ads; we optimized the ad
buying as well
Why Canada?• Because it was convenient
Results• Our Alpha group connected all their data but the Canadians did not? Privacy?• Retention wasn’t very good
Still Pre-Launch: Moved Focus to Activation
Canada Taught Us 2 Things
1. We need to improve the “don’t connect anything” education in Tempo2. Users acquired via cheap mobile ads was not the right segment for us;
mostly gamers
We improved education in Tempo but we didn’t have enough time to run another Canadian test (running out of money)• Focused on blind Craigslist testing and heuristics usability testing• Heuristics Usability = Asking experts for feedback
Still Pre-Launch: Moved Focus to ASO (App Store Optimization)
• Identified what we could A/B test‒ App Store description‒ App Store images‒ App Title‒ App Keywords
• Created 10s of permutations and tested• Couldn’t measure acquisition to retention (unfortunately)
Buying Ads – We Tested A Lot!
We even created a fake HTML App Store landing page to test different descriptions
Results• Not enough statistical variance between words:
“Smart,” “AI,” “Intelligent,” “Magic” etc (not conclusive)
• Users preferred icons that looked like calendars• App screenshots with descriptions outperformed
screenshots without descriptions
Living Cal
Calendar Assistant
Useful Calendar
Smart Cal
Smart Calendar
Smart Calendar #3
Cal Assistant
Intelligent Calendar 0.50
0.50
Actionable Cal
0.60
Smart Calendar #4
Magic Calendar
0.51
0.52
Smart Calendar #1
0.61
0.53
Living Calendar
0.55
0.63
Smart Calendar #2
Learning Cal
0.50
0.40
0.43
0.45
0.46
0.59
We’re Ready – Gearing Up For Launch
My Pre-Launch Blueprint• Identify Core Product Value (Hooks & Must-Haves: UI, Conference Calls &
I’m Late)• Optimize Activation / On-Boarding (Reduced the # of screens and clicks)• Identify Peaks and Valleys from Alpha data (Improved in-app education for
the more “private user”)• Optimize Acquisition / Discovery (ASO: A/B test keywords, app title, icon,
description)
(See slide 10 for the timeline)
Launch Day
• Exceeded our goal of 10K users• What do we focus on now???
‒ Look for surprises: Users were 24X more data than our Canada Alpha!‒ New Peaks & Valleys ‒ Activation, On-Boarding
• It takes 3 months before you have enough data to really understand engagement and retention
Post-Launch: Studied On-Boarding
• 30-40% bounce rate on registration :/• Possible A/B Tests:
‒ Facebook or LinkedIn registration‒ Skip
• Users confused between Tempo registration and connecting their calendar
More tests planned for future (but not the priority at present)
100%
70%
62%
62%
60%
59%
Downloads*Account Creation At-temptedAccount Creation SuccessfulView Calendar/Contacts Screen
Post-Launch: Studied Initial Bounce Rate
• 34% of users left after the 1st day • Data did not answer Day 1 bounce rates, so:
‒ Identified 100 users from support who bounced after 1-3 days‒ Sent a personal email and spoke to a handful on the phone‒ Considered a survey but it doesn’t work with dis-engaged users. Surveys yield responses
from your fans!
Results From Discussions• Email connectivity issues• Sync issues• Privacy• Confusion / Education
We can fix these things!
Post-Launch: Seeking Peaks & Valleys
Examples• The Settings button was the most popular button? (Peak)• FB connect errors as a % of FB actions were really high (Peak)• Adding Event was the most popular action (Peak)• Few customized their Agenda view image (Valley)• Few clicked on Foursquare or Yelp (Valley)
Post-Launch: Seeking Peaks & Valleys (Cont)
Valleys• Discovery/education or uninteresting feature?• Create simple A/B tests to answer
Example A/B TestHypothesis: Users didn’t customize Agenda View photos because they didn’t know they couldTest: Sent an email with tips after Tempo user registrationResult: Was no longer a valley. Test succeeded
Post-Launch: Seeking Peaks & Valleys (Cont)
Peaks• Unexpected or a feature• Create simple A/B tests to answer
Example A/B TestHypothesis: Users clicked Settings because they didn’t know how to change viewsTest: Move the menu from the top-middle to the top-leftResult: Settings was no longer a peak. Test succeeded
Post Launch: Seeking Peaks & Valleys (Cont)
Not all A/B tests are as cut and dry
A/B TestHypothesis: “Post Happy B-Day” is a popular actionTest: Let people “Post Happy B-Day” from the Calendar ViewResult: No change. Test was not successful?
Prioritize your A/B tests! Test, Measure, Repeat
3 Months After Launch: Understanding Retention
• Defined an internal goal of 20% after 3 months‒ Got comps from other apps in the category (eg Evernote etc)‒ It’s a great # and it’s good to have goals
• Tempo: 10-20% Retention‒ Current priority!
And so we started with the questions to answer:• How many days until we see the biggest drop-off in retention?• Why after using Tempo for X weeks, do you leave?• Are certain features in the app more or less used by more or less engaged
users?• Who is our user?
3 Months After Launch: Understanding Retention (Cont)
The process is exactly the same!
• Identify users from support and other channels and contact them!• Create a survey but it doesn’t substitute for speaking!• Looked for peaks & valleys but on a per active user and per session basis
over time (actionable stats)• Created A/B tests and executed
But one difference:• Look for correlations
3 Months After Launch: Looking For Correlations
Correlation = RatiosRatios ~= Actionable Data
Tempo ExamplesUsers with Exchange are more engagedUsers who use conference calling are more engagedUsers with Facebook events only are less engaged
Create hypothesis and measure the data over time
My Post-Launch Blueprint
• Look for surprises (Fixed settings button click rate)• Study activation workflow in aggregate (Fixed email registration confusion)• Identify, interview & talk to users (Improved education)• Find peaks and valleys (A/B tested Facebook birthday post)• Look for correlations (Lots of Excel!)• A/B test and prioritize your tests
What’s Next? (Conversation For Next Year?)
• Achieve our 20% 3 month retention goal• Then we focus on acquisition (Virality = Frequency X Reach X Yield)• Then we focus on monetization
No matter where you are in AARRR, the process is the same:Test, Measure, RepeatTalk to your usersPrioritize your A/B tests