daily active users don't matter! | raj singh, tempo ai

Daily Active Users Don’t MatterA Blueprint for Collecting & Analyzing Actionable Mobile Metrics

Raj SinghCEO, Tempo AI, Inc.510-282-4229 (M)raj@tempo.ai@mobileraj

About Me – Mobile First Dude

• 20+ mobile titles since ‘99• 7+ apps hit #1 in the App Store/Catalog (Vanity Metric!)• MobiTV, Kodak Mobile, Dell Mobile, Skyfire, Recipes, Tempo• Mobile blogger for O’reilly; guest blogger at numerous

2 Kinds of Metrics

Vanity or Actionable?

Examples# of webpages indexed by Google% of users uploading a video on YouTubeDAU/MAU (Daily Actives / Monthly Actives)Registration vs skip registration CTR (click-thru rate) over time

Growth Hacking

The process is EXACTLY the same on mobile or any screen for that matter!

• Test• Measure• Repeat

But Mobile Has Caveats

• 3-4 week App Review cycles suck!• No HTML; native code is slow!• Disconnected data across screens; stats from web to App Store to app are

hard to connect

Prioritize your A/B tests!

Start With the Framework

AARRR (Dave McClure’s Pirate Framework)• Acquisition• Activation• Retention• Referral• Revenue

Other frameworks like Acquisition, Activation, Engagement, Virality are the same.

Identify What Type of App You Are

• Free / Advertising• 1-time Paid• In-App (Trending)• Subscription (Trending)

Why App Type Matters?

Time

Rete

ntion

Time

Rete

ntion

Time

Rete

ntion

Time

Rete

ntion

Productivity Smile Curve?

Network Effect?

Common…Paid App?

Case Study: Tempo Smart Calendar

App Type: SubscriptionI care about longer-term retention

App Category: ProductivityI care about unique acquisition strategies since we are not social

Goal for Launch: MVPMVP = Minimum Viable ProductViable = Until 50% of our Alpha group were DAUs

Tempo’s Timeline

Brainstorming

12/10 8/11

Founded

3/12

First Proto

2/13

Public Launch

Seeking Core Product Value

7/12

Optimize Activation & Engagement

6/13

1. Digest Data2. Peaks & Valleys3. Acquisition4. Activation

Engagement / Retention

AARRR is a great framework but you don’t necessarily need to attack it in that order as demonstrated with our timeline.

Seeking Core Product Value

• Created an Alpha Group: ~150 users‒ Initially 50 users; moved to iOS Enterprise acct (~150 was what we could manage)‒ Friends, advisors, investors (too biased)‒ Added friends of friends, non-tech types, high school buddies, former colleagues

• Test, learn & iterate w/ Alpha Group‒ Monthly builds for 9 months (until Launch)‒ Interviewed each member of the Alpha group regularly‒ Except for identifying peaks and valleys, data wasn’t useful (sample too small)‒ Interviews were a MUST; lots of inductive learning

• Searching for Hooks and Must-Haves‒ Hook = 3 Days (of active use); initial delight and activation‒ Must-Have = 3 Weeks (of active use)

Seeking Core Product Value (Cont)

• Conducted surveys to identify Must-Haves‒ Started with fill-in-the-blank Qs (eg What do use Tempo for)‒ Ended with “Choose from the list” and rank 1-5‒ Must-Haves = When >50% of people in our survey said they “Can’t live without this feature”

Survey #11. Why do you use Tempo? [Fill in the Blank]I like how it looks; The auto-dialing feature is great; 100+ responses2. Why don’t you use Tempo? [Fill in the Blank]It doesn’t have the date on the icon; I can’t set 2 alerts; 100+ responses

Survey #2• Distill results to just 5-8 things1. What features you can’t live without (Rate 1-5)

Seeking Core Product Value (Cont)

• Perform Craigslist UX testing‒ 50/50 split between target demographic and random‒ Testing focused on Hooks & Education‒ Always completed a test with a simple Q&A and a word game (ie What words amongst the

following adjectives best describe Tempo for you)

Accessible Desirable Gets in the way Patronizing Stressful

Appealing Easy to use Hard to use Personal Time-consuming

Attractive Efficient High quality Predictable Time-saving

Busy Empowering Inconsistent Relevant Too technical

Collaborative Exciting Intimidating Reliable Trustworthy

Complex Familiar Inviting Rigid Uncontrollable

Comprehensive Fast Motivating Simplistic Unconventional

Confusing Flexible Not valuable Slow Unpredictable

Connected Fresh Organized Smart Usable

Consistent Frustrating Overbearing Stimulating Useful

Customizable Fun Overwhelming Straight Forward Valuable

We Identified Core Product Value?

Tempo Alpha Group Users loves Auto-Conference Calls and I’m Late!But… Alpha data can be misleading• Small # of Users = Small Learnings• Big # of Users = Big Learnings• Goal for Launch was to get Big Users (10K) to learn big things• Why 10K?

‒ That’s a big # in the productivity category!‒ Productivity apps are generally not social‒ We thought we could achieve it!

• Actionable Stats‒ Funnel drop-off rate by screen?‒ Time spent on each screen (speed is important)‒ Download to conversion rate

• Alpha group wasn’t good data b/c they all activated! (Don’t let friends test apps)

So how did we test activation pre-launch?

Performed an out-of-market test in Canada• Used a different application name: Teempo Smart Calendar• Spent $500 (total) to acquire ~2-3K users with mobile ads; we optimized the ad

buying as well

Why Canada?• Because it was convenient

Results• Our Alpha group connected all their data but the Canadians did not? Privacy?• Retention wasn’t very good

Still Pre-Launch: Moved Focus to Activation

Canada Taught Us 2 Things

1. We need to improve the “don’t connect anything” education in Tempo2. Users acquired via cheap mobile ads was not the right segment for us;

mostly gamers

We improved education in Tempo but we didn’t have enough time to run another Canadian test (running out of money)• Focused on blind Craigslist testing and heuristics usability testing• Heuristics Usability = Asking experts for feedback

Still Pre-Launch: Moved Focus to ASO (App Store Optimization)

• Identified what we could A/B test‒ App Store description‒ App Store images‒ App Title‒ App Keywords

• Created 10s of permutations and tested• Couldn’t measure acquisition to retention (unfortunately)

Buying Ads – We Tested A Lot!

We even created a fake HTML App Store landing page to test different descriptions

Results• Not enough statistical variance between words:

“Smart,” “AI,” “Intelligent,” “Magic” etc (not conclusive)

• Users preferred icons that looked like calendars• App screenshots with descriptions outperformed

screenshots without descriptions

Living Cal

Calendar Assistant

Useful Calendar

Smart Cal

Smart Calendar

Smart Calendar #3

Cal Assistant

Intelligent Calendar 0.50

0.50

Actionable Cal

0.60

Smart Calendar #4

Magic Calendar

0.51

0.52

Smart Calendar #1

0.61

0.53

Living Calendar

0.55

0.63

Smart Calendar #2

Learning Cal

0.50

0.40

0.43

0.45

0.46

0.59

We’re Ready – Gearing Up For Launch

My Pre-Launch Blueprint• Identify Core Product Value (Hooks & Must-Haves: UI, Conference Calls &

I’m Late)• Optimize Activation / On-Boarding (Reduced the # of screens and clicks)• Identify Peaks and Valleys from Alpha data (Improved in-app education for

the more “private user”)• Optimize Acquisition / Discovery (ASO: A/B test keywords, app title, icon,

description)

(See slide 10 for the timeline)

Launch Day

• Exceeded our goal of 10K users• What do we focus on now???

‒ Look for surprises: Users were 24X more data than our Canada Alpha!‒ New Peaks & Valleys ‒ Activation, On-Boarding

• It takes 3 months before you have enough data to really understand engagement and retention

Post-Launch: Studied On-Boarding

• 30-40% bounce rate on registration :/• Possible A/B Tests:

‒ Facebook or LinkedIn registration‒ Skip

• Users confused between Tempo registration and connecting their calendar

More tests planned for future (but not the priority at present)

100%

70%

62%

62%

60%

59%

Downloads*Account Creation At-temptedAccount Creation SuccessfulView Calendar/Contacts Screen

Post-Launch: Studied Initial Bounce Rate

• 34% of users left after the 1st day • Data did not answer Day 1 bounce rates, so:

‒ Identified 100 users from support who bounced after 1-3 days‒ Sent a personal email and spoke to a handful on the phone‒ Considered a survey but it doesn’t work with dis-engaged users. Surveys yield responses

from your fans!

Results From Discussions• Email connectivity issues• Sync issues• Privacy• Confusion / Education

We can fix these things!

Post-Launch: Seeking Peaks & Valleys

Examples• The Settings button was the most popular button? (Peak)• FB connect errors as a % of FB actions were really high (Peak)• Adding Event was the most popular action (Peak)• Few customized their Agenda view image (Valley)• Few clicked on Foursquare or Yelp (Valley)

Post-Launch: Seeking Peaks & Valleys (Cont)

Valleys• Discovery/education or uninteresting feature?• Create simple A/B tests to answer

Example A/B TestHypothesis: Users didn’t customize Agenda View photos because they didn’t know they couldTest: Sent an email with tips after Tempo user registrationResult: Was no longer a valley. Test succeeded

Post-Launch: Seeking Peaks & Valleys (Cont)

Peaks• Unexpected or a feature• Create simple A/B tests to answer

Example A/B TestHypothesis: Users clicked Settings because they didn’t know how to change viewsTest: Move the menu from the top-middle to the top-leftResult: Settings was no longer a peak. Test succeeded

Post Launch: Seeking Peaks & Valleys (Cont)

Not all A/B tests are as cut and dry

A/B TestHypothesis: “Post Happy B-Day” is a popular actionTest: Let people “Post Happy B-Day” from the Calendar ViewResult: No change. Test was not successful?

Prioritize your A/B tests! Test, Measure, Repeat

3 Months After Launch: Understanding Retention

• Defined an internal goal of 20% after 3 months‒ Got comps from other apps in the category (eg Evernote etc)‒ It’s a great # and it’s good to have goals

• Tempo: 10-20% Retention‒ Current priority!

And so we started with the questions to answer:• How many days until we see the biggest drop-off in retention?• Why after using Tempo for X weeks, do you leave?• Are certain features in the app more or less used by more or less engaged

users?• Who is our user?

3 Months After Launch: Understanding Retention (Cont)

The process is exactly the same!

• Identify users from support and other channels and contact them!• Create a survey but it doesn’t substitute for speaking!• Looked for peaks & valleys but on a per active user and per session basis

over time (actionable stats)• Created A/B tests and executed

But one difference:• Look for correlations

3 Months After Launch: Looking For Correlations

Correlation = RatiosRatios ~= Actionable Data

Tempo ExamplesUsers with Exchange are more engagedUsers who use conference calling are more engagedUsers with Facebook events only are less engaged

Create hypothesis and measure the data over time

My Post-Launch Blueprint

• Look for surprises (Fixed settings button click rate)• Study activation workflow in aggregate (Fixed email registration confusion)• Identify, interview & talk to users (Improved education)• Find peaks and valleys (A/B tested Facebook birthday post)• Look for correlations (Lots of Excel!)• A/B test and prioritize your tests

What’s Next? (Conversation For Next Year?)

• Achieve our 20% 3 month retention goal• Then we focus on acquisition (Virality = Frequency X Reach X Yield)• Then we focus on monetization

No matter where you are in AARRR, the process is the same:Test, Measure, RepeatTalk to your usersPrioritize your A/B tests