in-market experiments: the science of placing small bets fast
TRANSCRIPT
Meeting Logistics
› 35 minute presentation
› 25 minutes for Q&A
› Submit questions through Q&A widget at anytime—some answered during webinar and some post-webinar
› Slides and replay of event available after webinar
› Download our first webinar, Design Thinking for the Business Case: http://peerinsight.com/downloads/dt4bcwebinar.pdf
ABOUT TODAY’S WEBINAR
1
Tim Ogilvie is co-founder of Peer Insight. He is keen on helping large organizations create new experiences, services, and business models.
Tim is co-author of Designing for Growth but spends most of his time practicing in the field, not writing or teaching. Tim is passionate about entrepreneurship and placing small bets fast.
Jessica Dugan is a Senior Design Consultant at Peer Insight with a passion for people and their stories. Drawing on her background in design research and strategy, Jessica uses this passion to collaborate in the creation of innovative services that improve the quality of the life. She holds a Masters of Design and MBA from the Institute of Design in Chicago.
TODAY’S PRESENTERS
2
HOW DO LEAPS OF FAITH OCCUR?
Progressive: Rate Comparison “Ticker” (2001)
HP: Snapfish 1-hour print (2006)
The Hartford: FleetAhead (2009)
AARP: Life Reimagined (2011)
Will side-by-side quotes drive away customers we want to win? If we refer our
customer to a nearby retail photo finisher, do we lose her for life?
Will our fleet customers trade privacy for lower insurance premiums?
If we offer a different kind of service, can we expand our customer base?
3
Asked to describe design, Tim Brennan of Apple’s Creative Services drew the following picture:
DESIGN THINKING IS OFTEN TREATED AS A MYSTERY
? $
5
OUR BUSINESS CASE METHOD FOLLOWS FIVE STEPS
*Adapted from Rita McGrath, “Discovery-driven Planning”
1. Formulate a novel value proposition
2. Define future success as an income statement*
3. Spell out the assumptions
› What would have to be true?
4. Test the make-or-break assumptions
› Thought experiments
› In-market experiments
5. Based on test results, refine the model, refine the value proposition, or re-assess the investment potential of this opportunity
6
1. Formulate a novel value proposition
2. Define future success as an income statement*
3. Spell out the assumptions
› What would have to be true?
4.
5. Based on test results, refine the model, refine the value proposition, or re-assess the investment potential of this opportunity
4. Test the make-or-break assumptions
› Thought experiments
› In-market experiments
OUR BUSINESS CASE METHOD FOLLOWS FIVE STEPS
*Adapted from Rita McGrath, “Discovery-driven Planning”
7
Thought Experiments
Hypothetical experiments that use logic and existing data.
› Envisioned experiences, require imagination
› Set in the future
› 2D and 3D
In-Market Experiments
Experiments conducted in the marketplace with actual partners and customers.
› Live experiences, meant to feel real (even if some elements are faked)
› Set in the present
› 4D, persist over time (even in brief)
THERE ARE TWO WAYS TO TEST ASSUMPTIONS
8
Progressive: Rate Comparison “Ticker” (2001)
HP: Snapfish 1-hour print (2006)
The Hartford: FleetAhead (2009)
AARP: Life Reimagined (2011)
REMEMBER THESE LEAPS OF FAITH?
Will side-by-side quotes drive away customers we want to win? If we refer our
customer to a nearby retail photo finisher, do we lose her for life?
Will our fleet customers trade privacy for lower insurance premiums?
If we offer a different kind of service, can we expand our customer base?
9
tantalizingpossibility
A B
PARALYSIS
what lies beneath
chasm of unintended consequences
potentialreward
POUNCE
A B
PARALYSIS POUNCE
ANOTHER INNOVATOR’S DILEMMA
10
SIX SINS OF IN-MARKET EXPERIMENTS
1. Hi-Fidelity Believing it has to look and feel real/polished
2. Quantifying Thinking it must yield quantitatively significant (and repeatable) results
3. Going Solo Using only your firm’s resources and IP to stand up the experiment—even if it takes longer
4. Business Model Next Focusing on the user experience and technical solution now and worrying about defensibility and scalability later
5. IP Protection Keeping all the ideas concealed until they are patent-protected
6. 100% Belief Committing to a specific vision of the solution and looking only for signs of success
11
tantalizingpossibility
A B
PARALYSIS
what lies beneath
chasm of unintended consequences
potentialreward
POUNCE
A B
PARALYSIS POUNCE
TWO POSSIBLE RESULTS – BOTH BAD
67%Unexplored opportunities
33%Large bets placed slowly (too-big-to-fail)
12
tantalizingpossibility
PROBE
what lies beneath
chasm of unintended consequences
potentialreward
B1 B2A B
PARALYSIS POUNCE
THE REMEDY? IN-MARKET EXPERIMENTS
13
SIN #1: HI-FIDELITY
Hi-Fidelity
Quantifying
Going Solo
Business Model Next
IP Protection
100% Belief
Low-Fidelity Fake-it-before-you-make-it
instead of try this
14
INSTEAD OF HI-FIDELITY TRY LOW-FIDELITY
Key Assumption
Customers who see a slightly lower price from a competitor will still favor Progressive.
PROGRESSIVE
484AM. FAMILY
562000
STATE FARM
525ALLSTATE
644000000 000
15
SIN #2: QUANTIFYING
Hi-Fidelity
Quantifying
Going Solo
Business Model Next
IP Protection
100% Belief
Build One (then 10, then 100)
instead of try this
16
INSTEAD OF QUANTIFYING TRY BUILD ONE
Key Assumption
Customers who try the kiosk for 2-hour service will return to Snapfish.
17
INSTEAD OF QUANTIFYING TRY BUILD ONE
Tool: Alpha-Beta Planning Table
Number of Installations
Number of Participants
Dollar Investment
Build Time
Test Time
Fulfillment Partners
Testing What?
Is it Scalable?
Is there a Hard Stop?
What is the Feature Set?
Is the Technology viable? No
none
Key assumptions
No
Yes
Hypothesized Features
0
<$500
5 days 30 days
3 weeks
Beta Feature Set +
Same
...
...
IN-‐MARKET EXPERIMENTS
Yes (back-‐end platform+year 1 biz model)
Yes
Same as Beta, plus
NoFull Scale
No
Specific, Limited Features
Yes
No
Key assumptions
Small only
5 months
9 weeks
Branded, if possible
150 days (+option to extend)
Yes
Initial Feature Set
Full Initial Feature Set
No
COMMERCIAL LAUNCH
30X
30X
$5M+
DETAILS ALPHA
2 sites
$750k-‐$1.5M
150-‐200
THOUGHT EXPERIMENTS
15-‐20
<$500k
10 sites
BETA
0 sites
18
SIN #3: GOING SOLO
Hi-Fidelity
Quantifying
Going Solo
Business Model Next
IP Protection
100% Belief
Other People’s Money (and IP)
instead of try this
19
INSTEAD OF GOING SOLO TRY OTHER PEOPLE’S MONEY
Key Assumption
Fleet customers will incur the inconvenience of telematics to get a lower premium.
20
INSTEAD OF GOING SOLO TRY OTHER PEOPLE’S MONEY
Key Assumption
Customers will engage in peer-to-peer mentoring in an environment hosted by AARP and its partners.
21
SIN #4: BUSINESS MODEL NEXT
Hi-Fidelity
Quantifying
Going Solo
Business Model Next
IP Protection
100% Belief
Control Points Nowinstead of try this
22
UBIQUITOUS CONSTRAINED
Free, available to all Commodity
Many providers exist
Several providers exist
Possible to observe + copy
Branded + expensive to copy
Producible by only a few firms
Protected by contractual secrecy
Protected by patent
Protected by concealment
Brand
Device
Data transmission
Display/cues to drivers
Display/cues to manager
Historical data
Customized interventions
Re-pricing algorithms
Revenue sharing model
Architecture control
INSTEAD OF BUSINESS MODEL NEXT TRY CONTROL POINTS NOW
Tool: Control Point Grid
?
?
?
23
OTHER COMMON SINS
Hi-Fidelity
Quantifying
Going Solo
Business Model Next
IP Protection
100% Belief
IP Connection Services IP table; 3-ring template
instead of try this
Innovation Accounting Split tests Reverse income statement
instead of try this
24
RECAP: SIX SINS OF IN-MARKET EXPERIMENTS
1. Hi-Fidelity
2. Quantifying
3. Going Solo
4. Business Model Next
5. IP Protection
6. 100% Belief
25
BUT THEN THE WORLD LOOKS LIKE THIS
tantalizingpossibility
A B
PARALYSIS
what lies beneath
chasm of unintended consequences
potentialreward
POUNCE
A B
PARALYSIS POUNCE
26
INSTEAD: USE IN-MARKET EXPERIMENTS
the science of placing small bets fast
Hi-Fidelity
Quantifying
Going Solo
Business Model Next
IP Protection
100% Belief
instead of
Low-Fidelity
Build One
Other People’s Money
Control Points Now
IP Connection
Innovation Accounting
try this using these tools!
Key assumptions list
Alpha-Beta table
Control point grid
Services IP table 3-ring template
Split tests Reverse income statement
27
IN-MARKET EXPERIMENTS IN PERSPECTIVE
IN-MARKET EXPERIMENTS
TECHNICAL BUILD + QUANTITATIVE RESEARCH
WHAT HOW“building the right thing” “building things right”
TIM OGILVIE [email protected] 703 314 3123
JESSICA DUGAN [email protected] 312 532 8729
CHECK OUT OUR BLOGpeerinsight.com/musings
FOLLOW US ON TWITTER@peerinsight
Thank you!