experimenting in the enterprise agile 2013 conference

Experimenting

Enterprise

© 2013 BigVisible Solutions

in the

© 2012 BigVisible Solutions 2

David J BlandStartups consulted: 18Twitter followers: 2179Iterations of Coaching: 215

Speaker Vanity Metrics

Brian BozzutoBlogs & Articles Written: 77Presentations Delivered: 46Teams coached: 54


This session doesn’t guarantee a 100 million dollar success, but it should help prevent a 100 million dollar failure.


A business model used to be static and survived many years.

Radio

TV

Internet

iPod

Facebook

0 5 10 15 20 25 30 35 40

38

13

4

3

2

Years to Reach a Market of 50 Million


The Lean Startup provides a scientific approach to creating and managing startups and get a desired product to customers’ hands faster.

-Eric Ries


Entrepreneur Intrapreneur


Entrepreneur Intrapreneur

Skills


This is great in theory, but it’s not for B2B.

This is great in theory, but I’m a visionary.

EntrepreneurIntrapreneur


Define the value proposition for:

Source: Adapted from Steve Blank

Buyer

User

Decider

Buyer

User

Decider

B2C B2B

11© 2013 BigVisible Solutions

Are your users/customers:

Source: Adapted from Steve Blank

Aware of problem?Actively seeking a solution?Hacking together a solution?Have $$$ for your solution?


Can We Build It?

Should We Build It?


Requirements = Hypotheses

© 2013 BigVisible Solutions 14Source: Adapted from Cynefin

Simple“Everyone Knows That”

Complicated“Let’s consult the expert”

Complex“Whoa, it all makes sense now…”

Chaos“That still doesn’t make sense”

Work with experts to inform our hypothesis

Run experiments to gain new knowledge


Safe Fail != Fail Safe


Experiment withproduct & process


If you experiment just to see what happens, then you’ll succeed at seeing what happens.


Be careful how you structure your experiments

© 2013 BigVisible Solutions 19Image Source: Eric Ries The Lean Startup

Then here

Start hereLast here

© 2013 BigVisible Solutions 20Image Source: Eric Ries The Lean Startup

NowExecuteQuickly


Experiment Template

For <customer segment> I predict <outcome>

when I run <experiment>


Experiment Example

For new customers, I predict 10% increase in

conversion when I offer a “no questions asked” cancellation policy


Experiment Guidelines

• Customer Segment (who)• Up to / Limit Of (how many)• Length of Time (how long)• Financial Exposure (how much)• Amplification & Dampening Measures


Experiment DashboardID Hypothesis Description Status Next Steps

01a

We predict that adding

a refund feature will

increase conversions

Story 13c adds refund label and call to

action button that puts

customer in contact with call center

agent.

Complete

0.2% increase in conversion on cohort. Meeting

with VP of Ops to discuss

strategy.


“This is all great, but how do we know what

experiments to run?”


Iterative Product+ Static Business Model

Failed Organization


? ? ???

?

??

?

?

??

“Let’s map out our existing business model”

Source: Adapted from Alex Osterwalder

tip: tell a story


? ? ???

?

??

?

?

??

“Let’s validate our riskiest assumptions”



This is most likely your Riskiest Assumption



This is most likely your other Riskiest Assumption



What happens when we subtract costs from revenue?



“Okay, now we have a big list of assumptions, how do we decide where to start?”


experiment

experiment

experiment

experiment

experiment

experiment

experiment

“Let’s dot vote!”(you could, but that’s pretty subjective)


time

lear

ning

experimentexperiment

experiment

experiment

experiment

experiment

experiment

experiment

experiment

experiment

experiment

experiment

experiment

(ok better, but what are they tied to?)


Now we have a focusing mechanism

focustime time

lear

ning

lear

ning

experiments experiments



And assumptions that need validation

focus

assumption assum

ption

assumption assumption



time time

lear

ning

lear

ning


This is Experiment Mapping

focus

assumption assum

ption




time time

lear

ning

lear

ning


Customer Segment is our focus

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


Will they pay for our product?

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment

They willpay for our

product


Can we scale to meet demand?

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product

We can scaleto meetdemand


Can we use our distribution channels?

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product

We can useexisting

distributionchannels



Are we solving a real need?

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product

Our productsolves a need

they have

We can useexisting




Which of these is the riskiest?

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product


they have

We can useexisting




Can we do a concierge test?

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product


they have

We can useexisting



Conciergetest with 10customers


We could build a prototype

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product


they have

We can useexisting




Build aprototype


Maybe run a focus group?

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product


they have

We can useexisting




Build aprototype

Run a focusgroup with80 people


Prototype will take time

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product


they have

We can useexisting




Build aprototype



Focus groups take a long time too

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product


they have

We can useexisting




Build aprototype



Concierge test is quick & we’ll learn

focus

assumption assum

ption




time time

lear

ning

lear

ning

CustomerSegment


product


they have

We can useexisting




Build aprototype



“Let’s swarm on this experiment”

highlearning

shorttimeframe


Just because you can measure it, does not mean you should measure it.


Auditable

Actionable

Accessible

Source: Eric Ries The Lean Startup


Auditable

“Be able to defend your metrics with confidence”


Auditable?

zombieproject

metrics


Actionable

“Does this metric incentivize a change in behavior?”


Actionable?

Facebook likes won’t influence your dog


Actionable

Number of steps should influence

your lifestyle


Actionable?

What does this tell you about your

product?


Accessible

“Can you easily access and share this data?”


Accessible?

Not accessible


Accessible

Accessible

Image Source: Lean.st


“Metrics are people, too”

Source: Eric Ries The Lean Startup


Aware

Hopeful

Satisfied

PassionateSource: Adapted from Cooper & Vlaskovits


Aware

Hopeful

SatisfiedPassionate

Acquisition

Activation

RetentionReferral


Start small.


When you have stories around an uncertain feature, add a hypothesis that states the expected outcome on a specific metric.


If you have a Kanban Board, add a Validate Column to

the very right of it.


Problem Solution Process

Known Known Waterfall orAgile

Known Unknown Agile

Unknown Unknown Lean Startup


Where are the unknown / unknown’s in your enterprise?

experimenting in the enterprise agile 2013 conference

Technology

body mass

pedometer

increased

physical activity

step program

important

step goal

scaleto meetdemand