Jared Waxman

Fast & Furious: 10 Principles for

Rapid A/B Testing

Questions? Email jared@benchmark-analytics.com

Jared Waxman presents…

Presented at Internet Marketing Conference

Sept 16-17

Vancouver, CANADA

E X P E R I E N C E S A N D L E S S O N S W H I L E R U N N I N G A / B T E S T I N G F O R E C O M M E R C E S I T E S AT :

M a k e r s o f Q u i c k e n , T u r b o T a x , Q u i c k T a x a n d Q u i c k b o o k s

3

Intuit Small Business – Web Business

How much: On order of ~$100 M annual revenue transacted online

What: Desktop software, Services, Software-as-a-Service (subscription model)

Where: 45 Websites, 10 Shopping CartsTraffic: QuickBooks.com ~ 1 M monthly

unique cookies

Jared Waxman

3

4

QuickBooks.com – Various Timepoints

Jared Waxman

4

5

I’ve tested…

I’ve run 1 or MORE tests against each of these hypotheses:How much does a Free product cannibalize Paid

products?What’s the optimal way to position a Free product in

the lineup to maximize all units while minimizing paid cannibalization?

Does one Free offering cannibalize less than another Free offering?

What is the most effective “voice” used in describing the product?

What category of images sells the product the best?

Jared Waxman

5

6

…and tested…

Do videos sell better than static images?Do expanded versus collapsible product specs perform

better?Does having a special price for upgraders yield higher

overall sales than a flat pricing model? Does it improve customer sentiment?

Does exposing retail options significantly shrink direct online sales?

Which discount % level of promotional price discounts yields highest revenue?

Does adding cart CTA on home and line-up pages increase conversion or change pathing?

What is the optimal color, size, placement, and copy on CTAs?

Jared Waxman

6

7

…and tested…

Where is the optimal place in the purchase funnel for customer to configure their product?

Where is the optimal place in the purchase funnel to ask for credit card info?

How many pages in the checkout flow is optimal?What home page weighting of offerings moves us to

the desired platform mix while continuing to grow revenue?

How effective are certain product chooser tools, demo, or guides in moving conversion rates?

How effective is a certain cart abandonment tactic in moving conversion rates?

Jared Waxman

7

8

…and tested…

Where is the right step in the purchase funnel to cross-sell other offerings?

How many cross-sell offerings are optimal for a given funnel step?

Should cross-sells be passive or intrusive?How effective and efficient is proactive chat in

boosting over conversion?How many CTAs and/or featured offerings are

optimal on the home page?How many offerings is optimal within the line-

up?

Jared Waxman

8

9

…and tested…

Are bullet points or prose styles more effective on landing pages?

Are PPC ad group specific links on landing pages more effective than generic links?

What’s the likelihood of successful for a new product offering if added to the online store?

How would a New User vs Existing User focused site perform against overall online revenue?

Can lift conversion rates by guessing at New vs Existing and then sending them down different paths?

How effective would a simple User Contribution System be at engaging and converting prospects?

Jared Waxman

9

10

…and tested.

Does simplifying the home page by removing promotions on the homepage to other auxiliary Company sites/product offerings reduce the net traffic that gets sent?

Do menu titles perform better if they are job-oriented or product-oriented?

Do promos perform better as %s or as $s off?Do promos perform better as %/$ off vs Free

Shipping?Does a certain messaging approach consistently

outperform another regardless of specifics?

Jared Waxman

10

11

What has been the benefit of testing?

Half our growth in the online business comes from testing the pure site experience. The other half from acquisition programs.

Testing is highly leveraged growth – a handful of employees. Produces 3-digit ROI.

Acquisition programs requires serious cash and overall produces 2-digit ROI.

Jared Waxman

11

12

Today’s Focus: Organization Design

These gears must all work together.

Each deserves separate treatment.

Focus of this presentation

Jared Waxman

12

Jared Waxman 13

Fast & Furious

My Testing Team throughput in just 12 business days:• 5 tests (3 of which shown on next page) including 1

microsite• 11 Factors Tested• 26 Recipes In All

1 Big Winner: 20% increase on the success formulaAnnual impact ~ $10M

Plus…

1 Loser (but killed quickly)Some breakevens

14

Rapid Testing Team

Go Broad (8 recipes)

2/20/09

Then Narrow (8 Recipes)

3/4/09

And Broad, Again!

3/9/09

Formula: Index 1.29 Weekend, .91 Weekday

Formula: Index 1.20 WE/WD

Control

1

2

4

Jared Waxman

Formula: Index 0.50

3

Narrow Again (4 Recipes)

4/9/09

Product A: Index 2.00

Product A: Index 1.60

Jared Waxman 15

Top 10 “Go Fast” Testing Principles

10. Testing “Strike Force” with no outside dependencies

9. Exactly one “Approver” 8. One straight-forward

formula for success 7. Inform at go-live, then

again with results 6. Fully-baked ideas as the

starter, then testing team tightens it. After phase 1, iterate with true MVT and refine as appropriate.

5. Test ideas based in theory and customer data. Then go test, not talk.

4. Manage to deadline with quality. Reduce scope as needed!

3. Take in ideas from anywhere and everywhere. Don’t take in orders.

2. Open, flexible test calendar basing next tests on data from existing ones. No pre-slotted commitments

and finally…If you follow the prior 9 principles you’ll be able to deliver…

1. One-Day Test Sprints! From schedule-it to go-live in 8 hrs.

16

Process & Traps

Traps:

• The “Evil And”

• Management not aligned – competing goals

Traps:

• Ownership issues

• Re-tests, small tests

Traps:

• Debate

Traps:

• Perfection

Jared Waxman

16

Jared Waxman 17

IN THE ORG, YOU’LL ENCOUNTER…

Digging Deeper into Organization Design

18

HiPOs

Jared Waxman

18

19

HiPOs: Tips

1. Get Management to understand why testing success probability means it’s in their interest to have a healthy distance from individual test designs. It’s in your interest too.

2. Get Management engaged at the right level – setting hierarchy of goals and resourcing for highly leveraged testing.

Jared Waxman

19

Jared Waxman 20

Targets20

21

Targets: Tips

1. Set by Management, not testing team.2. Testing Team Lead responsible to push

back if needs more constraints.3. Put in place a hierarchy of success metrics.

Should be consistent across seasons, managers, businesses. Revenue per visitor (RPV) typically at top of hierarchy. Can assign each success metric it’s own team.

Jared Waxman

21

22

Design by Committee

Jared Waxman

23

Design By Committee: Tips

1. Hire a Test Lead whose business instinct and creative eye you know and trust.

Keep him/her in the kitchen on business strategy. Then let him/her be the one and only approver on what

goes live.

2. Ideas get thrown “over the wall” to the testing team.

3. Testing Teams set up as neutral, with no built-in biases towards specific businesses, ideas, or managers.

4. Idea submitter and/or related business unit have no official role during the execution phase.

Jared Waxman

23

24

Sacred Cows

Jared Waxman

25

Sacred Cows: Tips

1. Assign Executives to Goals Hierarchy. Assign Segment Marketers job of throwing ideas “over the wall”. Assign the Test Lead the job of selecting and executing.

2. Separate Segment Marketers from Testing.3. Get a clear list of constraints, a consistent success

metric, and then challenge everything.4. Outside viewpoint. Much easier to get people think

outside the box when they know some crazy-sounded idea worked somewhere else.

5. Test big. And if you can’t, test small. Remember you can also pull if it’s losing money.

6. Influence through your framing. Testing can be framed not as a challenge, but a way to quantify a “known” lever so it can be expanded or further optimized.

Jared Waxman

25

26

Stampedes

Jared Waxman

27

Stampedes: Tips

1. Cordon off your Testing Teams to focus your highly leveragable resources

2. Set clear, long-term goals for the Teams.3. Empower the Team Leads to decide

priorities, timing, workload.4. Leverage outside agencies to rapidly expand

capabilities and expertise as needed.

Jared Waxman

27

Jared Waxman

Fast & Furious: 10 Principles for

Rapid A/B Testing

Want to learn more? Email jared@benchmark-analytics.com