Presentation NameSpeaker’s Name

Title Ecommerce Test Ideation: What to Optimize and Why

• Optimizely's top solutions partner for enterprise brands & retailers

• Partner to over 30 clients including Adidas, Tory Burch, BluDot, Whole Foods, Cost Plus World Market, Express, Patagonia & many more

• Proud winner of 2015 Opticon Partner of the Year & Most Dramatic Business Impact

hi. i’m matty.founder & ceo at

SPRING INTO ACTION

Matty Wishnow& Leo Wishnow

• 15+ years testing & validating as:• SVP, Direct to Consumer at

Warner Music Group• Founder & CEO of Insound.com

• Loves “Less and More” design principles & advanced baseball stats

• Hates dabbling

so, you want test ideas.

hypotheses learnings investmentsproblemsgoals

wouldn’t you rather a repeatable framework for producing & prioritizing great test ideas?

SPRING INTO ACTION

the ideas will come. but, let’s start with the framework.

goals problems hypotheses learning

UX

product

merchandising

marketing

prioritized by data validated by data

investments

1

2

3

Ps

pS

ps

PS

2

3

4

1

2

3

4

1

SPRING INTO ACTION

unifying. wait like this applies to everything?

#1 Everything is an experiment

Banner/hero changes for new seasons/campaigns.

Merchandising changes.

Copy changes.

Redesigns & replatforms.

New SaaS additions.

Everything.

#2 Questions to live by:

What problems are the changes solving?

How will you know if the change was successful?

SPRING INTO ACTION

how to communicate goals

good example:In 2016, we want to increase smart phone purchase conversion rates by 10% y/o/y.

bad example:We want to improve the mobile user experience.

Ask yourself:• Are they widely understood?• Are they actionable?• Are they measurable?• Do they have refined baselines/targets?• Are they time-based?

SPRING INTO ACTION

how to capture problems

good example:Our customers have a problem understanding the difference between and value proposition of products on PLPs.

bad example:I think we need a responsive website.

Ask yourself:• Why is it a problem?• How does the problem impact the

business or end user?• Is it a root problem? • Do you have data to confirm that it is

a significant problem?

SPRING INTO ACTION

how to articulate a hypothesis

good example:I believe that if we add geo-targeted shipping commitments on the shipping page of checkout, customers will have more confidence in ordering from us.

If I am right we will see increased conversion & reduced abandonment at the shipping page of checkout.

bad example:I think we need to redesign checkout.

Try to:

• Be specific about the change you are contemplating.

• Describe what you expect the result of that change to be.

• Clearly define the metric you can most confidently measure success by.

• Start with:• I believe that…• If I am right then…

SPRING INTO ACTION

Be consistent about requiring all changes get mapped to problems & hypotheses.

2

what do you do once it’s all mapped?

Determine if additional problem research is needed.

1

hypotheses learnings investmentsproblemsgoals

Determine what should be tested vs “just do it”.

3

Review & iterate regulary.4

SPRING INTO ACTION

I believe that if we remove the “New Customers” section on the interstitial page between the Shopping Bag and checkout, we will reduce friction to continue to checkout for non-logged-in users.

If I am right, order conversions will increase overall with minimal impact to new account sign-up rates.

Primary Test KPIConversion to Order

Data Story MetricsNew Account Creation

ProblemCurrently, new customers can create a new account either on the My Account interstitial page, or on the Order Confirmation page. I believe having the “New Customers” section at the beginning of the checkout funnel is overwhelming to visitors overall, as they have to decide between logging into an account, checking out as a guest, and registering a new account.

reducing checkout frictionso let’s give some real examples

SPRING INTO ACTION

surface product options earlier so let’s give some real examples

Primary Test KPIConversion to Order

Data Story MetricsPLP Bounce Rate

ProblemPreviously, we have tried showing every color as its own product on the PLP grid (thus creating many PLPs to scroll through) or shown only the “parent product” in a single color on the grid (thus requiring people to click into PDPs for more color selections). Neither solutions have been satisfactory for our business of our customers.

I believe that if we denote which additional colors are available for each “parent product” on PLPs, customers will be more likely to identify a product color selection that appeals to them earlier in the funnel.

If I am right, we will get more people to PDPs and see increased conversion rates.

SPRING INTO ACTION

I believe that if we offer ShopRunner’s(SR) free two-day shipping service to visitors, then we will increase the incentive to buy when able to get items faster at no additional cost.

If I am right, we will see an increase in overall order conversion rates and revenue, driven by users taking advantage of the ShopRunner option.

testing saas benefit claimso let’s give some real examples

Primary Test KPIConversion to Order

Data Story MetricsBounce at Point of ShopRunner Option

ProblemCurrently, we do not have ShopRunner as a shipping option on the site. We offer free 4-7 day shipping on all orders (no threshold). Visitors can upgrade to faster shipping, but it comes as an additional cost.

SPRING INTO ACTION

I believe if we placed an inline email signup in the header of our desktop site, we will materially increase email signups.

If I am right, we will increase email signups and no decrease in conversion rate.

inline email sign-up so let’s give some real examples

Primary Test KPIEmail Newsletter Sign-Up Rate

Data Story MetricsConversion to Order

ProblemCurrently, we have links to sign-up for newsletters at the header & footer of our websites. These links go to a second page with several forms to fill in. The extra steps, we believe, create friction for something that can/should be very simple.

SPRING INTO ACTION

test idea process x

SPRING INTO ACTION

goals & problems establishedDo start with the critical problems you want to solve for your customers, mapped to the measurable goals that define success.

Do not sneak solutions into your goal or problem statements.

hypothesis generation hypothesis prioritization

customer service data

video session recordings

drop off analysis

web analytics

customer service feedback

customer survey data

1

2

3

4

3

1

2

effort to test

potential impact

likelihood belief is true

ability to adopt

and more

X

key takeaways

The magic of optimization & personalization is not simply creating experiments or campaigns. It is the magic of solving a customer experience or business problem. Metrics rise in correlation to your ability to solve problems.

1

Don’t lead with solutions. Don’t chase ideas simply because others have succeeded with it or because you are able to test the change. Prioritize your experiments based, initially, on the problems you expect them to solve.

2

Develop a unifying problem & solution mapping framework to ensure that you are working in concert to solve the biggest problems and that you are not dabbling. Getting winning tests is not winning if the biggest problems go unsolved and the biggest hypotheses unvalidated.

3

SPRING INTO ACTION

pages.optimizely.com/spring-into-action

Upcoming Spring Into Action Events