#epicfail! a/b test results vs. ux best practices (boston upa 2012)

#EpicFail!

A/B Test Results vs. UX Best Practices

Susan Rice & Kirk Doggett

Global UX Center of Excellence

UPA Boston Conference 19/12/2013

Overview

• Introduction

• Examples– Pop Up Windows

– Progress Indicators

– Homepages

– Buttons

– Website Purchase Flow

– People Don‘t Read

• Summary

• Questions?


Who We Are

Vistaprint provides high-impact personalized products and electronic services that empower micro businesses and consumers at an affordable price.

Company Facts

• 25 localized websites

• 3,700 employees

• 3 state of art manufacturing facilities

• 13 offices worldwide


―Test Before You Invest‖

We have dozens of multivariate ―split run‖ site tests live at any given time during our 3 week release cycles.

• Scalable: Most site changes are measured before rollout

• Low Risk: Multiple tests can be run simultaneously with a sample of the population

• Accurate: Test sample is representative of user base

4

Random

Sample

Price Control

Price Test

UIControl 5% 5%

UITest 5% 5%

User

Population

UPA Boston Conference9/12/2013

UX & A/B Testing

• UX designs the UI and

helps inform the test plan.

• Test runs and is analyzed.

• Layer the quantitative with

the qualitative to create a

holistic picture to inform

future design decisions.

UPA Boston Conference 5

But what happens when A/B testing results challenge tried

and true usability principles and best practices?

9/12/2013

POP UP WINDOWS

6UPA Boston Conference9/12/2013

What the experts say…

• Avoid popup windows: [1]

– Extra windows keep users from getting to your site content, and even if such windows contain valuable information, users are likely to dismiss them immediately as ads.

– The other drawback to popup windows is that they go away — once the user gets rid of them, they're gone, so users often can't find the information again even if they want to.

• Do not have unsolicited windows or graphics ―pop up‖ to users.[2]

– Users have commented that unsolicited windows or graphics that ‗pop up‘ are annoying and distracting when they are focusing on completing their original activity.

• Selective attention is very powerful, and Web users have learned to stop paying attention to any ads that get in the way of their goal-driven navigation. [3]

– “pop-up purges mean that users close pop-up windoids before they have even fully rendered; sometimes with great viciousness‖

[1] Jakob Nielsen, excerpt from 113 Design Guidelines for Homepage Usability, 2001[2] US Government, Research-Based Web Design & Usability Guidelines, 2006[3] Jakob Nielsen, Top 10 Mistakes in Web Design, updated 2011



• You’re going to lose my business if your web site experience is annoying.

• According to preliminary information from the survey… major peeves about commercial Web sites were: [1]

– Pop-up advertising (34.9 percent)

– Registration log-on pages (16.7 percent)

– Software installation (15.7 percent)

– Slow-loading pages (9.1 percent)

[1] John P. Mello Jr. http://www.ecommercetimes.com/story/44966.html, July, 2005


http://www.ecommercetimes.com/story/44966.html

http://www.ecommercetimes.com/story/44966.html

In 2007 we added a pop-up…

Control Test

UPA Boston Conference9/12/2013 9

Results of adding the pop-up

• Significant increase in conversion rate across

most countries and across most products, from

5% to 11%.

• More business cards were created

Most Popular

Most Popular

Beauty/Massage/Spa

Beauty/Massage/Spa

Construction & Trade

Construction & Trade

Flowers & Candles

Flowers & Candles

Music & Entertainment

Music & Entertainment

0

5000

10000

15000

20000

25000

3700 3701

Sum of Documents Created

Feature Value Id

Category Name

Control ExperimentTested in August, 2007


Today:

In 2008, Nielsen declares the lightbox as the interaction design technique of the year [1]

[1] Jakob Nielsen, 10 Best Application UIs, Alertbox, 8/12/2008

UPA Boston Conference9/12/2013 11

PROGRESS INDICATORS



• Show the steps of the process [1]

– Recommended for checkout

• Guideline: Provide feedback to let users know where they are in the Web site. [2]

– Feedback provides users with the information they need to understand where they are within the Web site, and for proceeding to the next activity.

• In this example, note the right-pointing arrows. Nielsen discusses his changeover from colons to arrows. The rationale for the arrows instead of colons or other indicators was because ―the arrows seem to be slightly easier to understand as an indication of moving deeper and deeper into the site.‖ [3]

[1] Nielsen Norman Group, E-Commerce User Experience, Vol. 4: Shopping Cart, Checkout and Registration, 2011[2] US Government, Research-Based Web Design & Usability Guidelines, 2006[3] Beth A. Martin, Breadcrumb Navigation, November 2006


We tried removing the progress indicator

• Removing progress bar from Options improves

conversion

Test

ControlResults:• We observed a

significant increase of +0.47% of the conversion rate overall.

• The test significantly increased the number of sessions that make it to the Cart.


http://vistawiki.vistaprint.net/wiki/File:Test_group_2_Remove_MPSOcheckoutprogress_bar.png

http://vistawiki.vistaprint.net/wiki/File:ControlCheckout_Progress_bar.png

We tried adding a progress indicator…

• Adding a progress bar to the design pages hurts

conversion:

15

Test condition sessions CR CR Change

Control 50175 7.33%

Display Nav 49362 7.06% -3.69%


Progress Bar visual design…

Control

Test

Visual redesign hurts conversion.

Results:• CR is down 0.9% for

New Customers


HOMEPAGES



• “Clearly designate one page per site as the official

homepage.” [1]

• For a decade, one of the primary homepage usability

guidelines has been to designate a single page as the

one and only official homepage for any given website.

Users are confused when several pages are referred to

as ―home.‖ [2]

• [3]

[1] Jakob Nielsen, 113 Design Guidelines for Homepage Usability, 2001[2] Jakob Nielsen, Overloaded vs. Generic Commands, 12/19/2011[3] US Government, Research-Based Web Design & Usability Guidelines, 2006


http://www.useit.com/homepageusability/guidelines.html

http://www.useit.com/homepageusability/guidelines.html

Two Home pages?

• It worked for HomeRuns in 2001:

19

First Time Home Returning Home


A decade later…

• For returning customers, conversion rate is significantly higher for the Returning Home page than from the First Time Visitor home page.

First Time Home Returning Home


BUTTONS



• ―Ensure that a pushbutton‘s label clearly indicates its

action.‖ [1]

• ―The text on the button should begin with a verb. [2]

• ―Trigger words are the words and phrases that trigger a

user into clicking. They contain the essential elements

to provide the motivation to continue with the site.‖ [3]

• Fitts’s Law: ―The time required to move to a target is a

function of target size and distance to the target.‖ [4]


[1] US Government, Research-Based Web Design & Usability Guidelines, 2006[2] UX Booth, Good Call-to-Action Buttons, 2009[3] Jared Spool, The Right Trigger Words, 2004[4] Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design, 2010

9/12/2013

E-Commerce Examples

Staples

Amazon


Cart Buttons


Significant decrease in CTR to ShippingSignificant increase of CTR to Shipping

Control

Test 1 Test 2

• We tested larger target areas with improved visual

hierarchy and verbs with security info like Staples.

Increased cognitive friction?

9/12/2013

E-Commerce Examples

Amazon


Checkout Buttons


Could simpler buttons actually decrease cognitive load and prevent overthinking?

• We tested distinctive trigger words on our checkout

buttons and similar language to Amazon.

CR decreased significantly in FRCTR decreased significantly from shipping to confirmation in top 5 locales

Shut off after 9 days in testing

Control Test 1

9/12/2013

http://vistawiki.vistaprint.net/wiki/CR

http://vistawiki.vistaprint.net/wiki/CTR

Product Page Buttons

• We tested rollover states, more prominent call-to-actions

and larger targets.


Control Test 1

Significant increase in CR of 1.5% was seen.

Law of diminishing returns?

Test 2

Test 2: No impact.

9/12/2013

WEBSITE PURCHASE FLOW



• Offer ―sufficient information about your products to answer users‘ pre-purchase questions.‖ [1]

• ―Helping customers see what they want, when they want it, and in the way they want to see it, is substantially preferable to forcing them through a hub-and-spoke, back-and-forth path.‖ [2]

• ―The number of clicks isn't what is important to users, but whether or not they're successful at finding what they're seeking.‖ [3]

And internally we‘ve learned…

• When a customer becomes invested in one of our designs, the more likely they‘ll convert.

• Multiple screens/clicks are OK if customers know they‘re making progress.


[1] NN/g, E-Commerce User Experience[2] Bryan Eisenberg et. al., Call to Action: Secret Formulas to Improve Online Results, 2005[3] Jared Spool, Testing the Three-Click Rule, 2003

9/12/2013

Design / Flow


Get started Choose Design Template

9/12/2013

Design / Flow


PreviewTemplate

Personalize! Checkout Process

9/12/2013

That seems to work well. But what if…?


Significant increases to Website CR for new customers. • DTI CR increased 9.58%• Acquisition campaigns increased 5.18% Less friction?

9/12/2013

PEOPLE DON’T READ


Good ol‘ Gary Larson…



• ―We've known since our first studies of how users read on the

Web that they typically don't read very much.‖ [1]

• ―Satisficing describes the situation where people settle with a

solution to a problem that is "good enough".‖ [2]

• “Users don‘t make optimal choices…Neither do they scan

web-page in a linear fashion, going sequentially from one site

section to another one.‖ [3]


[1] Jakob Nielsen, How Little Do Users Read?, 2008[2] Interaction-Design.org, Satisficing, 2004[3] Smashing Magazine, 10 Principles of Effective Web Design, 2008

9/12/2013

Product Page


Control

Test 2Significant increase in paper stock upgrades

9/12/2013

Relevant copy is helpful for decision making.

More Product Page Testing


Control Test

Marginal increase in GM/order

Significant shift in take rate of 100 lb. premium and recycled paper and a decrease in Matte and Glossy.

So copy is making a difference…

9/12/2013

Sign In Button


Significantly hurt overall CR by 1.25%

Sign in page exit rate increased significantly.

Shut off after 7 days of testing

Control

Test

Even changing 2 letters can have an impact.

9/12/2013

Summary

• Best practices are a great starting point but they‘re

general.

• A/B testing can help you zero-in on specific changes

that can help you make changes that benefit your

site and your customers.

• Context and relevancy matters.

• Even small changes can make a significant

difference in site performance.

• Nothing is straightforward and simple.


Summary

• There‘s a healthy tension between Standardization

and Innovation.

• This isn‘t the end state. We‘ll continue to optimize

and learn from our customers.

• It‘s all about balance… and learning more about

your customers through various avenues…

• What the experts say: It depends… [1]


[1] Jared Spool, User Interface Engineering

9/12/2013

Summary

• Consistency is one of the most powerful usability

principles…

• A foolish consistency is the hobgoblin of little

minds…

- Ralph Waldo Emerson

• A/B testing helps us learn when context is more

important than consistency.


QUESTIONS?


Thanks!

Special thanks to Dave Yuknat for his fantastic editing

expertise… and for gathering supporting examples for the

presentation.

Susan Rice leads global UX design and

Kirk Doggett leads global UX research at Vistaprint.

And just a plug… we’re hiring!


#epicfail! a/b test results vs. ux best practices (boston upa 2012)

Technology