implementing an a/b & usability testing process

Implementing an

A/B & Usability testing process

@guido | userlegion.com

http://userlegion.com

@guido userlegion.com

Guido?

2

@guido userlegion.com3

Netherlands: May 27 & 28, Utrecht


Usability?

5


Usability?

6

Learnability

Efficiency

Memorability

Errors

Satisfaction


Usability versus Persuasion

7


A/B Testing?

8

A B C D E

A B C D E

A B C D E

Multivariate Testing (MvT)


So why even bother…?

9


Selling online is tough!

10



11

Average conversion rates

Offline: 20 - 25%

Online: 5 - 15%



12

Average conversion rates by product category

Source: MarketingSherpa Benchmark 2014

0% 5% 10% 15% 20% 25%



13

Some shop examples

Source: MarketingSherpa Benchmark 2014

8% - 11%

Average order value: €430,-



14

Some shop examples

Source: Marketingcharts.com

15.8%



15

Some shop examples


http://www.google.nl/imgres?imgurl=http://cdn.bargainez.com/womanwithin/store/600/womanwithin-‐logo&imgrefurl=http://bargainez.com/womanwithin&h=168&w=600&tbnid=Md9NfBl9JPFALM:&zoom=1&docid=uCe3emUjdu4dkM&ei=RVosVZv1IcXXPY6cgbAE&tbm=isch&ved=0CDYQMygCMAI

25.3%



16

Some shop examples


40.6%

But Still…

What are users missing out on?


What are online users missing out on?

19

Smell

Feedback Commitment

Personalization


2 x 5 tips

20

My experience with testing process

Direct optimization tips

5 Testing Tips

Learnings from the process

behind optimization


Learnings first

22

Proper hypotheses

Time & resources are limited

Brute force vs smart testing

1


Get yourself a team

23

2


Optimization Team

24

Create a well balanced team

CM

Web/ Business Analist

Functional Tester

(Frontend) Developer

Content

(UX/UI) Designer Psychologist


Supporting roles

25

Your team will need support

Team

Marketing

Customer support

(Backend) Development

Traffic (SEO/SEA)

Business Sponsor (shit umbrella)


Optimize for the journey

26

Not just single elements

3

Experience Ordering

ShippingInspiration Detailed research

Service


Optimize for the journey

27

Not just single elements

Communicate consistently

Segmentation

Offsite behaviour


Optimization Workflow

28

4



29



30

Input Hypotheses Analysis

TestingPriority Results



31

Don’t forget:

Share results

Get employees and management involved


Involve your users

32

5


When to test?

33

Competitors

Old/ New Design Sketch

Test Preparation

Prototype

Live site


What to test?

34

Setting priorities with the PIE Framework

Potential Heuristics Voice of Customer

P

I

Importance Business Traffic

EEase

Development Cost


What to test?

35

Setting priorities with the PIE Framework

PIE Scoring Example

Idea A Idea B Idea C

Potential .50 .20 .80

Importance .20 1 .50

Ease .40 .80 .20

PIE Score x 100

4 16 8


How many users to include?

36

Test Preparation

Source: nngroup.com


What kind of users to include?

37

Test Preparation

Source: nngroup.com

Everyone


Where to test?

38

Test Preparation

Remote On-site

Test Day


Let the user do the talking

40

Help getting comfortable

Help explaining the case

Help thinking aloud

Other than that: shut up :)


Review day

41

Lots of ideas > A/B testing

Lots of bugs > fix it

Better estimate A/B test potentials


Sidenote: Toolbox

42

Project management

Web analytics

BI

A/B testingDevelopment

Raw data

Visualisation

Feedback

UX Testing

5 Direct Optimization TipsLearnings about users

from watching Usability and A/B testing


Get customers to return

44

Small next steps to the next micro conversion

1

Almost 50 % of users do not come back if they found it hard to find

relevant information on the web site (Forrester research)


Negative search / filter results

45

2


Discontinued products

46

3


Sidestep: context matters!

47

Perceptual contrast / framing

Web only

Print only

Web & print

$59

$125

$125

16%

0%

84%

68%

32%Source: Dan Ariely


Direct checkout

48

+ 4,77% CTR

4

Source: Kieskeurig


Mark visited products

49

or even make it a checklist

5

Source: Kieskeurig


Mark visited products

50

or even make it a checklist

Checkmark in button

+ 3,97% CTR

Source: Kieskeurig


That’s it!

51


So… will it make you money?

52

Conclusion


Case: UX sprint

53


Conversion is not a stable factor

54


Important off-site factors

55

Weather

Calendar

Stock/ Interest rates

(National) Holidays

Sport events

Competitors

Thanks For Listening

Guido [email protected]

+31 6 42093976@guido


Simple, Affordable Usability Testing in your Local Language

57

Business card = Free 10 user trial :) Agency accounts available

implementing an a/b & usability testing process

Internet

guido userlegion

online users

shop examplessource

comab testing

comoptimization team24create

comoptimization workflow284

comoptimization workflow29

comoptimization workflow31dont