who, how much, and how often? the difficult questions answered around your a/b testing strategy

The Past, Present and Future of Website OptimisationIn association with Qubit

Linus Gregoriadis Research Director@LinusGreg

| The Past, Present and Future of Website Optimisation| 6

Introduction and Methodology

The Business Case

Foundations and Resourcing

Moving Towards Personalisation



The Business Case




Aims of the report● To find out how companies are approaching the issue of

testing and optimisation.

● To identify the barriers to optimisation.− Technological− Cultural

● To understand the processes and approaches used.

● To highlight any potential future challenges.



The Business Case




Photo credit: 123dan321 on stock.xchng

Why test anyway?


Testing is key to improving website performance

Source: Conversion Rate Optimisation Report 2014


● Companies seeing significant increases in sales typically run more A/B and multivariate tests

Testing is key to improving website performance



“If it is going to fail, fail fast…

“We see a positive benefit from around a third of tests.”

Chris Howard, Head of Digital, Shop Direct


Case Study: Topshop

● Wanted to improve US experience

● Tested using size formats more familiar to North American shoppers

● Measured not on conversion but revenue optimisation

● Increased revenue by 4.5% with 99% statistical significance


Photo credit: 123dan321 on stock.xchng

What to test?

What to aim for?


“The onus is on optimising the funnel and we can’t do it all at once. We’ve started having grown-up conversations about what is the best point to focus optimisation on.”

Daniel Sale, Head of Digital Marketing, Investec Private Bank


Advanced Website Optimisation is yet to get over the adoption curve…



The Business Case




How much to test?


“We’ve learned what degrees of success you can have. You can chip away and only move the dial so far unless you are doing something significantly wrong.”

Paul Wishman, Group Ecommerce Director, LV


Size matters


“Realistically we can run three to four tests a month simultaneously… we never get the volume to the extent of retail banks for example. We need statistically significant samples and that takes time.”

Daniel Sale, Head of Digital Marketing, Investec Private Bank


“Apart from choosing the right tools, most of the obstacles were related to internal culture and getting buy-in from internal stakeholders.”

David Paice, Group Ecommerce and Admissions Director, Merlin Entertainments


“Testing has been an internal communications piece and especially because A/B testing involves so many different teams.”

Rachel Waller, Senior Digital Marketing and Social Manager, Farfetch


Decisions on technology


• Bought-in software, either off the shelf or customised?

• Built in-house? • Managed services

(when needs are constantly changing)?


Can your vendor grow with you?



The Business Case




“Personalisation is the most exciting challenge and the hardest. There’s so much we can do but it’s finding out what you should and what will add value.”

Rachel Waller, Senior Digital Marketing and Social Manager, Farfetch


Case Study: Farfetch

● Farfetch hypothesised that people convert more when they understand the business model

● Segment of users: new visitors with over 10 page views in their session

● A message layer slides in, prompting users to read FAQs

● Result: 17% increase in conversion


93% of companies saw an uplift in conversion rates

since implementing personalisation.



Ensure incremental gain


“Testing by revenue is critical to our business. Conversion tells only part of the story.”

Simon Pritchard, Group Digital Director, Arcadia


One to One?


“It’s the nature of fashion for consumers largely to behave like the herd in many ways. Perhaps one-to-one conversion isn’t necessary. If you can identify groups of people, that’s good enough.”

Stuart McMillan, Deputy Head of Ecommerce, Schuh


Case Study: DFS ● Hypothesised: European

customers may need more information when shopping on UK websites

● Targeted users from outside the UK with a layer informing them that DFS shipped to their country

● Result: 23% uplift in conversions for segment


Over time the majority will adopt best practice


Final Takeaways

● Testing has seen a surge of growth but barriers still remain.

● People elements are often as significant as technological ones.

● A pragmatic yet continual approach is most likely to pay future dividends.

PEOPLE, PROCESS, TECHNOLOGYNAOSHI YAMAUCHI, CPO@Yamalytics

Brooks BellThe premier firm focused exclusively on

enterprise-level A/B testing, optimization and personalization services.

SOME OF OUR CLIENTS

TALENT, CULTURE, PROCESS ISSUES CAUSING

‘UTILIZATION GAP’ Optimization tools increasing in sophistication

• Tracking across platforms• Personalization• Mobile• More data!

42

Capabilities

TALENT, CULTURE, PROCESS ISSUES CAUSING

‘UTILIZATION GAP’ Optimization tools increasing in sophistication

• Tracking across platforms• Personalization• Mobile• More data!

43

Insufficient Team & Talent• Business savvy analysts• Technical talent• Data savvy executive influencers

Broken Process• Lack of standards• Minimal documentation

Resistant Culture• Gut-driven decision

making• Minimal training & sharing

UTILIZATION GAP

Capabilities

Use

DRIVING ISSUES

44

Utilization ranks 20th out of 24 total categories in Brooks Bell Strategic AssessmentAssessment

Categories

Top Scoring

Worst Scoring

24

4 TIPS ON HOW TO CLOSE THE UTLILZATION GAP

So, What can you do?

46

ASK YOURSELF:“Do we really have the winning team?”

If the answer is no, you’re not alone ☺.

tip1

OPTIMIZATION TEAM

Strategy Design

Engineering

Analytics

Project Management

Creative

EVANGELIZE & BUILD CULTURE

GET INFLUENCERS RECOGNIZED IN TEST WINS

tip2

WHO IS BROOKS BELL?

WE RING A GONG EVERY TIME WE LAUNCH A TEST

CREATE A REPEATABLE PROCESS Standardize and then document

campaign, dev, QA, test archives

YOU CANNOT SCALE AND MAINTAIN MOMENTUM

WITHOUT PROCESS & STRUCTURE

tip3

51

THE STORYOF TESTINGAT BRAND X

2011 2012 2013 2014

DEMAND

COMPLEXITY

TEAM SIZE

# OF TESTS

PROCESS

CULTURE

DOWNLOAD! Campaign QA Checklist – bbi.bz/qa-2014

tip4START TARGETING SEGMENTS FOR TESTING.

IT WILL MAKE IDEATION MORE EFFECTIVE.

DOWNLOAD! Campaign QA Checklist – bbi.bz/qa-2014

Case Study: New v. Returning

Segment: New customers

Goal: Increase orders

Issue: Confused as to what the different fits mean

CONTROL CHALLENGER

54

TIP 1: Ask yourself “Do we really have a winning team?”

TIP 2: Get influencers recognized in test wins

TIP 3: Create a repeatable process

TIP 4: Start targeting segments for testing

who, how much, and how often? the difficult questions answered around your a/b testing strategy

Presentations & Public Speaking