10 Tactics for Building an Optimization CultureOPTIMIZELY & BROOKS BELL

3.4.2014

Welcome!

Gregory NgChief Marketing OfficerBrooks Bell@GregoryNg

Kyle RushHead of OptimizationOptimizely@kylerush

10 Tactics for Developing a Successful Experimentation Culture

What we’ll cover ...1. Best practices from top testing

organizations2. The people behind testing

3. How to organize your program4. How to keep the process

manageable5. How to include others and build

excitement

Successful experimentation culture must be data-driven.

What does it mean to be data-driven?

Progress is compelled by data, rather than by intuition or personal experience.

Evidence-based decision making.

Using data to inform key decisions.

How data-driven are you, actually?

• Do you dictate strategy without referencing data?

• Do you rely on the person with the most experience to make the ultimate call?

• Do you use words like “think” and “feel” when talking about strategy?

• Does your team trust the data?

How do you establish a successful testing culture?

10 Tactics1. Clear organizational structure

2. Identify the right talent3. Establish stakeholders

4. Solicit test ideas from everyone5. Prioritize tests

6. Share results and successes7. Build a knowledge base

8. Don’t rush into MVT tests9. Balance learning and “wins”

10.Go beyond CRO

#1 – Choose the right organization structure

• No “one-size-fits-all”• One or multi-person

team• Centralized or

decentralized?

Members of the Obama for America Team

#1 – Choose the right organization structure

• Army of One • Ringleader

• Fellowship • Kingdom

#2 – Pick the Right Talent

• Qualitative and quantitative combination

• Curiosity, creativity• People skills• Analytical mindset

VP of Marketing

Program Manager,RetailProduct Manager

E-Commerce Director

Optimizely’s customers come from a variety of backgrounds

#3 – Establish Stakeholders Early On• Identify the correct

stakeholders• Secure an executive sponsor• Develop an approval process

for tests• Communicate and share

successes

#4 – Solicit Test Ideas from Everyone

• Make it easy to submit ideas

• Capture interest after sharing results

• Follow up for every idea• Create a scoreboard• Attribute experiments to

team members Example: Optimizely ScorecardVote on predicted winning variations

#5 - Know how to Prioritize Tests

ESTABLISH A SCORECARD•Relevance to business goals (KPIs)•Potential impact•Potential for learning•Level of effort•Time to significance and available traffic•Likelihood of implementation

#6 – Share Results and Successes

• Make it easy to understand• Clarify the primary goal• Share 100 percent of

results, regardless of outcome

• Standing meetings• Keep score

Presenting A/B test findings at company all-hands

#7 – Build a Knowledge Base• Gain broader insights• Lay a foundation for growth• Eliminate single point of failure• Increase accountability• Standardize processes and

documentation

#8 – Don’t Rush Into MVT Tests• Ideal for fine-tuning

optimization• More complicated but less

effective• Fewer learnings

#9 – Balance Learning and “Wins”• Always start with a hypothesis• Never settle for a flat result• Invitation for deep analysis • Segment results, target

accordingly

#10 – Go Beyond CRO

Optimization doesn’t need to occur in isolation. Almost anyone can test.

For instance ...

#10 – Go Beyond CRO• Designers

Variation A Variation B

Walk designers through A/B tests so they can inform their designs with data

#10 – Go Beyond CRO• Engineers

Launch of OFA 2012 New Donation API

A/B testing the new platform revealed an architectural bug that was missed in QA

#10 – Go Beyond CRO• Email and Content Marketers

A/B Tested Emails and Landing Pages for Subscribers

#dataculture

In summary ...

1. Clear organizational structure

2. Identify the right talent3. Establish stakeholders

4. Solicit test ideas from everyone5. Prioritize tests

6. Share results and successes7. Build a knowledge base

8. Don’t rush into MVT tests9. Balance learning and “wins”

10.Go beyond CRO

Thank you for reading!