opticon 2015- benchmark your optimization program

Benchmark Your Optimization Program Shana RusonisContent Marketing Manager, Optimizely

@[email protected]

#opticon2015

mailto:[email protected]

#opticon2015@srusonis

optimize.ly/optimizationbmrkSlides will be posted, and this session is also being recorded.

http://optimize.ly/optimizationbmrk

“What gets measured, gets managed.”

—Pete Drucker

Stories

Data


Overview of today’s session

• Debuting industry research from Optimizely

• Key findings: Benefits & budget

• Key findings: People behind optimization

• Key findings: Processes for optimization

• Implications & takeaways


Let’s collect some data.

How does optimization impact businesses?


Benchmark Methodologyn = 417

1/4 Large Enterprise

1/2 Mid-Size to Small Enterprise

1/4 Small Businesses

Company Size

1/4 Executives

1/2 Managers

1/4 Associate, Consultants

Level

Marketers

Product Managers

Engineers

Titles


94%Improves my understanding of my customer’s needs and desires

Helps me to deliver relevant content to my customers6%

Do you agree or disagree with the following statements about optimization?

93%Helps me to deliver relevant content to my customers

Increases my conversion and engagement rates7%


87%Allows me to take action on data quickly

Has created a culture of learning and experimentation at my company13%


87%Has created a culture of learning and experimentation at my company

Improves the creative output of my team13%


91%Increases my conversion and engagement rates

9%


88%Improves the ROI of my marketing campaigns

Allows me to take action on data quickly12%


80%Improves the creative output of my team

Improves my team’s speed to execute on ideas20%


98%

94%

93%

91%

88%

87%

87%

80%

76%

Is valuable to the success of my marketing campaigns and/or product strategy

Improves my understanding of my customer’s needs and desires

Helps me to deliver relevant content to my customers

Increases my conversion and engagement rates

Improves the ROI of my marketing campaigns

Allows me to take action on data quickly

Has created a culture of learning and experimentation at my company

Improves the creative output of my team

Improves my team’s speed to execute on ideas

6%

7%

9%

12%

13%

13%

20%

24%

Agree

Disagree


98% of optimizers agree that it’s valuable to their success

98%Is valuable to the success of my marketing campaigns and/or product strategy



… Along with many other qualitative and quantitative benefits

94%Improves my understanding of my customer’s needs and desires

Helps me to deliver relevant content to my customers6%


87%Has created a culture of learning and experimentation at my company

Improves the creative output of my team13%


91%Increases my conversion and engagement rates

9%


88%Improves the ROI of my marketing campaigns

Allows me to take action on data quickly12%


80%Improves the creative output of my team

Improves my team’s speed to execute on ideas20%


Quantitative Qualitative

#opticon2015@srusonisIncrease

57%61%

43%

How do you plan to change the amount of budget allocated to optimization in the next 12 months?

Optimization budgets are set to increase

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

55% 53%

4%

43%

Increase

Decrease

No change


#opticon2015@srusonisIncrease

57%61%

43%

How do you plan to change the amount of budget allocated to optimization in the next 12 months?Executives & Managers are more certain of budget increases

• 57% of Executives reported a planned increase

• 61% of Managers reported a planned increase

Executive

Manager

Individual


“Optimization has become even more crucial in today's market,

so more budget is required to make a position in the market.”

– Anonymous Survey Respondent


The human capital behind optimization.


Who optimizers are planning to hire

Web Analyst UX DesignerMarketerFront-end Developer

Program Manager


0%

5%

10%

15%

20%

25%

30%

One, full-timeMultiple, full-time

One, part-timeMultiple, part-time

An agencyNone

23%

29%

17%

29%

2% 2%

How many members of your team are focused on optimization?

Multiple,full-time

Multiple,part-time

52% of optimization teams have 1 or more full-time hires

“I think the single best thing to do is to allocate headcount towards optimization.

If you can swing it, hire a dedicated person. If you can’t, get everyone on the team

(designers, engineers, product managers) evangelized. Let everyone run an experiment

so they can see how mission critical it is.”– Kyle Rush, Director of Frontend

Engineering and Optimization, Hillary for America


Optimization draws from a wide range of teams and skills

Marketing62%

Design53%

UI or Front-End Development40%

Product36%

Engineering35%

E-commerce31%

Sales30%

Customer Service or Support23%

Executive Staff20%

Which teams are involved with optimization at your company?

Marketing32%

E-commerce14%

UI or Front-End Development12%

Product12%

Engineering9%

Design8%

Executive Staff5%

Other3%

Customer Service or Support3%

Sales

Which team leads optimization at your company?


Move, Inc. Team Structure


The processes needed for optimization.

How do we scale testing to maximize our impact?


OPTIMIZATION LEAD

Build, QA, and launch

experiments

Conclude and analyze

experiments, share results

Prioritize backlog of

experiment ideas

Research to create solid experiment

hypotheses

Processes help scale optimization efficiency



hypotheses


OPTIMIZATION LEAD

“If you can’t state your reason for running a test simply, then you probably need to examine

why and what you are testing.”

.”

– Brian Schmitt, CROmetrics


Analytics are the top input for optimization

Analytics72%

Results from Previous Experiments57%

Customer Surveys51%

User Research Studies50%

A/B Testing Case Studies44%

Support Tickets or Live Chat Feedback38%

Heat Maps28%

Which of the following inform your optimization strategy?

Analytics72%




Customer Surveys51%










Heat Maps28%



… Along with other qualitative inputs and sources of inspiration

Customer Surveys51%






Heat Maps28%






Qualitative InputsInstitutional &

Industry Knowledge



hypotheses



experiment ideas

OPTIMIZATION LEAD


Optimizers prioritize for revenue, customer satisfaction

Potential to increase the value of each customer interaction, or increase overall revenue

65%

The impact on ease of use and satisfaction for my customer

58%

Improvement to website or app stability or performance

48%

The inefficiencies that currently exist in my marketing and product funnels

33%

The cost to acquire customers on various platforms

31%

We don’t do this kind of prioritization

2%

How do you prioritize which digital interactions with your audience require optimization?


Optimizers care deeply about the customer experience, in addition to positive ROI.

The values we use to prioritize our tests are a

snapshot of why we choose to optimize in the first place.

“The way your customers use your product and the way they experience it … is everything.”

”

.”

—Lev Tatarov, Marktplaats, eBay



hypotheses



experiment ideas


experimentsOPTIMIZATION LEAD


Optimizers are testing a variety of elements, each with different degrees of complexity

User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)63%

Landing Pages61%

Messaging and Copy57%

Forms54%

Offers and Promotions42%

Recommended Content Functionality38%

Social Sharing Functionality37%

Payment Functionality37%

Other

What do you test when you are optimizing?

User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)63%

Landing Pages61%

Messaging and Copy57%

Forms



What you’re likely to test depends on which industry you’re in

• B2B more likely to test landing pages

E-Commerce B2B Media


65%74%

67%

65%

Landing Pages

Messaging and Copy


51%46%

53%

44%

Offers and Promotions



32%36%

42%

50%

Social Sharing Functionality

Payment Functionality


• E-commerce more likely to test offers, payments

• Media more likely to test social sharing

#opticon2015@srusonisMultiple Times per Week

WeeklyMultiple Times per Month

MonthlyMultiple Times per Year

37%

21%

15%

12%

9%

Approximately how often do you run A/B or multivariate tests to opti-mize your website, mobile app, or other digital channels?

Multiple Timesper Week

Multiple Timesper Month

58% of optimizers are testing at least weekly



hypotheses



experiment ideas


experiments

Conclude and analyze experiments,

share results OPTIMIZATION LEAD


Measuring optimization success is multi-faceted

Conversion Improvements from Experiments

58%

Revenue Generated from Experiments

52%

ROI (Combination of Conversion Improvement and Revenue)

49%

Output or Velocity of Experiments

37%

Number of Team Members or Departments Involved

25%

Other

1%

0 5 10 15 20 25 30 35 40 45 50 55

How do you measure the success of your optimization program?


What does this mean for optimization as

an industry?


An emerging skills market for optimization


Widening gap between optimizing companies and those who have yet to start


How to leverage these findings to improve your optimization.

What can I do today?


• Evaluate the makeup of your team; focus on communication and removing bottlenecks

• Diversify your inputs for optimization beyond analytics

• Maintain a relentless focus on process to improve experiment quality and quantity

Conversation starters and next steps


Focus on building cross-functional teams


optimize.ly/optimizationbmrkDownload the full Benchmark Report

http://optimize.ly/optimizationbmrk

#opticon2015

Thank you!Shana RusonisContent Marketing Manager Optimizely

@[email protected]

mailto:[email protected]


Template Slides

Appendix

“The goal is to turn data into information, and

information into insight.” – Carly Fiorina, former CEO of HP

“As revenue and budget allowance increases, we plan

to put this money back into better optimization.”

– Anonymous Survey Respondent


Optimization is a cycle of continuous steps:

• Conduct research, uncover conversion points to optimize

• Create hypotheses, suggest variations and improvements

• Build, QA, and launch experiments

• Analyze experiment results

• Share findings and determine next steps


We all have challenges …

• Maximizing ROI

• Delivering custom content

• Making quick decisions

• Understanding, taking action on data

0 10 20 30 40 50 60 70 80 90 100

20%

24%

24%

27%

30%

31%

31%

Maximize the ROI of my resources

Deliver customized experiences to customer segments

Make quick decisions

Take action on company data

Embrace the creative talents of my team

Make data-driven decisions

Understand company data

37%

26%

35%

34%

35%

39%

35%

31%

26%

23%

31%

24%

21%

27%

12%

15%

15%

6%

10%

8%

6%

9%

Very Good Good OK Not Good Very Bad


Color palette

Primary

Secondary

AaAdelle Sans

AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz

0123456789

opticon 2015- benchmark your optimization program

Business

following statements

optimization key ndings

optimization impact

relevant content

creative output

engagement rates

culture of learning

stories data