Presented by Disney & SAS

October 2012

Disney Marketing ROI Case Study

DMA Conference

Defining A Marketing ROI Solution

Reach the right audience

Through the right channel

At the right time

With the right frequency

At the right price

Maximize Return

on investment for

marketing spend

Stand-alone studies often fail to achieve long-term

success—trying to implement a project instead of a process!

Presentation Agenda

• Introduction

– Disney Management Science & Integration

– SAS

– The Science Behind Marketing ROI

• Case Study Overview

– Project Goals & Organization

– Data Management

– Science Integration

– Tool Development

• Lessons Learned

• Questions & Answers

Disney Management Science and Integration

• Consulting support for analytics, data and reporting needs

• Technology integration for reporting and data tools

• Development and management of decision science tools

4 employees - 2008 30 employees - 2012

SAS® Company Overview

• 2011 & 2010 Fortune

Magazine: #1 Place to Work

• 2011 Revenue: $2.73 billion

• SAS® reinvests ~25% of annual

revenue into R&D

• 90 of top 100 companies on

FORTUNE Global 500® use

SAS®

SAS® is the largest

independent software vendor

in the world

SAS Annual Revenue 1976-2011

Science Behind Marketing ROI – Modeling

Marketing Spend

Sale

s

TV

Radio

Marketing Effort

For Each Channel (spend, impressions, etc.)

vs.Response

Variable (sales, leads, etc.)

Measurement Model

More effective

Lesseffective

Science Behind Marketing ROI – The “Right” Model

Regression / Time Series Model

R2 = 97%

Econometric / Panel Model

R2 = 67%

Sales (t) = …

+0.7 * Sales (t-1)

-0.2 * Price

+0.06 * TV

-0.005 * Online

+…

Sales (t) = …

+0.2 * Sales (t-1)

-1.0 * Price

+0.1 * TV

+0.02 * Online

+…

Heavy weight on lagged sales; sales not

responsive to price & media changes

Less weight on lagged sales; price &

media elasticities more reasonable

Selecting the right modeling approach is critical for success!

Better for FORECASTING Better for MEASUREMENT

Science Behind Marketing ROI – Measurement

Analysts pay careful attention to data considerations and

choice of models to robustly fit the data for measurement

Saturation Curves

GoodwillImpressions by

Media Type

Model

Cable

Print

Radio

Spend

Rati

ng

s

Rati

ng

s

Time

Imp

res

sio

ns

Time

Model Input Model Output

Saturation Curves

GoodwillImpressions by

Media Type

Model

Cable

Print

Radio

Spend

Rati

ng

s

Rati

ng

s

Time

Imp

res

sio

ns

Time

Radio

Spend

Cable

Print

Radio

Spend

Ra

tin

gs

Science Behind Marketing ROI – Optimization

Imp

res

sio

ns

Time

Optimal Media Mix

Optimal Flighting

Planners leverage model output and their insights to adjust

and optimize marketing plans per business constraints

Case Study Overview

• How effective is our current marketing spend?

• Which shows should get more marketing dollars?

• Which channels are the most effective? Most efficient?

• Based on current practices, where are we over-saturated?

A television network is seeking decision science

support to improve return on investment for the

marketing of primetime television shows

Case Study Challenges

• Data is warehoused in multiple systems, with few connection points

• Impression-level data is extremely difficult to capture, with actualized

data existing in combinations of spreadsheets, e-mails, and faxes

• Given the state of the data, common reports can take days to generate

Limited data availability prevents the network from getting

accurate measures of performance for marketing efforts

Previous attempts to answer these questions have yielded

valuable insights, but have not created sustained changes

• Avoid the temptation to answer all questions with a single model

• Ensure inputs into the solution are readily available and cost effective

• Avoid bundling decisions that are controlled by separate teams

Disney and SAS® Partnership

Project Management 15% 15%

Data Management 30% 15%

Science Integration 30% 30%

Tool Development 25% 40%

Project Timeline

Established a separate timeline for each work stream, inclusive

of milestone and reports out to key stakeholders

Data Collection Overview

• Identified over 30 potential data sources and almost 250 variables

• Data sources ranged from databases, spreadsheets, e-mails, and faxes

• Established weekly meetings with key stakeholders and implemented

dashboards to review data collection progress

• Placed an analyst in the media agency office for four weeks to speed

data collection and improve understanding of the data

Data collection ultimately took four times longer than

originally planned, due in large part to data quality issues

Data collection is never really over—continue to find errors

or missed opportunities even months later!

Data Collection Challenges

Model database changed 17 times during a 1-year span,

most often due to missing data or data collection errors

Bad circulation

estimate for

Entertainment

Weekly

Misclassified OOH

support as Events

“Week 53”

Issue

Nielsen P3

vs. C3

Duplication from

SQL Errors

Magazine Cume based

on all publications

instead of purchased

Data Visualization

Showing clients the relationship between impressions and costs helped

to identify likely errors in the data (e.g., misclassification of spending)

Data Visualization (cont.)

Exploring flights enabled us to recognize the need to model

certain media types differently than others

15% 70% 15%

Sunday

Monday

Tuesday

Wednesday

Thursday

Friday

Saturday

S

S M

S M T

S M T W

S M T W R

S M T W R F

S M T W R F S

Episode

Air Date

Promos in

Calendar Week

Promos in

Past 7 Days

Transform to a full week

Data Transformation

Often necessary to transform the data for measurement

variables in our models to avoid creating misleading

insights or recommendations

M T W R F S

T W R F S

W R F S

R F S

F S

S

S

S M

S M T

S M T W

S M T W R

S M T W R F

S M T W R F S

Program Name

Air Date

Start Time

Duration

Program Type

Program Rating

Lead-in Rating

Competition

Survey Respondents

Aware Respondents

% AwareUnaided & Aided

Intent to ViewTop Box, Top 2 Box, Non Committed, Bottom Box

On-Air PromosTRPs, Seconds, # of Spots

Digital Impressions & Clicks

Cinema Impressions, Seconds Per Spot

National Cable TRPs

Newspaper Impressions & Circulation

Magazine Total & Weekly Impressions

Spot CableTRPs & Impressions

Spot Radio TRPs

OOH Impressions

NIELSEN AWARENESSPROMOS & MARKETING

Data Handoff to Science

Key milestone was the go/no-go decision on beginning

the development of the measurement model

Network Radio

Synergy Cable

Synergy Online

Emails & Newsletters

Public Relations

Affiliate Promotions

On-Air PromosDay-of-Week, Promo Length

NielsenReach, Share, HUT, PUT

PrintSize, Placement, Inserts

National CableChannel, # of Spots, Promo Length

Spot Cable & Radio# of Spots, Seconds of Promo

OOH# of Units, Size, Media Form

DigitalSize, Placement, Pillar

Social MediaFacebook, Twitter, Blog Mentions

Geo-Panel DataLocal Market Ratings and Marketing

On-Air Promo PrecisionMinute-by-Minute Ratings

EfficiencyCosts for Marketing & Promotions

On-Air Promos

Digital Impressions

MISSING DATA MISSING COMPLEXITY

MODEL EXPANSION

DATA RECONCILIATION

Data Handoff to Science (cont.)

Future iterations of the model will incorporate new data that is either

unavailable right now or represents a higher level of complexity

Science Integration

Critical to integrate science team with tool

developers to ensure alignment with the

expected input and outputs of the models

Integration between the team managing

data collection and model development

is critical to the success of the project

When it doesn’t work well—each revision of

the data model would delay the science

timeline by 3 weeks!

Science Data

Science Tool

Overview of Planning & Optimization Tool

The tool is designed to become self-sustaining to support updates

to the measurement model and to allow media plan comparisons

Data Model

Agency Media Plans

Recommended Media Plans

Measurement Model

Optimization Model Goals &

Constraints

Actualized Media Plans

HistoricalData

(one time)

ApprovedMedia Plans

Model Adjustments

Optimization Goals

Objective is to maximize total ratings for the premiere episodes

of all shows within a marketing campaign portfolio

• Provide recommended spending by channel for each show/week

combination

• Allow users to input constraints on total spending by show/channel/week

• Define spend thresholds that reflect minimum purchase amounts for

each channel

• Compare optimal recommendations against manually created plans

Critical to understand relationship between spend and

impressions; some channels have a significant delay between

purchase and delivery!

Evaluating Media Plans

Ability to compare different plans by measuring the number of

new households generated for each incremental unit of spend

Week Cable Radio Print Outdoor Cinema

t = -5 20 N/A 20 20 20t = -4 20 N/A 20 20 20t = -3 20 N/A 20 20 20t = -2 20 20 20 20 20t = -1 20 20 20 20 20t = 0 20 20 20 20 20

Recommended Plan:

Media Agency Plan: Week Cable Radio Print Outdoor Cinema

t = -5 70 N/A 80 110 10

t = -4 105 N/A 5 170 15

t = -3 160 N/A 5 5 30

t = -2 240 5 5 10 80

t = -1 355 75 25 30 125

t = 0 25 10 50 50 150

(balanced by optimization)

1 2

3

4

5

(incremental opportunities)

Key Lessons Learned

Creating Clear Requirements

Having a Test Environment

Designing a Structured QA

Process & Team

“Shadow” Implementation

Questions and Answers