optimizely experience - la - disney-abc
DESCRIPTION
Learn how Disney-ABC uses Optimizely and experience optimization to drive content consumption of hit shows like Modern Family and The BacheloretteTRANSCRIPT
![Page 1: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/1.jpg)
Bringing A/B Testing to Entertainment Disney-ABC Digital Media
The Optimizely LA Experience 8.19.14
Proprietary. Do not distribute without permission (Khai Tran).
Khai Tran Sr. Mgr, Analytics & Customer Insights Disney-ABC Digital Media
![Page 2: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/2.jpg)
Building A/B Testing within Disney-ABC Digital Media
Background on our business
A case study
Final thoughts
2Proprietary. Do not distribute without permission (Khai Tran).
Why A/B test?
![Page 3: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/3.jpg)
3
Disney-ABC Television – WATCH Products & Platforms
Proprietary. Do not distribute without permission (Khai Tran).
Own, operate, and stream full-episodes, short-form, live video content and games. Serve as a marketing channel for our shows and brands.
What’s our Business?
Supported Platforms
Supported Brands
![Page 4: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/4.jpg)
4Proprietary. Do not distribute without permission (Khai Tran).
![Page 5: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/5.jpg)
5Proprietary. Do not distribute without permission (Khai Tran).
![Page 6: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/6.jpg)
6Proprietary. Do not distribute without permission (Khai Tran).
![Page 7: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/7.jpg)
7Proprietary. Do not distribute without permission (Khai Tran).
![Page 8: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/8.jpg)
8
Where A/B Testing Lives in our Org
WATCH Products & Platforms
Product Engineering Video Programming MarketingAnalytics &
Customer Insights PMO
Data "Data Design
Requirements Tagging
Data Integrity
Analytics "A/B Testing
Path Analysis Case Studies KPI Reporting
Customer Insights "Support Agents
Customer Feedback
Escalation Mgmt Voice of Customer
Proprietary. Do not distribute without permission (Khai Tran).
![Page 9: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/9.jpg)
Why A/B Testing Makes Sense in Entertainment
9
1. Controls for external factors"• Traffic is randomly split across test variations during
time period of test"• Based on real users, not focus groups "
2. Launch and iterate faster"• Eliminates guesswork in evaluating different options"• Reduces time and resources "
3. Drive innovation and growth"• Whether the test wins or loses, it enables to learn more
about your users"• Speeds up innovation by minimizing risk and fear in
trying out new ideas
Controls for External Factors
"Originals vs. Repeats
Show Popularity Seasonality Windowing
Competitive Offerings Macro Factors
Proprietary. Do not distribute without permission (Khai Tran).
![Page 10: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/10.jpg)
10
Driving Actionable Insights – Testing in the Larger Picture Track Key Success Drivers
Analyze User
Interactions
Segment for Deeper Insight
Test & Optimize
Drive Actionable
Insights
1. Track key success drivers • Keep pulse of the business • See if we’ve made a difference
"2. Analyze user interactions • Heat mapping, path analysis,
etc. • Identify areas for improvement "
3. Segment for deeper insight • Sub-groups behave differently • Solve for key segments "
4. Test and Optimize • Form hypotheses and tests based
on insights • Validate different variations
based on cause-and-effect
Proprietary. Do not distribute without permission (Khai Tran).
![Page 11: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/11.jpg)
11
Full-Episodes Main Page""• Top page of our user path"• High traffic page"• High potential impact"• Bounce rate: 20-35%
* Omniture: Jan – Jun 2011, FEP visits only (excluded iPad)
40%
35%
0.2% 1% 1%
0.5%1%
2% 4%
7%
4%3%
What’s Interesting "35% of users clicked “All Shows” in bottom nav. Why? "Insight: Users want to get directly to the show that they’re interested in.
ABC.com Case Study – What and How We Tested
![Page 12: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/12.jpg)
12Proprietary. Do not distribute without permission (Khai Tran).
Variation A: "Control Version""Full-Episode Home Page "www.abc.com/watch
![Page 13: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/13.jpg)
13
Variation: Display all shows with logos "Hypothesis: "Will increase video starts because it: " • Reduces click to all shows • Easier to access non-recent content "
Potential Concerns: • Requires lots of scrolling. • Choice overload may lead users to leave. • Increases page load time.
Proprietary. Do not distribute without permission (Khai Tran).
Variation B: "All Shows with Show Logos
![Page 14: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/14.jpg)
14
Variation: Display all shows text "Hypothesis: "Will increase video starts because it: " • Is easier to scan and find shows • Loads faster, requires less scrolling "
Potential Concerns: • Choice overload • Too much text. Unattractive.
Proprietary. Do not distribute without permission (Khai Tran).
Variation C: "All Shows with Text
![Page 15: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/15.jpg)
15
Variation: Display top 8 noncurrent shows with link to “All Shows” "Hypothesis: "Will increase video starts because it: " • Highlights most popular Fall shows • Extends long tail for non-recently aired content • Distinct from current shows displayed in the
slideshow
Proprietary. Do not distribute without permission (Khai Tran).
Variation D: "Top Shows Fall Season "(Noncurrent Shows)
![Page 16: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/16.jpg)
16
Variation: Display top 8 shows that are in season at time of test "Hypothesis: "Will increase video starts because: " • These shows account for 80% of current video
views • Easy to find and access these shows
Proprietary. Do not distribute without permission (Khai Tran).
Variation D: "Top Shows Summer Season "(Current shows)
![Page 17: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/17.jpg)
17
4.24% 3.74% 3.93% Video Starts 4.94%
Chance to beat 99.3% 98.4% 97.8% 97.9%
So Which One Won vs. the Control?
A Control
B All Shows Logos
C All Shows Text
D Top Shows Noncurrent
E Top Shows Current
![Page 18: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/18.jpg)
18
0.5 Days "• Define goals,
success metrics, scope, etc. ""
2 Days "• Assess dev needs
• Build test versions
0.5 Day "• Configure
campaign in Optimizely
Planning Design Development
Campaign Configura
tion0.5 Day "• QA
• Launch test
1 Day "• Hypothesize test
variations and design comps
Setup time – 4.5 days
7 Days "• Advised by
T&T to run at least 1 week ""
Run
A/B Test
Analysis
2 Days "• Analyzed real-
time results throughout
Run + Analysis time – 9 Days
A/B Test Details ""• Ran for one week (June 28 – July 5) "• 660k+ unique visitors"• 20% traffic allocation "• Test groups maintained across sessions
A/B Testing Steps and Actual Time Spent
![Page 19: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/19.jpg)
19
"Even with a slight page tweak, which appears below the fold, we can produce a sizable revenue lift for the business.
Video Starts Driven from Homepage Revenue
146 MM $13.7 MM
+ 7.2 MM + $677 KLift from
Test + 4.94%
Projected Contribution
Connect A/B Testing Results to Revenue Impact
Actual numbers have been masked, but results are in the ballpark. This test ran in 2011.
x $0.0938 =
x $0.0938 =
Avg. Ad Revenue per Start
Before Test
![Page 20: Optimizely Experience - LA - Disney-ABC](https://reader031.vdocuments.site/reader031/viewer/2022020217/554f1f3fb4c9058b388b4f43/html5/thumbnails/20.jpg)
20Image from Entrepreneur magazine (Dec 2012). http://www.entrepreneur.com/article/224967
Final Thoughts