kaboose.com – valentine’s day 2009 ab testing campaign - omniture cafe april 2009
TRANSCRIPT
![Page 1: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/1.jpg)
corporate.kaboose.com
KABOOSE Valentine’s Day 2009 A/B Testing Campaign
Case StudyApril 01, 2009
![Page 2: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/2.jpg)
corporate.kaboose.com
Introduction:
Jose DavilaWeb Analytics - A/B Testing Strategist
Kaboose Inc.
![Page 3: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/3.jpg)
corporate.kaboose.com
About Kaboose Inc.
![Page 4: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/4.jpg)
corporate.kaboose.com
Business GoalsOnline Publisher:
• Increase the reach of our network of web properties
• Engage more our audience: • View more content, • Stay longer • Use our services• Interact.
• Business Model:• Advertisement...
![Page 5: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/5.jpg)
corporate.kaboose.com
Questions & ChallengesBefore A/B Testing:
• Testing process was lengthy and cumbersome.
• Most of the decisions were made with limited testing.
• Potentially great ideas were not being tested.
• Organization ready for further optimization.
![Page 6: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/6.jpg)
corporate.kaboose.com
Questions & ChallengesValentine’s Day Traffic:
• Valentine’s Day is an important seasonal source of traffic.
• Previous optimization for traffic and engagement (Marketing, Editorial, SEO, Web Analytics).
• Question: Can we further improve engagement on our Valentine’s page? (PVs/Visit, stickiness)
• List of “improvement” ideas to test: Valentine’s Day landing page.
![Page 7: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/7.jpg)
corporate.kaboose.com
Valentine’s Day Page• Control Recipe: The original page.
![Page 8: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/8.jpg)
corporate.kaboose.com
Valentine’s Day Page• Control Recipe: The original page.
![Page 9: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/9.jpg)
corporate.kaboose.com
Test StrategyValentine’s Day Landing Page:
• Test variations of the center content.
• Incorporate multiple “improvement” ideas.
• Content area broken down in three modules/blocks.
• Variations for each module/block (based on previous tests)
• Success metrics: PVs/Visit, stickiness
![Page 10: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/10.jpg)
corporate.kaboose.com
Test Strategy• Modules/blocks of content...
![Page 11: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/11.jpg)
corporate.kaboose.com
Recipes: Module 1 -Top ModuleA: Feature
Rotator• Flash based rotator.
B: Static Image and Links
• Minimalist version.
C: Tab Based Hub
• Flash based. • Static.
![Page 12: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/12.jpg)
corporate.kaboose.com
Test Strategy• Modules/blocks of content...
![Page 13: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/13.jpg)
corporate.kaboose.com
Recipes: Module 2 - Middle ModuleA: Small Index• List of linked categories
and short description.
B: Extended Index• List of categories with
additional links.
![Page 14: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/14.jpg)
corporate.kaboose.com
Test Strategy• Modules/blocks of content...
![Page 15: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/15.jpg)
corporate.kaboose.com
Recipes: Module 3 - Bottom ModuleA: Empty space B: Bottom content
element• Group of three additional links
with images and a short descriptions
![Page 16: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/16.jpg)
corporate.kaboose.com
The Results...!Can you guess?
• From the 12 possible combinations (+ control) the winner was….
… nobody on our team was able to guess it …
First some interesting findings….
![Page 17: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/17.jpg)
corporate.kaboose.com
The Results...!Some interesting findings…• All the new versions performed better.
• One recipe showed the highest performance across the board(+ 54% Lift – PVs/Visit)
• More influential modules/elements identified.
• Behavioral patterns from key visitor segments extracted
![Page 18: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/18.jpg)
corporate.kaboose.com
The Results...!Which recipe won?
• Module 1: Tabbed image features (C)• Module 2: Small Index (A)• Module 3: Blank (A)
• Recipe CAA had the highest performance: engagement + stickiness
• Recipe CAA pushed live
• Over 45% increase in engagement (PVs/Visit) during the whole campaign (including test).
![Page 19: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/19.jpg)
corporate.kaboose.com
Long Term Outcomes.From our A/B testing experience.
• Start small + Value + Allocate resources
• Build awareness based on a sequence of success stories
• Learning process (“failures” can be extremely valuable)
• Don’t assume “best practices” are the best solution. Test them.
• Create testing business culture
• A/B Testing is now part of our process.
![Page 20: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/20.jpg)
corporate.kaboose.com
Questions?
![Page 21: KABOOSE.com – Valentine’s Day 2009 AB Testing Campaign - Omniture Cafe April 2009](https://reader035.vdocuments.site/reader035/viewer/2022062523/58ef32241a28ab02448b4589/html5/thumbnails/21.jpg)
corporate.kaboose.com
Thank you!
Jose DavilaWeb Analytics - A/B Testing [email protected]
Kaboose Inc.