namp conference - a/b testing your way to success
Post on 22-Jan-2018
193 Views
Preview:
TRANSCRIPT
What is A/B testing? 1. Goal 2. Hypothesis 3. Segment 4. Split 5. Show 6. Measure 7. Change
Version A
Version B Winner
✓Each shown to Limited Group Shown to Full Group
Week 1: Tes?ng Image Week 2: Tes?ng Headline Week 3: Release Winner
Headline Winner
Image Winner
Headline A
Image A
✓
Image B
Headline B
Each Shown to Limited GroupEach shown to Limited Group Shown to Full Group
✓
Goal: Click on the ad Segment: Audience interested in “contemporary classical” on Facebook Hypothesis: Focus on the ar#st vs. focus on the arMorm
Week 1: Tes?ng Image
Image A Image B
Goal: Buy a #cket Segment: Audience interested in “contemporary classical” on Facebook Hypothesis: audience knows the ar#sts vs. audience wants background about the ar#sts
Week 2: Tes?ng Headline
Headline A Headline B
Week 2: Tes?ng Headline
Headline A Headline B
Cost per #cket sold: $3.06Cost per #cket sold: $4.70
✓
Tips for A/B testing social ads: • Make a campaign plan with a list of tests you want to run • Let ads run for at least a week before upda=ng • Capture what you learned each week • Different audience segments will respond differently to tests • Strive for cost per conversion (but be wary of bidding on it)
From: Name A
Tes?ng “From” Name Send Winner
From: Name A
From: Name B
✓
Each sent to Limited Group Sent to Full Group
Tes?ng “Subject line”
Goal: Open the email Hypothesis: Be specific about email content vs. general invita#on to an unspecified event.
A St. Patrick’s Day Celtic Sojourn A St. Patrick’s Day Celtic Sojourn Tickets Now On Sale!
2/10/17
Version B
A St. Patrick’s Day Celtic Sojourn Join us this March!
2/10/17
Version A
Tes?ng “Subject line”
A St. Patrick’s Day Celtic Sojourn A St. Patrick’s Day Celtic Sojourn Tickets Now On Sale!
2/10/17
Version B
A St. Patrick’s Day Celtic Sojourn Join us this March!
2/10/17
Version A
Open rate: 13.2%
Open rate: 12.6%
✓
Tes?ng Email Image
Goal: Click on the “nominate” buVon Hypothesis: Many faces of diversity vs. personalizing the impact on one recipient
Image A Image B
Image A: 1.2% began nomina#on process
Image B: 5.9% began nomina#on process
Tes?ng Email Image
Conversions ultimately matter more than opens or clicks.
Tips for A/B testing emails: • Your total list size needs to be at
least 4,000 people • Don’t expect same results to apply
to every campaign • Test “bad ideas” too • Don’t get overwhelmed by all the
tesAng opAons • Strive to measure on-site
conversions, but use the data you can get quickly
Open Rate
Clickthrough Rate
On-site Conversion
Date: ___________ To: ___________ From: ___________ Subject: ___________ Pre-Hear Text: ___________
Headline
Website
DONATE
✓
DONATE
Website A/B Tes?ng Implement Winner
DONATE
Each Shown to Half of Users Shown to All Users
Website A/B Tes?ng
Original (A) Variant (B)
Goal: Donate Hypothesis: Users pay more aVen#on to the le\ side of the page
Tips for A/B testing on your website: • Start with your most important conversion goals and
areas of your site • Try experimenAng with audience targeAng opAons • Elements on the page you can test: text content, image
content, text size/styling in an element, size of element, locaAon of an element, existence of an element
• TesAng tool will tell you when you’ve reached staAsAcal significance (usually 2+ weeks)
• Don’t forget you have to roll out winning soluAon across your site permanently
Instructions: Part 1
1. Head to your group’s corner 2. Reflect on the exisAng Version A 3. Develop a hypothesis for what to change 4. Sketch your own version B on your card
Version B
Sketch your own version B.
Facebook Ad Email Website
Page
My hypothesis…
Audience: Boston area prior #cket buyers
A Christmas Celtic Sojourn Join us for the 15th season of A Christmas Celtic Sojourn!
12/11/17
Example Index Card
“My hypothesis: an email from a named person will have a higher open rate than an email from an
organiza?on”
Robert Lue Join us this March!
12/10/17
Need help? slideshare.net/devonvsmith
Facebook Ad
Website Page
Instructions: Part 2 1. Turn to someone in your color group 2. Say your hypothesis to each other 3. Play Rock, Paper, Scissors to idenAfy
the “winning version” of the test 4. Winner finds a new person to play 5. Loser becomes the cheering secAon
behind their partner 6. Repeat
Check out the NAMP blog for the results of these tests…hUps://namp.americansforthearts.org/get-smarter/arts-marke?ng-blog
Eva Rosenberg | eva_rosenberg@harvard.edu | edportal.harvard.edu Devon Smith | @devonvsmith | measurecrea#ve.com
Q&A
top related