testing is not cro but it’s still darn important
TRANSCRIPT
TESTING IS NOT CROBUT IT’S STILL DARN IMPORTANT
FIRST LET’S COVER SOME
A/B TESTING BASICS
Types Of Tests
Synchronous Asynchronous
Sequential Testing A/B/…/N TestingMultivariate
Testing
Test Types
Synchronous Testing
July 1st - July 15thJuly 16th - July 30th
A/b/…/n Split Testing
$$$$$$$$$$$$
$$$
A/b/…/n Testing
You’ll Get Page ConversionData (Not Element By Element)
Better For Low Traffic Sites
A/b/…/n Testing
Some Rules:Isolate Individual Elements
If You Don’t Isolate Elements, DO NOT attribute the lift or loss to any single element
Multivariate Testing
You’ll Get Element By Element Data
Don’t Run If Less Than 100k Pageviews Per Month
NOW LET’S GIVE A/B TESTING SOME LOVE
I BUILT MY CAREER AROUND SPLIT TESTING
IT’S THE ONLY WAY TO GET DEFINITIVE ANSWERS IN A
CLEAN ENVIRONMENT
TESTING HELPS GUIDE DECISIONS AND IMPROVE
THE USER EXPERIENCE WHILE BOOSTING PROFITS
TESTING GOT THE CONVERSATION
STARTED!
A/B TESTS ARE SEXY
AWESOME STUFF RIGHT?
THERE’S ONE BIG PROBLEM
EVERYONE WANTS TO TEST
EVEN WHEN THEY CAN’T
SIMPLY PUT: NOT EVERYONE CAN OR
SHOULD BE SPLIT TESTING
NOT BEING ABLE TO TEST DOESN’T MEAN…
…NOT BEING ABLE TO OPTIMIZE
THERE ARE PLENTY OF WAYS TO BOOST CONVERSIONS
OUTSIDE OF SPLIT TESTING
TODAY IN A NUTSHELL1. When You Can Test2. Should You Actually Test3. What To Do When You
Can’t/Shouldn’t Test
Part PART 1Can I Actually Run A Test
Testing Eligibility
• ~100 Conversion Per Variation• Run for at least a full week • Not run longer than 8 weeks• Test can reach 95% confidence in this time frame
Testing Eligibility
• ~100 Conversion Per Variation• Run for at least a full week • Not run longer than 8 weeks• Test can reach 95% confidence in this time frame
Conversion Per Day Chart
Minimum Number Of Conversions Per Day
Test Period (Days)
2 Variants 3 Variants 4 Variants 5 Variants
7 29 43 58 72
14 15 22 29 36
21 10 15 20 24
28 8 11 15 18
35 6 9 12 15
42 5 8 10 12
49 5 7 9 11
56 4 6 8 9
Problem: Not Enough Conversions
Solution:Run Longer
Secondarily:If Outside Of 8 Week Testing Window - Don’t Test It
VARIABLE NUMBER ISN’T THE ONLY THING THAT DICTATES
TIME!
THE PERCENT LIFT IS A HUGE TIMING
FACTOR
In order for this test to get statistical results, it would require a 10% lift
AND SO IS THE CONFIDENCE RATE
But we’ll talk more about that later
Does Your Page Get Enough Daily Conversions?
YES No
Can’t Test - Sorry!Is There A Big Enough Lift Potential?*
YES No
This Is A Testable Page
Can’t Test - Sorry!
*With An Appropriate Confidence Rate/Index
Part PART 2Should I Actually Run That
Test
THE CONVERSION HIERARCHY
FunctionalDoes the page work and do what it’s supposed to?
Accessible Can people of all skill levels on any device use the site?
What are the barriers on the site?
UsableCan people understand your site, i.e., is it user friendly?
IntuitiveDoes the site have a logical flow?
Does the offer match the source?
PersuasiveIs your site convincing people to buy or take an action?
“For every test you run, there are an infinite number of other tests that you could have run” - Justindamus 120 AD
Part PART 3How To Optimize If I Can’t/Shouldn’t Test
You Have A Few Options
• Evaluate Your Offer• On-Site Retargeting• Changes Based On Qualitative Research
EVALUATE YOUR OFFER
It’s not a traffic problem, it’s an offer problem
It’s not a testing problem, it’s an offer problem
You might have to head back to the drawing board completely.
Does your product/service provide real value?
You also might be going after the wrong traffic source.
Identify if your ideal customer is ‘hanging out’ on the traffic medium
TRY ON SITE RETARGETING
What is On Site Retargeting?
Evaluating current interactions and providing a relevant offer based on these interactions.
Time On Site
Number Of Pageviews
Type Of Content Consumed
Exit Intent
Referring Source
Just Under Half Of All Offer Pageviews Sent By Optimonk
Typical Campaign Conversion:69%With Exit pop:80.7%
Using this method we got 2,689 more leads in just 14 days!
“Respondents targeting more than 20% of their visitors with personalized content achieved a 5% conversion rate, which was 2x the average conversion rate of 2.5% in their study” [Adobe 2014 Survey]
PAGE CHANGES BASED ON
OPTIMIZATION RESEARCH
Here are a few things to try
• Click/Scroll Tracking• Usability Testing• User Surveys• Session Recordings
CLICK/SCROLL TRACKING
People who actually scroll to the bottom clickNot enough people are scrolling that far down
~15% of all clicks on this page are on the nav bar
Try a shorter variation to get people to scroll to the CTA.
Add multiple CTA buttons at areas where scrolling is highest.
Hide the nav bar on this page
USABILITY TESTING
USER SURVEYS
• Was it was easy to find what you were looking for?• Were the words/vocab used to define categories/sub-categories clear to you?• Do you have any suggestions to improve our website navigation?• Does our website look credible to you?• Is our pricing clear?• Is there anything else you’d like to know before signing up with us?• Will you shop with us again? Why/why not?• Do you think the form has any confusing/unnecessary input fields?
QUESTION IDEAS
SESSION RECORDINGS
Shows individual recordings of a user interacting with your site.
Unlike heatmaps, this isn’t aggregated data
Really blow theirmind with a real life user recordingwith audio
TRICKS TO TEST WHEN YOU DON’T HAVE THE TRAFFIC
USE MICRO CONVERSIONS AS
INDICATORS
Test LegnthMinimum Conversions Per Variation
Per Day
7 Days 29
14 Days 15
21 Days 10
28 Days 8
35 Days 6
42 Days 5
49 Days 5
56 Days 4
LOWER THE CONFIDENCE THRESHOLD
Lift: 20%Confidence Rate: 95%_________________
Required Sample Size:7,682
Lift: 20%Confidence Rate: 90%_________________
Required Sample Size:6,052 21% Decrease
REMEMBER - SPLIT TESTING IS NOT OPTIMIZATION…
IT’S AN OPTIMIZATION TOOL
Split Testing
PersonalizationUser Surveys
Funnel Analysis
Web AnalyticsPersonas
Neurotesting Advanced Segments
Eye Tracking
Scroll Tracking
CRO
SPLIT TESTING CAN ACTUALLY HURT A
COMPANY
SO PRACTICE SAFE TESTING WITH A PAGE THAT
MATTERS
• Email: [email protected]• Twitter: @jtrondeau• Website: digitalmarketer.com
Justin RondeauOptimization Manager,
Digital Marketer
Let’s Talk More!