digital impact 2014 - oh boy these ab tests sure look like bullshit to me
DESCRIPTION
The main slide deck from Digital Impact - which goes with the deck uploaded just before this one (with the same cover page)TRANSCRIPT
Oh Boy!
@OptimiseOrDie
These A/B tests appear
to be bullshit!
@OptimiseOrDie
• UX and Analytics (1999)
• User Centred Design (2001)
• Agile, Startups, No budget (2003)
• Funnel optimisation (2004)
• Multivariate & A/B (2004)
• Conversion Optimisation (2005)
• Persuasive Copywriting (2006)
• Joined Twitter (2007)
• Lean UX (2008)
• Holistic Optimisation (2009)
Was : Working at Spareroom.co.ukNow : Doing workshops & consulting again!
Oh Boy!
@OptimiseOrDie
These A/B tests appear
to be bullshit!
Top F***ups for 20141. Testing in the wrong place2. Your hypothesis inputs are crap3. No analytics integration4. Your test will finish after you die5. You don’t test for long enough6. You peek before it’s ready7. No QA for your split test8. Opportunities are not prioritised9. Testing cycles are too slow10. You don’t know when tests are ready11. Your test fails12. The test is ‘about the same’13. Test flips behaviour14. Test keeps moving around15. You run an A/A test and waste time16. Nobody ‘feels’ the test17. You forgot you were responsive18. You forgot you had no traffic19. You ran the wrong test type20. You didn’t try all the flavours of testing @OptimiseOrD
ie
slidesha.re/1wBbZ9c
#fail
@OptimiseOrDie
@OptimiseOrDie
@OptimiseOrDie
Oppan Gangnam Style!
@OptimiseOrDie
@OptimiseOrDie
@OptimiseOrDie
@OptimiseOrDie
The 95% Problem
• Many people use 95, 99% ‘confidence’ to stop
• This value is unreliable• Read this Nature article : bit.ly/1dwk0if• You can hit 95% early in a test• If you stop, it could be a false positive• Tools need to be smarter about inference• This 95% thingy – it’s last on your list for
reasons to stop testing
@OptimiseOrDie
Business & Purchase Cycles
@OptimiseOrDie
• Customers change• Your traffic mix changes• Markets, competitors• Be aware of all the waves• Always test whole cycles• Minimum 2 cycles
(wk/mo)• Don’t exclude slower
buyers
Start Test Finish Avg Cycle
16
When to stop?• Know the cycles • Run a test length calculator -
bit.ly/XqCxuu• Set your test run time • Yup, just like a retail store• Run it• Stop it• Analyse the data• That’s it. No peeking and waiting for the
numbers to look good so you can stop.• Go again @OptimiseOrD
ie
No QA testing for the AB test?
QA Test or Die!• Over 40% of tests have had QA issues.• It’s very easy to break or bias the testing
Browser testing www.crossbrowsertesting.comwww.browserstack.comwww.spoon.netwww.saucelabs.com
www.multibrowserviewer.com
Mobile devices www.appthwack.comwww.deviceanywhere.comwww.opendevicelab.com
Article bit.ly/1wBccsJ
@OptimiseOrDie
Gamble the Company AWAY!
• I get 60-70% right• UX and Copywriters good at
picking!• C level execs are easy marks• Ironically, many decide ‘designs’• You need collaborative test
design• It’s a team game, with
customers• Flip a coin, anyone?
WE’RE ALL WINGING IT
2004 Headspace
What I thought I knew in 2004
Reality
2014 Headspace
What I KNOW I know
Me, on a good day
Guessaholics Anonymous
Rumsfeldian Space
@OptimiseOrDie
Rumsfeldian Space
@OptimiseOrDie
23 : Business Future Testing?
“Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy.”
The 5 Legged Optimisation Barstool@OptimiseOrD
ie
#1 Smart Talented Polymath People
Flexible and Agile teams
@OptimiseOrDie
Fittest? Agile!
@OptimiseOrDie
#2 : Analytics Investment (tools, people, dev time)
@OptimiseOrDie
#3 : User research and insight
@OptimiseOrDie
#3 : THE BEST IDEAS COME FROM?
“On the average, five times as many people read the headline as read the body copy. When you have written your headline, you have spent eighty cents out of your dollar.”David Ogilvy
“In 9 years and 40M split tests with visitors, the majority of my testing success came from playing with the words.”@OptimiseOrDie
#4 : GREAT COPYWRITING
• Google Content Experiments bit.ly/Ljg7Ds
• Optimizelywww.optimizely.com
• Visual Website Optimizerwww.visualwebsiteoptimizer.com
• Multi Armed Bandit Explanationbit.ly/Xa80O8
• New Machine Learning Toolswww.conductrics.comwww.rekko.com
@OptimiseOrDie
#5 : Split Testing Tools
@OptimiseOrDie
#1 Culture & Team#2 Toolkit & Analytics investment#3 UX, CX, Service Design, Insight#4 Persuasive Copywriting#5 Experimentation (testing) tools
The 5 Legged Optimisation Barstool
READ STUFF
READ STUFF
READ STUFF
#5 : FIND STUFF
@OptimiseOrDie
@danbarker Analytics@fastbloke Analytics@timlb Analytics@jamesgurd Analytics@therustybear Analytics@carmenmardiros Analytics@davechaffey Analytics@priteshpatel9 Analytics@cutroni Analytics@avinash Analytics@AschottmullerAnalytics, CRO@cartmetrix Analytics, CRO@Kissmetrics CRO / UX@Unbounce CRO / UX@Morys CRO / Neuro@UXFeeds UX / Neuro@Psyblog Neuro@Gfiorelli1 SEO / Analytics
@PeepLaja CRO@TheGrok CRO@UIE UX@LukeW UX / Forms@cjforms UX / Forms@axbom UX@iatv UX@Chudders Photo UX@JeffreyGroks Innovation@StephanieRieger Innovation@BrianSolis Innovation@DrEscotet Neuro@TheBrainLadyNeuro@RogerDooley Neuro@Cugelman Neuro@Smashingmag Dev / UX@uxmag UX@Webtrends UX / CRO
#5 : LEARN STUFF
@OptimiseOrDie
Baymard.comLukew.comSmashingmagazine.comConversionXL.comMedium.comWhichtestwon.comUnbounce.comMeasuringusability.comRogerDooley.comKissmetrics.comUxmatters.comSmartinsights.comEconsultancy.comCutroni.com
www.GetMentalNotes.com
#12 : The Best Companies…
• Invest continually in analytics instrumentation, tools, people• Use an Agile, iterative, cross-silo, one team project culture• Prefer collaborative tools to having lots of meetings• Prioritise development based on numbers and insight• Practice real continuous product improvement, not SLEDD*
• Are fixing bugs, cruft, bad stuff as well as optimising• Source photos and content that support persuasion and utility• Have cross channel, cross device design, testing and QA• Segment their data for valuable insights, every test or change• Continually reduce cycle (iteration) time in their process• Blend ‘long’ design, continuous improvement AND split tests• Make optimisation the engine of change, not the slave of ego
* Single Large Expensive Doomed Developments
THE FUTURE OF TESTING
43
Thank You!
@OptimiseOrDie
Slides
:slideshare.com/sullivac
: linkd.in/pvrg14