13 Ways You're Screwing Up

Your A/B Tests

by Peep Laja

Peep Laja, ConversionXL @peeplaja

Typical scenario:

A company runs 100 tests

/ year. After 12 months

their conversion rate is still

the same.

Testing mistake #1:

Precious time wasted

on stupid tests

Peep Laja, ConversionXL @peeplaja

Testing mistake #2:

You don’t have a data-

driven, learning-

oriented hypothesis

Peep Laja, ConversionXL @peeplaja

Learn more: http://conversionxl.com/how-to-come-up-with-more-

winning-tests-using-data/

Peep Laja, ConversionXL @peeplaja

Testing mistake #3:

You think you know

what will work

Peep Laja, ConversionXL @peeplaja

Testing mistake #4:

Your absolute

conversions are way

too low

Peep Laja, ConversionXL @peeplaja

Peep Laja, ConversionXL @peeplaja

Calculate needed sample

size in advance:

Peep Laja, ConversionXL @peeplaja

I won’t even look at results until

there’s at least 250-350

conversions PER variation.

Way more if you want to segment the data

Peep Laja, ConversionXL @peeplaja

Testing mistake #5:

You run tests when

you have no traffic

Peep Laja, ConversionXL @peeplaja

Learn more: http://conversionxl.com/how-to-do-conversion-

optimization-with-very-little-traffic/

Peep Laja, ConversionXL @peeplaja

Testing mistake #6:

Your tests don’t run

long enough

Peep Laja, ConversionXL @peeplaja

Peep Laja, ConversionXL @peeplaja

Testing mistake #7:

You don’t test full

weeks at a time

Peep Laja, ConversionXL @peeplaja

Peep Laja, ConversionXL @peeplaja

Testing mistake #8:

Test data is not sent

to third-party

analytics

Peep Laja, ConversionXL @peeplaja

Peep Laja, ConversionXL @peeplaja

Testing mistake #9:

You give up after your

first test for a

hypothesis fails.

Peep Laja, ConversionXL @peeplaja

Peep Laja, ConversionXL @peeplaja

Full case study:

http://conversionxl.com/case-study-how-we-improved-landing-page-conversion/

79.3% uplift in conversions

Peep Laja, ConversionXL @peeplaja

Testing mistake #10:

You’re not aware of

validity threats

Peep Laja, ConversionXL @peeplaja

History effect

Instrumentation effect

Selection effect

Peep Laja, ConversionXL @peeplaja

Testing mistake #11:

You’re ignoring small

gains

Peep Laja, ConversionXL @peeplaja

5% monthly increase in

your conversion rate will

result in 80% uplift for

the year. That’s how

math works.

Peep Laja, ConversionXL @peeplaja

Testing mistake #12:

You copy competitors

/ leaders / other

people’s tests

Peep Laja, ConversionXL @peeplaja

Testing mistake #13:

You’re not running

tests at all times

Peep Laja, ConversionXL @peeplaja

Follow me @peeplaja

Thank you!

Join 150,000+ monthly readers to learn from the best conversion

optimization blog on the planet:

Peep Laja