Testing, Proving, and DiggingMicah Fisher-Kirshner | Senior SEO Manager

www.become.comSeptember 13th 2011

Quick Background• Become.com: price comparison website

• Competitors: US: NexTag, Shopping, Shopzilla UK: PriceRunner, Twenga DE: Idealo, Billiger JP: Kakaku

• Why are we able to run multiple tests? We are a vertical search engine

Public Correlations• How to tell good versus bad presentations?

Good Correlation Presentations Use…• Scatter plots

Helps visualize the correlation

• Non-integer numbers Generally integers make it

harder to visualize correlations

A Thing (Or Three) About Correlations1. Correlations are a place to start, not an end.2. Sanity check the results with an SEO3. Ask yourself, “Does this make sense? What if we factor

for…”

Likes?

Comments?

When To Go Linear (Regression)• Amount of effort

Do you have enough data to prove it is statistically significant?

How long with data collection take?

What is the return on your investment to collect and test this data?

• Quality data Do you trust where the data

comes from? Do you have enough metrics that

factor into the algorithm?

• Facebook Shares Enough data and factors to run a

linear regression

Panda and Average Time On Site

• Sample set Correlation: 42%

Panda Correlation Digging Does the correlation hold up by

site type?

Punishment by group?1 = How To’s2 = CSEs3 = Marketplaces4 = News

Testing and Understanding Your Data

Day over day

Week over week

Watch out for seasonality

Best Of The Tests• Subdirectories, subdomains,

and domains Find ways to split your site in half along

standard site architecture to run tests Don’t have these? Use numbers in URL or

even/odd

• Test and control Randomize what your data sets are to

avoid bias Two controls are better than one

• Length of time The smaller the data set, the longer the

test needs to run

Common Correlating Pitfalls• Other marketing channels

Did your brand team launch something during the same time period? Did your UX group modify the layout affecting your test?

• Extraneous online events Did Google update their algorithm? Was there a change in how your analytics tracks events? Did something break? Did you launch a change that affected your own tests?

• Various offline events Was there a holiday that skewed week over week data? Did a world or national event occur around the same time?

Proving Requires Digging

• DC Earthquake lowered traffic by 2% Odd traffic increases on Wednesday gets magnified by a weak Tuesday

Thanks!

Question EverythingQuestion Everything

Learn more at: www.become.comwww.linkedin.com/in/micahfk