big data, big changes: data-driven product development at etsy

Big Data, Big Changes: Data-Driven Product Development at Etsy

Strata London, 2012

Dr. Jason DavisDirector of Search & Datajdavis@etsy.com / @jasondavis

Monday, October 1, 12

(The change)

Monday, October 1, 12

6 months from inception to launch3 engineering teams3 product launchesDozens of experimentsLots and lots of data analysis

Monday, October 1, 12

The switch to relevancy

Search, advertising, analytics, and more

Building big data products at Etsy

Invest in tooling

Understand tradeoffs

Appreciate complexity

Monday, October 1, 12

Monday, October 1, 12

By the Numbers

1.4B pageviews/month

$525MM sales in 2011

40MM unique visitors/month

800k shops / 150 countries

Monday, October 1, 12

SELECT * FROM listings  WHERE title LIKE '%silver%' ORDER BY creation_tszDESC;

Monday, October 1, 12

Solving the recency problem

Monday, October 1, 12

Better buyer experience

Improved search relevancy

Monday, October 1, 12

Improved advertising

Search Ads

Monday, October 1, 12

More search insight for sellers

Shop Stats

Monday, October 1, 12

Building big data products

Monday, October 1, 12

Build lots of tools

Monday, October 1, 12

Go to the sourceRaw data

visit logsraw query logs

Descriptive statistics

search depthquery conversion rates

Monday, October 1, 12

Invest in tooling

Lunr

Monday, October 1, 12

Side by side testing

Monday, October 1, 12

A/B Analyzer

Monday, October 1, 12

Shop Stats with Referrals

Monday, October 1, 12

Understand tradeoffs

(most tradeoffs don’t look like this)

Monday, October 1, 12

The bike is 20% lighter!Monday, October 1, 12

Queries a, b, & c are better. Queries x & y are not.

Monday, October 1, 12

¡Hola! வணக்கம்!

Guten Tag 你好 ألّس'م عليكم

Hello नम#$ 今日は

Monday, October 1, 12

How will the change affect behavior?

Affect expectations?

User incentives?

Improved data?

Thinking short-term vs long-term

Monday, October 1, 12

Appreciate complexity

Monday, October 1, 12

Vs

Algorithmic Complexity

Monday, October 1, 12

4 months30 experiments11 wins

Success / Complexity Tradeoff

Monday, October 1, 12

June 2005

November 2006

May 2008

Today

UI Complexity

Monday, October 1, 12

Communications & Support Complexity

Monday, October 1, 12

(The change)

Monday, October 1, 12

Etsy - Strata NYCContinous Experimentation with Continuous DeplyomentSteve Mardenfeld

Scala + Cascading = ScaldingAvi Bryant

Case Study: What’s a Customer Worth?Roberto Medri

October 23-25

Monday, October 1, 12

ResourcesCode as Craft bloghttp://codeascraft.etsy.com/

Slides (including these)http://www.slideshare.net/etsy

Github http://github.com/etsy/

Join in the fun http://www.etsy.com/careers/

Monday, October 1, 12

Monday, October 1, 12