experimentation platforms: build or buy?

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

Experimentation Platforms: Build or Buy?RecSys 2014Mike Lo, PayPal

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

2

• Product

• Test Ops

• Merchant Testing

• Knowledge Mgt and Insights

• Experimentation Analytics

Hello!Mike Lo, Sr. Manager - Experimentation, PayPal

PayPal

Experimentation Program

• Manage PayPal’s Experimentation Program

• Formerly part of eBay’s Program

• Build or Buy?

• We did both at PayPal

Expectations for today’s talk

Focus on Experimentation Platform

• PayPal’s ‘Testing Story’

• Some comparison with eBay program.

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

3

Words words words

Big DATA

AN

ALY

TIC

S

Statistics

Ag

ile Mobile

Beta

Ramp

InsightsR

and

om

Alp

ha

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

4

Becoming Data-Driven is difficult!It doesn’t happen overnight. It took us years.

Product

Process

People

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

5

We bought.

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

6

Technology Adoption eBay-focused

PayPal = eBay right? What about the existing platform?Rationale why we didn’t implement the eBay Experimentation Platform

• PayPal’s tech stack

• Fix Tracking

• Can’t wait

• Quick wins

• Endorsement

• Program, not the Product.

• Built ground up for eBay

• Tightly coupled with eBay systems

• Need to decouple features

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

7

Outcome?Ran experiments successfully for 12 months… then …

Couldn’t Scale

Had to cut the cord

Result: Implemented eBay Platform

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

8

No.

It’s all about the TESTING NEEDS of your organization.

So are you advocating to build rather than buy?

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

9

No Product Tests Keep it separate? PayPal’s Needs

What happened?Like many companies, the need started from Marketing but didn’t end there.

• Product Pages

Marketing Pages Product Pages

• Testing different content variants • Testing different feature variants

• Technical Limitations • Is there a clean distinction? • 20 experiments is nice. But how

do we get to 2,000?

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

10

Why it made sense for PayPal to build instead of buy3 main reasons PayPal is investing to build

1 Analytical

2 Technical

3 Scale

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

11

1 - High Fidelity AnalyticsOur site is an ecosystem

Single Enterprise Testing Solution

Scientific rigor in the results

Other sites have natural ‘fences’

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

12

Agile Product Lifecycle Who does your dev? Look beyond

2 - Technical agilityIt’s all about going Agile!

• PayPal’s tech stack

• Master of your domain

• Data

• Site speed performance

• Rolling Full-Bright

Test Plan Design Code Validate

Smoke Test

• Coupled with PDLC

• Code once, push in minutes

• Allows MVT

Biz Req Product Req

Eng Req QA Release

Alpha Beta AB Test Collect Analyze Ramp

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

13

Volume Sophistication Deeper Insights

3- Scaling to the next levelUtilizing the platform to its fullest potential

• Thousands of experiments

• Patented mechanism

• Utilizing PayPal’s platforms • Entire data pipeline

• Automated design

• Real-time analytics

• Data modeling

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

14

Beware2 potential unforeseen effects of building a robust platform, if left un-checked…

Platform is reduced to a CMS tool

Ramping / Throttling / Release tool

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

15

If you build it, will they come? Again, it is not just about the Product… follow through with Adoption!

Product

Process

People

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary.

16

Questions?Thanks for listening!