Slide 1

2014 PayPal Inc. All rights reserved. Confidential and proprietary. Experimentation Platforms: Build or Buy? RecSys 2014 Mike Lo, PayPal Slide 2 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Product Test Ops Merchant Testing Knowledge Mgt and Insights Experimentation Analytics 2 Hello! Mike Lo, Sr. Manager - Experimentation, PayPal PayPal Experimentation Program Manage PayPals Experimentation Program Formerly part of eBays Program Build or Buy? We did both at PayPal Expectations for todays talk Focus on Experimentation Platform PayPals Testing Story Some comparison with eBay program. Slide 3 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 3 Words words words Slide 4 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 4 Becoming Data-Driven is difficult! It doesnt happen overnight. It took us years. Slide 5 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 5 We bought. Slide 6 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 6 TechnologyAdoptioneBay-focused PayPal = eBay right? What about the existing platform? Rationale why we didnt implement the eBay Experimentation Platform PayPals tech stack Fix Tracking Cant wait Quick wins Endorsement Program, not the Product. Built ground up for eBay Tightly coupled with eBay systems Need to decouple features Slide 7 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Outcome? Ran experiments successfully for 12 months then Couldnt Scale Had to cut the cord 7 Result: Implemented eBay Platform Slide 8 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 8 No. Its all about the TESTING NEEDS of your organization. So are you advocating to build rather than buy? Slide 9 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 9 No Product TestsKeep it separate?PayPals Needs What happened? Like many companies, the need started from Marketing but didnt end there. Product Pages Marketing Pages Product Pages Testing different content variants Testing different feature variants Technical Limitations Is there a clean distinction? 20 experiments is nice. But how do we get to 2,000? Slide 10 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Why it made sense for PayPal to build instead of buy 3 main reasons PayPal is investing to build 1 Analytical 10 2 Technical 3 Scale Slide 11 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 11 1 - High Fidelity Analytics Our site is an ecosystem Single Enterprise Testing Solution Scientific rigor in the results Other sites have natural fences Slide 12 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 12 Agile Product Lifecycle Who does your dev?Look beyond 2 - Technical agility Its all about going Agile! PayPals tech stack Master of your domain Data Site speed performance Rolling Full-Bright Test PlanDesignCodeValidate Smoke Test Coupled with PDLC Code once, push in minutes Allows MVT Biz Req Product Req Eng Req QARelease AlphaBetaAB TestCollectAnalyzeRamp Slide 13 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 13 VolumeSophisticationDeeper Insights 3- Scaling to the next level Utilizing the platform to its fullest potential Thousands of experiments Patented mechanism Utilizing PayPals platforms Entire data pipeline Automated design Real-time analytics Data modeling Slide 14 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 14 Beware 2 potential unforeseen effects of building a robust platform, if left un-checked Platform is reduced to a CMS tool Ramping / Throttling / Release tool Slide 15 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 15 If you build it, will they come? Again, it is not just about the Product follow through with Adoption! Slide 16 2014 PayPal Inc. All rights reserved. Confidential and proprietary. 16 Questions? Thanks for listening!