optimizing ebay - architecture · ebay inc. confidential ebay manages … – over 276,000,000...
TRANSCRIPT
Optimizing eBay Improving customer experience at the world’s online marketplace
Elissa Darnell, Director, User Experience Research
Deepak Nadig, eBay Principal Architect
eMetrics Marketing Optimization Summit May 06, 2008
eBay Inc. confidential 2
In the beginning…
“I started eBay as an experiment” — Pierre Omidyar
eBay Inc. confidential
An Amazing Story
1996 Registered eBay
users: 41,000
2008 Registered eBay users: 276 Million
eBay Inc. confidential
eBay manages … – Over 276,000,000 registered users – Over 1 Billion photos
– eBay users worldwide trade more than $2039 worth of goods every second – eBay averages well over 1 billion page views per day – At any given time, there are over 113 million items for sale on
the site in more than 50,000 categories – eBay stores over 2 Petabytes of data – over 200 times the size of the Library of Congress! – eBay analytics processes over 25 Petabytes of data on any day – The eBay platform handles 4.4 billion API calls per month
The eBay context
Over ½ Million pounds of Kimchi are sold every year!
A sporting good sells every 2 seconds
In a dynamic environment – 300+ features per quarter – We roll 100,000+ lines of code every two weeks
In 39 countries, in seven languages.
>44 Billion SQL executions/day!
An SUV is sold every 5 minutes
eBay Inc. confidential 5
Site Statistics: in a typical day…
June 1999
Q1 2007
Growth
Outbound Emails 1 M 41 M 41x
Total Page Views 54 M >1 B 19x
Peak Network Utilization 268 Mbps 16 Gbps 59x
API Calls 0 150 M N/A
Availability ~97% 99.94% 50x
43 mins/day 50 sec/day
eBay Inc. confidential 6
Meet the Buyers and Sellers
Some people buy on eBay and other people sell, and some do both
eBay users trade in more than 50,000 categories
eBay’s buyers want a fun shopping experience that provides a great deal
Approximately 1.3M people around the world make all or part of their living selling on eBay
eBay Inc. confidential
Personal Accounts
7
Who is eBay’s Community?
“eBay Community” is everyone who has a relationship with eBay, Inc.
New Buyers
Collectors
Frequent buyers
One-time buyers
Small Businesses
Experienced Buyers
In-active buyers
Store Sellers
Power Sellers
New Sellers
Casual Sellers
Business Sellers
Hobby Sellers
Merchant Accounts
Pay a friend
Buy online
Top Buyers
eBay Inc. confidential 8
What we mean by the whole user experience
It starts by being useful…
Functionally, people must be able to use it…
The way it looks must be pleasing…
This helps create an overall brand experience
It is useful to me. It meets my needs.
I am able to use the product easily (I can perform the tasks it supports).
I like the way the product looks and feels.
My overall feeling about a brand/ product (in the abstract) is good.
Executing well on all of these areas is what creates a great user experience. Research is
needed for each.
eBay Inc. confidential 9
Assortment of User Research methods
Participatory Design
Card sorting
Field Visits Lab Testing
Eye Tracking Surveys
eBay Inc. confidential 10
Lab-based Usability Testing
• Usability or “lab-testing” – We bring in representative users
individually to our usability labs
– Observe them while they perform assigned tasks
– Use prototypes or the live site
• Enables direct observation of target users as they interact with our web site or a design prototype
– Observe what users ACTUALLY do, rather than relying on what they SAY they do
– Understand WHY people do what they do, not just what they do
• Identify areas that are confusing and potential fixes
• Usability testing done iteratively, throughout the design process
eBay Inc. confidential 11
Variations on the standard lab study
Some variations on standard lab tests
• Low Fidelity (Paper) Lab Testing
– Designs are shown on paper
– Researcher or Designer acts as the computer
– Participant uses their finger as the mouse
• RITE Lab Testing
– Stands for Rapid Iterative Test and Evaluation
– Focuses more on rapidly iterating and re-testing the design based on very small sample
– Can be performed on lo-fi (paper) or high fidelity interactive prototypes
eBay Inc. confidential 12
Visits
Seeing through the eyes of our customers
eBay Inc. confidential 13
Visits: What is it?
• Also known as a Field Study or Ethnography
• Involves going into people’s home, office, where ever they use eBay
• Spending two to three hours with them, observing them use eBay
(or shop online) and listening to them, and sometimes conducting
structured interview
• Unlike Lab Testing, typically formal tasks are not given
• Data is collected through videotaping and taking extensive notes
• Findings are summarized across participants
eBay Inc. confidential 14
Visits allow us to observe how people use eBay in a natural context
eBay Inc. confidential 15
We get to see people use eBay…
• On their own computers (PCs, laptops, old, new) • With various connection speeds (some dial-up)
• To perform their own tasks (such as selling a camera or book)
• With their own cameras and workspace (living room, office)
• With life’s normal interruptions (talking parrot, cluttered desk)
They tend to let their guard down and we learn things we might not otherwise learn online or in the lab
eBay Inc. confidential 16
The focus of Visits
• Questions we focus on in Visits are: – “What is the larger context of use?” – “What issues exist, and WHY?” – “What can we do to address the issues?”
– Visits are NOT about the numbers, or the question, “How many users experienced that?”
– If pervasiveness of a particular issue is of interest, we supplement with survey or analytics data to help quantify the issue
eBay Inc. confidential 17
Research methods span the Product Development
Lab Testing RITE Testing Paper Prototypes Eye Tracking Card sorting
Strategic research to inspire Design research to inform Assessment research to track
Understand Conceive Design Develop Launch
Surveys Live-to-site Testing Longitudinal Studies Diary Studies
Field Visits Participatory Design Competitive Evaluation
eBay Inc. confidential
Research methods
Scripted or lab-based use of product
Natural use of product
Qualitative Quantitative
Self-reported (stated)
Observed Behavior
Mixture
APPROACH
DAT
A SO
UR
CE
Combination / hybrid De-contextualized / not using product
KEY – Context of data collection with respect to product use
Focus Groups / “Voices” Phone Interviews Cardsorting Diary/Camera Study
Intercept Surveys Desirability studies Email Surveys
Message Board Mining
“Visits” / Ethnographic Field Studies Usability Lab Studies (task-based)
Quantitative user experience assessments (e.g., Keynote/Vividence)
(Onsite interviews)
(Extended observation) /
Eyetracking
Usability benchmarking (in lab)
Experimentation
Clickstreams
Data mining
Why How to fix
Behaviors | What people do
What How much
Attitudes | What people believe
eBay Inc. confidential 19
Case Study Using Qualitative and Quantitative Research to create a New “View Item” Page
Why redesign the “View Item” Page? • Increase BID/BIN efficiency
• Improve the user experience: Reduce complexity and clutter and inspire buyers to convert
Research Approach • Qualitative and quantitative research to understand user needs and inform design decisions
• Incorporate User Experience Research, Market Research, and Analytics
• Institute research as an integral component of the redesign process (from start to finish)
View Item 2008 View Item 2000 View Item 2003 View Item 2006
eBay Inc. confidential
Understanding User Needs (Qualitative) • “Compelled” lab study (US) to understand the current experience and how it may be improved
• Participatory design (US, UK, DE) to understand the “ideal” experience to gather user requirements and inform product design and innovation
Concept Testing (Qualitative)
• Focus groups (US) to gauge user reaction to early design concepts and isolate aspects of alternative designs that resonate with users
Iterative Design (Qualitative & Quantitative)
• Rapid Iterative Testing (RITE) (US, DE, IT) to gauge user reaction to an early View Item prototype and make rapid improvements to the design based on user feedback and behavior
• Maximum Differential Survey (US) to determine the relative perceived usefulness of features to Buyers in an effort to streamline and simplify the experience
• Analytics (US) to understand current usage of existing features
20
Research Overview
Visual Design Research (Quantitative)
• Desirability study (US) to determine which visual design approach best conveys target brand attributes (e.g. convenient, clean, safe, fun)
• Tab Visual Design Eyetracking study (US) to explore different visual design treatments for View Item page tabs and determine which best captures and maintains attention as measured by eyetracking data
Longitudinal Research (Q2 2008) (Qualitative)
• Longitudinal Diary Study (US) with functional prototype to understand “real world” usage of View Item over time and determine areas for improvement prior to launch
eBay Inc. confidential 21
Compelled Lab Study
Objective
Surface usability issues and perceived strengths and weaknesses of the current View Item page and larger transaction flow
Approach
A “compelled” lab study with users genuinely interested in purchasing items of interest on eBay. Users were asked to find and purchase items of interest on eBay with their real account and payment information in the lab setting
Participants
12 - Mix of experienced and less experienced eBay buyers
All participants were pre-screened to ensure they had a genuine interest in purchasing specific items of interest on eBay in the coming days
eBay Inc. confidential 22
Feature Prioritization Maximum Differential Survey
• A key goal was site simplification: Provide buyers with the most important information they need to make their purchase decision
• Traditional surveys focused on understanding feature usefulness typically do not effectively differentiate among features (e.g. majority of features considered useful by 90% or more of buyers)
• A Max Differential survey was conducted in the U.S. to determine the relative usefulness of View Item features to understand which features to include and the relative prominence that should be assigned
• Approximately 5000 buyers and 5000 sellers completed the survey which involved asking users to choose the most useful and least useful feature among different combinations of features to yield a score for each feature
Current bid
Auction countdown
End time
Time left
Real-time updating of bid
Questions and Answers
Your maximum bid
Number of unique bidders
Total Buyers Top 30% Buyers Lower 70% Buyers EXAMPLE
Background and Objectives
eBay Inc. confidential 23
Tab Visual Design Eyetracking Study
Goal: To explore different visual design treatments for View Item page tabs and determine which best captures and maintains attention as measured by eyetracking data
On Hover Off
On Hover Off
On Hover Off
Sample Heat map summarizing overall viewing pattern
Sample scanning pattern
eBay Inc. confidential 24
How do we know the new “View Item” page will be a success?
Research was conducted throughout the product lifecycle to evaluate the current strategy and design at each stage (prior to further dedication of design or engineering resources)
• Focus groups on early concepts
• Rapid Iterative Testing of early prototype
• Max Differential survey (and Analytics data) to ensure that reduction of page complexity targeted the least useful features
• Quantitative “Desirability” study to ensure that the chosen design approach best conveyed target brand attributes (such as “fun” and “unique”)
• Quantitative Eyetracking study of tab design alternatives to ensure the chosen treatment would best capture and maintain user attention
• A longitudinal diary study (planned in Q2 2008) to gather qualitative real-world usage feedback to ensure a good user experience prior to launch (to be interpreted in conjunction with A/B Test data)
eBay Inc. confidential 25
Experimentation
• Setup Experiment Samples Treatments, Factors • Implementation
• User (Experiment, Treatment) • Serve Treatment
• Tracking • Monitoring
• DOE • Define Samples, Treatments, Factors
• Metrics • Reporting • Idea (!)
• Learning
1. Hypothesis
2. Experimental Design
3. Setup Experiment
4. Launch Experiment
5. Measurement
7. Analysis & Results
eBay Experimentation
Platform
eBay Inc. confidential 26
Automation
• Dynamically adapt experience – Choose page, modules, and inventory which provide best experience for that user and
context – Order results by combination of demand, supply, and other factors (“Best Match”)
• Feedback loop enables system to learn and improve over time – Collect user behavior – Aggregate and analyze offline – Deploy updated metadata – Decide and serve appropriate experience
• Best Practices – “Perturbation” for continual improvement – Dampening of positive feedback
eBay Inc. confidential
Experimentation Platform
27
Experience
Page, Module
Experiment Metadata Experience
Response
Metrics / Experience
Results Observations
Design
Access
Results Observations
Access
Metrics / Experience
eBay Inc. confidential 28
What we think about
28
Type/Level of Experiment
Signal/Noise Ratio
Concurrency
Iteration time
Cost of Experiments
Fidelity of Experiments
The total cost of designing, building, running, and analyzing an experiment
The time from hypothesis to when the analyzed results are available for planning the next iteration
Number of experiments that can be run at the same time
The extent to which the metrics are obscured by experimental noise
Extent to which the model and its conditions represent the final feature or product
Supporting different types (A/B, 1-FAT, DOE) and levels (page, module, page flow) of experiments
eBay Inc. confidential 29
Challenges …
• Sticky-ness to user
• What, not why
• Duration and long term effects
• Minor vs. major differences
• Extent of generalization
eBay Inc. confidential 30
Qualitative research such as lab tests and field visits give us rich data about • Usability problems • Discoverability • Navigation • Terminology • More complex problems
Quantitative research such as surveys, usage studies, and log analysis tells us • How pervasive a problem is • What features on the site are actually being used
The combination of qualitative and quantitative data give us a more complete picture that is most powerful
The Power of Combining Data
eBay Inc. confidential 31
How we get our data differs…
Analytics Measure what customers do through serving experiences and tracking responses
Research Work directly with customers by…
• Talking to them
• Observing them
• Surveying them
The real power is from looking at our user’s from both sides!
eBay Inc. confidential 32
Summary
• Understanding the customer experience requires insights into – What they do – Why they do it – Attitudes, motivations and behaviors
• A variety of research methods, both qualitative and quantitative, can be used.
• Each of the methods – qualitative and quantitative - have their advantages and limitations.
• Using them together continues to help eBay gain a holistic understanding of the user experience and aspects that need to be improved.